Add goldeneye, a module that generates and checks the dialect seeds

.github/workflows/gen.yml

@@ -1,13 +1,13 @@
-name: sqlc-pg-gen
+name: gen
 on:
   workflow_dispatch:
 jobs:
-  gen:
-    name: sqlc-pg-gen
-    runs-on: ubuntu-22.04
+  generate:
+    name: generate dialects
+    runs-on: ubuntu-24.04
     services:
       postgres:
-        image: postgres:15.0-alpine
+        image: postgres:16
         env:
           POSTGRES_USER: postgres
           POSTGRES_PASSWORD: postgres
@@ -20,20 +20,18 @@ jobs:
     - uses: actions/checkout@v7
     - uses: actions/setup-go@v7
       with:
-        go-version-file: go.mod
+        go-version-file: internal/goldeneye/go.mod
         check-latest: true
-    - run: go build -o sqlc-pg-gen ./internal/tools/sqlc-pg-gen
-    - run: mkdir -p gen/contrib
-    - run: ./sqlc-pg-gen gen
+    - run: go run ./cmd/goldeneye install clickhouse
+      working-directory: internal/goldeneye
+    - run: go run ./cmd/goldeneye generate
+      working-directory: internal/goldeneye
       env:
-        PG_USER: postgres
-        PG_HOST: localhost
-        PG_DATABASE: postgres
-        PG_PASSWORD: postgres
-        PG_PORT: ${{ job.services.postgres.ports['5432'] }}
+        POSTGRESQL_SERVER_URI: postgres://postgres:postgres@localhost:${{ job.services.postgres.ports['5432'] }}/postgres?sslmode=disable
     - name: Save results
       uses: actions/upload-artifact@v7
       with:
-        name: sqlc-pg-gen-results
-        path: gen
-
+        name: dialects
+        path: internal/engine/*/dialect
+    - name: Fail if the committed dialects differ
+      run: git add -N internal/engine && git diff --exit-code --stat -- internal/engine

CLAUDE.md

@@ -143,6 +143,24 @@ SQLC_TEST_CORE=1 go test ./internal/endtoend -run 'TestReplay/core'
 Go aborts a test binary on panic, so a case that panics the core analyzer ends
 the run early. Run a subset to get past one (`-run 'TestReplay/core/^select'`).
 
+### Dialect Checks
+
+The dialect seeds under `/internal/engine/<engine>/dialect/` are generated
+from a live database by `/internal/goldeneye`, a nested module, and its tests
+verify the committed files against one byte for byte. Engines whose database
+is not available skip.
+
+```bash
+cd internal/goldeneye
+go run ./cmd/goldeneye install clickhouse   # download the pinned clickhouse binary once
+POSTGRESQL_SERVER_URI="postgres://postgres:postgres@127.0.0.1:5432/postgres?sslmode=disable" go test ./...
+go run ./cmd/goldeneye generate postgresql  # rewrite the files after a change
+```
+
+The checks are not part of CI. The `gen` workflow
+(`.github/workflows/gen.yml`) generates the files on demand and uploads
+them as an artifact.
+
 ### Example Tests
 
 - **Location:** `/examples/` directory
@@ -216,10 +234,13 @@ MYSQL_SERVER_URI="root:mysecretpassword@tcp(127.0.0.1:3306)/mysql?multiStatement
   - `/dolphin/` - MySQL parser (uses TiDB parser)
   - `/sqlite/` - SQLite parser
   - `/duckdb/` - DuckDB 2.0 parser (uses darkwing, the pure Go port of
-    DuckDB's PEG parser); its dialect seeds are generated by
-    `/internal/tools/sqlc-duckdb-gen` from a live DuckDB CLI
+    DuckDB's PEG parser)
   - `<engine>/dialect/` - The engine's type system and standard library, as
-    JSONL read by `/internal/core/seed`
+    JSONL read by `/internal/core/seed`; the generated parts come from
+    `/internal/goldeneye`
+- `/internal/goldeneye/` - Nested module that generates the dialect seeds
+  under `/internal/engine/<engine>/dialect/` from a live database and checks
+  the committed ones against it, one package per engine; see its README
 - `/internal/core/` - The analysis core: catalog, analyzer and dialect seeds
 - `/internal/compiler/` - Query compilation logic
 - `/internal/codegen/` - Code generation for different languages

Makefile

@@ -1,4 +1,4 @@
-.PHONY: build build-endtoend test test-ci test-examples test-endtoend start psql mysqlsh proto
+.PHONY: build build-endtoend test test-ci test-examples test-endtoend test-goldeneye start psql mysqlsh proto
 
 build:
 	go build ./...
@@ -26,8 +26,11 @@ test-ci: test-examples build-endtoend vet
 sqlc-dev:
 	go build -o ~/bin/sqlc-dev ./cmd/sqlc/
 
-sqlc-pg-gen:
-	go build -o ~/bin/sqlc-pg-gen ./internal/tools/sqlc-pg-gen
+goldeneye:
+	cd ./internal/goldeneye && go build -o ~/bin/goldeneye ./cmd/goldeneye
+
+test-goldeneye:
+	cd ./internal/goldeneye && go test ./...
 
 sqlc-gen-json:
 	go build -o ~/bin/sqlc-gen-json ./cmd/sqlc-gen-json

internal/engine/clickhouse/dialect/types.jsonl

@@ -1,40 +1,66 @@
-{"name": "UInt8", "category": "N"}
-{"name": "UInt16", "category": "N"}
-{"name": "UInt32", "category": "N"}
-{"name": "UInt64", "category": "N"}
-{"name": "UInt128", "category": "N"}
-{"name": "UInt256", "category": "N"}
-{"name": "Int8", "category": "N"}
-{"name": "Int16", "category": "N"}
-{"name": "Int32", "category": "N"}
-{"name": "Int64", "category": "N"}
-{"name": "Int128", "category": "N"}
-{"name": "Int256", "category": "N"}
-{"name": "Float32", "category": "N"}
-{"name": "Float64", "category": "N"}
-{"name": "BFloat16", "category": "N"}
-{"name": "Decimal", "category": "N"}
-{"name": "Decimal32", "category": "N"}
-{"name": "Decimal64", "category": "N"}
-{"name": "Decimal128", "category": "N"}
-{"name": "Decimal256", "category": "N"}
-{"name": "Bool", "category": "B"}
-{"name": "String", "category": "S"}
-{"name": "FixedString", "category": "S"}
-{"name": "UUID", "category": "S"}
-{"name": "Date", "category": "D"}
-{"name": "Date32", "category": "D"}
-{"name": "DateTime", "category": "D"}
-{"name": "DateTime64", "category": "D"}
-{"name": "IPv4", "category": "S"}
-{"name": "IPv6", "category": "S"}
-{"name": "JSON", "category": "U"}
-{"name": "Enum8", "category": "U"}
-{"name": "Enum16", "category": "U"}
-{"name": "Nullable", "category": "U"}
-{"name": "LowCardinality", "category": "U"}
-{"name": "Array", "category": "A"}
-{"name": "Map", "category": "U"}
-{"name": "Tuple", "category": "U"}
-{"name": "Nested", "category": "U"}
-{"name": "Nothing", "category": "U"}
+{"name":"AggregateFunction","category":"U"}
+{"name":"Array","category":"A"}
+{"name":"BFloat16","category":"N"}
+{"name":"Bool","category":"B","aliases":["boolean"]}
+{"name":"Date","category":"D"}
+{"name":"Date32","category":"D"}
+{"name":"DateTime","category":"D","aliases":["TIMESTAMP"]}
+{"name":"DateTime32","category":"D"}
+{"name":"DateTime64","category":"D"}
+{"name":"Decimal","category":"N","aliases":["DEC","FIXED","NUMERIC"]}
+{"name":"Decimal128","category":"N"}
+{"name":"Decimal256","category":"N"}
+{"name":"Decimal32","category":"N"}
+{"name":"Decimal64","category":"N"}
+{"name":"Dynamic","category":"U"}
+{"name":"Enum","category":"U"}
+{"name":"Enum16","category":"U"}
+{"name":"Enum8","category":"U"}
+{"name":"FixedString","category":"S","aliases":["BINARY"]}
+{"name":"Float32","category":"N","aliases":["FLOAT","REAL","SINGLE"]}
+{"name":"Float64","category":"N","aliases":["DOUBLE","DOUBLE PRECISION"]}
+{"name":"IPv4","category":"S","aliases":["INET4"]}
+{"name":"IPv6","category":"S","aliases":["INET6"]}
+{"name":"Int128","category":"N"}
+{"name":"Int16","category":"N","aliases":["SMALLINT","SMALLINT SIGNED"]}
+{"name":"Int256","category":"N"}
+{"name":"Int32","category":"N","aliases":["INT","INT SIGNED","INTEGER","INTEGER SIGNED","MEDIUMINT","MEDIUMINT SIGNED"]}
+{"name":"Int64","category":"N","aliases":["BIGINT","BIGINT SIGNED","SIGNED"]}
+{"name":"Int8","category":"N","aliases":["BYTE","INT1","INT1 SIGNED","TINYINT","TINYINT SIGNED"]}
+{"name":"IntervalDay","category":"T"}
+{"name":"IntervalHour","category":"T"}
+{"name":"IntervalMicrosecond","category":"T"}
+{"name":"IntervalMillisecond","category":"T"}
+{"name":"IntervalMinute","category":"T"}
+{"name":"IntervalMonth","category":"T"}
+{"name":"IntervalNanosecond","category":"T"}
+{"name":"IntervalQuarter","category":"T"}
+{"name":"IntervalSecond","category":"T"}
+{"name":"IntervalWeek","category":"T"}
+{"name":"IntervalYear","category":"T"}
+{"name":"JSON","category":"U"}
+{"name":"LineString","category":"U"}
+{"name":"LowCardinality","category":"U"}
+{"name":"Map","category":"U"}
+{"name":"MultiLineString","category":"U"}
+{"name":"MultiPolygon","category":"U"}
+{"name":"Nested","category":"U"}
+{"name":"Nothing","category":"U"}
+{"name":"Nullable","category":"U"}
+{"name":"Object","category":"U"}
+{"name":"Point","category":"U"}
+{"name":"Polygon","category":"U"}
+{"name":"Ring","category":"U"}
+{"name":"SimpleAggregateFunction","category":"U"}
+{"name":"String","category":"S","aliases":["BINARY LARGE OBJECT","BINARY VARYING","BLOB","BYTEA","CHAR","CHAR LARGE OBJECT","CHAR VARYING","CHARACTER","CHARACTER LARGE OBJECT","CHARACTER VARYING","CLOB","GEOMETRY","LONGBLOB","LONGTEXT","MEDIUMBLOB","MEDIUMTEXT","NATIONAL CHAR","NATIONAL CHAR VARYING","NATIONAL CHARACTER","NATIONAL CHARACTER LARGE OBJECT","NATIONAL CHARACTER VARYING","NCHAR","NCHAR LARGE OBJECT","NCHAR VARYING","NVARCHAR","TEXT","TINYBLOB","TINYTEXT","VARBINARY","VARCHAR","VARCHAR2"]}
+{"name":"Time","category":"D"}
+{"name":"Time64","category":"D"}
+{"name":"Tuple","category":"U"}
+{"name":"UInt128","category":"N"}
+{"name":"UInt16","category":"N","aliases":["SMALLINT UNSIGNED","YEAR"]}
+{"name":"UInt256","category":"N"}
+{"name":"UInt32","category":"N","aliases":["INT UNSIGNED","INTEGER UNSIGNED","MEDIUMINT UNSIGNED"]}
+{"name":"UInt64","category":"N","aliases":["BIGINT UNSIGNED","BIT","SET","UNSIGNED"]}
+{"name":"UInt8","category":"N","aliases":["INT1 UNSIGNED","TINYINT UNSIGNED"]}
+{"name":"UUID","category":"S"}
+{"name":"Variant","category":"U"}

internal/engine/clickhouse/seed.go

@@ -7,6 +7,15 @@ import (
 	"github.com/sqlc-dev/sqlc/internal/core/seed"
 )
 
+// The dialect directory describes ClickHouse's type system. types.jsonl is
+// generated from system.data_type_families of the pinned ClickHouse release
+// by goldeneye (internal/goldeneye), which also checks it against one;
+// dialect.json and functions.jsonl are authored by hand, since ClickHouse
+// publishes no function signatures. Regenerate from internal/goldeneye with:
+//
+//	go run ./cmd/goldeneye install clickhouse
+//	go run ./cmd/goldeneye generate clickhouse
+//
 //go:embed dialect
 var dialectFS embed.FS
 

internal/engine/duckdb/seed.go

@@ -9,10 +9,10 @@ import (
 
 // The dialect directory describes DuckDB's type system. types.jsonl,
 // functions.jsonl and operators.jsonl are generated from a live DuckDB 2.0
-// CLI by sqlc-duckdb-gen (internal/tools/sqlc-duckdb-gen); dialect.json is
-// authored by hand. Regenerate with:
+// CLI by goldeneye (internal/goldeneye), which also checks them against one;
+// dialect.json is authored by hand. Regenerate from internal/goldeneye with:
 //
-//	DUCKDB=/path/to/duckdb go run ./internal/tools/sqlc-duckdb-gen
+//	DUCKDB=/path/to/duckdb go run ./cmd/goldeneye generate duckdb
 //
 //go:embed dialect
 var dialectFS embed.FS

internal/engine/postgresql/catalog.go

@@ -16,7 +16,7 @@ func NewCatalog() *catalog.Catalog {
 }
 
 // genPGCatalog and genInformationSchema build the schemas sqlc knows
-// PostgreSQL by. Both read the dialect directory, which sqlc-pg-gen writes,
+// PostgreSQL by. Both read the dialect directory, which goldeneye writes,
 // and which the analysis core seeds its catalog from as well.
 func genPGCatalog() *catalog.Schema {
 	return systemSchema("pg_catalog", pgCatalogFuncs())

internal/engine/postgresql/dialect/functions.jsonl

@@ -1695,11 +1695,6 @@
 {"name":"pg_export_snapshot","returns":"text"}
 {"name":"pg_extension_config_dump","args":[{"type":"regclass"},{"type":"text"}],"returns":"void"}
 {"name":"pg_extension_update_paths","args":[{"name":"name","type":"name"}],"returns":"record"}
-{"name":"pg_file_rename","args":[{"type":"text"},{"type":"text"}],"returns":"boolean"}
-{"name":"pg_file_rename","args":[{"type":"text"},{"type":"text"},{"type":"text"}],"returns":"boolean"}
-{"name":"pg_file_sync","args":[{"type":"text"}],"returns":"void"}
-{"name":"pg_file_unlink","args":[{"type":"text"}],"returns":"boolean"}
-{"name":"pg_file_write","args":[{"type":"text"},{"type":"text"},{"type":"boolean"}],"returns":"bigint"}
 {"name":"pg_filenode_relation","args":[{"type":"oid"},{"type":"oid"}],"returns":"regclass"}
 {"name":"pg_function_is_visible","args":[{"type":"oid"}],"returns":"boolean"}
 {"name":"pg_get_backend_memory_contexts","returns":"record"}
@@ -1772,7 +1767,6 @@
 {"name":"pg_lock_status","returns":"record"}
 {"name":"pg_log_backend_memory_contexts","args":[{"type":"integer"}],"returns":"boolean"}
 {"name":"pg_log_standby_snapshot","returns":"pg_lsn"}
-{"name":"pg_logdir_ls","returns":"record"}
 {"name":"pg_logical_emit_message","args":[{"type":"boolean"},{"type":"text"},{"type":"bytea"}],"returns":"pg_lsn"}
 {"name":"pg_logical_emit_message","args":[{"type":"boolean"},{"type":"text"},{"type":"text"}],"returns":"pg_lsn"}
 {"name":"pg_logical_slot_get_binary_changes","args":[{"name":"slot_name","type":"name"},{"name":"upto_lsn","type":"pg_lsn"},{"name":"upto_nchanges","type":"integer"},{"name":"options","type":"text[]","mode":"v","has_default":true}],"returns":"record"}

internal/engine/postgresql/extension.go

@@ -9,7 +9,7 @@ import (
 )
 
 // loadExtension reads the named extension's function list from the dialect
-// directory, where sqlc-pg-gen writes one directory per extension. An
+// directory, where goldeneye writes one directory per extension. An
 // extension sqlc does not know is nil, which CREATE EXTENSION treats as
 // nothing to add.
 func loadExtension(name string) *catalog.Schema {

internal/engine/postgresql/seed.go

@@ -17,10 +17,11 @@ import (
 // "WHERE bigint_col = 1" depends on.
 //
 // functions.jsonl and relations.jsonl are pg_catalog's functions and the
-// tables and views of pg_catalog and information_schema, all written by
-// sqlc-pg-gen. Both the analysis core and the catalog the legacy compiler
-// builds read them. Each contrib extension sqlc knows is a directory under
-// extensions/ holding the types and functions CREATE EXTENSION adds.
+// tables and views of pg_catalog and information_schema, all generated from
+// a live PostgreSQL 16 server by goldeneye (internal/goldeneye), which also
+// checks them against one. Both the analysis core and the catalog the legacy
+// compiler builds read them. Each contrib extension sqlc knows is a directory
+// under extensions/ holding the types and functions CREATE EXTENSION adds.
 //
 //go:embed dialect
 var dialectFS embed.FS

internal/goldeneye/README.md

@@ -0,0 +1,69 @@
+# goldeneye
+
+`goldeneye` generates the dialect seeds under `internal/engine/<engine>/dialect`
+from a live database, and verifies the committed ones against it. A dialect
+is the JSONL that gives an engine its type system and standard library —
+`types.jsonl`, `functions.jsonl`, `operators.jsonl`, `relations.jsonl` and
+the `extensions/` bundles — read by `internal/core/seed`. Each engine package
+here asks the database what it knows, writes the answer in that shape, and
+the tests compare it with what is committed, byte for byte. A difference
+means the committed dialect has drifted from the database.
+
+It is a nested Go module, so its only dependency beyond the standard library
+is the PostgreSQL driver, and it never shares code with the analysis that
+reads the files: the files are the contract. Run it from this directory:
+
+```bash
+go run ./cmd/goldeneye install clickhouse   # download the pinned clickhouse binary once
+go run ./cmd/goldeneye check                # check every engine whose database is available
+go run ./cmd/goldeneye check postgresql     # check one engine
+go run ./cmd/goldeneye generate [engine]    # rewrite the generated files from the database
+go test ./...                               # the same checks as tests; engines without a database skip
+```
+
+`generate` and `check` say which engines they skipped for lack of a
+database; naming an engine makes its database required.
+
+## What is generated, and what is not
+
+A generator owns only the files it produces; `dialect.json` is always written
+by hand, and so are the lists an engine cannot describe. Both commands leave
+the hand-written files alone, and the checks do not look at them.
+
+- **`postgresql`** reads a live server named by `POSTGRESQL_SERVER_URI`:
+  `functions.jsonl` is `pg_catalog`'s functions, `relations.jsonl` the tables
+  and views of `pg_catalog` and `information_schema`, and each contrib
+  extension gets a directory under `extensions/` holding the types and
+  functions `CREATE EXTENSION` adds, so the server needs contrib installed.
+  A function that one of those extensions puts in `pg_catalog`, as
+  `adminpack` does, belongs to the extension's directory rather than the
+  catalog's list. The server has to be the major release pinned in
+  `postgresql.Major`, since every release adds to the catalogs; the top-level
+  `types.jsonl` and `operators.jsonl` are hand-written.
+- **`duckdb`** reads the DuckDB CLI named by `DUCKDB`, or `duckdb` on `PATH`:
+  `types.jsonl`, `functions.jsonl` and `operators.jsonl` come from
+  `duckdb_types()` and `duckdb_functions()`. The CLI has to be the DuckDB 2.0
+  build darkwing is pinned against, which has no release to download yet.
+- **`clickhouse`** needs no server: `types.jsonl` comes from
+  `system.data_type_families` of an ephemeral `clickhouse local` process,
+  every family that is not an alias becoming a type carrying the spellings
+  that alias it, with a category decided by its name. The binary is
+  downloaded once per pinned release by `install` into the user cache
+  directory, or supplied through the `CLICKHOUSE` environment variable; the
+  pinned release and the SHA-512 of each platform's download live in
+  `clickhouse/install.go`, and a download that does not match is discarded.
+  ClickHouse describes its functions no further than their names, so
+  `functions.jsonl` is hand-written.
+
+## Layout
+
+- `dialect/` — the record types the files are made of, mirrored from
+  `internal/core/seed`, and the helpers that write a generated set of files
+  into an engine directory or diff it against what is committed.
+- `postgresql/`, `duckdb/`, `clickhouse/` — one package per engine, each
+  exposing `Locate`, `Version` and `Generate`, and a test that runs the check.
+- `cmd/goldeneye/` — the command.
+
+The analysis checks — verifying the `analyze_*` cases under
+`internal/endtoend/testdata` against what each database itself reports — are
+meant to live here too, alongside the dialect checks.