Replace the sql dependency with our own lexer and parser

[jessestimpson]

We only ever used elixir-dbvisor/sql for lexing and parsing, and we'd piled up a fair number of token rewrites to get around it. ... and x is not null and any column named after a reserved word (where day = 'x') made it hang, and order by day desc quietly flipped to ascending. On top of that, the latest version brings C NIFs and a runtime we had to keep from starting. So this writes a small front end for the dialect efsql actually speaks and drops the dependency.

What's in lib/efsql/sql/

• Lexer. Hand-written over binaries. Unquoted names fold to lower case like Postgres, quoted ones keep their case. It handles ''/"" escapes, nested /* */ comments, CRLF, Unicode names and whitespace, and exponent floats. Every error carries a line and column: syntax error at line 3, column 10: expected an expression, got '='.
• Grammar. src/efsql_sql_grammar.yrl, compiled by yecc (Mix does this on its own, nothing to install). Output is a documented AST (Efsql.SQL.AST). day, user, date and value all work as bare column names. The handful of words that can't (order, select, limit, and, ...) are listed, and the error tells you to quote them.
• Error messages. yecc only tells you where it stopped. To say what it wanted, the parser replays the tokens up to that point with every terminal and keeps the ones the grammar accepts. So messages come from the grammar itself (expected BY, got 'a', expected NOT or NULL, got 'true'), and new rules need nothing extra to get good errors. On top of that there's a short list of hints for things we don't support yet: GROUP BY, joins, WITH, functions, subqueries, DISTINCT. Each hint goes away when its feature lands.

Efsql.Parser is now just the AST → Efsql.Logical translation, and all the old workarounds are gone. It also catches two inputs that used to crash: field names too long to be atoms, and versionstamps outside 0..2^96-1.

Why yecc for the parser

I wrote the parser twice, once by hand (recursive descent) and once with leex/yecc, and got them to give identical output on all the tests and 60k fuzzed inputs before comparing. For today's dialect the hand-written one was a bit smaller and twice as fast. But CTEs, joins and GROUP BY are coming, and that's where a grammar yecc checks for conflicts pays off. (select …) vs (expr) vs a tuple, aliases vs keywords, and join chains are all ambiguities that yecc reports at build time and a hand parser just hides. Speed doesn't matter at this size: about 50µs for a typical query, 0.4s for a 550KB one.

The lexer stays hand-written. leex can't do nested comments, Unicode classes or character columns, and it behaves differently on OTP 25 and 28, which broke CI once. Grammar growth only adds keywords to the lexer anyway.

Smaller things

• Typed literals now work as BETWEEN bounds, and the README is updated to match.
• The CLI prints exception messages instead of inspected structs, so syntax errors are readable.
• Efsql.lex_and_parse/1 is gone. It returned the old library's parse tree and nothing called it.
• sql, unicode, unicode_set and nimble_parsec are out of mix.lock. No more C toolchain needed to build.

Behaviour changes worth knowing

• Clauses have to come in standard order (WHERE, ORDER BY, LIMIT).
• The reserved words above need quotes when used as column names.

Tests

• Lexer and parser suites covering every token, construct and error, with exact messages and positions.
• The existing translation tests pass unchanged. There are new ones for the translator errors, plus every query on the TUI help page.
• Randomized tests:
  • Random trees get printed with randomized case, comments, whitespace, quoting and alternate spellings (!=, CAST(...), ISNULL, timestamp with time zone), and must parse back to the exact tree.
  • Mangled queries and random bytes must either parse or give a positioned SyntaxError, never anything else.
  • A 20,000-condition query has to parse and translate quickly.

The fuzzing found one real crash: float literals too big for a double (1e400), now a syntax error.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PFBZUrerge6bbWVCHpS5Ko

---

Generated by Claude Code