Negated conditions and ILIKE; operators that stream

[jessestimpson]

Two independent changes, one commit each.

1. NOT, <>, NOT IN, NOT BETWEEN, ILIKE, NOT ILIKE

These already parsed, but the translator refused them. Now they work:

where status <> 'paid'                      -- or !=
where status not in ('paid', 'refunded')
where total not between 10 and 20
where name ilike 'al%'                      -- also not ilike
where not (status = 'paid' or total > 100)

• NOT is pushed into the condition it negates, and SQL's NULL rules keep each opposite exact:
  • not (a < 1) is a >= 1 (both false when a is NULL), and not (a = 1) is a <> 1;
  • IN, LIKE, ILIKE, BETWEEN and IS NULL flip, and not not cancels;
  • not (a or b) is not a and not b;
  • not (a and b) would need OR, so it gets OR's "not supported" message.
• New predicate shapes: {:cmp, :!=, …}, {:not_in, …}, {:not_range, …}, {:ilike, …} and {:not_ilike, …}. An index can't serve any of them, so they're checked on the rows read (Predicate.pushdown/1 is :filter). The rest of the WHERE still narrows what's read.
• NULLs: a NULL field matches none of them, as in SQL. A NULL status is neither = 'paid' nor <> 'paid'.
• Rewrite: a single-value NOT IN becomes <>.
• Primary key _: a condition an index can't serve, like _ <> 'x', now says what _ does support (=, <, <=, >, >=, BETWEEN, IN). Before, it said "cannot be combined with this query shape".

The README, TUI help and completion cover the new forms.

2. Operators that process rows as they arrive (Efsql.Pipeline)

Operators used to take the whole list of rows. A batched cross-tenant query (\set tenant_batch N) concatenated every batch before the first operator ran, so it held every row it read, and a LIMIT still read every batch.

Efsql.Pipeline runs a plan's operators a chunk at a time, and each operator keeps only what it needs:

• LIMIT without ORDER BY: stops reading once it has its rows. Unread batches are skipped, and ones already running are cancelled.
• ORDER BY … LIMIT n: keeps only the best n rows so far.
• GROUP BY and aggregates: keep a running total per group. Efsql.Aggregate is now new/2, add/2 and finish/1, and run/3 is unchanged.
• Filter and project: keep nothing.
• A sort with no limit: still keeps every row, as it has to.

The executor feeds every plan through a pipeline; a normal read is one chunk. Batches become a stream (Executor.stream_concurrently/3, replacing map_concurrently/3), so a halt stops fetching.

One behaviour change: sum/avg switch to Decimal at the first Decimal value, instead of converting every value when any is a Decimal. That only matters if one field mixes floats and Decimals, which an Ecto field can't.

Tests

• Property test: random rows, random operator chains shaped like the planner's, and random chunking must give exactly the same result as running the operators over all rows at once, early halts included. 500 cases per run, passing on 11 seeds.
• Unit tests:
  • every NOT translation;
  • evaluating the new conditions (NULL handling, case-insensitive and non-ASCII ILIKE) and the new rewrite;
  • pipeline halting, top-n memory and running aggregates;
  • a halted stream starting no further work.
• FoundationDB tests:
  • the negations and ILIKE on the seeded users;
  • a plan check that an index still serves >= while <> is filtered afterwards;
  • the primary-key message;
  • ORDER BY … LIMIT and LIMIT across real batches.

All 308 tests that don't need FoundationDB pass locally on the combined branch, with no new compiler warnings. This CI run is the first time the FoundationDB tests run on either change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PFBZUrerge6bbWVCHpS5Ko

---

Generated by Claude Code