Skip to content

feat(store): v1 SQLite store with init migration, queries and pinned query plans (AFDEV-12) - #4

Merged
foae merged 2 commits into
mainfrom
feat/AFDEV-12-store-init-migration
Sep 28, 2026
Merged

foae merged 2 commits into
mainfrom
feat/AFDEV-12-store-init-migration

Conversation

@foae

@foae foae commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator

What

The v1 store: internal/store is the SQLite package the v1 API's core will run on.

  • internal/store/migrations/001_init.sql is the only migration: one submissions table for every kind, uid (UUIDv7) beside the integer id, the friction payload's category and fix_status as VIRTUAL generated columns, and ten indexes (keyed uniqueness on (kind, key), the hash + created_at pair for the keyless dedupe window, the open queue, and the filter columns).
  • internal/store/queries.go: inserts (API and id-preserving restore), lookups by id, uid, (kind, key) and the 24 h same-hash window that skips processed rows, List/Count with every filter of GET /api/v1/submissions and both cursors, the stats pieces (StatsTotals, StatsGroups for up to three keys, StatsRecurring, StatsSeries by day or ISO week), set-based mark/unmark, redaction tombstones, ascending export iteration.
  • internal/store/store.go: the pools and migration loader as before, plus a PRAGMA application_id stamp written before the first migration. A non-empty database without the stamp is refused at open with ErrForeignDatabase naming the path and the way out (a fresh DATABASE_PATH). Both the previous store and this one number their init migration 1, so the schema version alone cannot tell them apart; without the stamp a binary on the old file would fail on its first query instead of at start.
  • q is implemented as a registered deterministic SQLite function (af_lower, backed by pkg/schema.LowerSimple) over summary and every string value in the payload via json_tree, so "case-insensitive" means the same Unicode mapping the contract uses for tokens; keys, numbers and booleans are not searched.
  • internal/store/plan_test.go pins EXPLAIN QUERY PLAN for 45 statements: the queue and its pages, the identity lookups, every indexed list filter, q alone and with a filter, each stats grouping, recurring, series and export. Where the schema has an index the named index must appear and the table is not scanned; where it has none the scan is recorded as expected.

Coexistence with the running v3 service

The v3 service (internal/core, internal/api, cmd/agentfeedback) still compiles against the previous store, and scripts/e2e.sh exercises it in CI. That package moves unchanged to internal/v3/store (import paths only; its own tests move with it) so the new store can take the path the v1 layout names. The tickets that replace the core, HTTP and tooling packages remove internal/v3/ when its last consumer goes. The e2e suite proves the move changed nothing (55/55).

Docs and contract

  • docs/develop.md: layout lines for both stores, the write-once rule now names the processing fields and the tombstone, the migration rule names the stamp.
  • docs/openapi.yaml: the by and q descriptions state the rules the store enforces (rows without a grouping key are in no group; what q searches). Descriptions only; no shape change.

Out of scope, by design: identity decisions (200/201/409), warnings, the HTTP layer. The store exposes the unique index, the window lookup and the snapshot read; the core decides.

Verification

Locally, all green: just check with a clean tree, go test -race -count=1 ./..., just fuzz, live e2e 55/55, shellcheck (CI form and the develop.md form), skill tests 124/124, just contract (89 fixtures, Redocly valid), staticcheck 2026.2.1 clean.

Store tests cover: byte-exact payload and context round trips (spacing, member order, large integers, exponent spelling, U+FFFD, escapes), "" ⇔ NULL, keyed uniqueness and the uid index, the dedupe window (inside/outside, processed skipped, newest wins), every list filter with its count, both cursors including after_id=0, count and list inside one read snapshot, every grouping key and multi-key groups, numeric category grouped as text, recurring, day and week series with Sunday/Monday and year-turn boundaries, mark keep/replace/clear and unmark, tombstones (hash and identity kept, category gone, still markable), export ordering and filters, q from more reader connections than the pool holds, and the foreign-database and interrupted-first-open cases.

…query plans (AFDEV-12)

The v1 store at internal/store: single init migration with the application_id stamp, every query the v1 core needs (lookups, dedupe window, list with filters and cursors, stats, marks, tombstones, export), and a test that pins the query plans. The v3 store moves unchanged to internal/v3/store so the running service and its e2e gate stay green until the v1 core, HTTP and tooling packages replace them.
…p and plan checks (AFDEV-12)

Review findings: the stamp check and the schema_version bootstrap run in one IMMEDIATE transaction so concurrent first opens agree on one row; the series bucket floors microseconds to whole seconds instead of passing a fractional unixepoch that date() rounds to milliseconds (the last microsecond of a day crossed midnight and year 9999 became NULL); a foreign database is refused for any schema object, not only tables; the interrupted-first-open test writes the real stamp; the plan test matches whole index names; the q clause documents the driver's NUL limit and a test pins it.
@foae

foae commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator Author

Review round (cross-model panel, 4 reviewers) addressed in the second commit:

  • Fixed: the schema_version bootstrap and the stamp check now run in one IMMEDIATE transaction, so concurrent first opens of a fresh file agree on one row (test added). The series bucket floors microseconds to whole seconds: a fractional unixepoch is rounded to milliseconds by date(), which carried 23:59:59.999999 over midnight and turned 9999-12-31 into NULL (boundary tests added, including year 0001 and pre-1970). A foreign database is refused for any schema object, not only tables. The interrupted-first-open test now writes the real stamp value. The plan test matches whole index names.
  • Documented, not changed: the driver hands TEXT arguments to Go functions as C strings, so q does not see payload text after an embedded NUL; SQLite's replace() stops at the NUL too, so it cannot be masked in SQL. Pinned by a test. The previous major version's binary cannot detect a database of this version (its code is frozen and on its way out). The tombstone is stored compacted, as every payload is.

@foae
foae merged commit c5624f5 into main Sep 28, 2026
4 checks passed
@foae
foae deleted the feat/AFDEV-12-store-init-migration branch September 28, 2026 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant