test: the pooler soak's clients survive a chaos cancel that lands outside pg_warren

DESIGN.md

@@ -1926,7 +1926,7 @@ Test it by installing as a non-superuser in CI and running the suite. **Make the
     - Not run: the large tier. *Deferred (owner, 2026-10-25), then moved to M12 (owner, 2026-10-26),* not dropped: keepset's fetch holds a dataset in memory, so the large datasets don't download on the development machine. They are performance runs, not on the critical path.
   - A head-to-head with igraph and Neo4j (with and without GDS) on the same machine, on graphs from the same suite, published with its caveats. *Done (M11):* `suite.py head-to-head` (engines behind a port, the runner tested against a fake engine), run on the medium tier and four tiny graphs; results, caveats and where pg_warren loses in §12 *Head-to-head* and `bench/results/corpus/head_to_head/`. The large tier's head-to-head waits on the large tier (M12).
 - Robustness:
-  - a soak of several days behind a connection pooler, watching for memory growth. *The harness and CI are done; the multi-day run is not* (`test/pooler_soak.py`, `test/harness/pooler_soak/`; CI on PG 16, preloaded, 2 min, about 2.5 min with setup). PgBouncer in transaction and in session pooling at once, each a login role in the graphs' owner role with 48 clients over 8 server connections (a pool hands out its last released connection first, so a few backends see most clients); the clients mix traversals on a `bigint` and a `text` graph, `last_stats()` in the traversal's transaction, transient vectors on a version named by id, persisted vectors written, read and dropped, prepared statements (SQL `PREPARE` in session pooling, the protocol's unnamed statement in transaction pooling), introspection and calls cut short by `statement_timeout`; writers edit the source and refresh, stage then promote or drop, and compact, so versions retire under the readers; a chaos thread cancels calls and makes clients disconnect idle in a transaction or in a call. Sampled: each pooled backend's `smaps_rollup`, its memory contexts' total (`pg_log_backend_memory_contexts`, read back from the log) and its `backends()` private and transient bytes; the shared cache, the DSM segments the cluster maps, the activity rows, the progress rows and those that outlived their call (a row whose command began before its backend last went idle), the versions, segments, vectors and chunks and the catalog tables' sizes; and the poolers' `SHOW` counters. A series *grew* when its robust slope (Theil–Sen) over the steady window, after the warm-up, and over each half of it, adds more than its tolerance (an absolute amount plus a fraction of its level; it does not scale with time, so a longer run is stricter); a backend's warm-up starts at its first sample, and each pool is also judged by the median of its backends' slopes, since pools replace the server connections clients broke. Before the clock starts, every server connection of each pool, pinned by an open transaction, runs every kind of call the clients make three times (`warm.py`): a backend's anonymous memory is a high-water mark that rises in steps the first few times it runs each kind of call (a first `refresh` about 4 MB, the next ones less) and is then flat, and only one client in 48 writes, so without it a backend first served the writer well into the steady window and its warm-up read as growth (CI runs 40372279107, a session backend, 6.0 MB against 5.2 allowed; in 6 local runs (PG 16 and 18) the most a backend grew was 3.5–6.8 MB, in 7 runs after warming 0.1–1.1 MB). A backend the pool opens later, in place of one a client broke, starts cold and keeps its own warm-up. The progress rows are reported, not judged: they count the calls in flight as each sample is taken, about 5, a gauge of the load whose slope over 18 samples exceeded a bound of 2 in up to one run in twenty with no trend (0.1–5% across 11 local runs, by shuffling each run's samples; CI run 40390992808); a progress row that outlives its call is what would leak, and none is allowed, in a sample or once the clients stop. It also fails on an SQLSTATE an operation does not allow, on anything left once clients and then poolers stop (a progress row that outlived its call, activity and progress rows, pins, staged versions, orphans, retention, the DSM segments, and, after `evict`, the shared bytes), on session state that session pooling loses or leaks, and on trouble in the server log; it writes `samples.csv` as it goes and `summary.json`. It probes what each mode keeps of a session: transaction pooling loses a `SET`, a transient vector, a SQL prepared statement and `last_stats()` from one transaction to the next and leaks them to other clients, and session pooling carries them and, through `DISCARD ALL`, leaks nothing (docs/limits.md, "Behind a connection pooler"). *Runs:* the CI mode on PG 16 and 18 (debug builds, 2 min); and 2.5 h on PG 16 (debug, 64 clients per pool, 0.2 s think time, a 30-minute warm-up): 7.0M statements and transactions, every failure an allowed one, and over the 7,140 s steady window the median pooled backend's anonymous memory grew 0.26 MB/h (session) and 0.40 MB/h (transaction) at 15.5 MB, its memory contexts and pg_warren's private and transient bytes were flat, and the shared cache, DSM segments, versions (5), segments (45), vectors and the catalog's sizes were flat or fell. *Found and fixed:* `DISCARD ALL` left a backend's transient vectors and `last_stats()` (§6.2), so in session pooling the next client could read them and they stayed until the backend exited (about 10 MB per session backend in the first runs); and a persisted vector dropped on another backend stayed in the cache of the backend that wrote it whenever the shared cache served that backend's reads, since only a private load pruned (§5.5), so a transaction-pooled backend gained about 8 MB a minute, up to `warren.max_cache_memory`. The harness's own clients first left a persisted vector behind when a cancel kept a persist from its drop or a compaction moved the vector to the compacted version (§7), each keeping its version from retention; an application that drops a vector by the version it wrote has the same hazard. *Moved to M12 (owner, 2026-10-26):* the multi-day run, on the owner's machine. A GitHub-hosted job stops after 6 hours, so it needs a machine of our own, running `--hours 48` (or longer) on a release build in a private tree with `--no-lock`, its `summary.json` and the slopes recorded here. Chained 6-hour scheduled jobs on hosted runners would not be one run: each starts a new cluster and new backends, so no backend lives long enough to show a slow leak;
+  - a soak of several days behind a connection pooler, watching for memory growth. *The harness and CI are done; the multi-day run is not* (`test/pooler_soak.py`, `test/harness/pooler_soak/`; CI on PG 16, preloaded, 2 min, about 2.5 min with setup). PgBouncer in transaction and in session pooling at once, each a login role in the graphs' owner role with 48 clients over 8 server connections (a pool hands out its last released connection first, so a few backends see most clients); the clients mix traversals on a `bigint` and a `text` graph, `last_stats()` in the traversal's transaction, transient vectors on a version named by id, persisted vectors written, read and dropped, prepared statements (SQL `PREPARE` in session pooling, the protocol's unnamed statement in transaction pooling), introspection and calls cut short by `statement_timeout`; writers edit the source and refresh, stage then promote or drop, and compact, so versions retire under the readers; a chaos thread cancels calls and makes clients disconnect idle in a transaction or in a call. Sampled: each pooled backend's `smaps_rollup`, its memory contexts' total (`pg_log_backend_memory_contexts`, read back from the log) and its `backends()` private and transient bytes; the shared cache, the DSM segments the cluster maps, the activity rows, the progress rows and those that outlived their call (a row whose command began before its backend last went idle), the versions, segments, vectors and chunks and the catalog tables' sizes; and the poolers' `SHOW` counters. A series *grew* when its robust slope (Theil–Sen) over the steady window, after the warm-up, and over each half of it, adds more than its tolerance (an absolute amount plus a fraction of its level; it does not scale with time, so a longer run is stricter); a backend's warm-up starts at its first sample, and each pool is also judged by the median of its backends' slopes, since pools replace the server connections clients broke. Before the clock starts, every server connection of each pool, pinned by an open transaction, runs every kind of call the clients make three times (`warm.py`): a backend's anonymous memory is a high-water mark that rises in steps the first few times it runs each kind of call (a first `refresh` about 4 MB, the next ones less) and is then flat, and only one client in 48 writes, so without it a backend first served the writer well into the steady window and its warm-up read as growth (CI runs 40372279107, a session backend, 6.0 MB against 5.2 allowed; in 6 local runs (PG 16 and 18) the most a backend grew was 3.5–6.8 MB, in 7 runs after warming 0.1–1.1 MB). A backend the pool opens later, in place of one a client broke, starts cold and keeps its own warm-up. The progress rows are reported, not judged: they count the calls in flight as each sample is taken, about 5, a gauge of the load whose slope over 18 samples exceeded a bound of 2 in up to one run in twenty with no trend (0.1–5% across 11 local runs, by shuffling each run's samples; CI run 40390992808); a progress row that outlives its call is what would leak, and none is allowed, in a sample or once the clients stop. It also fails on an SQLSTATE an operation does not allow, on anything left once clients and then poolers stop (a progress row that outlived its call, activity and progress rows, pins, staged versions, orphans, retention, the DSM segments, and, after `evict`, the shared bytes), on session state that session pooling loses or leaks, and on trouble in the server log; it writes `samples.csv` as it goes and `summary.json`. It probes what each mode keeps of a session: transaction pooling loses a `SET`, a transient vector, a SQL prepared statement and `last_stats()` from one transaction to the next and leaks them to other clients, and session pooling carries them and, through `DISCARD ALL`, leaks nothing (docs/limits.md, "Behind a connection pooler"). *Runs:* the CI mode on PG 16 and 18 (debug builds, 2 min); and 2.5 h on PG 16 (debug, 64 clients per pool, 0.2 s think time, a 30-minute warm-up): 7.0M statements and transactions, every failure an allowed one, and over the 7,140 s steady window the median pooled backend's anonymous memory grew 0.26 MB/h (session) and 0.40 MB/h (transaction) at 15.5 MB, its memory contexts and pg_warren's private and transient bytes were flat, and the shared cache, DSM segments, versions (5), segments (45), vectors and the catalog's sizes were flat or fell. *Found and fixed:* `DISCARD ALL` left a backend's transient vectors and `last_stats()` (§6.2), so in session pooling the next client could read them and they stayed until the backend exited (about 10 MB per session backend in the first runs); and a persisted vector dropped on another backend stayed in the cache of the backend that wrote it whenever the shared cache served that backend's reads, since only a private load pruned (§5.5), so a transaction-pooled backend gained about 8 MB a minute, up to `warren.max_cache_memory`. The harness's own clients first left a persisted vector behind when a cancel kept a persist from its drop or a compaction moved the vector to the compacted version (§7), each keeping its version from retention; an application that drops a vector by the version it wrote has the same hazard. And a cancel acts on a backend, not a statement: a call chaos found could end before the cancel arrived, which then stopped the next statement there, in transaction pooling another client's session setup, which failed that client; a client now runs its own setup, `BEGIN` and `ROLLBACK` again after such a cancel, counted as a chaos event, while a cancel anywhere else is still its operation's outcome. *Moved to M12 (owner, 2026-10-26):* the multi-day run, on the owner's machine. A GitHub-hosted job stops after 6 hours, so it needs a machine of our own, running `--hours 48` (or longer) on a release build in a private tree with `--no-lock`, its `summary.json` and the slopes recorded here. Chained 6-hour scheduled jobs on hosted runners would not be one run: each starts a new cluster and new backends, so no backend lives long enough to show a slow leak;
   - many backends refreshing and querying one graph at once. *Done* (`test/concurrent_refresh.py`, CI on PG 16, 20 s preloaded and 20 s at the default configuration): 8 readers (`reachable_set`, `bfs`, `neighbours`, `dijkstra`, `walk`, `degree`, on the current version or one named by id; `spmv` and `pagerank` persisted, read back and dropped; `cache()`, `shared_cache()`, `backends()`, `progress()`), 3 refreshers (source edits, `refresh` alone and in the edit's transaction, rolled back, staged then promoted or dropped, `compact`; one with a `lock_timeout`) and a backend that writes and drops persisted vectors. The source moves through numbered generations, each a deterministic edit of the last, and the graph's queries add a marker node that names the generation, so every read is checked against an oracle of the generation its version was built from, read in the same statement. Checked: every read matches some complete version; every failure is a documented one, by SQLSTATE and message (a version retired after its id was read, 42704; a lock timeout, 55P03; a staged version superseded by a compaction, 55000; a vector moved by a compaction meanwhile, 42704 or 40001); afterwards no backend left in `backends()`, nothing pinned or mapped in the shared cache and, after `evict`, no more shared bytes than at the start, a fresh backend's private memory 0, the current version the one the last committed promotion made, no staged version, vector or partial segment left, every remaining version matching its generation, retention back to `retain_versions` after a refresh, and nothing in the server log but those SQLSTATEs. *Found and fixed:* a vector written to a version retention deleted meanwhile failed with a raw foreign-key violation (now 42704, §7); `compact` failed with `data_corrupted` when a vector it was moving was dropped meanwhile (now skipped, §5.2); and, preloaded, a read that revived a shared version another backend was evicting failed with an internal error, "cannot pin a segment that is already pinned" (the segment is now unpinned under the registry's lock, §5.6). 240 s preloaded and 45 s in each mode then ran clean;
   - physical streaming replicas, which §5.1 claims and nothing tested. *Done* (`test/replicas.py`, CI on PG 16, preloaded and at the default configuration, about 25 s; also run on PG 18): a primary, a hot standby made by `pg_basebackup -R` and a cascading standby of the standby. On the primary: graphs with `bigint` and `text` ids, weighted, several relations, directed and undirected; staged, promoted, dropped and compacted versions; persisted, dropped and moved vectors (`pagerank`, `spmv`, `eigen`, a matrix), Louvain, a contraction and a projection; an owner, a reader and a quota. On each standby, once it has replayed the primary's WAL (replay LSN at least the primary's flush LSN): every read gives the same rows as on the primary (traversals, `walk`, `dijkstra`, `components`, vector and matrix reads, `diff`, `distance`, `modularity`, similarity, `info`, and the compute that persists nothing: transient vectors from every function that has `persist`, `gather`, `eigen` without stored eigenvectors), a reader reads and a stranger is refused (42501); the server's own `cache()`, `shared_cache()`, `backends()`, `progress()`, `report_progress`, `prewarm`, `evict` and `last_stats` work; two backends share a version where the cluster shares, and `evict` leaves nothing pinned; every function that writes (a test fails on any function of the SQL surface that is neither read nor refused) is refused with 25006 and pg_warren's message. A version retired on the primary (retention after a refresh, then `VACUUM`) while one standby session holds it in an open REPEATABLE READ transaction and another reads it in a running statement: each read is the old version's answer or cancelled by the recovery conflict (40001), with and without `hot_standby_feedback`; once replayed, a new statement does not find it (42704) and reads the new one as the primary does. Then the standby is promoted: its graphs and vectors read as the primary's, it refreshes and persists a vector, and the cascade follows it. The server logs hold only refusals, the stranger's 42501, the 42704, recovery conflicts (40001, 25P02) and walreceivers ended at promotion or shutdown (57P01). *Found:* every write on a standby failed with an internal error (XX000, "cannot assign TransactionIds during recovery"); now 25006 (§5.1, *Standbys*). §5.1 now says what a standby can and cannot do;
   - crash recovery during `refresh`. *Done* (`test/crash_recovery.py`, CI on PG 16, preloaded and at the default configuration, about 75 s): a refresh of a 200K-node, 2M-edge graph is stopped waiting to read its edges (its edge query takes a shared advisory lock the test holds), while it writes its segments (the segments table has grown past 1 MB; where another backend's progress shows, its phase is checked to be `building` or `writing`) and after it returned, before its commit; each by `pg_terminate_backend`, by SIGKILL of the backend (the postmaster reinitialises shared memory, the registry with it) and by an immediate shutdown and restart (WAL recovery). No test-only hook was needed. A version is built in its refresh's transaction (§5.1), so a staged version left by a crash is rolled back with it, and nothing needs cleaning up. Checked after each: the catalog (versions, segments by version and name, vectors and their chunks, the current version) and the graph's answers exactly as before; the interrupted version not recorded; no row of the stopped backend in `backends()` or `progress()`, and nothing pinned; the shared cache loads the current version again and another backend reads it from there; the server log holds only what the interruption explains (57P01, 57P02, 57P03, the killed backend's signal 9); and after each way, a refresh succeeds and builds the source (its distinct edges counted by sorting: a parallel HashAggregate of the unanalysed edge rows now and then spilled into some 10^5 batches and ran past the statement timeout, on PG 16 and 18 with or without pg_warren). Found nothing. *Extended (0.9.0, §13 0.9.0 "Lifecycle"):* `pagerank` stopped while it iterates, and a read stopped while it loads the current version (into the shared cache where the cluster shares; for a terminate or a cancel, with a second reader waiting on that load), and `pg_cancel_backend` at every point where a statement runs. A stopped compute leaves no vector. A cancelled call fails with 57014, and its backend then reads and lists no progress. The waiting reader loads the version itself and reads it. Run on PG 15–18 (CI's lifecycle job). Found nothing;