diff --git a/DESIGN.md b/DESIGN.md index a2b71aa8..9b5e7af4 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -1926,7 +1926,7 @@ Test it by installing as a non-superuser in CI and running the suite. **Make the - Not run: the large tier. *Deferred (owner, 2026-10-25), then moved to M12 (owner, 2026-10-26),* not dropped: keepset's fetch holds a dataset in memory, so the large datasets don't download on the development machine. They are performance runs, not on the critical path. - A head-to-head with igraph and Neo4j (with and without GDS) on the same machine, on graphs from the same suite, published with its caveats. *Done (M11):* `suite.py head-to-head` (engines behind a port, the runner tested against a fake engine), run on the medium tier and four tiny graphs; results, caveats and where pg_warren loses in §12 *Head-to-head* and `bench/results/corpus/head_to_head/`. The large tier's head-to-head waits on the large tier (M12). - Robustness: - - a soak of several days behind a connection pooler, watching for memory growth. *The harness and CI are done; the multi-day run is not* (`test/pooler_soak.py`, `test/harness/pooler_soak/`; CI on PG 16, preloaded, 2 min, about 2.5 min with setup). PgBouncer in transaction and in session pooling at once, each a login role in the graphs' owner role with 48 clients over 8 server connections (a pool hands out its last released connection first, so a few backends see most clients); the clients mix traversals on a `bigint` and a `text` graph, `last_stats()` in the traversal's transaction, transient vectors on a version named by id, persisted vectors written, read and dropped, prepared statements (SQL `PREPARE` in session pooling, the protocol's unnamed statement in transaction pooling), introspection and calls cut short by `statement_timeout`; writers edit the source and refresh, stage then promote or drop, and compact, so versions retire under the readers; a chaos thread cancels calls and makes clients disconnect idle in a transaction or in a call. Sampled: each pooled backend's `smaps_rollup`, its memory contexts' total (`pg_log_backend_memory_contexts`, read back from the log) and its `backends()` private and transient bytes; the shared cache, the DSM segments the cluster maps, the activity rows, the progress rows and those that outlived their call (a row whose command began before its backend last went idle), the versions, segments, vectors and chunks and the catalog tables' sizes; and the poolers' `SHOW` counters. A series *grew* when its robust slope (Theil–Sen) over the steady window, after the warm-up, and over each half of it, adds more than its tolerance (an absolute amount plus a fraction of its level; it does not scale with time, so a longer run is stricter); a backend's warm-up starts at its first sample, and each pool is also judged by the median of its backends' slopes, since pools replace the server connections clients broke. Before the clock starts, every server connection of each pool, pinned by an open transaction, runs every kind of call the clients make three times (`warm.py`): a backend's anonymous memory is a high-water mark that rises in steps the first few times it runs each kind of call (a first `refresh` about 4 MB, the next ones less) and is then flat, and only one client in 48 writes, so without it a backend first served the writer well into the steady window and its warm-up read as growth (CI runs 40372279107, a session backend, 6.0 MB against 5.2 allowed; in 6 local runs (PG 16 and 18) the most a backend grew was 3.5–6.8 MB, in 7 runs after warming 0.1–1.1 MB). A backend the pool opens later, in place of one a client broke, starts cold and keeps its own warm-up. The progress rows are reported, not judged: they count the calls in flight as each sample is taken, about 5, a gauge of the load whose slope over 18 samples exceeded a bound of 2 in up to one run in twenty with no trend (0.1–5% across 11 local runs, by shuffling each run's samples; CI run 40390992808); a progress row that outlives its call is what would leak, and none is allowed, in a sample or once the clients stop. It also fails on an SQLSTATE an operation does not allow, on anything left once clients and then poolers stop (a progress row that outlived its call, activity and progress rows, pins, staged versions, orphans, retention, the DSM segments, and, after `evict`, the shared bytes), on session state that session pooling loses or leaks, and on trouble in the server log; it writes `samples.csv` as it goes and `summary.json`. It probes what each mode keeps of a session: transaction pooling loses a `SET`, a transient vector, a SQL prepared statement and `last_stats()` from one transaction to the next and leaks them to other clients, and session pooling carries them and, through `DISCARD ALL`, leaks nothing (docs/limits.md, "Behind a connection pooler"). *Runs:* the CI mode on PG 16 and 18 (debug builds, 2 min); and 2.5 h on PG 16 (debug, 64 clients per pool, 0.2 s think time, a 30-minute warm-up): 7.0M statements and transactions, every failure an allowed one, and over the 7,140 s steady window the median pooled backend's anonymous memory grew 0.26 MB/h (session) and 0.40 MB/h (transaction) at 15.5 MB, its memory contexts and pg_warren's private and transient bytes were flat, and the shared cache, DSM segments, versions (5), segments (45), vectors and the catalog's sizes were flat or fell. *Found and fixed:* `DISCARD ALL` left a backend's transient vectors and `last_stats()` (§6.2), so in session pooling the next client could read them and they stayed until the backend exited (about 10 MB per session backend in the first runs); and a persisted vector dropped on another backend stayed in the cache of the backend that wrote it whenever the shared cache served that backend's reads, since only a private load pruned (§5.5), so a transaction-pooled backend gained about 8 MB a minute, up to `warren.max_cache_memory`. The harness's own clients first left a persisted vector behind when a cancel kept a persist from its drop or a compaction moved the vector to the compacted version (§7), each keeping its version from retention; an application that drops a vector by the version it wrote has the same hazard. *Moved to M12 (owner, 2026-10-26):* the multi-day run, on the owner's machine. A GitHub-hosted job stops after 6 hours, so it needs a machine of our own, running `--hours 48` (or longer) on a release build in a private tree with `--no-lock`, its `summary.json` and the slopes recorded here. Chained 6-hour scheduled jobs on hosted runners would not be one run: each starts a new cluster and new backends, so no backend lives long enough to show a slow leak; + - a soak of several days behind a connection pooler, watching for memory growth. *The harness and CI are done; the multi-day run is not* (`test/pooler_soak.py`, `test/harness/pooler_soak/`; CI on PG 16, preloaded, 2 min, about 2.5 min with setup). PgBouncer in transaction and in session pooling at once, each a login role in the graphs' owner role with 48 clients over 8 server connections (a pool hands out its last released connection first, so a few backends see most clients); the clients mix traversals on a `bigint` and a `text` graph, `last_stats()` in the traversal's transaction, transient vectors on a version named by id, persisted vectors written, read and dropped, prepared statements (SQL `PREPARE` in session pooling, the protocol's unnamed statement in transaction pooling), introspection and calls cut short by `statement_timeout`; writers edit the source and refresh, stage then promote or drop, and compact, so versions retire under the readers; a chaos thread cancels calls and makes clients disconnect idle in a transaction or in a call. Sampled: each pooled backend's `smaps_rollup`, its memory contexts' total (`pg_log_backend_memory_contexts`, read back from the log) and its `backends()` private and transient bytes; the shared cache, the DSM segments the cluster maps, the activity rows, the progress rows and those that outlived their call (a row whose command began before its backend last went idle), the versions, segments, vectors and chunks and the catalog tables' sizes; and the poolers' `SHOW` counters. A series *grew* when its robust slope (Theil–Sen) over the steady window, after the warm-up, and over each half of it, adds more than its tolerance (an absolute amount plus a fraction of its level; it does not scale with time, so a longer run is stricter); a backend's warm-up starts at its first sample, and each pool is also judged by the median of its backends' slopes, since pools replace the server connections clients broke. Before the clock starts, every server connection of each pool, pinned by an open transaction, runs every kind of call the clients make three times (`warm.py`): a backend's anonymous memory is a high-water mark that rises in steps the first few times it runs each kind of call (a first `refresh` about 4 MB, the next ones less) and is then flat, and only one client in 48 writes, so without it a backend first served the writer well into the steady window and its warm-up read as growth (CI runs 40372279107, a session backend, 6.0 MB against 5.2 allowed; in 6 local runs (PG 16 and 18) the most a backend grew was 3.5–6.8 MB, in 7 runs after warming 0.1–1.1 MB). A backend the pool opens later, in place of one a client broke, starts cold and keeps its own warm-up. The progress rows are reported, not judged: they count the calls in flight as each sample is taken, about 5, a gauge of the load whose slope over 18 samples exceeded a bound of 2 in up to one run in twenty with no trend (0.1–5% across 11 local runs, by shuffling each run's samples; CI run 40390992808); a progress row that outlives its call is what would leak, and none is allowed, in a sample or once the clients stop. It also fails on an SQLSTATE an operation does not allow, on anything left once clients and then poolers stop (a progress row that outlived its call, activity and progress rows, pins, staged versions, orphans, retention, the DSM segments, and, after `evict`, the shared bytes), on session state that session pooling loses or leaks, and on trouble in the server log; it writes `samples.csv` as it goes and `summary.json`. It probes what each mode keeps of a session: transaction pooling loses a `SET`, a transient vector, a SQL prepared statement and `last_stats()` from one transaction to the next and leaks them to other clients, and session pooling carries them and, through `DISCARD ALL`, leaks nothing (docs/limits.md, "Behind a connection pooler"). *Runs:* the CI mode on PG 16 and 18 (debug builds, 2 min); and 2.5 h on PG 16 (debug, 64 clients per pool, 0.2 s think time, a 30-minute warm-up): 7.0M statements and transactions, every failure an allowed one, and over the 7,140 s steady window the median pooled backend's anonymous memory grew 0.26 MB/h (session) and 0.40 MB/h (transaction) at 15.5 MB, its memory contexts and pg_warren's private and transient bytes were flat, and the shared cache, DSM segments, versions (5), segments (45), vectors and the catalog's sizes were flat or fell. *Found and fixed:* `DISCARD ALL` left a backend's transient vectors and `last_stats()` (§6.2), so in session pooling the next client could read them and they stayed until the backend exited (about 10 MB per session backend in the first runs); and a persisted vector dropped on another backend stayed in the cache of the backend that wrote it whenever the shared cache served that backend's reads, since only a private load pruned (§5.5), so a transaction-pooled backend gained about 8 MB a minute, up to `warren.max_cache_memory`. The harness's own clients first left a persisted vector behind when a cancel kept a persist from its drop or a compaction moved the vector to the compacted version (§7), each keeping its version from retention; an application that drops a vector by the version it wrote has the same hazard. And a cancel acts on a backend, not a statement: a call chaos found could end before the cancel arrived, which then stopped the next statement there, in transaction pooling another client's session setup, which failed that client; a client now runs its own setup, `BEGIN` and `ROLLBACK` again after such a cancel, counted as a chaos event, while a cancel anywhere else is still its operation's outcome. *Moved to M12 (owner, 2026-10-26):* the multi-day run, on the owner's machine. A GitHub-hosted job stops after 6 hours, so it needs a machine of our own, running `--hours 48` (or longer) on a release build in a private tree with `--no-lock`, its `summary.json` and the slopes recorded here. Chained 6-hour scheduled jobs on hosted runners would not be one run: each starts a new cluster and new backends, so no backend lives long enough to show a slow leak; - many backends refreshing and querying one graph at once. *Done* (`test/concurrent_refresh.py`, CI on PG 16, 20 s preloaded and 20 s at the default configuration): 8 readers (`reachable_set`, `bfs`, `neighbours`, `dijkstra`, `walk`, `degree`, on the current version or one named by id; `spmv` and `pagerank` persisted, read back and dropped; `cache()`, `shared_cache()`, `backends()`, `progress()`), 3 refreshers (source edits, `refresh` alone and in the edit's transaction, rolled back, staged then promoted or dropped, `compact`; one with a `lock_timeout`) and a backend that writes and drops persisted vectors. The source moves through numbered generations, each a deterministic edit of the last, and the graph's queries add a marker node that names the generation, so every read is checked against an oracle of the generation its version was built from, read in the same statement. Checked: every read matches some complete version; every failure is a documented one, by SQLSTATE and message (a version retired after its id was read, 42704; a lock timeout, 55P03; a staged version superseded by a compaction, 55000; a vector moved by a compaction meanwhile, 42704 or 40001); afterwards no backend left in `backends()`, nothing pinned or mapped in the shared cache and, after `evict`, no more shared bytes than at the start, a fresh backend's private memory 0, the current version the one the last committed promotion made, no staged version, vector or partial segment left, every remaining version matching its generation, retention back to `retain_versions` after a refresh, and nothing in the server log but those SQLSTATEs. *Found and fixed:* a vector written to a version retention deleted meanwhile failed with a raw foreign-key violation (now 42704, §7); `compact` failed with `data_corrupted` when a vector it was moving was dropped meanwhile (now skipped, §5.2); and, preloaded, a read that revived a shared version another backend was evicting failed with an internal error, "cannot pin a segment that is already pinned" (the segment is now unpinned under the registry's lock, §5.6). 240 s preloaded and 45 s in each mode then ran clean; - physical streaming replicas, which §5.1 claims and nothing tested. *Done* (`test/replicas.py`, CI on PG 16, preloaded and at the default configuration, about 25 s; also run on PG 18): a primary, a hot standby made by `pg_basebackup -R` and a cascading standby of the standby. On the primary: graphs with `bigint` and `text` ids, weighted, several relations, directed and undirected; staged, promoted, dropped and compacted versions; persisted, dropped and moved vectors (`pagerank`, `spmv`, `eigen`, a matrix), Louvain, a contraction and a projection; an owner, a reader and a quota. On each standby, once it has replayed the primary's WAL (replay LSN at least the primary's flush LSN): every read gives the same rows as on the primary (traversals, `walk`, `dijkstra`, `components`, vector and matrix reads, `diff`, `distance`, `modularity`, similarity, `info`, and the compute that persists nothing: transient vectors from every function that has `persist`, `gather`, `eigen` without stored eigenvectors), a reader reads and a stranger is refused (42501); the server's own `cache()`, `shared_cache()`, `backends()`, `progress()`, `report_progress`, `prewarm`, `evict` and `last_stats` work; two backends share a version where the cluster shares, and `evict` leaves nothing pinned; every function that writes (a test fails on any function of the SQL surface that is neither read nor refused) is refused with 25006 and pg_warren's message. A version retired on the primary (retention after a refresh, then `VACUUM`) while one standby session holds it in an open REPEATABLE READ transaction and another reads it in a running statement: each read is the old version's answer or cancelled by the recovery conflict (40001), with and without `hot_standby_feedback`; once replayed, a new statement does not find it (42704) and reads the new one as the primary does. Then the standby is promoted: its graphs and vectors read as the primary's, it refreshes and persists a vector, and the cascade follows it. The server logs hold only refusals, the stranger's 42501, the 42704, recovery conflicts (40001, 25P02) and walreceivers ended at promotion or shutdown (57P01). *Found:* every write on a standby failed with an internal error (XX000, "cannot assign TransactionIds during recovery"); now 25006 (§5.1, *Standbys*). §5.1 now says what a standby can and cannot do; - crash recovery during `refresh`. *Done* (`test/crash_recovery.py`, CI on PG 16, preloaded and at the default configuration, about 75 s): a refresh of a 200K-node, 2M-edge graph is stopped waiting to read its edges (its edge query takes a shared advisory lock the test holds), while it writes its segments (the segments table has grown past 1 MB; where another backend's progress shows, its phase is checked to be `building` or `writing`) and after it returned, before its commit; each by `pg_terminate_backend`, by SIGKILL of the backend (the postmaster reinitialises shared memory, the registry with it) and by an immediate shutdown and restart (WAL recovery). No test-only hook was needed. A version is built in its refresh's transaction (§5.1), so a staged version left by a crash is rolled back with it, and nothing needs cleaning up. Checked after each: the catalog (versions, segments by version and name, vectors and their chunks, the current version) and the graph's answers exactly as before; the interrupted version not recorded; no row of the stopped backend in `backends()` or `progress()`, and nothing pinned; the shared cache loads the current version again and another backend reads it from there; the server log holds only what the interruption explains (57P01, 57P02, 57P03, the killed backend's signal 9); and after each way, a refresh succeeds and builds the source (its distinct edges counted by sorting: a parallel HashAggregate of the unanalysed edge rows now and then spilled into some 10^5 batches and ran past the statement timeout, on PG 16 and 18 with or without pg_warren). Found nothing. *Extended (0.9.0, §13 0.9.0 "Lifecycle"):* `pagerank` stopped while it iterates, and a read stopped while it loads the current version (into the shared cache where the cluster shares; for a terminate or a cancel, with a second reader waiting on that load), and `pg_cancel_backend` at every point where a statement runs. A stopped compute leaves no vector. A cancelled call fails with 57014, and its backend then reads and lists no progress. The waiting reader loads the version itself and reads it. Run on PG 15–18 (CI's lifecycle job). Found nothing; diff --git a/test/harness/pooler_soak/workload.py b/test/harness/pooler_soak/workload.py index 145a196b..247fe6ee 100644 --- a/test/harness/pooler_soak/workload.py +++ b/test/harness/pooler_soak/workload.py @@ -19,7 +19,15 @@ Chaos (`Chaos`), once per period per pool: `pg_cancel_backend` of a pooled backend running a pg_warren call, or a client that disconnects in a transaction, idle or in a call (the pooler then closes that server -connection). Every failure must be `allowed` for its operation.""" +connection). Every failure must be `allowed` for its operation. + +A cancel acts on a backend, not a statement: the call chaos found can end +before the cancel arrives, which then stops whatever that backend runs +next, under transaction pooling perhaps another client's. Where that is a +statement of the client's own that calls no pg_warren function and can be +run again (its session's setup, `BEGIN`, `ROLLBACK`), the cancel is a +chaos event and the statement runs again; anywhere else it is the +operation's outcome, as before.""" import random import re @@ -29,7 +37,7 @@ from dataclasses import dataclass, field from ..concurrent_refresh import LOCK_TIMED_OUT, MOVED, RETIRED, SUPERSEDED, WRITTEN_MEANWHILE, joined -from ..session import BackendLost, Session +from ..session import BackendLost, Session, canceled_by_request from .fixture import DATABASE, edit_sql CANCELED = ("57014", r"^canceling statement due to (user request|statement timeout)$") @@ -48,6 +56,8 @@ CARELESS = 0.3 PERSISTED_NAMES_PER_CLIENT = 2 # p and to; a client reuses them STAGED_DROP_ATTEMPTS = 10 +STRAY_CANCEL = "cancel stopped a statement outside pg_warren" +STRAY_CANCEL_ATTEMPTS = 5 def allowed(operation, sqlstate, message): @@ -153,6 +163,8 @@ def _client(self, number): def _connection(self, rng, number): """One client connection: a few dozen transactions, or fewer if chaos asks it to abandon one.""" s = Session(self.pooler, DATABASE, user=self.role) + for _ in range(s.canceled_setup): + self.tally.event(self.mode, STRAY_CANCEL) prepared = False try: for _ in range(rng.randint(10, 60)): @@ -182,20 +194,38 @@ def _record(self, operation, done): def _transaction(self, s, operation, statements, end="COMMIT"): """Runs `statements` in one transaction, ended by `end`; their outcomes, or None if one failed (recorded; the transaction rolled - back).""" - s.run("BEGIN") + back). A COMMIT is not run again: once canceled, its transaction + is aborted, and a second COMMIT would roll it back and succeed.""" + begun = self._again(s, "BEGIN") + if not begun.ok: + self._record(operation, begun) + return None outcomes = [] for sql in statements: done = s.run(sql) if not done.ok: self._record(operation, done) - s.run("ROLLBACK") + self._again(s, "ROLLBACK") return None outcomes.append(done) - done = s.run(end) + done = self._again(s, end) if end == "ROLLBACK" else s.run(end) self._record(operation, done) + if not done.ok: + self._again(s, "ROLLBACK") return outcomes if done.ok else None + def _again(self, s, sql): + """Runs `sql`, a statement of the client's own that calls no + pg_warren function and can be run again; its outcome. A cancel + meant for an earlier call on the same backend can stop it: that is + a chaos event, and `sql` runs again (up to `STRAY_CANCEL_ATTEMPTS` times).""" + for _ in range(STRAY_CANCEL_ATTEMPTS): + done = s.run(sql) + if not canceled_by_request(done): + break + self.tally.event(self.mode, STRAY_CANCEL) + return done + def _current(self, s, operation, graph="g"): done = s.run(f"SELECT warren.info('{graph}') ->> 'current_version'") return done.value() if self._record(operation, done) else None diff --git a/test/harness/session.py b/test/harness/session.py index 72b6fa69..b68c72f6 100644 --- a/test/harness/session.py +++ b/test/harness/session.py @@ -25,6 +25,8 @@ SESSION_SETTINGS = ("client_min_messages = warning", "statement_timeout = '120s'", "client_connection_check_interval = '1s'") CLOSE_SECONDS = 10 +CANCELED_BY_REQUEST = ("57014", "canceling statement due to user request") +SETUP_ATTEMPTS = 5 class BackendLost(RuntimeError): @@ -51,6 +53,12 @@ def value(self): return self.rows[0][0] +def canceled_by_request(done): + """Whether `done` failed because its backend was asked to cancel + (`pg_cancel_backend`), not by a timeout.""" + return (done.sqlstate, done.message.strip()) == CANCELED_BY_REQUEST + + def script(sql): """The text that runs `sql` and then prints its outcome's markers. A statement that a meta-command sends (`... \\bind 1 \\g`, by the @@ -80,15 +88,30 @@ def __init__(self, cluster, database, settings=(), user=SUPERUSER): # Notices go to stderr, which nothing reads: discarded, so it never fills. self.process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, bufsize=1) + self.canceled_setup = 0 for setting in (*SESSION_SETTINGS, *settings): self._set(setting) - self.pid = int(self.run("SELECT pg_backend_pid()").value()) + self.pid = int(self._setup("SELECT pg_backend_pid()").value()) def _set(self, setting): - done = self.run(f"SET {setting}") + done = self._setup(f"SET {setting}") if not done.ok: raise RuntimeError(f"SET {setting}: {done.message}") + def _setup(self, sql): + """Runs `sql`, one of the session's own setup statements; its + outcome. Nothing cancels a setup statement on purpose, but + `pg_cancel_backend` cancels a backend, not a statement: through a + pooler, a cancel meant for another client's call can stop this + client's setup on the same backend. Such a statement is run again, + up to `SETUP_ATTEMPTS` times, and counted in `canceled_setup`.""" + for _ in range(SETUP_ATTEMPTS): + done = self.run(sql) + if not canceled_by_request(done): + break + self.canceled_setup += 1 + return done + def __enter__(self): return self diff --git a/test/tests/test_pooler_soak_workload.py b/test/tests/test_pooler_soak_workload.py index 7506df20..055a7556 100644 --- a/test/tests/test_pooler_soak_workload.py +++ b/test/tests/test_pooler_soak_workload.py @@ -92,3 +92,39 @@ def test_a_timeout_call_that_finishes_in_time_is_rolled_back_so_its_vector_keeps workload._timeout(session, random.Random(1), 0) assert session.ran[-1] == "ROLLBACK" assert "COMMIT" not in session.ran + + +STRAY = Outcome([], "57014", "canceling statement due to user request") + + +def test_a_cancel_that_stops_begin_or_rollback_is_an_event_and_the_statement_runs_again(): + """Chaos cancels a backend whose pg_warren call may just have ended: the + cancel then stops the next statement there, perhaps no pg_warren call.""" + tally = Tally() + workload = Workload(None, "transaction", "soak_transaction", 1, 0, 100, 1, tally, None, 0, 0) + session = ScriptedSession({"BEGIN": [STRAY], "ROLLBACK": [STRAY]}) + workload._timeout(session, random.Random(1), 0) + assert session.ran[:2] == ["BEGIN", "BEGIN"] + assert session.ran[-2:] == ["ROLLBACK", "ROLLBACK"] + assert tally.chaos[("transaction", "cancel stopped a statement outside pg_warren")] == 2 + assert tally.unexpected == [] and tally.mismatches == [] + + +def test_a_transaction_whose_begin_never_succeeds_runs_none_of_its_statements(): + tally = Tally() + workload = Workload(None, "session", "soak_session", 1, 0, 100, 1, tally, None, 0, 0) + session = ScriptedSession({"BEGIN": [Outcome([], "25001", "odd")]}) + assert workload._transaction(session, "stats", ["SELECT warren.last_stats()"]) is None + assert session.ran == ["BEGIN"] + assert tally.unexpected == [("session", "stats", "25001", "odd")] + + +def test_a_canceled_commit_is_the_operations_outcome_and_its_transaction_is_rolled_back(): + """A COMMIT is not run again: once canceled, its transaction is aborted + and a second COMMIT would roll it back and succeed.""" + tally = Tally() + workload = Workload(None, "session", "soak_session", 1, 0, 100, 1, tally, None, 0, 0) + session = ScriptedSession({"COMMIT": [STRAY]}) + assert workload._transaction(session, "stats", ["SELECT 1"]) is None + assert session.ran == ["BEGIN", "SELECT 1", "COMMIT", "ROLLBACK"] + assert tally.outcomes[("session", "stats", "57014")] == 1 diff --git a/test/tests/test_session.py b/test/tests/test_session.py index 1d63f08c..ea0cdfae 100644 --- a/test/tests/test_session.py +++ b/test/tests/test_session.py @@ -5,7 +5,7 @@ import pytest -from harness.session import END, MESSAGE, SESSION_SETTINGS, Outcome, Session, outcome, script +from harness.session import END, MESSAGE, SESSION_SETTINGS, Outcome, Session, canceled_by_request, outcome, script def test_the_script_runs_the_statement_and_then_echoes_both_markers(): @@ -55,3 +55,40 @@ def test_closing_a_session_whose_psql_does_not_end_kills_it(): session.close(seconds=0.2) assert session.process.returncode is not None assert time.monotonic() - started < 10 + + +CANCELED = Outcome([], "57014", "canceling statement due to user request") + + +def test_only_a_requested_cancel_is_canceled_by_request(): + assert canceled_by_request(CANCELED) + assert not canceled_by_request(Outcome([], "57014", "canceling statement due to statement timeout")) + assert not canceled_by_request(Outcome([["1"]], "00000", "")) + + +def scripted_session(outcomes): + """A session whose statements answer `outcomes` in turn, then succeed.""" + session = Session.__new__(Session) + session.canceled_setup, session.ran = 0, [] + + def run(sql): + session.ran.append(sql) + return outcomes.pop(0) if outcomes else Outcome([["4242"]], "00000", "") + session.run = run + return session + + +def test_a_setup_statement_stopped_by_a_cancel_meant_for_another_client_runs_again_and_is_counted(): + """Through a pooler, a cancel of another client's call can land on this + client's setup: pg_cancel_backend cancels a backend, not a statement.""" + session = scripted_session([CANCELED, CANCELED]) + session._set("statement_timeout = '120s'") + assert session.ran == ["SET statement_timeout = '120s'"] * 3 + assert session.canceled_setup == 2 + + +def test_a_setup_statement_that_fails_otherwise_still_fails_the_session(): + session = scripted_session([Outcome([], "22023", "invalid value")]) + with pytest.raises(RuntimeError, match="invalid value"): + session._set("statement_timeout = 'x'") + assert session.canceled_setup == 0