[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework (merged) · PR 6 the catalog index · PR 7 configuration, the store identity file and backup purge · #700 the CreateBackup job · this PR: the restore job from the store, every backend, the cross-backend matrix and extenddb backup remove-target (the design's PR 9a and PR 9b as one PR, with PR 10's remove-target pulled forward) · next: PR 10 backup sync.
Closes #739 (part of #624).

On main at 3033e090, which carries #671 (merged as ff6f74eb), #672 (4a42c5e0) and #700 (d35d8601). The rebase after #700 merged had no conflicts; it adds main's #847 and #868. The rebase onto #700's head after #672 merged renumbers restore_progress to migration 8: main's #819 took 7 for PostgreSQL's stream_pending. The SQL file is 0008_restore_progress.sql. Each backend's RestoreProgress arm follows StreamPending, NotApplicable on SQLite, MongoDB and Cassandra as before. CONTRIBUTING's worked example (table_note) is now migration 9. The sample outputs say "migration 8". MongoDB's migration tests count one more not-applicable step and one more pending migration, and their later-binary row is id 9. The rebase conflicted in 13 files: the registry and the four backends' migration tables; SQLite's two migration tests and Cassandra's pending-migration cell, where main's general form is kept; CONTRIBUTING, the upgrade manual and design item 32 (#819's text and #700's round-four text beside this PR's); the admin guide; and heartbeat.rs. In heartbeat.rs, ClaimCheck now holds #700's flag and calls in_flight_confirms(in_flight, parks), so a restore's pre-write check takes #700's rule: a beat in flight confirms the claim on SQLite only. #700's new between-beats cell takes this PR's HeartbeatMetric::None argument. The rebase after #671's merge conflicted only in crates/storage-sqlite/src/lib.rs's module list, where main's #704 added mod password_hash_store; beside this PR's mod restore_engine; (both kept). (Rebased for this review round onto #700's current head; crates/storage/src/lib.rs took main's #776 pub mod secret_cipher; beside this PR's pub mod restore_engine;.) This PR adds restore_progress to the PostgreSQL data database (migration 8, not applicable on SQLite, MongoDB and Cassandra); docs/schema/postgres.txt carries the table and the other three schema files are unchanged, each checked against a fresh init. It binds PR 7's store, spool and health and PR 6's claim, row and reconciler to the restore: check_catalog_row_for_restore no longer refuses a store-backed backup; the request takes require_healthy_store and the job permit and registers the target under the backup row's lock; the reconciler's cycle gains the dead-restore resume and the restore_progress and spool sweeps beside #700's backup duties; the backup commands' catalog refusals and named registrations are PR 7's, which remove-target uses. Since the last push main merged #335, #666 and #681 under the stack: restore_progress is migration 7 (#335 took 5, PR 6 took 6) and is declared not applicable in #681's MongoDB and Cassandra registries, and main's migration tests that count the ids now count it; the reconciler runs in #666's restartable worker set, which a lost serving registration stops while the jobs go on, so a reconciler whose own token has fired spawns no resume (a resume is a job); a restore job meets #700's drain in #666's shutdown path as a backup job does; and remove-target releases its serving registration on exit and carries its name on MongoDB and Cassandra as well as PostgreSQL. The rebase onto #700's head conflicted in twelve files: cmd_backup.rs (PR 7's registration release on this PR's split of the catalog opening from the store's), the Cassandra crate's Cargo.toml and backup-engine imports, the MongoDB backup live tests' imports and the GSI backfill worker's doc, the PostgreSQL migration table (migration 7 beside main's 5), and the design, differences, limits, usage, admin and upgrade documents, each resolved by keeping both sides' intent, main's text where #335 removed point-in-time recovery; Cassandra's claim_restore no longer calls the catalog check #681 removed. The rebase onto the review rounds of #671, #672 and #700 conflicted in 23 files, resolved so that the three lower PRs' review changes stand beside this PR's: the three fail_restore_job writes of the resume arm (OWNER_DEAD, TARGET_NOT_EMPTY under the claim, OTHER_STORE) now pass the owner_generation and heartbeat_at the listing read, #671's fence, so a heartbeat that lands after the listing keeps the record its owner's (the engine's Fake applies the fence to its restore records); the restore request takes take_permit().await, #671's 30 s wait, before the manifest read; the target name is checked after the catalog row's state (and, on the legacy path, after the source resolves), #671's measured order, and before the backup's in-use rule and the manifest read (check_catalog_row_status, then check_restore_target_name, then check_catalog_row_for_restore; the_manifest_request_refuses_in_order_before_the_registration reads one row read and no store read before a taken name's refusal), so a restore of a backup into its own CREATING target is TableInUseException, as the fidelity suite measured on DynamoDB, and the same backup into another name BackupInUseException (a_backup_being_restored_is_in_use_until_its_target_is_active carries the three cases, with another backup into the name being restored into; a_store_backed_restore_checks_the_target_name_after_the_row_and_before_the_store is #700's cell adapted: the free name goes on to the manifest read). The first CI run of this head had the in-use rule before the name on the store-backed path and failed test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_refused on the three store-backed jobs; the in-use text is #671's service text naming the backup (backup_in_use_message), which the SQLite, PostgreSQL and Cassandra registration cells assert; HeartbeatHandle keeps #700's heartbeat_within and recheck with this PR's HeartbeatMetric and ClaimCheck; the reconciler's state is the Arc<Mutex<..>> #671's serve.rs shares, its store check carries #672's defaulted-root rule, and the resume arm takes the configured store as the Arc #700's JobEnv::new needs (BackupRuntime::configured_store_arc); SQLite's in-catalog restore slot (#671, by table id) is legacy_restore_slot_key beside this PR's restore_slot_key on the target name, and the dead listing skips a record whose slot of either kind a task holds; every backend's list_running_restore_jobs (#672) reads the restore record through this PR's wider row reader, and list_dead_restore_jobs keeps #671's skip-and-log of a record this build cannot read; cmd_backup.rs opens the catalog for remove-target ahead of #672's store-requiring open, under the same signal listener; devtools/run-mongodb-tests keeps one store mechanism, this PR's --backup-store KIND (filesystem by default), with #700's --backup-store-root DIR naming the root of that store (a named root is created and made owner-only, never cleared); tests/test_backup_job.py keeps #700's MongoDB deployment, on which this PR's seven restore cells skip (_restore_cell), and the cell that seeds a target_not_empty record at the catalog edits SQLite's and PostgreSQL's as before; MAINTENANCE is back in job.rs for the restore job's reason (the backup job, since #700's round, checks the claim before its row exists). #671's review item took revision item 28, #672's are 29 and 30 and #700's 31 and 32, so this PR's is 33 (every reference in the tree renumbered).

What

Core (crates/core). No new rule; validate_restorable_item, RestorableLimits and the manifest reader (ExtenddbManifest::reader, reader_prehashed, the per-file KeyRange check) are PR 4's and PR 8's, bound by the restore. DataFileDecoder and DataFileReader expose bytes_decoded(), the uncompressed bytes decoded so far, which the load's chunk bound counts.

Storage trait (crates/storage). A new RestoreEngine trait in StorageEngine's bound: bulk_load(target, claim, BulkLoadFile { file_index, start_at, reapplying }, items) -> applied, finish_restore(target, claim, item_count, size_bytes) -> Finished { Active, Pending, NotMatched }, count_restore_target, read_restore_progress (default []), sweep_restore_progress (default 0); RESTORE_BATCH_ITEMS (10,000) and RESTORE_BATCH_BYTES (32 MiB), the SQLite and Cassandra batch bounds, live beside it. BackupCatalog gains claim_restore(account_id, input, &RestoreRegistrationRow) -> (TableDescription, Box<dyn JobClaim>) (the registration creates the CREATING target and the RUNNING record under the backend's own generation), claim_restore_dead(key, stale_after, spool_host) (the restore analogue of claim_dead_backup; it writes a fresh heartbeat_at on the database's clock in the same conditional write as the generation and spool_host, so a second taker finds the record live at once, where the backup takeover leaves the heartbeat for its FAILED-prefix clock), read_restore_job and list_restore_jobs; RestoreJobRow carries the store, spool_host, file_index, items_applied, failure_reason and completed_at. restore_input_from_manifest builds a CreateTableInput from the manifest's source_table and applies the overrides, one implementation for both restore paths. restore_pause_point(name) is a pub async fn under test-hooks, a no-op without it, and the four backend pause-point names are pub consts beside it (RESTORE_SQLITE_MID_BATCH, RESTORE_MONGODB_AFTER_INSERT_MANY, RESTORE_CASSANDRA_BEFORE_PROGRESS, RESTORE_POSTGRES_BEFORE_PROGRESS_DELETE), one spelling each: the engine's KillPoint holds the engine's five points alone. JOB_HEARTBEAT_INTERVAL's doc says the store-backed jobs run at max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL) and the in-catalog paths and the maintenance claim at the constant.

PostgreSQL (crates/storage-postgres). restore_engine.rs: bulk_load opens one data transaction per file, multi-row INSERTs of 500 base and index rows (or 4 MiB of item text) through insert_restore_rows, writes the restore_progress row in the same transaction (a plain INSERT for a file applied from its start, an upsert on (table_id, file_index) for a file resumed past it, so a row a decoder panic left short is completed by the resume rather than failing on the row's key), and fails a duplicate on the base table; nothing in it conditions on the generation (the base table's key and the claim's liveness are its fence). finish_restore flips the table and every index ACTIVE in one catalog transaction and then deletes the progress rows on the data pool (the pause point restore_postgres_before_progress_delete between the two); that delete's error is logged and swept by the reconciler, never the restore's failure, since the table is ACTIVE and the record DONE once the flip commits. claim_restore locks backup_index FOR UPDATE (the lock start_backup_delete takes; its comment says so); claim_restore_dead takes the name's advisory lock and a stale RUNNING manifest record, refreshing heartbeat_at with the generation and spool_host. sweep_restore_progress deletes from a positive list (the table ids the progress rows name, read first, minus the RUNNING records read after and the caller's list), so a registration that commits between the two reads is never a candidate. Migration 8 adds restore_progress(table_id, file_index, items_applied). remove_restore_target's to_regclass('restore_progress') check and its DELETE now run on the data pool; on 09c881f both ran on the catalog pool, so with the table in a separate data database the PR 6 code found no table and deleted nothing. remove_restore_target drops the vector index's _ddb_vec_ relation with the data and index tables (table_storage_relations); the record's heartbeat_at is the registration transaction's now(), refreshed by the engine's first beat before the reply.

SQLite (crates/storage-sqlite). bulk_load writes batches of at most 10,000 items or 32 MiB of serialized item text, each a BEGIN IMMEDIATE holding the base, index and vector rows and the record's (file_index, items_applied), conditioned on the generation (the pause point restore_sqlite_mid_batch between batches); claim_restore takes the name's slot; claim_restore_dead takes over a record whose slot no task holds, whatever its heartbeat (a restart frees every slot and ages every RUNNING record), refreshing heartbeat_at. The startup line for an aged RUNNING record names the reconciler's duty per kind (a store-backed restore is resumed; an in-catalog one is removed). remove_restore_target no longer probes for a restore_progress table (SQLite never has one; a store-backed restore's progress is on its table_restores row, file_index and items_applied). The flip's backfilling = NULL on the vector rows is now observed by a fixture that sets it before the flip. A table with secondary indexes restores in more resident memory than the no-index figures the docs quote, and the excess is SQLite's own per-connection page cache for the index b-trees plus the allocator's arenas, not the restore's buffers, which stay bounded by the batch's 32 MiB (the module doc, the admin guide and the record state the shape beside the figures; a smaller batch for an indexed table was built, measured ineffective and not kept). A job's heartbeat queues behind the engine write lock like any writer, so the restore engine's and the heartbeat's module docs state what holds a beat, the batch times the bench record measured, and what a beat late by a period costs (the job stops as a lost claim and the reconciler resumes it from its last batch).

MongoDB (crates/storage-mongodb). bulk_load writes each batch of up to 1,000 base documents in one session transaction with the record's progress write: the transaction first find_one_and_updates the record (RUNNING under the claim's generation, setting restore_job.file_index, restore_job.items_applied to the file's running total and heartbeat_at to $$NOW) and ends the file as a lost claim when that write matches nothing, then insert_manys the batch and commits, so a frozen owner whose record a takeover moved can write no batch after the takeover (the Blocking fence, [Lee 4468802456]: the fence is in the write). Because the progress commits atomically with each batch, a crash mid-file leaves no documents the record does not account for and a resume continues from items_applied with nothing re-applied; reapplying is therefore vestigial on MongoDB (a transaction cannot do ordered=false duplicate suppression), and target_not_empty arises only from the engine's count-above-total check, not from a mid-file crash. A duplicate key (E11000, the unique _id) is the backup's and fails the file. The pause point restore_mongodb_after_insert_many now fires after each batch's transaction commits. finish_restore writes DONE and restore_backfill_pending and returns Pending, the backfill worker flipping the table ACTIVE. A record carries restore_job.ddl_done, set false at the registration's insert and true once the target's collections and index metadata exist; the reconciler's restore_target_ready refuses a record left unmarked (its owner died between the commit and the data DDL) and fails it rather than resuming into a target missing its indexes (the Blocking registration-order fix, [Lee 4468802443]). claim_restore_dead refreshes heartbeat_at with $$NOW, so a second taker right after the first is refused; the registration stamps the record's heartbeat from the server's clock inside its transaction. An N sort key past Decimal128 fails the file with the write path's text. bulk_load holds the GSI backfill test gate (test-hooks builds only) before the first insert of the restore's first file, as the in-catalog copy does before its $out, so restore_worker_waits_for_base_copy_before_backfilling_indexes runs on the store path. The snapshot open that a store-backed CreateBackup takes reads the majority-committed snapshot, which can trail DescribeTable's local read, and a CreateBackup sent as soon as DescribeTable said ACTIVE was refused TableNotFoundException (5 of 80 in a probe beside a write load; 0 of 80 on the in-catalog path); the open now waits, at most 2 s, while a local read shows the table ACTIVE (0 of 150 after), pinned by an_open_waits_for_the_snapshot_to_show_a_table_a_local_read_shows_active, which writes the flip unjournaled so the snapshot trails it.

Cassandra (crates/storage-cassandra). bulk_load writes each item's base row and every GSI and LSI row in one logged batch USING TIMESTAMP a reading of the coordinator's clock taken before the batch's claim check (put_restored_item_at, no lightweight transaction, eight in flight), in batches of at most 10,000 items or 32 MiB of serialized item text; the claim check before each batch is a lightweight transaction on the record (confirm_restore_claim), and the timestamp is the fence against a batch a stalled writer lands after a takeover: it carries a timestamp below every write and delete the successor made, so Cassandra keeps the successor's cells and tombstones and a key a client deleted stays deleted (the second round's thread item; test_a_batch_landed_after_a_takeover_loses_to_the_successors_writes_and_deletes pauses a writer after its check, lets the successor write, flip ACTIVE and delete two items, and releases it); it records progress with a lightweight transaction per batch (the pause point restore_cassandra_before_progress); finish_restore is one LWT (Active, never Pending; index rows created ACTIVE); count_restore_target checks the catalog first, as the other three do. The count is the only duplicate check. An unknown outcome of the progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore); a registration whose record was taken over during its DDL removes nothing and answers the lost claim; the test-hooks feature gains fail_progress_writes (Landed, Lost). claim_restore_dead refreshes restore_heartbeat_at in its LWT and re-applies the backup's active_restores entry for the target when a crash between the target's DDL and the registration left none, so the in-use rule holds through the resume. bulk_load's index rows carry the batch's timestamp, which is also the base row's version, so ADR-0027's rule holds (a later ordinary write's version is above it) and a re-applied restore batch keeps every GSI and LSI row under a later stamp (#424's in-place rewrite of an unchanged index row is what the legacy copy's put_restored_item relies on; the store-backed load no longer goes through it). restore_target_ready reads each GSI's and LSI's table in system_schema.tables as well as the data table, since the registration creates them in separate statements after the data table (test_restore_target_ready_requires_every_index_table drops the GSI's table). A crash before the first batch's progress record leaves the batch's items with no progress; the engine resumes that record with the first file re-applied (upserts) when the target holds no more than the backup's total (test_a_first_batch_applied_without_progress_resumes_to_the_exact_count: the progress write's outcome unknown twice, the load failed with 3 of 7 items landed and no progress, the resume to 7 with both index-row counts exact). Two concurrent registrations of one backup leave one target: the registration's LWT (IF backup_status = 'AVAILABLE' AND active_restores = null) is the fence, linearized on the backup's partition; the in-use pre-read is advisory. New index-row tests: a re-applied bulk_load batch, whose GSI and LSI rows keep their count and carry their base rows' versions as write times, and the two-registration race.

Engine (crates/engine/src/backup/). restore_table_from_backup and the reconciler take &Arc<S> (the manifest branch and the resume spawn tasks). restore.rs and restore/{spool,download,load}.rs: the request verifies the sidecar and manifest (each read bounded, 4 KiB and 16 MiB; an object over its bound is a ValidationException naming a corrupt prefix), the vector gate, the overrides, the key schema and the target, registers under timeout(RESTORE_REGISTRATION_TIMEOUT) (60 s, every backend), and runs the job (download with a 15-minute transport retry counted from the job's first attempt, per-file bulk_load, the count, finish_restore). The load's chunk bound (LOAD_CHUNK_BYTES, 32 MiB) counts the decoded line's bytes, and the feeder checks the claim at every chunk boundary beside the shutdown token; a decoder that dies fails the file. .verified markers hold the file's sha256 and a stale entry (a record that was another host's) is removed before use; a job whose record is another's (a lost claim, a DONE or FAILED write that matched no record) leaves its spool entry to the sweep. The spool sweep lists the spool before the records and keeps the entry of a restore registered between the two reads; the spooled manifest and sidecar are written .partial → sync → rename and a spooled pair that fails verification yields to the store's; a finish_restore error is re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); the resumed job's first beat is the takeover's return, taken by the reconciler beside claim_restore_dead; BackupStoreErrors{get} on every get the store failed; the restored table's ItemCount and TableSizeBytes are 0 at ACTIVE, as the service reports them (measured: 0 and 0 on 15 restored tables, one of them ten minutes after ACTIVE; [Lee 4468802685]), the statistics pass rewriting them later where the backend has one; a registration that fails after the request's 60 s bound is logged with the ARN, the target and the account. The job sends its first heartbeat right after claim_restore returns and before the reply, and claimed_at is that beat's return, so a slow registration does not end the job at its first check and the record's heartbeat_at is fresh at the reply (on PostgreSQL now() is the transaction's start, so a registration stalled on the backup_index lock or the DDL would otherwise commit a stale stamp); a first beat that finds the claim lost fails the record ("the claim was lost before the job started") and answers InternalServerError; the maintenance refusal records RestoreJobs{failed}. remove-target refuses or removes a RUNNING record's target by list_dead_restore_jobs on the database's clock (no host clock), and the purge's wait reads the backend's heartbeat listing the same way; the reconciler takes nothing over once the shutdown began, spawns no resume once its own token has fired while the jobs' token has not (the server stopping its workers while it holds no serving registration; a_stopping_reconciler_takes_nothing_over), and applies the target_not_empty guard under a maintenance claim (a dead record with no progress over a target that holds more items than the backup's row counts is failed target_not_empty and left for remove-target, not OWNER_DEAD and removed); the lost-claim stop line carries the heartbeat timer's reason, which the timer also logs at WARN; the spool entry is removed on a failed spool document write and on a resume that fails before it runs; a DONE write whose answer was lost and whose record re-reads DONE ends with complete (DONE); the two target_not_empty lines print extenddb backup remove-target --account <account> <table>. The reconciler checks the store first in its cycle (a store whose first check has not recorded is not yet checked, not unhealthy), resumes a dead store-backed restore from its progress under claim_restore_dead, fails one of another store (OTHER_STORE) or no progress into a target holding more items than the backup's total (target_not_empty; at or below the total the first file is re-applied with reapplying, exceeds_backup_total reading the backup row's item_count), re-reads the maintenance claim after the takeover and leaves the record unresumed when a command took the claim during the cycle, bounds its own takeovers of one record (RESTORE_TAKEOVERS, 5, counted per table id in ReconcilerState; the sixth takeover fails the record TAKEOVERS_EXHAUSTED and removes its target as OWNER_DEAD does, so a record whose every resume ends RUNNING again is not retaken every stale window for as long as the server runs; per instance, reset by a restart), counts a maintenance-held dead record in a field of its own, and sweeps restore_progress rows and spool entries. The final count is attempted three times with a wait between; three failures stop the job with the record RUNNING and no FAILED write (a_count_that_fails_three_times_leaves_the_record_running). Metrics RestoreJobs, RestoreJobDuration, RestoreBytesDownloaded (each data-file download attempt's bytes); KillPoint gains five restore variants (RestoreAfterDownload, RestoreAfterFileCommit, RestoreMongodbBeforeWorkerFlip, RestoreBetweenFailedAndRemoval, RestoreFencedOut; the engine's list; the engine's points log job reached kill point <name> and the backends' restore job reached kill point <name>). crates/server/Cargo.toml's backup-test-hooks adds extenddb-storage/test-hooks so the backends' pause points compile in.

App (crates/app). extenddb backup remove-target --account <id> <table> (PR 10's command, pulled forward): under the maintenance claim it removes a FAILED restore's CREATING target and record (target_not_empty included, the state a catalog restored from a dump leaves when it names a restore that finished and was written to since), a RUNNING restore's target whose owner is dead, or a legacy record's target, through remove_restore_target; it refuses a live RUNNING restore, a DONE restore, a table no record names, and, on PostgreSQL, a CREATING restore target whose owner's session still holds the target's advisory lock, reported as its own case (the record is unchanged after the backend refused the removal: "its owner's lock is still held ... wait for the owner to stop, or stop it, and run the command again"; remove_target_reports_a_held_owner_lock_as_its_own_case). It needs no store and opens the catalog without creating the default root or an identity file, prints the claim it took and one line naming what it removed, and logs the removal under extenddb::audit; on PostgreSQL, MongoDB and Cassandra its serving registration is named extenddb backup remove-target (PR 7's per-backend name) and released when the command exits, whatever the outcome, and a catalog under migration or behind the binary ends it with exit 6 or 3, as list and purge do. backup purge on SQLite fails a dead store-backed restore and removes its target rather than resuming it inside the command, since its cycle runs as the claim's holder. The command's logic and its tests are the engine's (maintenance::remove_target).

Harness and CI. The 24 restore cells of tests/test_backup_restore_fidelity.py run on a server with a store, and test_create_backup_of_a_vector_table_is_refused is test_vector_table_backup_refused_in_catalog_and_restored_from_store, since it asserts the round trip with SearchVectors on a server with a store; tests/test_auth_permissions.py's restore cell and tests/rust's backup suite run against the store; tests/test_backup_job.py gains the tamper, SIGTERM, store-outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments (the multi-part cell enables allow_multipart_table_keys on its own deployment and compares the restored item set with the source's Scan; the hand-built prefixes rewrite the two DynamoDB export manifests too, so they are whole backups) and, on SQLite, the backup remove-target cell: a finished restore, with one item written to the target since, is seeded at the catalog back into the target_not_empty state (a RUNNING manifest record with no progress over a CREATING target that holds more items than the backup, the shape a catalog restored from a dump leaves), the restarted server's reconciler fails it target_not_empty and DeleteTable, DeleteBackup and a second restore are refused, the command removes the target and its record with the documented output, and the restarted server answers ResourceNotFoundException for the name, DeleteBackup succeeds and a new restore runs; tests/test_mongodb_backup_restore.py gains the vector-manifest refusal, the 35-digit Decimal128 failure and the 38-digit intact cells, and its hand-built prefixes rewrite the two export manifests as the job suite's do; devtools/run-mongodb-tests serves a filesystem store and exports its root, its log and EXTENDDB_TEST_BACKUP_STORE, and its --backup-store none serves no store for the legacy LSI fallback cell (the one wire cell of the in-catalog restore path on MongoDB, which no CI job runs; the test guide and the cell's docstring give the command); the runner starts from an empty store root, since a reused --output directory's extenddb-store.json names the previous run's catalog. The PostgreSQL CI job starts moto, so the store-outage cell runs there as on the SQLite S3 job, whose workflow comments now say that cell is the one S3 restore the job runs. The matrix is devtools/backup-restore-matrix and .github/workflows/matrix-nightly.yml, which also runs every restore kill point of every backend in four restore-kill-points-<backend> jobs (the round-two section): one target directory and four sequential builds, the artifact action pinned, one run at a time, requirements-matrix.txt for pymongo and cassandra-driver, an apt-get install postgresql-client step for the host psql the script's PostgreSQL seed and teardown call, and a step that checks psql --version and the two drivers' import before the cells; the SQLite Rust-integration job's comment in integration.yml says the backup tests restore the store-backed backups (it had said they expect the restore's refusal); its duplicate-key prefix writer chmods its files 0600 as copy_prefix does, and its docstring's exit codes are the script's (1 on an assertion failure or a setup error); the nightly's rust-cache step saves only on main, as #680 made every other step, and its four actions are pinned to the commits main pins (#752). The job suite gains a SQLite purge cell (a stopped server's RUNNING store-backed restore record with progress: purge --confirm fails it and removes the target, and no taken over by this instance line is logged), and the store-config suite a remove-target cell that leaves no default root and no identity file; the PostgreSQL exit-code cell covers remove-target. The engine's a_file_replaced_between_the_two_passes_fails_in_the_load runs at a 1 s heartbeat: its hashing pass over both files runs on the test's one runtime thread before the first load's claim check, and under load it outlasted two 50 ms periods.

Docs. The differences document (vector, multi-part-key, how-restore-runs with its timings labelled as the in-catalog restore's beside the store-backed figures, concurrent-restore lock with Cassandra's two-registration guarantee, restore-interruption resume with the target_not_empty state, its remedy and the measured first-reach outcomes, a restored table's TableSizeBytes, a new Decimal128 row), the limits document (concurrent-restore spool and batch bound and Cassandra COUNT, backup contents), the usage guide and upgrade manual (the restore lands this release; migrations 1 to 7 breaking and 8 declared so on PostgreSQL for want of the additive-declaration job, not applicable on the other backends; the rollback paragraph; backup remove-target for the target_not_empty state), the admin guide (the spool's contents and sweep, the restore's memory terms and measured figures with the shape and the input order they were measured on, the two-GSI and the unordered figures beside them, backup list lists no restore, the backup remove-target command with its output and refusals, the owner's liveness on the database's clock, and the [backup.store] paragraph no longer saying a store-backed restore is a later release's), the developer test guide and CONTRIBUTING (the job suite, the MongoDB cells' variables and the runner's --backup-store none, the moto cells' credentials, the matrix and the nightly), docs/design/15-backup-restore-bench-pr9.md (the restore memory and duration record, with the two-GSI and the store-backed 60,000-item figures) and design revision item 33 with the body sentences it corrects. With the rebase: the admin guide's stale_job_after_secs row (a dead store-backed restore is resumed) and purge paragraph (under the serve lock such a restore is failed and its target removed), the upgrade manual's SQLite startup sentence (a store-backed restore is resumed) and its restore_progress sentence (migration 8, declared breaking on PostgreSQL, not applicable on SQLite, MongoDB and Cassandra), design 4(e) and the reconciler's module doc (what a lost serving registration does, citing registration.rs's module doc: the reconciler stops with the server's workers and spawns no resume once stopping, the running jobs go on until they end or the server stops; #755), CONTRIBUTING's worked example (migration 8, 7 being restore_progress), the PR 9a load paragraph (500 rows or 4 MiB), and the differences document's consistency row (the MongoDB snapshot open's wait).

Why

#396 section 5.3 and the PR 9a and 9b sections. PR 9a and 9b ship as one PR by the maintainer's decision (item 30). Decisions, each in the module doc of the code it concerns and in item 30: bulk_load is a RestoreEngine method, not a StorageEngine one; progress is recorded inside bulk_load in the write that applied the items, so there is no record_progress and PostgreSQL's progress row (in the data database, no job record) is fenced by the base table's key and the claim's liveness rather than the generation; reapplying makes MongoDB's duplicate suppression explicit; the manifest registration locks backup_index (the lock start_backup_delete takes) so a store-backed restore and a DeleteBackup of its backup order wholly; RESTORE_REGISTRATION_TIMEOUT is 60 s on every backend; a filesystem store is verified in place (two get passes), an S3 store spooled; the restore takeover refreshes the heartbeat where the backup takeover leaves it; the memory bounds are in bytes of text (the chunk in line bytes, the SQLite and Cassandra batch in serialized item bytes); the download retry bound is the job's; the matrix seed stands in for backup sync --adopt-store until PR 10. extenddb backup remove-target ships here rather than in PR 10 because the target_not_empty rule, written for a catalog restored from a dump, has as its common trigger on MongoDB and Cassandra an ordinary crash before the first progress record (both record progress after the write), and without the command this release had no way to remove such a target. The spool sweep lists the spool before the records, as the progress sweep lists its rows first, so neither removes what a registration between its two reads created. A finish_restore error is an unknown outcome re-read through the claim, as the status writes are, since on PostgreSQL the reachable case (the progress delete after the flip's commit) is a finished restore.

Testing done

After the rebase onto the review rounds of #671, #672 and #700, on this head: cargo clippy --all-targets with extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks,extenddb-engine/backup-test-hooks under -D warnings, clean; cargo test -p extenddb-engine --lib --features backup-test-hooks -- backup:: 262 passed (the cells adjusted to #671's order and text named above); the extenddb-core (797), extenddb-storage (233), extenddb-storage-sqlite (276, the restore_job_tests among them), extenddb-server (307), extenddb-app (47) and extenddb-config (24) lib suites; backup_restore (56) and restore_job (12) on PostgreSQL at 127.0.0.1:55432; backup_engine::live_tests:: with test-hooks on the MongoDB replica set at 127.0.0.1:27117, 42 passed; direct_integration -- backup_engine:: with test-hooks on a fresh Cassandra 4.1 node, 60 passed (test_the_catalog_listings_are_scoped_and_ordered failed once on a keyspace the node had not agreed on yet, "Keyspace ... doesn't exist" inside the migration runner, and passed when rerun). The wire suites (tests/test_backup_job.py's restore cells among them), the kill points and the memory runs below are from the previous heads and were not repeated; CI runs the wire suites.

The first bullets, down to the restore kill points' SIGTERM cells, are measured on this tree after the move onto main at e8ca7df, except the parts a bullet scopes to 03baa321 (the tree before the move: #700's head 42c4f6d8 plus this commit); the bullets after them are scoped "Before the rebase" and are the tree on #700's head before the stack's two rebases (09c881f4), where "new" is against that head.

• cargo fmt --all -- --check clean; cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings, the same --features extenddb-engine/backup-test-hooks, and -p extenddb-storage-cassandra and -p extenddb-storage-mongodb with --all-targets --features test-hooks, all clean; cargo +1.88.0 check --locked --workspace --all-targets clean with no warning; on 03baa321, cargo doc --no-deps of the engine, app and storage crates warned in none of this PR's files and the licence notices (default, MongoDB, Cassandra, dev) were current (the move changes no Cargo.lock or Cargo.toml: main changed neither between 03baa321 and e8ca7df3).
• cargo test --locked --workspace --exclude extenddb-storage-cassandra --exclude extenddb-storage-mongodb --exclude extenddb-backup-store with PostgreSQL required: bin 3, app 47, auth 274, cache 27, config 23, core 796 (1 ignored), engine 389, server 305, storage 220, PostgreSQL lib 78 and every test binary green (backup_restore 50, restore_job 11, the migrations_* binaries, table_snapshot 89 with 2 ignored among them), SQLite 267 (1 ignored) with restore_job_resume 1 and backup_job_panic 1. -p extenddb-backup-store with moto: lib 37, conformance 123. MongoDB: lib 169 (62 ignored), interpreter_selftests 18, migrations_fresh 8, migrations_guards 7, migrations_holders 10, migrations_locks 11, migrations_released 5, pushdown_parity 20, --lib -- --ignored --test-threads=1 live_ 62, and the test-hooks fence test 1. Cassandra under the shared container's lock: lib 115, migrations 22, migrations_stopped 8, --features test-hooks --test direct_integration 219 (three parts of 72, 74 and 73 tests named --exact; in the first part's first run two cells timed out on the node, one with "no answer within 60 s" on a keyspace's baseline file and one with "Connection timeout after 10s", and passed when rerun), ttl_integration 26 passed, 1 ignored and test_ttl_enable_rejects_asynchronous_gsi failing as it does on main.
• Migration 7 met main's tests that count the migrations, which now count it: SQLite's reconverge cell counts the convergent migrations that apply on SQLite; MongoDB's fresh-run and ledger lists record 7 not_applicable, its "later binary" rows move to id 8, its lock test counts eight steps and its released-catalog test eight pending; MongoDB's and Cassandra's registry tests list 7 as not applicable; Cassandra's backfill cell allows 7 pending beside 6.
• On 03baa321: the engine's backup::tests 20 times under taskset -c 0-1 nice -n 5 beside one busy loop: 20 of 20 green (214 tests, 33 to 34 s each). Mutant: a_stopping_reconciler_takes_nothing_over (new) fails with manifest_restores_resumed: ["MusicRestored"] when the guard on the reconciler's own token is disabled.
• Through the binary, from a catalog at main's head (release builds of e8ca7df3) on SQLite and PostgreSQL: migrate --list shows 1 to 7 (7 not applicable on SQLite, catalog, data on PostgreSQL); serve, backup list and backup remove-target on the catalog behind exit 3 naming 6 and 7; migrate --yes takes it from 5 to 7 (PostgreSQL: restore_progress in the data database and 7 in both ledgers; SQLite: 7 recorded not_applicable); verify HEALTHY; remove-target of an unknown table exits 1, leaves no backups/ on SQLite and no extenddb backup% session on PostgreSQL; serve starts, extenddb stop returns in 0.26 s, and main's serve refuses the migrated catalog with exit 4. On 03baa321, on MongoDB and Cassandra (a replica set of its own; the shared node), list, the dry run and remove-target leave no extenddb_servers entry, migrate --yes right after them does not wait, under a planted migration lock each exits 6 naming the holder, and on a catalog migrate --forget 6 put behind each exits 3 with the migrate --yes advice.
• CI jobs run the CI way on this tree, the three workflows that failed on d27c9bb9 first. test.yml: cargo test --locked --workspace --exclude extenddb-backup-store with no service environment, as on its runner, 59 binaries and 3,264 tests passed (the Cassandra contact point pointed at a closed port, so the Cassandra suites found no node and returned as on the runner), the 1.88 row's check above, and the store crate with moto. integration-mongodb.yml: mongodb-rust (devtools/run-mongodb-tests --rust-live -- --rust-integration with filesystem on the mongodb-test-hooks build): records 6, the migration binaries, tests/rust 514 with restore_worker_waits_for_base_copy_before_backfilling_indexes among them on the store path, live 19; mongodb-pytest (-- --pytest --comprehensive --parallel): 1638 passed, 51 skipped, 8 xfailed, 4 xpassed, comprehensive 502, test_abac.py and test_rfc0003_concurrency.py among the passes; mongodb-catalog: the schema check, check-mongodb-catalog-lock 29, -auth 16, -released 53, -older-binary 28, each "all checks passed". integration.yml: run-integration's check-postgres-catalog-lock 16 and check-postgres-older-binary 26, and its backup suites step with moto (17 passed, 15 skipped); run-integration-sqlite: both lock checks, the older-binary check, the schema check, run-tests --pytest --comprehensive --parallel --filter "not import_export" (1666 passed, 24 skipped, 1 xfailed; comprehensive 502) and the by-name suites (26 passed, 6 skipped); run-rust-integration-sqlite's tests/rust 514; run-integration-sqlite-s3: the S3 store on moto, --filter "backup" 65 passed, 6 skipped, the by-name suites 28 passed, 4 skipped. integration-cassandra.yml: cassandra-storage (the Cassandra rows above), cassandra-rust tests/rust 514, cassandra-catalog-checks (the schema check; check-cassandra-catalog-lock 42 and check-cassandra-older-binary 33 against v0.1.14 built from its tag, each on a node of its own), cassandra-two-nodes migrations 22. docs/schema/*.txt against a fresh init: SQLite 48 lines, PostgreSQL 416, Cassandra 320, identical; MongoDB's check passed. Not run: cassandra-pytest and cassandra-authorizer (this rebase changes no Cassandra request or auth path; the pytest leg is host-limited on this machine), the PostgreSQL pytest and Rust-integration jobs (their default extenddb_catalog exists on the shared PostgreSQL), and the nightly matrix.
• On 03baa321: the restore kill points' SIGTERM cell on PostgreSQL and SQLite (24,000 items, backup-test-hooks builds): the restore stopped by the server's shutdown inside feat(backup): CreateBackup job, worker and reconciler duties [backup/restore 8] #700's job drain, no FAILED written, 16 workers drained, the record RUNNING across the restart and the restore resumed to ACTIVE with the base table's and the GSI's counts exact.
• Before the rebase: cargo fmt --all -- --check clean; cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings, the same --features extenddb-engine/backup-test-hooks, and -p extenddb-storage-cassandra --features test-hooks --all-targets all clean; cargo check --locked -p extenddb-storage --lib with and without test-hooks clean; cargo check --locked --workspace --all-targets clean.
• Before the rebase: extenddb-engine 336 passed (62 new test attributes against 09c881f, one removed, 61 net; 7 of them added last: the maintenance-held target_not_empty guard, no takeover once the shutdown began, the loss raised inside a step named by the next check, the first beat refreshing the registration's heartbeat before the reply, the request stopping at the 60 s bound with the task going on, a resume that fails before it runs removing its spool entry, a spool document write failure removing the entry; the 28 before them: the slow registration, the stopped timer before each write and before a download attempt, the DONE write's NotMatched, the lost-claim entry left, Pending, the job-wide retry bound, a decoder that dies, the same-host spool resume, a marker with another digest, a crash's spool states, a file replaced between the passes (the replacement matching the original in compressed length, uncompressed length and item count and failing on the sha256 alone), a file over one chunk by bytes, the feeder's claim check, an unhealthy or unchecked store with a known identity, SSE on both paths, a restore registered between the sweep's two reads, a failed DONE write re-read, the resume's first beat, a truncated spooled manifest, a resume whose store documents fail verification, Progress::from_records on each backend's shape, a record gone after the takeover, a registration that fails after the request stopped waiting, the bounded document reads, remove_target over a FAILED, a dead RUNNING, a live RUNNING, a DONE and a legacy record); extenddb-core 728, extenddb-server 246, extenddb-app 44. extenddb-storage 200. extenddb-storage-postgres with the PG env: 353 passed over every binary before the last two cells were added, and then restore_job.rs 10 passed (the takeover test asserting the refreshed heartbeat, the progress delete failing after the flip, the resumed file's upsert) with every other binary clean up to the slow table_snapshot.rs, which the run's timeout cut short without a failure. extenddb-storage-sqlite: lib and tests 245 passed (restore_job_tests 10, a_batch_closes_at_the_byte_bound_before_the_count_cap new, the flip test setting backfilling before the flip). extenddb-storage-mongodb: unit 182 (the lib binary's 144 rerun after the last change; no MongoDB code changed in it), --lib -- --ignored live_ 43 in the earlier run (37 backup live_ cells, live_bulk_load_stops_at_a_lost_claim_between_two_insert_many_calls new), and the test-hooks fence test a_takeover_during_bulk_load_fences_the_file_index_write (--features test-hooks -- --ignored --test-threads=1) 1; the 43 live_ cells pass under the shared replica set with --test-threads=1: run in parallel, three legacy-path copy cells can time out waiting for a GSI-backfill worker within the 90 s deadline (an environment limit, not a logic defect). extenddb-storage-cassandra: lib 78, --features test-hooks --test direct_integration 213 passed in the full run before the last three cells were added (tests/direct/backup_engine.rs 48 tests, five new: the active_restores re-apply, the delete between the registration's pre-read and its fence, the composite-key resume with index rows and an empty file, the byte bound, the lost claim between two batches), and the three new cells and two changed ones ran twice and passed (tests/direct/backup_engine.rs: two concurrent registrations leaving one target, the re-applied batch keeping every index row over a composite GSI+LSI target; tests/direct/index.rs: an unchanged item's re-put keeping its GSI and LSI rows, the GSI worker keeping the row), the full binary not rerun end to end on the shared container; --test ttl_integration 25 passed, 1 ignored, test_ttl_enable_rejects_asynchronous_gsi the known pre-existing failure and test_backfill_cursor_resume passing alone.
• Before the rebase: mutants, each built by a targeted edit, caught at the assertion it targets and reverted by the reverse edit: 52 in the engine and core (22 of them added last: the shipped spool-sweep order, finish_restore's error as a failure, the resume's instant at its entry, the spooled documents written in place, the fallback to the store removed, no metric on a refused get, the unlogged registration failure, the Cancelled arm recording a worker error, a zero duration, the read bound removed, the in-place pass not hashing again, from_records ignoring the MongoDB and PostgreSQL shapes, the sidecar and ARN re-checks dropped, the gone record as an error, the compressed size as the table's, and four on remove_target: the DONE refusal skipped, every owner live, the claim dropped unreleased, a zero window not read as every owner dead), 21 across the storage trait and the four backends (five added last: the Cassandra index-row skip removed, the GSI worker bypassing the shared writer, PostgreSQL's progress delete propagated, the plain INSERT for a resumed file, the SQLite flip's backfilling clause dropped).
• Before the rebase: the four release binaries (--features postgres|sqlite|mongodb|cassandra, -p extenddb), the four hooks binaries (<backend>,extenddb-app/backup-test-hooks) and the mongodb-test-hooks build built on the final tree; CARGO_TARGET_DIR=target/t188 cargo +1.88.0 check --locked --workspace --all-targets clean. docs/schema/postgres.txt and sqlite.txt match a fresh init of the release builds byte for byte (no schema changed since); cargo test -p extenddb-storage-postgres --test doc_version_literals passes with the record in the tree.
• Before the rebase: wire legs on the PostgreSQL and SQLite release builds with a filesystem store: tests/test_backup_restore_fidelity.py test_auth_permissions.py (the 24 restore cells and the vector round trip with SearchVectors) — SQLite 100 passed, PostgreSQL 100 passed, no skip (the multi-part-key cell moved to the job suite); and tests/test_backup_store_config.py test_backup_job.py by name with EXTENDDB_BINARY and the moto variables on both builds — SQLite 25 passed, 3 skipped (the PostgreSQL-only store cells skipped; test_backup_job.py 14 of 14, the remove-target cell included), PostgreSQL 16 passed, 12 skipped (the SQLite-only store cells and the SQLite-only remove-target cell skipped; test_backup_job.py 13 of 14), the outage cell against moto on both. The multi-part-key cell runs on both (the restored item set equals the source's Scan; on SQLite the preview's item paths keep one item per first HASH key value in the source itself, so the set is 5 of the 60 items written). The MongoDB cells through devtools/run-mongodb-tests: the CI way (filesystem) tests/test_mongodb_backup_restore.py 5 passed, 1 skipped (the legacy LSI fallback cell skipped), and with --backup-store none 3 passed, 3 skipped (the legacy LSI fallback cell and the two throughput cells passed; the three store-backed cells skipped).
• Before the rebase: the restore's memory and duration (docs/design/15-backup-restore-bench-pr9.md; the admin guide's max_concurrent_jobs row and the design's section 7 quote it): release binaries, filesystem store, VmHWM over idle from a fresh start, 24 items of 27.7 MB (the worst shape) restore with a peak of 848 MiB on SQLite and 778 MiB on PostgreSQL under the default allocator (848 and 1,013 MiB at 12 and 48 such items on SQLite) and 392 and 427 MiB under MALLOC_ARENA_MAX=1; 1,200 items of 350 KB in one data file 270 MiB on SQLite and 156 MiB on PostgreSQL; 1M items of 200 bytes under a composite key and no secondary index, the backup's lines in key order as the server's own backups are, 89 MiB on SQLite and 16 MiB on PostgreSQL (100k: 38 and 10.7 MiB), restored in 26.8 s and 9.5 s; the same 1M items as an unordered backup (every MongoDB-sourced backup and every multi-part-key table) 144 to 164 MiB on SQLite in 66 to 90 s (scattered inserts dirty the whole base b-tree; the restore's buffers are the same) and 16.4 MiB on PostgreSQL in 17.5 s; at 3M rows SQLite 109 to 116 MiB key-ordered and 171 to 174 MiB unordered, the final COUNT(*)'s share 22 to 28 MiB at both sizes, bounded by the counting connection's cache_size (32 MiB) rather than growing with the table, as the record had said from one measurement. The same 200-byte items with two full-projection GSIs, measured on the pushed head's and on the final binaries: SQLite 143, 225 and 143 MiB over idle at 100k in three runs (10.2 to 13.9 s to ACTIVE) and 165 and 172 MiB at 300k (51.7 and 51.9 s), PostgreSQL 29.3 and 22.9 MiB at 100k (9.9 and 6.2 s) and 30.8 and 33.5 MiB at 300k (19.7 and 21.4 s); the excess on SQLite is the backend's per-connection page cache for the index b-trees (cache_size, 32 MiB) plus the allocator's arenas, not the restore's buffers, and a smaller batch for indexed tables was measured ineffective (91.7 against 92.7 MiB) and not kept; every GSI's Scan count equalled the item count after ACTIVE. The 60,000-item shape the differences document's in-catalog timings were measured on, restored from the store on the final binaries: response 0.051 s (PostgreSQL), 0.016 s (SQLite), 0.055 s (MongoDB), ACTIVE after 0.67 s, 1.85 s and 3.74 s (the MongoDB job's DONE at 1.8 s, the backfill worker's flip after), 9.3, 38.3 and 30.1 MiB over idle; the differences document's in-catalog timings for that shape now carry their label. The record carries the terms the code bounds and the sizing sentence (about 450 MiB per running restore of large items under one arena, times max_concurrent_jobs).
• Before the rebase: the cross-backend matrix from devtools/backup-restore-matrix, one invocation per cell, on the final binaries: the twelve plain cells (PostgreSQL, SQLite, MongoDB sources into all four targets) PASSED (items, description, GSI and LSI queries match the source; 10 to 44 s each, the Cassandra targets the slowest), with the seed's timestamps as the job writes them; the four --duplicate-key cells and the eight --vector cells PASSED on the final binaries too (the --duplicate-key cells into Cassandra by the count alone, restored 12 items, the manifest lists 13; the --vector cells into PostgreSQL and SQLite with SearchVectors answering, into MongoDB and Cassandra refused before any table). The nightly workflow has had no run: workflow_dispatch needs the workflow file on the default branch, so it cannot be dispatched until this PR merges, and its first run (the schedule's, or a dispatch after the merge) is the proof of the runner image. The workflow installs postgresql-client and checks psql --version and python3 -c "import pymongo, cassandra" before the cells, since the matrix script seeds and tears down its PostgreSQL deployments through the host's psql, which no other workflow uses.
• Before the rebase: the restore kill points (kill-points.py, posted with this PR; every ACTIVE outcome asserts the base table's Scan count and the GSI by-g's Scan count both equal the item count), each killed or signalled at the point, restarted and asserted, on the final hooks binaries where the row says so and on the pushed head's otherwise (nothing on those paths changed between the two):

point	SQLite	PostgreSQL	MongoDB	Cassandra (SQLite source, adopted)
restore_after_download	ACTIVE, 12,000, GSI 12,000	ACTIVE, 12,000, GSI 12,000	ACTIVE, 12,000, GSI 12,000	ACTIVE, 12,000, GSI 12,000
restore_after_file_commit	ACTIVE, 12,000, GSI 12,000	ACTIVE, 12,000, GSI 12,000	ACTIVE, 12,000, GSI 12,000 (resumed from file 1 after 0 of its items, 3,771 applied by the resume; rerun on the rebuilt binary)	ACTIVE, 12,000, GSI 12,000 (3,771 applied by the resume)
restore_sqlite_mid_batch (24,000 items, one file)	killed after batch 1's progress; ACTIVE 24,000, GSI 24,000	—	—	—
restore_postgres_before_progress_delete	—	ACTIVE across the restart, 12,000, GSI 12,000, the progress rows swept	—	—
restore_mongodb_after_insert_many, first reach	—	—	the resume found written items and no progress: FAILED target_not_empty, the target left CREATING (DeleteTable ResourceInUseException); extenddb backup remove-target --account 123456789012 <table> printed the four lines and removed it; DescribeTable ResourceNotFoundException, DeleteBackup allowed	—
restore_mongodb_after_insert_many, second reach (--second-reach, 24,000 items in three files)	—	—	file 0 recorded, killed after file 1's insert_many; resumed from file 1 under reapplying, ACTIVE 24,000, GSI 24,000	—
restore_mongodb_before_worker_flip	—	—	DONE before the kill; the restarted worker flipped the table, the reconciler left the DONE record alone, ACTIVE 12,000, GSI 12,000 (rerun on the rebuilt binary)	—
restore_cassandra_before_progress, first reach (12,000 items, one file)	—	—	—	10,000 written, no progress: FAILED target_not_empty, the target left CREATING; remove-target removed it, DescribeTable ResourceNotFoundException, DeleteBackup allowed
restore_cassandra_before_progress, second reach	—	—	—	10,000 recorded, killed before batch 2's progress write; resumed after 10,000, ACTIVE 12,000 (2,000 applied), GSI 12,000 (one batch short before #742 was corrected; rerun on the rebuilt binary)
restore_between_failed_and_removal (a tampered file)	the FAILED-target duty removed the target after the stale window	gone; with --vector (a vector index on the source) gone with its data table and _ddb_vec_ relation (rerun on the rebuilt binary; the pushed head's build left the relation)	gone	gone
restore_fenced_out	—	the same server's reconciler took the fenced record over and finished it (12,000, GSI 12,000); the released job's DONE write matched no record and made no table access	—	—
pg_terminate_backend (the job paused at restore_after_file_commit)	—	the claim session terminated while the job was paused after file 0's commit (psql answered t, the last beat 1.1 s old); released 20 s later, the job stopped at its next check as a lost claim naming the terminated session, the record RUNNING with its (0, 8228) progress row; the same server's reconciler took it over 45.6 s after the termination (46.7 s after the last beat) and finished it: 24,000, GSI 24,000 (rerun on the rebuilt binary)	—	—
sigterm	RUNNING across the restart, ACTIVE 24,000, GSI 24,000	the same, 24,000	the same, 24,000	the same, 12,000

Every cell above ran on the pushed head's hooks binaries (4c35319); the cells marked "rerun on the rebuilt binary" ran again on the binaries built after the last corrections (the first beat before the reply, the Cassandra progress re-read and registration fence, the PostgreSQL vector-relation drop), the pg_terminate_backend cell as reworked (the shipped cell slept 2 s, terminated and asserted the final count alone, which a 2.6 s restore passed without the case running). The Cassandra points ran against a freshly recreated single-node container with replication_factor = 1 (no GCInspector, StatusLogger or "No nodes available" line in its log after the runs). Two MongoDB cells (restore_between_failed_and_removal, sigterm) failed once at setup with ResourceNotFoundException on the fixture's first write when a second deployment on the same replica set dropped the fixed databases under them, and passed when rerun alone.

Review round (2026-11-04): changes addressing Lee's review

Measured on this stack's head after rebase onto #700's current head. Counts here are re-measured on this head; the counts above are carried from the pre-rebase runs unless noted.

Code changes:

• MongoDB bulk_load fence (Blocking, 4468802456) and per-file progress (Major, 4468802471): each batch's insert_many and its (file_index, items_applied) record now commit in one session transaction whose find_one_and_update fence requires the record RUNNING under the claim's generation, so a frozen owner writes nothing after a takeover, and a mid-file crash resumes from items_applied to ACTIVE rather than target_not_empty. reapplying is vestigial on MongoDB (a transaction cannot do ordered=false duplicate suppression).
• MongoDB registration order (Blocking, 4468802443): the record carries restore_job.ddl_done, set true only after the target's collections and index metadata exist; the reconciler's new restore_target_ready refuses and fails an unmarked record instead of resuming into a target missing its GSIs/LSIs.
• Cassandra bulk_load fence (Blocking, 4468802456): each batch runs a LWT on the record (restore_heartbeat_at refreshed IF RUNNING under the generation) before its upserts. Round two completes the fence: the batch's writes carry a timestamp read before that check (the round-two section).
• Reconciler, target with no data tables (Major, 4468802481): a dead manifest record whose target's data tables are missing is failed and its target removed via restore_target_ready, not retaken every stale window; the reconciler re-checks the maintenance claim after claim_restore_dead (Minor, 4468802609).
• Reconciler, other stores (Major, 4468802503): an instance with no configured store leaves a store-backed record alone rather than failing it OTHER_STORE. The three fail_restore_job writes are conditioned on the owner_generation and heartbeat_at the listing read (feat(backup): catalog index, BackupCatalog, JobClaim, legacy reader [backup/restore 6] #671's signature), so a record whose owner heartbeated after the listing keeps its record.
• Registration retry vs the 60 s bound (Major, 4468802521): a retry that meets its own registration in flight waits for it and is answered the CREATING description; a different backup into the same name is TableInUseException.
• Final count (Minor, 4468802621): retried; a persistent failure leaves the record RUNNING for the reconciler rather than failing the restore and removing the loaded target.
• Spooled-file hashing (Minor, 4468802633): the load hashes the spooled file again (through ExtenddbManifest::reader, not reader_prehashed), so a file changed in the spool between runs is caught.
• Download skip (Minor, 4468802649): a resume downloads only the files its progress leaves to load.
• remove-target (Minors): the command prints the documented terms "store-backed"/"in-catalog" (4468802662); on PostgreSQL it takes the target name's advisory lock for the removal via a command-only remove_restore_target_exclusive, so a frozen owner keeps its target (4468802600); the account-scoped find, the maintenance-claim-lost pre-removal check, and the count-above-total path are pinned by new engine cells (4468802541, 4468802553, 4468802532).
• ItemCount/TableSizeBytes 0 at ACTIVE (Minor, 4468802685).

Tests run this round (re-measured on this head):

• cargo test -p extenddb-engine backup:: with extenddb-engine/backup-test-hooks: 241 passed (236 carried plus five new: the count-above-total cell, the account-scoped remove-target cell, the claim-lost remove-target cell, the dead-restore-with-no-tables cell, and the registration-retry cell).
• cargo test -p extenddb-storage-postgres --test restore_job against PostgreSQL: 12 passed (11 carried plus the new sweep-reads-order cell).
• cargo test -p extenddb-storage-sqlite --test restore_job_resume and --test backup_job_panic: 1 + 1 passed; SQLite --lib restore 57 passed.
• MongoDB live (--features test-hooks, replica set at mongodb://127.0.0.1:27117): the rewritten fence, lost-claim and resume cells plus live_finish_restore_hands_over_and_the_worker_flips_active, live_bulk_load_refuses_a_long_sort_key_and_keeps_a_long_number, live_restore_returns_creating_and_the_detached_copy_finishes — all passed.
• Mutants verified killed (mutate, run the one test, revert): the count !=→< (4468802532), the remove-target account comparison (4468802541), the claim.lost() pre-removal check (4468802553), the restore_target_ready guard (4468802481), the in-flight registration dedup (4468802521), PostgreSQL's sweep_restore_progress two-read order (4468802562), and the reconciler's spool/record sweep order (4468802548).
• cargo fmt --all clean; cargo clippy with -D warnings on the engine, storage, PostgreSQL, SQLite, MongoDB (test-hooks) and Cassandra (test-hooks) crates clean; cargo check --workspace --tests clean; shellcheck on devtools/run-mongodb-tests reports only the pre-existing SC2174 the MongoDB runner already carried.

Not run this round (no environment / CI only): the Cassandra live suite (shared container; the fence is compile-checked with test-hooks), the nightly matrix, and the new matrix-nightly.yml kill-points job (its devtools/backup-kill-points.py is committed by #700 and arrives by rebase).

CI wiring:

• devtools/run-mongodb-tests gained a test-hooks step running the restore fence cells.
• .github/workflows/matrix-nightly.yml gained a restore-kill-points job that builds the SQLite binary with backup-test-hooks and runs the four SQLite restore points through devtools/backup-kill-points.py (its call did not match the script's CLI; round two corrects it and adds the other backends' jobs).
• The Cassandra store CI item (4468802575) is not done: Cassandra is legacy_create_only (it cannot take a store-backed backup), so serving a filesystem store in its Rust job would not give store-backed-restore coverage; the nightly matrix (SQLite/PostgreSQL/MongoDB source → Cassandra target via adopt-store) is that coverage. See the reply.

Review round two (2026-11-05): Lee's second review

Measured on this head (the round-one head plus these changes). Counts marked re-measured were counted on this head; the rest are carried from the sections above.

Code changes:

• Kill-point job call (Blocking): the nightly job called devtools/backup-kill-points.py --backend sqlite --point <p>, which the script's parser refuses (exit 2). The call is now restore <backend> <point> --bin target/release/extenddb --work kill-point-logs/<backend>-<point>; the jobs' exact calls were parsed locally (--help, and runs with a bogus --bin that get past the parser into init), and two SQLite points (restore_after_file_commit, restore_sqlite_mid_batch) ran end to end with the job's exact call on a backup-test-hooks release build: both resumed and ACTIVE with 12,000 items, base and GSI exact.
• Kill-point jobs for every backend (thread): four jobs, restore-kill-points-{sqlite,postgres,mongodb,cassandra}, each building its backend's binary with extenddb-app/backup-test-hooks and starting its backend as the matrix job does (the PostgreSQL job installs postgresql-client for the script's psql; the MongoDB job starts the replica set and installs pymongo; the Cassandra job configures PasswordAuthenticator, installs cassandra-driver, and builds a plain SQLite binary as the --source-bin whose backup the Cassandra target restores). Each runs every point of the script's RESTORE_POINTS for its backend, plus restore_mongodb_after_insert_many --second-reach (24,000 items), restore_cassandra_before_progress --second-reach and restore_between_failed_and_removal --vector on PostgreSQL; logs are uploaded on failure. The script's first-reach assertion for the two no-progress points changed from target_not_empty to ACTIVE with the exact count (MongoDB since round one, Cassandra since this round), and its --remove-target flag is gone (the target_not_empty state is seeded by tests/test_backup_job.py's cell).
• Count retry (Major): a_count_that_fails_three_times_leaves_the_record_running: the fake's count fails three times, the job stops with the record RUNNING at (file 1, 2 items), no FAILED or DONE write, the loaded target kept, RestoreJobs{failed} recorded. Mutant: with the first count error failing the restore at once, the cell fails (count_calls 1, not 3); passes after the revert.
• Second maintenance read (Major): a_claim_taken_during_the_cycle_leaves_the_taken_over_record_unresumed: the fake's new takeover_hook sets the claim held inside claim_restore_dead, after the cycle's first read; the record is taken over (new generation) but not resumed (no bulk_load, RUNNING). Mutant: with the second read replaced by a constant "not held", the cell fails (manifest_restores_resumed: ["MusicRestored"]); passes after the revert.
• Cassandra batch fence (thread): put_restored_item_at writes each item's base row and index rows in one logged batch USING TIMESTAMP a reading taken before the batch's confirm_restore_claim; a stalled writer's batch lands below the successor's writes and deletes. test_a_batch_landed_after_a_takeover_loses_to_the_successors_writes_and_deletes (direct_integration, test-hooks, the new pause_after_restore_claim_check hook): the owner paused after its check, the record taken over, the successor loads, flips ACTIVE and deletes two items (GSI and LSI rows gone), the owner released: its progress write reports the lost claim, the two items stay deleted with their index rows gone, the survivors carry the successor's versions. Mutant: with the timestamp read after the pause (when the writes land), the cell fails (4 items, not 2: the deleted items resurrected); passes after the revert.
• Cassandra first-batch resume (thread): the reconciler's no-progress rule compares the target's count with the backup row's item_count (exceeds_backup_total): above it target_not_empty (the dump-restored catalog), at or below it the resume re-applies the first file with reapplying set (the resume already set it for the first file it loads). Engine cell a_half_applied_first_file_with_no_progress_is_resumed_with_upserts (one of two items landed, no progress: resumed, reapplying true, DONE with 2); storage cell test_a_first_batch_applied_without_progress_resumes_to_the_exact_count (3 of 7 landed with no progress through two lost progress writes, resumed to 7 with 4 GSI and 3 LSI rows). The target_not_empty cells now seed three items against a backup of two. The differences row, design 5.3 step 7 and the PR 9a load paragraph, the admin guide and the kill-point description are updated.
• Takeover bound (thread): ReconcilerState.restore_takeovers counts this instance's takeovers per table id; the sixth (RESTORE_TAKEOVERS = 5) fails the record owner dead: taken over 5 times by one instance without the restore ending and removes the target as for OWNER_DEAD. In-process rather than a column: a takeovers column would touch every backend's record and PostgreSQL's catalog table_restores (not this PR's migration 8, a data-database table), and the count per instance is what the thread allowed; stated in the module doc, the design and the differences row (a restart starts the count over). Cell the_takeover_after_the_fifth_fails_a_record_that_never_ends: the count fails on every attempt, five cycles take the record over and leave it RUNNING, the sixth fails and removes it with the reason in the FAILED write. Mutant: with the bound check disabled, the sixth cycle resumes again and the cell fails; passes after the revert.
• claim_restore doc (thread): no longer "in one transaction": the catalog rows commit first, the data tables (MongoDB: collections and index metadata) after, in the order restore_target_ready's doc gives, and the resume checks restore_target_ready.
• Cassandra in-use cell (thread): test_a_running_store_backed_restore_keeps_its_backup_in_use_until_active (direct_integration): with a store-backed restore RUNNING, backup_in_use names the target and a second claim_restore is BackupInUse with the service's text, twice; with the active_restores map entry deleted (the registration's window before the entry lands) the target rows alone keep it in use; after finish_restore the backup is free. The engine's order (the in-use rule before the override validation) is the engine's cells'; this cell is the Cassandra input to it.
• wait_for_items_seen deadline: 120 s, with the reason in the comment (the decoder's hashing pass and first chunk on the test's one runtime thread under a loaded CI machine).
• Comment ids (Minor): the ten (Lee 4468...) citations in the diff replaced with descriptions in the code's terms (engine tests and reconciler, Cassandra backup_catalog.rs and backup_engine.rs, PostgreSQL backup_catalog.rs and tests/restore_job.rs); grep 4468 over the tree finds none.
• Held owner lock (Minor): remove-target re-reads the record after the backend refuses the removal; an unchanged record means the table is still its CREATING target and the refusal was the lock, reported as "the restore into T (account A) from is a CREATING restore target (table id ) whose owner's lock is still held: its owner's session is alive though its heartbeat is stale; wait for the owner to stop, or stop it, and run the command again"; a changed or gone record is "not a restore target: it is no longer a CREATING restore target with table id ". remove_target_reports_a_held_owner_lock_as_its_own_case (the fake's owner_lock_held), the legacy cell's text adjusted; the admin guide lists the line.
• Cassandra readiness (Minor): restore_target_ready checks each GSI's and LSI's table in system_schema.tables beside the data table; test_restore_target_ready_requires_every_index_table drops the GSI's table and sees false.
• Retry wait is per server (Minor): the "How RestoreTableFromBackup runs" row says the in-flight list is per server process, so a retry waits only when it reaches the server that holds the registration, and what a retry on another server sees.
• Also: tests/test_backup_job.py's remove-target cell expected "its manifest restore from", where the command prints "its store-backed restore from" since round one; the cell and the admin guide's sample output now carry the printed text (the cell ran green on the hooks build).

Tests run this round (re-measured on this head): cargo test -p extenddb-engine --lib --features backup-test-hooks -- backup:: 267 passed (--list count 267; the five new cells above plus the held-lock cell); cargo test -p extenddb-storage-cassandra --features test-hooks --test direct_integration -- backup_engine:: on the shared Cassandra 4.1 node 64 passed (60 carried plus the four new cells; the fence cell run alone three more times, once with the mutant); cargo test -p extenddb-storage-postgres --test restore_job against PostgreSQL at 127.0.0.1:55432 12 passed; tests/test_backup_job.py -k remove_target_removes_a_target_not_empty on the SQLite backup-test-hooks release build 1 passed; devtools/backup-kill-points.py restore sqlite restore_after_file_commit and restore_sqlite_mid_batch with the job's exact call on that build, both ACTIVE with 12,000 items, base and GSI exact; cargo check --workspace --tests clean; cargo fmt --all -- --check clean; cargo +1.97.0 clippy --all-targets -- -D warnings on the engine, storage, Cassandra (test-hooks) and PostgreSQL crates with extenddb-engine/backup-test-hooks, and on the Cassandra crate without test-hooks, clean; shellcheck on the four jobs' run steps clean. Mutants verified (mutate, run the one cell, revert): the count error failing at once, the second maintenance read skipped, the takeover bound disabled, the Cassandra batch timestamp read after the check. Not run: the MongoDB live suite (no MongoDB code changed), the nightly workflow (it needs the workflow file on the default branch), and the PostgreSQL, MongoDB and Cassandra kill-point jobs' points (their calls were parsed locally; the SQLite points ran).

Review round three (2026-11-05): Lee's third review

Three inline Minors, the review body's request to bound the restore handle's in-flight clause with #700's, and one CI change.

• In-flight clause (review body): ClaimCheck carries parks (JobClaim::heartbeat_parks_before_its_bound, feat(backup): CreateBackup job, worker and reconciler duties [backup/restore 8] #700's new provided method) and passes it to in_flight_confirms, so the restore handle's check is feat(backup): CreateBackup job, worker and reconciler duties [backup/restore 8] #700's: a beat in flight confirms the claim for as long as it waits on SQLite, where the beat parks on this process's lock, and within one period of its start on PostgreSQL, MongoDB and Cassandra. feat(backup): CreateBackup job, worker and reconciler duties [backup/restore 8] #700's cell a_beat_in_flight_past_its_bound_confirms_the_claim_only_where_the_beat_parks runs on this head too.
• fetch_update (Minor): count_errors in the engine tests' fake is a compare-exchange loop, as crates/storage-cassandra/src/migrations/backend.rs does, with the reason (try_update is not on 1.88) in its comment.
• Cassandra clock (Minor): restore_engine.rs's module doc names the stamp's clock (the ExtendDB host's, data::version::now_micros) and the write times it must lose to (lightweight transactions and the deletes through the ACTIVE table, stamped by the Cassandra coordinator), so the rule needs the ExtendDB hosts' and the Cassandra nodes' clocks to agree.
• Section headers (Minor): the two engine test headers and the one in crates/storage-cassandra/tests/direct/backup_engine.rs name the cases their cells cover (the manifest total, remove-target's scope and lost claim, a target without tables, the registration retry; the count retry, the maintenance claim taken mid-cycle, the takeover bound, the first file resumed with upserts; the batch's timestamp fence, the first batch resumed without progress, the readiness check's index tables, the in-use rule of a running restore) instead of the review round they came from.
• Integration job bounds: the previous head's run-integration and run-integration-sqlite were cancelled at their 40-minute timeout-minutes after the backup store and job suites step, which this stack adds to the end of each job; both are now 55 minutes, with the reason in a comment. The other jobs keep 40.

Tests run this round on this head: cargo test -p extenddb-engine --lib --features backup-test-hooks -- backup:: 272 passed; extenddb-storage-sqlite write_lock_tests 15; cargo +1.97.0 clippy --all-targets with the Cassandra and MongoDB test-hooks and the engine's backup-test-hooks features, -D warnings, clean; cargo check --workspace --tests clean.

Review round four (2026-11-06): Lee's notes with the approvals of #700 and #740

None was a condition of either approval; all are fixed here.

• Design 5.1 step 4 (feat(backup): CreateBackup job, worker and reconciler duties [backup/restore 8] #700's approval): the late-first-put sentence no longer says "with no beat in flight". On PostgreSQL, MongoDB and Cassandra the put is refused whether or not a beat is in flight; on SQLite, unless a beat is in flight, parked on the write lock.
• Design 4(i) and the PR order paragraph: restore_progress is id 8 in both, with fix(storage-postgres): number stream records after their writes commit #819's stream_pending at 7.
• MongoDB not-applicable test: renamed every_body_is_code_and_three_four_seven_and_eight_do_not_apply. Its list now holds StreamPending (7) as well as RestoreProgress (8).
• Cassandra clock doc (thread on restore_engine.rs:16): the batch reads the ExtendDB host's clock for its write timestamp, not the coordinator's. The module doc's first sentence on it now says so, as the later sentence on the clocks did.
• CI timeout comment: each 55-minute job's comment gives the measured figures from this PR's CI: 39 min 49 s with 3 min 3 s of backup suites (run-integration), and 40 min 31 s with 4 min 16 s (run-integration-sqlite).

Tests run this round on this head: engine backup:: with backup-test-hooks 273 passed; storage-sqlite lib 281; storage-mongodb migrations::tests 5; storage-cassandra lib 129 and direct_integration backup_engine:: 64; fmt and clippy 1.97 clean with the three test-hooks features.

Not changed here

PR 10's backup sync --adopt-store, --move-store, --forget-backup and migrate-legacy; PR 11's CI finalization and documentation, with the MongoDB and Cassandra 1M restores and the vector cell still to be measured; PR 13; PR 14's drop of the in-catalog format. The matrix seeds the target's backup_index row by hand in place of backup sync --adopt-store. MongoDB's insert_many unit stays 1,000 documents with no byte bound of its own (the data file bounds it: bulk_load is per file). SQLite's restore memory for a table with indexes stays SQLite's page cache (cache_size) above the no-index figure; a smaller batch for indexed tables was measured ineffective and not kept.

Breaking changes

kind = "disabled" is no longer needed to keep a deployment able to restore the backups it takes. Migration 8 adds restore_progress to the PostgreSQL data database: run extenddb migrate --yes before starting the new binary. An S3 store needs spool_path on PostgreSQL, MongoDB and Cassandra for restores. The restore of a store-backed backup is asynchronous on every backend with a 60 s registration bound, as on the service.

DCO

Signed-off-by: Scott Robinson robinnsc@amazon.com

[LeeroyHannigan]

I built the server at this head for SQLite, PostgreSQL, MongoDB and Cassandra, and at the base for SQLite. Each ran in a container with a filesystem store. I sent the same RestoreTableFromBackup requests, and the calls around a running restore, to each backend and to Amazon DynamoDB in us-east-1. I sent each error case twice. I killed, froze and stopped servers (SIGKILL, SIGSTOP, SIGTERM) at the pause points in this PR and at two I added. The two are right after the registration's commit and, on MongoDB, between the claim check and the insert_many. For the takeover cases a second server ran on the same catalog. I ran the engine's backup:: module (236 tests), the SQLite and PostgreSQL restore job tests (11 each), and the MongoDB and Cassandra restore cells. I ran 51 mutants of the code those tests guard, and for 45 a test failed on an assertion about the change. Clippy 1.97.0 with the test-hook features, which CI does not lint, passes.

The restore response and DescribeTable matched the service on RestoreSummary, on what a restore carries and drops, and on the tables restored with accepted overrides. They also matched on the errors for a missing backup, another account's backup, an ACTIVE target name and data-plane calls on a restoring table. On SQLite, PostgreSQL and MongoDB a restore of a backup in use answers BackupInUseException before any override error, as on the service. The fenced DONE write, the shutdown path, and remove-target on a target_not_empty record behaved as described. So did SQLite's takeover at restart and the job's first beat after a 75 s registration. Main has no store-backed restore, and the base answers ValidationException once a restore reads a store-backed backup, so each store-backed answer here is new. Two Blocking items, twelve Major and nine Minor ones are inline, and the Minor ones are marked. Smaller items:

• From #671: backup_in_use_message gives Backup is being used to restore table <name>: <arn>, where Amazon DynamoDB gives Backup is being used to restore another table: <arn> (eight requests). This PR's registrations raise that text on all four backends.
• a_feeder_whose_timer_stopped_ends_the_stream_with_the_claim_lost_reason failed once in 40 runs of the backup:: module on a loaded host, at the 30 s deadline in wait_for_items_seen. It passed in the other 39 runs and in three reruns. Is 30 s enough on a loaded CI machine?

Once the Blocking and Major items are fixed, I will approve.

[LeeroyHannigan]

Blocking: this commits the RUNNING record before the target's data collection, its unique key index and its GSI and LSI metadata exist. I killed the server right after this commit and restarted it, and the reconciler resumed the record into a collection indexed on _id only. The table went ACTIVE with no GSIs or LSIs, and Query on by-g answered ValidationException: The table does not have the specified index: by-g where the source returned 15 items. Please make the record resumable only after the last index metadata is written. Also have the resume check the target's indexes against the manifest.

[LeeroyHannigan]

Blocking: this read checks the claim, and the insert_many that follows carries no condition, so a frozen writer can write into a finished restore. With two servers on one catalog, I froze the owner with SIGSTOP after this call, and the other server took the record over and finished. I removed 10 items through it and resumed the owner, and 20 s later all 10 were readable again with ConsistentRead and missing from by-g. Please put the fence in the write here, as the PostgreSQL and SQLite loads do. The Cassandra batch at crates/storage-cassandra/src/restore_engine.rs line 182 needs the fence too.

[LeeroyHannigan]

On MongoDB I froze the owner inside its batch transaction, the other server finished, and the 10 items I deleted stayed deleted. On Cassandra the batch writes still follow a separate check, and the module doc says a stalled writer can write one batch after a takeover. Please fence those writes too, for example by writing each batch at a timestamp read before the check, so that later writes and deletes win.

[robinnsc]

Fenced. The Cassandra batch now writes each item's base row and its GSI and LSI rows in one logged batch USING TIMESTAMP a reading of the coordinator's clock taken before the batch's claim check (put_restored_item_at, replacing the put path's lightweight transaction for the store-backed load; a CREATING target has no other writer). A writer that stalls between the check and its writes and lands them after a takeover lands them below every write and delete the successor made, so Cassandra keeps the successor's cells and tombstones and a key deleted through the ACTIVE table stays deleted; the base row's version is the timestamp, so a later ordinary write's version is above it. test_a_batch_landed_after_a_takeover_loses_to_the_successors_writes_and_deletes (direct_integration, test-hooks) pauses the owner after its check, takes the record over, has the successor load the batch, flip ACTIVE and delete two items, then releases the owner: the two items stay deleted, their GSI and LSI rows stay gone, the survivors carry the successor's versions, and the owner's progress write reports the lost claim. With the timestamp read after the pause instead (when the writes land) the cell fails with 4 items in place of 2; it passes after the revert. The module doc states the rule and that it needs the hosts' clocks to agree, as the version scheme already does.

[LeeroyHannigan]

Progress is recorded once per data file, so a one-file restore records none until its load ends, and data files are 64 MiB by default. A SIGTERM in the middle of a 5,000-item one-file restore left the record RUNNING, and 39 s after the restart it was FAILED target_not_empty. The table then stays CREATING, with DeleteTable ResourceInUseException and DeleteBackup BackupInUseException, until an operator runs remove-target. But the differences row says a SIGTERM resumes to ACTIVE. Please mark the record before any insert_many and resume a marked record with reapplying (upserts on Cassandra). Keep target_not_empty for a count above the total.

[LeeroyHannigan]

On MongoDB a SIGKILL after the first batch and a SIGTERM inside it both resumed to ACTIVE with every item and the full GSI. On Cassandra a crash before the first progress record still ends target_not_empty, as the differences row now says. Please resume that case with upserts, as my comment asked.

[robinnsc]

Done. The reconciler's no-progress rule now compares the target's count with the backup's total (the backup row's item_count, exceeds_backup_total): above it is target_not_empty (the dump-restored catalog); at or below it the record is resumed and the first file re-applied with reapplying set, which on Cassandra is the same idempotent upserts. The engine cell a_half_applied_first_file_with_no_progress_is_resumed_with_upserts resumes one landed item of two to DONE with 2; the storage cell test_a_first_batch_applied_without_progress_resumes_to_the_exact_count lands 3 of 7 with no progress (two lost progress writes) and resumes to 7 with both index-row counts exact. The differences row, 5.3 step 7, the PR 9a load paragraph, the admin guide and the kill-point script's first-reach assertion (now ACTIVE with the exact count) are updated; the Cassandra point's first reach runs in the nightly job.

[LeeroyHannigan]

When the target's tables do not exist, this returns after claim_restore_dead refreshed the heartbeat, so the record is retaken every stale window and never fails. I killed SQLite and PostgreSQL servers after the catalog commit, before the data DDL, and each record was retaken every 45 s for three minutes. The table stayed CREATING, and DeleteTable, DeleteBackup and a restore into the name answered ResourceInUseException, BackupInUseException and TableInUseException. Please fail the record and call remove_restore_target when the target's tables are missing, and bound the takeovers of one record.

[LeeroyHannigan]

A target without data tables now fails in one takeover on SQLite and PostgreSQL. Takeovers stay unbounded otherwise: when a resume fails on a read error or stops with LeftRunning, the reconciler claims it again every stale window. Please count the takeovers of a record and fail it after a small number.

[robinnsc]

Bounded. ReconcilerState counts this instance's takeovers of each record by table id (RESTORE_TAKEOVERS = 5); the takeover after the fifth fails the record owner dead: taken over 5 times by one instance without the restore ending and removes its target as for OWNER_DEAD. The count is in process rather than a column: a takeovers column would reach every backend's record and PostgreSQL's catalog table_restores, which this PR's migration does not touch, so the count is per instance and a restart starts it over (stated in the module doc, the design and the differences row). the_takeover_after_the_fifth_fails_a_record_that_never_ends: the count fails on every attempt, five cycles take the record over and leave it RUNNING, the sixth fails and removes it. With the bound check disabled the sixth cycle resumes again and the cell fails; it passes after the revert.

[LeeroyHannigan]

fail_restore_job matches any RUNNING record, and an instance without [backup.store] takes this arm for every dead store-backed record it lists. So this write can fail a record another instance took over after the listing, and on PostgreSQL I queued such a takeover ahead of it. The record ended FAILED under the new generation, the new owner's DONE write for its 50 items matched no record, and the target was removed. Please condition this write and those at lines 689 and 719 on the listed owner_generation and a stale heartbeat. Also leave other stores' records alone.

[LeeroyHannigan]

Minor: a spooled file with a .verified marker is skipped here and later read as prehashed (load.rs line 118). So the restore loads a file changed in the spool between two runs without checking its digest again. Please hash spooled files at load, as the in-place path does.

[LeeroyHannigan]

Minor: download_all fetches every data file before the progress skip in restore.rs (line 674). So a resume on a host without the spool entry also downloads the files its progress records as applied. Please download only the files the progress leaves to load.

[LeeroyHannigan]

Minor: this prints its manifest restore from ... was FAILED (target_not_empty), the internal record kind, where the docs call these store-backed and in-catalog restores. Please print the documented terms.

[LeeroyHannigan]

Minor: on Cassandra the engine's in-use pre-check, backup_in_use at line 1571, reads only the legacy backups row, so it misses the restores this function registers. While one restore of a backup ran, a second with a GlobalSecondaryIndexOverride naming an unknown index answered ValidationException: One or more parameter values were invalid: global secondary index nope is not in the backup on Cassandra. SQLite, PostgreSQL and MongoDB answered BackupInUseException, as Amazon DynamoDB did (Backup is being used to restore another table: <arn>) to six such requests. Please have backup_in_use read the store-backed registration as well, and add a Cassandra cell to the in-use precedence test.

[LeeroyHannigan]

With a restore of the backup running, Cassandra answers BackupInUseException to the unknown GSI override, the throughput override and a plain restore, twice each. No test covers this on Cassandra yet. Please add the cell.

[robinnsc]

Added test_a_running_store_backed_restore_keeps_its_backup_in_use_until_active to the Cassandra direct_integration suite: with a store-backed restore RUNNING, backup_in_use names the target and a second claim_restore is BackupInUse with the service's text, twice; with the backup's active_restores entry deleted (the registration's window before it lands) the target rows alone keep the backup in use; after finish_restore it is free. The order of the engine's checks (the in-use rule before the override validation) is pinned by the engine's cells; this is the Cassandra input to it.

[LeeroyHannigan]

Minor: this writes the manifest's item total and the source's size at ACTIVE, so a restored 20-item table showed ItemCount 20 on every backend. TableSizeBytes was 0 on SQLite and MongoDB, 49152 on PostgreSQL and 1385 on Cassandra, and 1385 on the in-catalog path of the same SQLite build. Amazon DynamoDB showed ItemCount 0 and TableSizeBytes 0 on all 15 tables I restored there, one of them 10 minutes after it went ACTIVE. Please write 0 and 0 here. Otherwise, add the service's 0 and 0 to the DynamoDB column of the "What a restore carries across" row.

The base branch was changed.

[LeeroyHannigan]

At 0763a19, the handle and the restore decoder check the claim through ClaimCheck, which calls in_flight_confirms with self.parks. When I replace self.parks with true, the two clause cells from #700 fail on parks = false. Without the clause, they fail on parks = true.

I stopped a test process for 1.5 s between two beats while a thread polled a ClaimCheck like the decoder's. No check passed after the resume in three runs. With true for self.parks, checks passed for 6.2 ms.

restore_progress is 8 in the registry, the SQL file, the MongoDB counts and the upgrade manual. Each backend has one arm for 7 and one for 8. CI shows 43 of 43 and ran the MongoDB, Cassandra and PostgreSQL migration tests and restore_job.rs against servers. The engine's 273 backup tests pass on 1.88.0. Rustfmt and clippy are clean.

The design doc still gives restore_progress id 7 in 4(i)'s first parenthetical (docs/design/15-backup-restore.md:109) and at line 476. The MongoDB test every_body_is_code_and_three_four_and_seven_do_not_apply names 7, and its list holds 3, 4 and 8. Neither is a condition for approval.

[robinnsc]

The notes with your approvals of #700 and #740 are fixed in this head (15f38c72, rebased onto main at 3033e090 after #700 merged; no conflicts):

• Design 5.1 step 4 (feat(backup): CreateBackup job, worker and reconciler duties [backup/restore 8] #700's approval): the late-first-put sentence no longer says "with no beat in flight". It now says the put is refused on PostgreSQL, MongoDB and Cassandra whether or not a beat is in flight, and on SQLite unless a beat is in flight, parked on the write lock.
• Design 4(i) and the PR order paragraph: restore_progress is id 8 in both, with fix(storage-postgres): number stream records after their writes commit #819's stream_pending at 7.
• MongoDB test: renamed every_body_is_code_and_three_four_seven_and_eight_do_not_apply, and its list now holds StreamPending as well as RestoreProgress.
• Cassandra clock sentence and CI comment: as on their threads.

Engine backup:: with backup-test-hooks 273 passed, storage-mongodb migrations::tests 5, storage-cassandra lib 129 and backup_engine:: 64, clippy 1.97 clean.