[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework (merged) · PR 6 the catalog index · PR 7 configuration, the store identity file and backup purge · #700 the CreateBackup job (merged) · #740 the restore job, the cross-backend matrix and backup remove-target (merged) · #825 backup sync, --prune-missing, --adopt-store · #855 the CI filters, the open measurement cells and the backup and restore manual · #880 backup sync --move-store, --forget-store, --forget-backup · this PR: backup migrate-legacy · next: #748 export and import through the store, #749 the in-catalog reader's removal.
Closes #821 (part of #624).

Stacked on #880 (feat/backup-store-maintenance, head 37fde534), itself on #855 (feat/backup-docs, 695dc66b) and #825 (b42dbfa9) on main at b8ad9cd5 (#740 merged as 76b9710d, then #792); opened as a draft against feat/backup-store-maintenance. Rebased from #880's earlier head 537097af with git rebase --onto 37fde534 537097af, after #825 gave HeldClaim the job's two-period rule, recheck() and the trait's heartbeat_within and heartbeat_parks_before_its_bound (SQLite's overrides), added the claim to purge_row, prune_missing, adopt and SyncTarget, and stopped plain SQLite sync's dead-owner resolutions, and #880 re-checked the claim before each step of the move and the forget options that cannot be undone. Five files conflicted. maintenance.rs: this PR's HeldClaim kept (a superset: the on-schedule restriction, lost_signal, acting_deadline()), with #825's module doc, purge_row's claim parameter and its two re-checks, and wait_for_quiet's doc taken from the base; of the two sides' HeldClaim cells, this PR's versions of the four both have, #825's recheck() assertion that a claim read as lost gets no beat added to this PR's, and #825's a_beat_in_flight_confirms_the_claim_only_where_the_beat_parks kept, its resumed SQLite beat now confirming nothing (it started late). The engine's Fake fields, SQLite's two maintenance-beat cells (both kept) and its trait docs, and the design's (c) paragraph and revision log (item 37 after #880's 36, no renumbering) took both sides. Without a conflict, sync.rs's new in-catalog skip needed .into() for #825's NotAdopted. The commit is 43 files, +12,619 −456. Before that, rebased from #880's head c96cb472 with git rebase --onto 537097af c96cb472. Two files conflicted: the admin guide's [backup] table, where main changed the heartbeat_interval_secs row (on PostgreSQL, MongoDB and Cassandra a job's beat in flight no longer counts) and this PR the stale_job_after_secs row, so each row is taken from the side that changed it; and the upgrade manual's [backup.store] paragraph, which keeps this PR's migrate-legacy sentences with main's "Migration 8" (restore_progress) twice. The design's revision log merged without a conflict; item 37 follows #880's 36, so no item is renumbered. The rebase left the commit the same size (38 files, +11,610 −311); with the changes the Testing section lists first, it is 43 files, +12,906 −363. One commit. No migration, no schema change, no new dependency; Cargo.lock is unchanged. This is the third of the three PRs the design's PR 10 ships as (revision item 34). Every binary and measurement below is a debug build of the tree before the rebase (c96cb472 plus this diff), unless a line says otherwise.

What

Format (crates/core). SnapshotDescriptor::KIND_LEGACY_IN_CATALOG = legacy_in_catalog, whose doc says the marker is empty, that the copy has the legacy create's consistency (a point-in-time snapshot on PostgreSQL and SQLite; on MongoDB an $out of the live collection and on Cassandra a paged read of the live table, neither point-in-time), and that source_backend and source_extenddb_version name the binary that wrote the copy (the in-catalog creator's version was never recorded). No format version change and no fixture byte change; a unit test parses and round-trips a manifest of that kind. is_valid_backup_name is exported, so the command refuses a name the manifest would refuse before it writes anything.

The definition (crates/storage/src/backup_definition.rs). source_table_from_legacy(source, vector_state, region, account), beside restore_input: the table the legacy restore creates, by one rule on every backend. It runs ensure_restorable, refuses LegacyVectorState::Unknown (a SQLite backup with no definition), and otherwise takes restore_input with no overrides (a backup with no definition keeps its keys, its key attribute definitions and the 5/5 fallback) plus the definition's table class, TTL and stream and SseDescription::from_specification of its SSE specification. The vector state is [] on PostgreSQL and SQLite with no names (PostgreSQL with no definition and a NULL or zero snapshot too) and null on MongoDB and Cassandra. Tests: an exhaustive property over every combination of key schema, billing, recorded throughput, GSI shapes, LSI, four SSE specifications, on-demand throughput, class, TTL and stream and no definition, times two vector states and four override sets, that restore_input_from_manifest(source_table_from_legacy(s)) restores what s.restore_input restores; the one difference, a KMSMasterKeyId not carried, has a test of its own.

Storage trait (crates/storage/src/backup_catalog.rs). LegacyBackupReader gains items(account, arn) -> LegacyBackupItems (the name, the recorded source, the vector state and a 'static item stream), list_all() -> Vec<LegacyBackupKey> (the in-catalog backups of every account that are not DELETED, by creation time then ARN; a row that does not decode is an error), list_deleted() (the DELETED ones, which are always a DeleteBackup's), holder_status(account, arn) -> Option<BackupStatus> (the status, DELETED included, of an in-catalog backup at the ARN that the replacement did not replace), finish_replacement(account, arn) -> bool (MongoDB's finish of a replacement that committed and stopped before its drop; false elsewhere, the trait's default), remove_deleted(account, arn) -> bool (the removal of a DELETED in-catalog backup that is not a replacement's, conditioned on both, once nothing is left under its prefix) and replace_with_index_row(row) -> Replaced { Replaced, AlreadyReplaced, NotAvailable(String), InUse(String) }; an index row of the id under another ARN or store is AlreadyExists with nothing written, whatever the in-catalog backup's status (Cassandra sees only the ARN's own partition, which the trait doc says, and the engine covers). The store trait (crates/backup-store) gains delete_written(&ObjectMeta) -> bool, the deletion of the object a put of the store answered with and of no other: S3 DeleteObject with If-Match on the put's ETag (412 leaves the object, a missing key is gone); a filesystem root renames the file to a hidden temporary name beside it, removes it if its size and modification time are the put's and links it back otherwise; the default, for test doubles, is unconditional. The engine's BoundedCalls forwards it; a conformance cell runs it on both stores. BackupCatalog gains row_holding_backup_id(account, backup_id), the row of the id whatever its ARN, table and store, the key the inserts conflict on; Cassandra, which indexes no backup id, answers None, and the doc says why. The trait doc names the replacement as its second write beside the restore's registration. MaintenanceClaim gains heartbeat_within(bound) and heartbeat_parks_before_its_bound(), with JobClaim's defaults; SQLite's starts the bound once the beat holds its write lock.

PostgreSQL, SQLite, MongoDB, Cassandra. Each stream is a frozen copy of the backend's restore read: PostgreSQL a REPEATABLE READ READ ONLY catalog transaction whose first statement re-checks AVAILABLE, owned by a spawned task sending through a one-slot channel; SQLite the restore's keyset batches (500 rows or 4 MiB), each in its own read transaction re-checking AVAILABLE; MongoDB a cursor on _backup_<backup_id> with batch_size(DEFAULT_FETCH_BATCH) and legacy_item_from_document, a copy of document_to_item, and Unsupported for a document with no backup_id; Cassandra the buckets in order, each read in pages of LEGACY_ITEMS_PAGE = 8 rows by item_index, at consistency ALL (legacy_items_page_params): the in-catalog create wrote each row at ONE, so a read of fewer replicas could end short, and a replica that does not answer (unavailable, timed out or failed) fails the backup with "a replica of the account keyspace did not answer; run again when every replica is up" (legacy_items_read_error). The replacement: PostgreSQL one transaction (the backups row FOR UPDATE, the CREATING-target check, the insert statement insert_backup_row used, now insert_unowned_row, then the items, the definition and the backups row deleted, so a DELETED backups row is always a DeleteBackup's); SQLite the same under the write lock and BEGIN IMMEDIATE; MongoDB start_replacement, one session transaction ($inc lock_seq while AVAILABLE, the CREATING-target check, the upsert, DELETING with migrated_to_store: true), then finish_legacy_delete's drop and DELETED, a DELETING document with an index row of the same ARN and store being finished as AlreadyReplaced, and a commit whose answer is lost settled by replacement_landed (the marked document and the index document read back: the call's own replacement answers Replaced); Cassandra replace_legacy_row, the row converted in place by one lightweight transaction IF backup_status = 'AVAILABLE' AND store_kind = null AND active_restores = null after creating_restore_target, then write_index_entries (sent once more on a failure) and cleanup_backup_payload, an unknown outcome settled by read_index_row_serial, NotMatched read again up to three attempts, and a converted row without its entries finished by the next call, its entries written only for a row the serial read finds AVAILABLE (one deleted since gets its buckets deleted and NotAvailable). holder_status and list_deleted: the backups rows on PostgreSQL and SQLite; the documents without the marker on MongoDB, whose finish_replacement finishes a marked DELETING document whatever its index row holds; the rows with store_kind null on Cassandra. remove_deleted: DELETE ... AND backup_status = 'DELETED' on PostgreSQL and SQLite (SQLite under the write lock), MongoDB's delete_one of a DELETED document without the marker, and on Cassandra a lightweight transaction IF backup_status = 'DELETED' AND store_kind = null, a lost answer settled by a read of the partition. row_holding_backup_id: the backup_index row by (account_id, backup_id) on PostgreSQL and SQLite, the index document by the same fields on MongoDB. PostgreSQL's maintenance take, with the advisory lock taken, now refuses while the backup_maintenance row's heartbeat is younger than stale_job_after_secs (the session closed, AlreadyExists naming the holder and the window), as the other backends' take does; the module doc says why. Cassandra's conversion writes the insert's columns, created_at among them, which a unit cell compares with the insert's column set. Four frozen paths change, each to order a request wholly before or after a replacement: PostgreSQL's delete reads the status under its FOR UPDATE lock and answers not found for anything but AVAILABLE; MongoDB's reads a DELETING document marked migrated_to_store as not found, in delete and in the lock's miss; Cassandra's DELETING claim carries AND store_kind = null, and so does Cassandra's legacy restore registration (register_restore(.., in_catalog)), whose false re-reads through load_backup and answers not found for an index row. SQLite's are unchanged: the command holds the serve lock. On PostgreSQL, SQLite and MongoDB the replacement of an in-catalog backup that is not AVAILABLE answers AlreadyExists when a row of its id under another ARN or store exists, as the AVAILABLE path does, rather than NotAvailable (after which the engine deleted the copy). Module docs name the stream and the replacement. Fake implements the seven reader methods and row_holding_backup_id with fault knobs, its holder_status answering None for a marked backup as MongoDB does and its AlreadyReplaced keeping a DeleteBackup's DELETED key as the backends do; PostgreSQL's test-hooks gain legacy_items_pause (after the stream's AVAILABLE re-check, before its first read); Cassandra's test-hooks gain fail_convert_writes, stop_convert_after_row, pause_before_convert and pause_before_legacy_restore_registration; MongoDB's test liveness gains replacement_commit_unknown.

Engine (crates/engine/src/backup/migrate_legacy.rs). migrate(storage, &Migration, Selection::{Arn, All}, on_backup) -> MigrateReport, written against BackupCatalog + LegacyBackupReader and the configured store. --all takes list_all, then list_deleted; <arn> is resolved through holder_status under its two spellings (no listing of every account), then finish_replacement, then the catalog row (AVAILABLE "already in the store", DELETING or DELETED "deleted", after the replacement's call; another store's or a CREATING or FAILED row refused). Per backup, in this order: the id (no instant: refused before anything is read, the text naming 0.1.x and the way out); the table name by the store-prefix rules and the partition; a DELETING backup handed to finish_replacement; a catalog row of the id under either spelling (read_catalog_row: one of the configured store under the stored spelling is finished, reported by its status; any other refuses the backup before anything is deleted, since the prefix carries no region), then a row of the id under any ARN (row_holding_backup_id), refused the same way; the status (CREATING and DELETING skipped, named); the open (Unsupported refused); the name; ensure_restorable ("the legacy restore refuses it too"); the unknown vector state. Then the copy: a listing of the prefix, whose objects with no row are an earlier run's when the manifest there, if any, names this backup (StoreCalls::manifest_arn; one that names another backup or does not read refuses the backup with the prefix left, which on Cassandra is what tells another row's prefix), and are deleted through the fence; the fence before the first put; writer::write_data_files with the item validator under the instance's limits; the count check (its text names PostgreSQL's case, a wrong recorded count, and the way out); the three documents and the manifest (unordered, legacy_in_catalog, no marker); verify_prefix and the second count check; the claim re-checked; replace_with_index_row with the row built as sync builds it (the manifest's name, size, item total and source_table, the id's instant as created_at, unowned). Every object of the copy is written with put_if_absent (Pipeline::written records each object stored, in order; a backup job's pipeline has none and keeps put), and an object already under a key stops the run with that object left (Stop::Occupied). A refusal or failure after the first put deletes the copy through the fence, object by object, the last written first, with delete_written, reading lost() before each (delete_copy; a signal's stop adds a note naming the deleted copy, and an object another run wrote is noted and left); AlreadyExists at the replacement keeps it, with a note. The prefix walk is kept for a leftover the steps above have shown no row's (delete_leftover): each pass follows a beat and is delete_prefix_before(prefix, HeldClaim::acting_deadline()), the instant two periods after the last confirmed beat, and is dropped when the timer records a loss. On Cassandra the count refusal names a repair of the account's keyspace before the rerun, not the legacy restore, which reads at ONE. After a stream error or a short count the command re-reads holder_status and reports Skipped ("deleted during the copy") when the backup is no longer AVAILABLE. Before each step that cannot be undone (the manifest's put, delete_copy, the leftover's deletion, clean_deleted's deletion, every replacement call, finish_replacement and the record's removal) it calls HeldClaim::recheck(); a failure stops the run with what it wrote left, and the stop says so. A DELETED backup goes to clean_deleted: a catalog row of the id under either spelling or any ARN refuses it (one under its own ARN and AVAILABLE means the backup is in the catalog again, after a backup sync of an earlier release adopted a stopped run's copy: the refusal names the row and the choice, DeleteBackup the ARN again or keep it, in --all too and with no audit line; any other may own the prefix), then the listing, the fence, the copy's deletion when its manifest, if any, names this backup, the listing again, and once that lists nothing remove_deleted, each with a note; so list_deleted shrinks, and a DELETED in-catalog row whose id carries an instant is a prefix not yet confirmed clean. --all reports only a deleted backup it did something for, and an id with no instant is never listed in the store nor removed. Outcome::{Migrated, Finished, InStore, Deleted, Refused, NotFound, Skipped, Failed}, carried on BackupOutcome with in_catalog_kept (false for a deleted backup, after a replacement call whose answer did not come, and once the claim is lost) and wrote (the run deleted or replaced anything for it; finish reads it from holder_status, clean_deleted from what it removed); MigrateReport::stopped for a lost claim, a signal, an unhealthy or fenced store, a failed listing, a catalog call and a replacement whose call fails, and for a signal received while the last replacement ran. The writer (job/writer.rs) runs under a ClaimWatch (confirms_the_claim, the loss token, the reason): Pipeline<'a, W: ?Sized = HeartbeatHandle>, so the job's Pipeline<'_> is unchanged, and impl ClaimWatch for HeldClaim, whose timer cancels a lost_signal token when it records the loss. maintenance.rs: #825 gave HeldClaim the job's rule (last_confirmed, in_flight, parks, each beat bounded by heartbeat_within(interval), a timeout a loss, lost() answering once the last confirmed beat is two periods old unless the beat parks before its bound, SQLite, and one is in flight, and recheck()); this PR keeps that shape as a superset: last_confirmed is dated when the beat's answer arrived, in_flight holds the timer's beat's start, and a parked beat in flight counts only when it is on schedule. A beat is on schedule when it started within 1.1 periods of the last confirmed one (the beats fall one period apart from the take, interval_at), so a process stopped and resumed starts a late beat that confirms nothing; acting_deadline() is the instant two periods after the last confirmed beat; the flight is marked by a drop guard (InFlight), cleared by a timer that ends mid-beat, and a confirmation is recorded before the flight ends. recheck() is one beat under the period's bound that counts as a confirmed beat or records the loss as the timer does; purge's loop and the waits and loops of backup sync's options read the same lost(). The writer's refusal of items is its own stop, Stop::Refused, which a backup job fails as it fails Stop::Fail. put_document is generic over the watch. Under backup-test-hooks the writer's after_data_file and the command's before_manifest and after_manifest kill points pause it as they pause a job.

Sync, Describe, Delete, Restore and ListBackups. sync.rs: SkipReason::InCatalog { backup_arn, status } ("an in-catalog backup (STATUS) holds this ARN; run extenddb backup migrate-legacy ", and for DELETED "an in-catalog backup with this ARN was deleted; this prefix is a copy a stopped migration left, which extenddb backup migrate-legacy deletes"), not transient, from holder_status under both spellings before the row read under them; sync, finish_group, adopt_prefix and startup_sync now need LegacyBackupReader, which every caller's storage has. backup.rs: DescribeBackup, DeleteBackup and RestoreTableFromBackup read the catalog row once more after a legacy not-found (RestoreTableFromBackup after the source read's and after the registration's, then taking the row's path, restore_from_row), and ListBackups reads the legacy reader before the catalog index, then drops an in-catalog summary whose ARN a listed catalog row has.

App (crates/app/src/cmd_backup/migrate_legacy.rs, a child of cmd_backup). BackupCommand::MigrateLegacy { arn, all } with a required ArgGroup (usage migrate-legacy <ARN|--all>, exit 2 for neither or both) and <arn> checked by a clap value_parser (backup_arn_arg, exit 2 before anything is opened), Mode::Write, the registration name extenddb backup migrate-legacy. The command needs Identity::Known, takes the claim on every backend through take_claim(storage, config, command), which purge and sync now share with it, starts HeldClaim, runs wait_for_jobs over the configured identity (on SQLite after resolve_dead_owners, with a zero window), runs the migration, releases the claim, prints one line per backup with its notes indented (outcome_lines; "Its in-catalog backup is kept" after a refused or failed line only where in_catalog_kept holds, so never for a deleted backup nor after a replacement call whose answer did not come) and the summary line (N migrated, N finished, N already in the store, N deleted, N refused, N skipped, N failed), logs one extenddb::audit line per backup the run deleted or replaced anything for (audited, by wrote; none for an outcome that wrote nothing), and exits 0 only when every backup reached is in the store or deleted. The refusal with no store configured says the command needs one. On SQLite the dead-owner pass logs its removal of an in-catalog backup left CREATING under extenddb::audit, for every command that runs it. backup_arn_arg returns the bare reason, so clap's error states it once. The subcommand's help names the item check's and the legacy restore's refusals and --all's clean-up after deleted backups; lib.rs's help and the module doc name the command.

Harness. tests/test_backup_job.py: four new cells, one extended (below); helpers _catalog_rows, _in_catalog_rows, _copy_in_catalog_backup, _keep_in_catalog_backup, _put_back_in_catalog_backup, _typed_canonical, _migrate. tests/test_backup_store_config.py: the serve-lock cell runs migrate-legacy beside a SQLite server.

Docs and texts. The backup manual's Legacy backups section: "Migrating legacy backups into the store" in place of "This release has no command that converts a legacy backup" (the two forms; what is kept; the check and the one-step replacement per backend; the cost: the claim, every backup and restore job answering LimitExceededException, the wait, SQLite stopped, a killed command's claim held until its heartbeat is older than stale_job_after_secs (PostgreSQL too; SQLite's serve lock takes it), PostgreSQL's held REPEATABLE READ transaction and the vacuum it holds back; <arn> to migrate in pieces; sample output, and that the log lines share standard output; the outcomes, the notes and exit (audit lines only for a write, "deleted" with the record's removal, "deleted during the copy", where "Its in-catalog backup is kept" is said, the claim re-checked before each step that cannot be undone); the refusals, every 0.1.x backup first, with the list's lead-in qualified and the list completed (the count check, a row of the id under another ARN or a manifest of another backup under the prefix, a CREATING or FAILED row of the configured store, the partition, AlreadyExists at the replacement, a deleted backup an AVAILABLE row holds again); the KMS sentence (the stored specification loses the key id, and no answer reports one before or after); the restore-and-back-up way out, kept from the earlier text; what a migrated backup reports; a stopped run, sync's two skips and a deleted backup's copy and record (the guarantee rests on the record, which --all removes once the prefix is clean, so --all is run after the last DeleteBackup of an in-catalog backup before the release that removes the reader); MongoDB's and Cassandra's stopped replacements, and running <arn> before deleting a Cassandra backup whose conversion stopped; Cassandra's recurring --all, which exits 1 while a create is in flight; memory, measured on this change's earlier revision, with Cassandra's table shape); the vector-index sentence corrected; the Strategy section's Cassandra paragraph and the catalog-lost paragraph point to it. The upgrade manual: the store paragraph, the 0.1.x paragraph (refused on their ids), the rollback paragraph (a backup migrated after the upgrade is adopted by --adopt-store like any other backup this release wrote) and --all run after the last DeleteBackup of an in-catalog backup, and why. The admin guide: the command's paragraph (the leftover rule by id and manifest, the record's removal, the re-check, the line kinds, the audit lines, memory) and sample output with its refused line, the registration names, and the stale_job_after_secs row (a dead maintenance command's claim, on PostgreSQL too). The differences doc: "Where backups live" and the maintenance-claim row. The manual and the admin guide also state the copy's conditional writes and its deletion of only what it wrote, what a store call under way at a stop can still do, Cassandra's reads at ALL and its count refusal, a second signal's exit 130, the SQLite dead-owner pass's removal of a CREATING in-catalog backup, the full list of maintenance commands a second one of which fails, the store check's log line before the command's first line, and that on MongoDB a DELETED document without the migrated_to_store marker is a DeleteBackup's. Design: revision item 37, with sections 3, 4(c), 4(f), 4(g) and 7, PR 10's claim paragraph (PostgreSQL's take needs the stale window), the PR 10 paragraph and transitions, the PR 14 lines (its migration also refuses while a DELETED in-catalog row of a DeleteBackup whose id carries an instant exists, naming --all; a DELETING one names DeleteBackup again) and section 10's upgrade-manual row edited with it.

Why

Design section 3, 4(f), 4(g) and the PR 10 section's migrate-legacy paragraph, and PR 14's precondition: before the in-catalog reader is removed, operators need a way to move the backups it holds into the store without losing their ARNs. Until now the only way was a restore and a new backup, which gives a new ARN, a new creation time and, for a table that still exists, an interim table. Cassandra keeps writing in-catalog backups until PR 14 settles its backup path, so there the command is a recurring step.

Choices to weigh

• One replacement step per backend, not "insert the row, then the reader's delete". DeleteBackup does not read the maintenance claim. In the design's order a DeleteBackup that landed between the AVAILABLE check and the insert was answered DELETED for a backup that then came back as the migrated row, and a restore target in CREATING made the delete refuse after the insert, leaving both rows and two ListBackups entries. On Cassandra the order cannot work at all: the legacy row and the index row share the backups_by_arn key, and the frozen delete removes the list rows the migrated row uses and leaves a DELETED row that every later insert answers false for. The cost is one write method on the frozen trait (as the restore's registration already is) and four small changes to frozen paths (three deletes and Cassandra's legacy registration), each pinned by a test and a mutant; PR 14 removes all of it with the trait.
• Every 0.1.x backup is refused. The released fixtures carry 13-digit ids (v0.1.14 is the newest tag, and the shared 14-digit id came after it), and the store format's ids carry their creation instant, which the manifest parser checks. So the command migrates only in-catalog backups taken by this release line: without a store, with kind = "disabled", and on Cassandra. The texts say so plainly, and the refusal names the way out. The pre-0.0.4 SQLite refusal is in practice a subset of this one; it stays as a defensive check, reachable through a hand-seeded row, with its own SQLite cell.
• The source table is what the legacy restore creates. One rule on every backend, restore_input with no overrides plus the features DescribeBackup reports, so a migrated backup restores the table the in-catalog backup restored. The design refused only the pre-0.0.4 SQLite case; a backup with no definition on PostgreSQL, MongoDB and Cassandra migrates with its keys and the 5/5 fallback, as its legacy restore creates it. What the legacy restore refuses (vector indexes, a multi-part key) is refused here too, so the migration lifts no refusal. The SSE specification's KMSMasterKeyId does not survive the manifest's SSE description; the manual names it.
• Extra refusals the design does not list: a table name the store-prefix rules refuse, a backup name the manifest refuses (the name survives with the ARN or not at all), a MongoDB backup with no item collection, and an ARN a row of another store holds. Each keeps the in-catalog backup and writes nothing.
• The claim on every backend, and the deletion fence. The design's invariants say every writing command takes the maintenance claim, and 4(f)'s list did not name this one. On SQLite it costs nothing under the serve lock and gives one code path. The fence runs before the first put, as CreateBackup's does, and before each deletion of a copy. The cost, stated in the texts: every backup and restore job answers LimitExceededException for the run, and on SQLite the server is stopped. <arn> migrates in pieces.
• Always rewrite after a crash. Objects under the prefix with no row are deleted through the fence and the copy is written again. Reusing a complete copy would save one copy after a crash and needs a second rule; one rule is simpler to state and to test. Meanwhile backup sync skips such a prefix, naming the command, so a sync between the crash and the rerun cannot give the backup two rows. Cassandra already skipped it through the insert's false.
• A deleted backup never comes back from a copy. A stopped run's copy can outlive its in-catalog backup: a DeleteBackup of the legacy ARN does not read the claim, and the copy's deletion can fail. So the in-catalog row in any status, DELETED included, bars sync's adoption, and the command deletes the copy of a DELETED in-catalog backup that no catalog row holds. Telling a DeleteBackup's DELETED row from a migration's needs one mark per backend: the replacement deletes the backups row on PostgreSQL and SQLite (the frozen delete keeps its tombstone), MongoDB's migrated_to_store marker already tells them apart, and Cassandra's converted row is an index row. list_deleted lists every DeleteBackup's tombstone, so --all lists one prefix per deleted in-catalog backup with an instant in its id (none of a 0.1.x release's). Once that prefix lists nothing and no catalog row holds the id, --all removes the tombstone (remove_deleted), so the list shrinks instead of growing with every DeleteBackup, and a tombstone left means a prefix not yet confirmed clean. The guarantee therefore rests on those tombstones until --all has cleaned them: PR 14's migration is to refuse while one with an instant-bearing id exists, and the manuals say to run --all after the last DeleteBackup of an in-catalog backup. A row of the id under any ARN keeps the prefix, since the inserts conflict on (account_id, backup_id) and a prefix carries no region: row_holding_backup_id on PostgreSQL, SQLite and MongoDB, and on Cassandra, which can read only an ARN's partition, the manifest under the prefix, which must name this backup.
• No maintenance command acts after another took its claim over. A maintenance claim whose heartbeat hangs (a half-open connection), or whose process was stopped and resumed, used to stay "confirmed" to the writer and to the check before the replacement while the stale window let a second command take the claim over and act on the same prefix; and PostgreSQL's take succeeded as soon as the advisory lock was free, so a command whose catalog session ended could still run until its next beat while a second replaced the backup (the first then put its manifest over the second's copy, failed its re-read and deleted the prefix, leaving an AVAILABLE row with an empty prefix). Three changes hold one bound: on PostgreSQL, MongoDB and Cassandra a holder starts no store or catalog call more than two periods after the answer of its last confirmed beat, which wrote the record's heartbeat at most one period before that answer, and a taker needs the heartbeat older than stale_job_after_secs, which the config holds to at least three periods, so a holder starts nothing once a take can succeed (a margin of the window less three periods, zero at the floor; on SQLite the serve lock keeps every other process out): HeldClaim bounds each beat by a period and reads a claim with no beat confirmed within two periods as lost, counting a beat in flight only where it parks, as the job's rule does (SQLite, and there only when it started on schedule); PostgreSQL's take refuses while the row's heartbeat is younger than the window, as the other backends' take does; and the command calls HeldClaim::recheck(), one live beat, before each step that cannot be undone, since the job's other fences (generation-conditioned writes, a prefix of its own) have no counterpart here. The cost: after a killed command on PostgreSQL the next one fails, naming the holder, until the window passes (Breaking changes). The bound and the on-schedule rule also apply to purge and the options of backup sync, which read the same lost(). What the bound does not cover is a store call already sent when the process stopped, which completes on resume; the next choice covers it for the copy.
• Two runs under one prefix change neither's objects. A backup job's prefix is its own; a migration's is not, since a run stopped past the stale window can resume after a second run took the claim and migrated the backup. Writing the copy with put and deleting it with delete_prefix let such a run, stopped at a data file's rename or at its deletion's first removal, replace or remove a file of the second run's AVAILABLE copy. So every object is written with put_if_absent, and a run deletes only the objects it wrote, each conditioned where the store can: on S3 on the put's ETag, which does not tell apart another object with the same bytes; on a filesystem root, with no conditional unlink, by moving the file aside and back unless its size and modification time are the put's, which leaves another run's file under a hidden name only if the process is stopped again between the two moves. The prefix walk remains for a leftover no row holds, bounded by the claim's deadline. The cost: one DeleteObject per object of a failed copy instead of a DeleteObjects of the prefix, and empty directories left on a filesystem root after a copy's deletion (listings ignore them).
• The count is checked twice, against the recorded count. A DeleteBackup during the read ends the SQLite, MongoDB and Cassandra streams short; PostgreSQL reads a pinned snapshot. The manifest's table_size_bytes is the sum of the copied items' sizes, as the job records it, so BackupSizeBytes and TableSizeBytes can differ from what the in-catalog backup reported; the name, ARN, id, creation times and item count do not.
• --all goes on past a refused, skipped or failed backup and exits 1 at the end with the counts; a global condition (claim lost, a failed re-check, signal, store unhealthy or fenced, listing or catalog failure, a replacement whose answer did not come) stops the run at once.
• A Cassandra conversion that stopped after its row is an index row, which list_all does not list, so --all does not finish it; migrate-legacy <arn> does (its "already in the store" path runs the replacement once more). Until then the backup is served by Describe, List and Restore, and backup list, which reads status entries, does not show it. The manual says so.
• ListBackups dedupes by ARN. It reads the legacy reader before the catalog index, so a replacement that commits between the two reads lists the backup in both, and the filter keeps the catalog row's summary. The other way to list both is a backup sync without the in-catalog skip (a feat(backup): backup sync, --adopt-store and --prune-missing [backup/restore 10] #825 or feat(backup): backup sync --move-store, --forget-store, --forget-backup [backup/restore 10b] #880 build during a rolling upgrade) adopting a stopped run's copy before the migration replaced it. DescribeBackup, DeleteBackup and RestoreTableFromBackup read the catalog row once more after a legacy not-found, for the same window.
• The stream reads are frozen copies, not calls into the data path, so the data path stays free to change before PR 14. Cassandra's bucket read is paged (8 rows of a 64-row bucket), where the restore's copy reads a whole bucket, about 1.7 GiB at the line cap; the restore's copy stays as it was.

Testing done

On the tree rebased onto 37fde534 (#880; #855 695dc66b, #825 b42dbfa9, main b8ad9cd5), debug builds, every tracked file touched: cargo fmt --all -- --check and cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, also with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; cargo +1.88.0 check --locked --workspace --all-targets clean; engine --lib 587 passed (nine HeldClaim cells: this PR's eight and #825's in-flight cell), app 64, server 344, storage 241; SQLite --lib backup 77 (#825's maintenance-beat cell and this PR's both); PostgreSQL backup_restore 74 (--test-threads=2); MongoDB live, this PR's nine cells and the insert cell, 10 passed; Cassandra direct on a node of its own, this PR's nine cells and the two insert cells, 11 passed, and the three unit cells of the read at ALL and the conversion; the wire suites tests/test_backup_store_config.py tests/test_backup_job.py on the SQLite build 49 passed, 14 skipped (-n 4) and on the PostgreSQL build 43 passed, 20 skipped (-n 2). Mutant: HeldClaim counting a parked beat in flight whatever its start, killed by a_beat_that_starts_after_a_gap_confirms_nothing and #825's a_beat_in_flight_confirms_the_claim_only_where_the_beat_parks.

Before that, on the tree rebased onto 537097af (main at b8ad9cd5), debug builds, every tracked file touched: cargo fmt --all -- --check and cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, also with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; engine --lib 565 passed, app 63, server 343, storage 241; SQLite --lib backup 75; PostgreSQL backup_restore 74 (--test-threads=2); the wire suites tests/test_backup_store_config.py tests/test_backup_job.py on the SQLite build 47 passed, 13 skipped (-n 4), as before the rebase.

With the changes since (the in-flight rule of HeldClaim, the copy's conditional writes and own-object deletions, the leftover walk's deadline, Cassandra's reads at ALL, the replacement's AlreadyExists in any status, the shared claim take and the texts), on the rebased tree, debug builds, every tracked file touched, the same host:

• cargo fmt --all -- --check and cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, also with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks.
• Engine --lib 575 passed (49 in backup::tests::migrate_legacy, 8 new: an object another run wrote stopping the copy with it left; a failed copy deleting only its own objects, by key, the last written first; a loss read during that deletion leaving the rest, with the in-catalog backup no longer said kept; the leftover walk under the claim's deadline, and one that makes no progress failing the backup; a short read of a backup gone since skipped as deleted; a deleted backup whose other spelling a row holds refused with no id index; the Fake's AlreadyReplaced keeping a DELETED key; the count refusal naming a repair on Cassandra only), and in HeldClaim a late beat in flight confirming nothing where it does not park (the paused-clock case: three periods after the stamp the claim reads lost) and the deadline two periods after the last confirmed beat; the signal cell now checks the note naming the deleted copy. App --lib 63, server 343, storage 241. Store: --lib 37, conformance 130 with moto (delete_written on a filesystem root and on S3: its own object removed, another of the same size under the key left and its bytes kept). SQLite --lib 294 (1 ignored), with the maintenance beat parked on the write lock past its bound and confirming, and the replacement of a deleted in-catalog backup answering AlreadyExists for another store's row; PostgreSQL backup_restore 74 (--test-threads=2), with the same replacement case and the frozen-delete cell renamed for two concurrent DeleteBackups; MongoDB's 9 live cells with the same case; Cassandra's 9 direct cells on a node of their own (a converted index row left DELETED not removed by remove_deleted) and 2 unit cells (the item pages read at ALL; Unavailable, ReadTimeout and ReadFailure reply bodies decoded and named).
• Wire, Python 3.10, moto variables: tests/test_backup_store_config.py tests/test_backup_job.py on the SQLite build 47 passed, 14 skipped (-n 4), on the PostgreSQL build 43 passed, 18 skipped (-n 2); the migrate and second-command cells on the MongoDB build (a replica set of their own) 3 passed, 2 skipped. New: test_a_second_maintenance_command_fails_at_once_beside_a_live_holder (PostgreSQL and MongoDB: backup sync --prune-missing waiting on a FAILED row's owner window; purge --confirm, migrate-legacy --all and sync --adopt-store refused within a second each, naming the holder; a SIGTERM stopping the holder before any change; a plain sync after).
• The two stops of a first run across a second run's migration, 45,000 items padded to 1,000 bytes, stale_job_after_secs = 45, PostgreSQL (the first's claim session ended) and MongoDB, the first run stopped whole under gdb: at the link of its first data file (a second run with 64 MiB data files migrated past the window; the first, resumed, stopped at "data/000001.json.gz is already under ...", exit 1, its temporary removed), and after a SIGTERM at the first object of its copy's deletion (both runs with 32 MiB data files; the second deleted the leftover and migrated; the first, resumed, moved the second's data/000001.json.gz aside, found it not its own, linked it back and noted "holds an object this run did not write, which is left"). In all four the second run's prefix, sizes and sha256 were unchanged by the first, the row AVAILABLE, and a restore held 45,000 items. With put and delete_prefix the same stops left a data file that did not match its manifest, and a prefix missing one.
• Mutants, each a targeted edit with the named cells run and the file written back by hand: 26, all killed (169 in all): HeldClaim counting a beat in flight where it does not park, a deadline of three periods; the copy written with put; its deletion by the prefix, or without reading the claim before each object; the leftover walk with no deadline, or by delete_prefix; a short read of a gone backup taken as AVAILABLE; clean_deleted reading one spelling; the item refusal as a plain failure; the signal's note dropped; the Fake removing a DELETED key; the Cassandra count text on every backend; the in-catalog backup said kept with the claim lost; delete_written matching on size alone or ignoring the modification time on a filesystem root, and without If-Match on S3; SQLite's beat with its bound over the lock wait, or not parking; the not-AVAILABLE replacement answering NotAvailable for another store's row on SQLite, PostgreSQL and MongoDB; Cassandra's pages at ONE, a timed-out or failed replica not named, remove_deleted without store_kind = null; the shared take with a zero stale window (the new MongoDB wire cell: purge --confirm took the live holder's claim and waited).

Measured on the tree before the rebase, debug builds, a 16-core Xeon Platinum 8124M host (30 GiB, Linux 5.10), the services local.

• cargo fmt --all -- --check clean; cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, and with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; cargo +1.88.0 check --locked --workspace --all-targets clean.
• Core: a_manifest_of_the_legacy_in_catalog_kind_parses_and_round_trips; --lib backup_format 181 passed, 1 ignored. Storage: --lib 237 passed, with the four source_table_from_legacy cells (the exhaustive property; the described features and the legacy identity; the KMS key id; the refusals and the unknown vector state).
• Backends, against the live services: SQLite legacy_migration 7 cells (the stream across both batch bounds with 1,201 small and three 3 MiB items; a stream ending at a DeleteBackup after its batch; list_all across accounts; the replacement's insert and removal in one step, the backups row gone, an adopted copy finished, another store refused, in-use and unavailable refused with nothing written; holder_status and list_deleted telling a delete from a replacement, remove_deleted of a DeleteBackup's row only and once, row_holding_backup_id by the id and the account); PostgreSQL legacy_migration 11 cells (the stream from its pinned snapshot while another transaction deletes every item row; the same replacement and holder cells; a legacy delete queued behind a replacement answers not found; a real replacement and a DeleteBackup queued on each other in both orders, the replacement's insert or the delete's item deletion held behind a table lock, each asserting the answers, so a replacement without FOR UPDATE fails at an answer; a DeleteBackup committed while the stream is paused after its AVAILABLE re-check, the stream reading every item of the pinned snapshot), and the dead-holder claim cell refusing the take while the row's heartbeat is young; MongoDB 9 live cells (2,500 items past a cursor batch; a drop ending a stream; the replacement; a replacement stopped before its drop finished by the next call, DeleteBackup leaving it; the same stop with the index row removed since, finished by finish_replacement; holder_status and list_deleted; a commit whose answer is lost answering Replaced; the in-use and unavailable refusals) and the unit cells of lock_miss (a migrated document) and legacy_key (a missing field is an error); Cassandra 9 direct cells on a node of their own (200 items over four buckets in pages of 8; list_all; the conversion in place, the legacy describe, list and delete no longer seeing it; in-use and unavailable; a lost answer either way, a stop after the row finished by the next call, and a DeleteBackup paused between its read and its claim answering not found; a legacy restore paused between its DDL and its registration while the row is converted, answering not found with its target removed; a registration landing between the conversion's read and its transaction, InUse; a stopped conversion of a backup deleted since, its buckets deleted and no entry written; holder_status and list_deleted), run with the 6 existing cells that reach the changed delete, registration, restore_source, insert and abandoned-create paths (15 passed), holder_status_and_list_deleted covering remove_deleted (an index row and an AVAILABLE row left, the DELETED one removed once), and the unit cell comparing the conversion's columns with the insert's. The PostgreSQL backup_restore suite: 74 passed; SQLite --lib: 289 passed; MongoDB live live_: 80 passed, --lib backup: 31 passed.
• Engine --lib: 563 passed, 41 of them in backup::tests::migrate_legacy: the copy under the ARN and its row; each refusal before the copy writing nothing; a refused item deleting the copy and --all going on; a short or failing read; the fence on a root another catalog adopted, before a leftover's deletion, at a copy's deletion after its manifest and at a deleted backup's copy; a copy that does not read back; a store serving another backup's copy (the manifest's ARN) or another count (the second count check); the items' size and the region and creation-time notes; a copy an earlier run left; a backup whose row is in the store; <arn> and the finish by the row's status; a row under the other spelling; AlreadyExists keeping the copy; a replacement that fails; deleted or in use at the replacement; CREATING and DELETING skipped; MongoDB's stopped replacement finished whatever its row holds; a deleted backup's whole and partial copies (the sync skip, --all and <arn> deleting them, a row's prefix left, nothing listed for an id with no instant); a signal, a lost claim, a stalled heartbeat before the replacement, a signal during a slow copy and during the replacement; the claim's loss token; the ARN's spellings; sync's skip under both spellings; Describe, Delete, Restore and ListBackups racing a replacement; the ListBackups dedupe; a claim taken over since its last beat, which stops each step that cannot be undone (the manifest, the leftover's deletion, delete_copy, clean_deleted's deletion, the record's removal, the replacement, finish's and finish_replacement), and one taken over during the copy, caught before the next such step; a lost claim read before each backup; a row of the id under any ARN keeping the prefix, and with no id index (Cassandra's case) a manifest of another backup keeping it; a deleted backup's record removed only once its prefix is clean (not while a deletion is refused, and after the listing again); a deleted backup an AVAILABLE row holds again, refused rather than called deleted; a read a DeleteBackup ended, and a backup deleted before its open, skipped as deleted while --all goes on; a restore whose registration meets a replacement taking the row's path; sync's skip of an in-catalog backup and its copy's manifest both under another region; Fake's holder_status answering None for a marked backup. Six cells of HeldClaim (a hanging beat lost after two periods and a parked one confirmed; a stopped timer lost after two periods; a beat that starts after a gap confirming nothing, parked or not; a beat on schedule confirming only within its bound; a beat that panics leaving no flight; recheck() counting as a beat or recording the loss). The job, writer, sync and purge cells pass unchanged; describe_backup_of_another_accounts_id_is_not_found now counts the second row read. App --lib: 63 passed (the outcome lines and notes, the audit lines, the ARN check, a refused or failed line of each kind saying the in-catalog backup is kept only where it is, an audit line only after a write).
• Wire, Python 3.10, moto variables: tests/test_backup_store_config.py tests/test_backup_job.py on the SQLite build 47 passed, 13 skipped (-n 4), on the PostgreSQL build 42 passed, 18 skipped (-n 2); the migrate cells on the MongoDB build (a replica set of their own) 2 passed, 1 skipped. New: test_migrate_legacy_carries_in_catalog_backups_into_the_store_under_their_arns (303 items with a GSI and NS, B, nested M and L, BOOL and NULL values; two in-catalog backups; <arn> spelt in another region, --all, reruns, "1 already in the store"; the manifests; the in-catalog backups rows gone with no item or definition row; ListBackups once each, DescribeBackup as before, the restore's items and GSI, DeleteBackup; beside the server on PostgreSQL and MongoDB), test_migrate_legacy_refuses_a_backup_with_no_instant_or_an_item_618_let_in_and_keeps_it (--all past a BatchWriteItem, TransactWriteItems and other operations store N values that fail number validation as written #618 backup and, on SQLite and PostgreSQL, a 13-digit copy of a backup's rows; the three refused items logged; both kept and AVAILABLE; the 13-digit one restored and backed up again into the store) and test_a_backup_deleted_after_a_stopped_migration_never_comes_back (SQLite, PostgreSQL: a malformed <arn> exits 2 with no claim taken; two migrations put back as a stop after the copy leaves the catalog, one prefix cut to its data files; sync adopts neither, before and after DeleteBackup of both; --all deletes both copies, each line with its note, "2 deleted", exit 0; both stay BackupNotFound). Extended: test_a_second_maintenance_command_fails_at_once (PostgreSQL: migrate-legacy --all refused while --prune-missing holds the claim, then holding it while it waits for a live job, backup purge --confirm refused beside it, a SIGTERM stopping it before any change) and the serve-lock cell (exit 6 beside a SQLite server).
• Binary-level checks before the wire cells (SQLite, PostgreSQL; this diff's first form over feat(backup): backup sync --move-store, --forget-store, --forget-backup [backup/restore 10b] #880's earlier head 11544ecd): init with kind = "disabled", two CreateBackups, the store configured, the command run (refused with exit 6 beside a SQLite server, nothing written), boto3 afterwards: 33 and 30 checks, none failed.
• SIGKILL at each kill point, on SQLite and PostgreSQL builds with backup-test-hooks, a 40,000-item table in two data files (this diff's first form): after the first data file, before the manifest and after it, the in-catalog backup was whole with no row; after the manifest a plain backup sync skipped the prefix naming the command and wrote no row; each rerun deleted the copy, migrated 40,000 items and removed the in-catalog backup; a restore of the last held every item. On PostgreSQL the killed command's advisory lock ended with its session, so that first form's rerun took the claim at once; on this tree the rerun takes it once the dead holder's heartbeat is older than stale_job_after_secs (the takeover runs below). On this tree's backup-test-hooks builds: SIGKILL after the manifest and after the first data file, then DeleteBackup of the in-catalog backup, then --all, <arn> and a plain sync: the copy deleted ("1 deleted"), nothing adopted, the backup BackupNotFound, on SQLite and PostgreSQL, and on PostgreSQL with an S3 store (moto); on PostgreSQL a DeleteBackup while the run was paused after its manifest, with the prefix made unwritable: the run skipped with its note and exit 1, sync adopted nothing, --all then deleted the copy; a SIGTERM at each point exits 1 (after the manifest the replacement completes first); the kill-after-manifest rerun migrates. Cassandra's stop after the conversion is the direct cell above (no binary kill point and no Cassandra wire leg).
• Mutants, each a targeted one-occurrence edit, the named tests run, the original bytes written back and checked by hash: 32 in core, the storage crate and the four backends (the kind's text; the definition untrimmed, the unknown state accepted, ensure_restorable ignored, null written as [], the class or the SSE description dropped; a batch or bucket or page boundary skipped, a stream truncated or read outside its snapshot, the per-batch AVAILABLE check dropped; the in-use check skipped, deletion before the insert, Cassandra's store_kind left null or the buckets deleted before the conversion, a lost answer read as applied; each frozen delete's new check dropped; DELETED rows or index rows listed), 10 in the engine (the id refusal, the first count check and both, the fence before the first put, --all stopping at a refusal, the leftover kept, the re-read skipped, the sync skip, the dedupe, the loss token), 3 on the wire (a run with refusals exiting 0, the <ARN>/--all group optional, the wait skipped); and since: 18 in the backends (the replacement leaving a DELETED row on PostgreSQL or SQLite, list_deleted or holder_status filtering wrongly on each backend, PostgreSQL's replacement without FOR UPDATE, MongoDB's holder_status ignoring the marker, finish_replacement a no-op, lock_miss without the marker, the lost commit not settled, list_all taking a default for a missing field, Cassandra's registration without store_kind = null, the conversion without active_restores = null, a deleted row's finish writing entries), 29 in the engine (a deleted backup's prefix adopted by sync, not cleaned, not listed, cleaned without the fence or over a row; MongoDB's stopped replacement not finished; HeldClaim with no bound and no two-period clause, either alone, or no parked clause; the row read under one spelling; the copy deleted on AlreadyExists; the finish ignoring the row's status or calling for a CREATING row; each of the four re-reads dropped; ListBackups reading the index first; the fence before a leftover's and a copy's deletion; the second count check; the claim check before the replacement; the manifest's ARN check; table_size_bytes not set; the copy kept on a signal; the region and the creation-time notes; the finish's InUse and NotAvailable answers; the signal after the last replacement), 3 in the app (the notes not printed, the in-store finish not audited, the in-catalog suffix on every refusal), 2 on the wire (<arn> not checked before the claim, deleted backups not cleaned): 97, all killed; and on this tree 46 more, each a targeted edit with the named cell run and the bytes written back and checked by hash, all killed: 4 in HeldClaim (any late beat confirming, the flight cleared without the guard, the re-check recording no loss or not counting as a beat), 8 re-checks dropped (before the manifest, the replacement, the leftover's deletion, delete_copy, clean_deleted's deletion, the record's removal, finish's replacement, finish_replacement), 4 of the id rule (the lookup dropped in the migration and in clean_deleted, the manifest check dropped at the leftover and at a deleted backup's copy), 3 of the record's removal (never, before the prefix is known clean, without the listing again), 3 of the kept and skipped reports (no re-read after a short or failed read, in_catalog_kept always true, a lost replacement answer not recorded), one each for the deleted backup an AVAILABLE row holds, the audit of a write, the restore's third read, the leftover's wrote, MongoDB's stopped replacement answering Deleted and the five engine cells of other findings, 3 in the app (the suffix on every line, an audit line without a write, the clap error doubled) and 11 in the backends (PostgreSQL's take of a young dead row, its stream's AVAILABLE re-check outside the snapshot, its replacement without FOR UPDATE; remove_deleted of any status on each of the four; row_holding_backup_id of any account on PostgreSQL, SQLite and MongoDB; Cassandra's conversion without created_at): 143 in all.
• Takeover runs on this tree's backup-test-hooks builds, 45,000 items padded to 1,000 bytes, stale_job_after_secs = 45, a second run on another config (32 MiB data files): on PostgreSQL a first run paused before its manifest had its claim session ended (pg_terminate_backend) right after a beat; the second was refused 12 times, naming the holder, and migrated 51.6 s after the beat; the first, released, exited 1 at its re-check; the row AVAILABLE with its six objects, a restore holding 45,000 of 45,000 items. On MongoDB (a replica set of its own) the first was stopped (SIGSTOP) 52 s at the same point, the second migrated, the replica set was paused (docker pause) as the first resumed and was released; the first exited 1 ("no heartbeat confirmed the maintenance claim within two periods"), the prefix unchanged throughout, the restore 45,000 of 45,000. On the head before this change both sequences ended with an AVAILABLE row whose prefix was empty. On SQLite: a whole copy left with the in-catalog backup AVAILABLE, DeleteBackup, sync adopting nothing, --all with the prefix made unwritable (exit 1, the record kept, no "Its in-catalog backup is kept"), --all again (the copy deleted, the record removed, "1 deleted"), a third --all ("0 deleted", no audit line), sync, <arn> refused (no record and no row), DescribeBackup BackupNotFound.
• Memory and time of the command (design section 7, the manual and the admin guide state it), measured on this change's earlier revision, whose read paths and writer this tree keeps (the re-checks add one claim beat per step that cannot be undone): in-catalog backups of the job's measurement shape (a composite key, two GSIs, a 200-byte pad), each migrated in its own run with the server stopped, ru_maxrss of the command through wait4, two runs each:

Backend	1,000 items	100,000 items	250,000 items
PostgreSQL	56.1 MiB, 0.9 s	56.1 MiB, 8.2 to 8.3 s
SQLite, a 224 MB database file	55.0 MiB, 0.1 s	110.8 to 110.9 MiB, 7.0 s
SQLite, a 557 MB database file	67.9 MiB, 0.1 s		116.9 to 117.1 MiB, 16.8 to 17.2 s

An independent measurement of the same shape on a host of the same description gave 68.4 to 74.9 MiB on PostgreSQL beside a running server and, on SQLite, 81.0 MiB for 1,000 items over a 562 MB file and 121.2 and 148.6 MiB for 250,000. A measurement on the rebased head before the changes above, through a fork and wait4 wrapper started outside Python, gave 48.3 and 48.4 MiB on PostgreSQL for 1,000 and 100,000 items. So PostgreSQL is flat in the item count at 48 to 75 MiB, and SQLite's excess (55 to 149 MiB measured) is the page cache of the command's connections (PRAGMA cache_size = -32000 per connection), bounded by about 32 MB times the pool's connections (pool_size, 10 by default), its small-backup figure following the database file's size. The earlier run of this PR measured MongoDB and Cassandra flat at 56.1 and 55.2 MiB for 1,000 and 100,000 items (12.4 and 10.6 s for 100,000; Cassandra without GSIs, whose asynchronous queue over 100,000 writes timed the node out); their read paths are unchanged by this revision.

Acceptance lines of #821 (the design's PR 10 acceptance list) and where each is tested:

Line	Tests
migrate-legacy on a backup the PR 0 binary created produces a prefix Describe, List and Restore serve under the original ARN, and the legacy rows are gone	engine an_in_catalog_backup_is_copied_under_its_arn_and_replaced_by_its_row; the backends' replacement cells (no non-DELETED row, no items, definition, collection or buckets); wire test_migrate_legacy_carries_in_catalog_backups_... (SQLite, PostgreSQL, MongoDB; the in-catalog rows read on SQLite and PostgreSQL). Those cells' backups are taken by this tree's in-catalog create, the frozen copy of the PR 0 (#384) code each backend keeps, so they stand in for the PR 0 binary's; an independent run on this change's earlier form (head 27070b6c) also migrated, on SQLite, a GSI table's backup with B and NS values that main 94b5f689 (catalog migration 5) took, after migrate --yes: 40 items, DescribeBackup differing only in BackupSizeBytes, the restore holding 40 items with 40 GSI hits
migrate-legacy refuses a backup with a 13-digit id, one holding a value #618 let in and a pre-0.0.4 SQLite backup, keeps their legacy rows and names the reason	engine each_refusal_before_the_copy_... and a_refused_item_deletes_the_copy_...; storage a_copy_refuses_what_the_legacy_restore_refuses_and_an_unknown_vector_state and SQLite's stream cell (the definition row removed: Unknown); wire test_migrate_legacy_refuses_... (the 13-digit id on SQLite and PostgreSQL, #618 on all three)
a second maintenance command while one holds the claim fails at once	wire test_a_second_maintenance_command_fails_at_once (PostgreSQL, both ways) and test_a_second_maintenance_command_fails_at_once_beside_a_live_holder (PostgreSQL and MongoDB, where only the stale window decides the take: a zero window in the shared take fails it)
a maintenance command started while a DELETING row and a RUNNING restore with a dead owner exist proceeds once the reconciler has finished the one and failed the other	not a cell of its own: migrate-legacy waits through the shared wait_for_jobs (cmd_backup.rs), the function whose wire cell test_a_maintenance_command_waits_out_a_deleting_row_and_a_dead_restore runs it from sync --forget-backup; the migrate cells' second-command cell shows the command waiting for a live job, and an independent binary run on PostgreSQL waited for a running legacy restore of a 60,000-item backup and migrated after it
on SQLite every writing command refuses to run beside a server	wire, the serve-lock cell (exit 6)
PR 10 paragraph: the re-read and count check before the row	engine a_read_short_..., a_copy_that_does_not_read_back_...
4(c): the fence before the first put	engine migrate_legacy_on_a_root_another_catalog_adopted_writes_nothing
transitions: legacy rows replaced by an AVAILABLE row	the backends' replacement cells (the row unowned, completed_at set)
4(g): sync skips a prefix an in-catalog backup holds	engine sync_skips_a_prefix_... and a_backup_deleted_after_a_stopped_copy_never_comes_back; wire test_a_backup_deleted_after_a_stopped_migration_never_comes_back; SIGKILL after the manifest, then backup sync (SQLite, PostgreSQL)

Cassandra has no wire leg (no pytest fixture runs it); its share is the direct cells above and the memory run.

Not in this PR

• Backup/restore: legacy reader removal #749: removing the in-catalog reader (PR 14), whose refusal counts the rows that are not DELETED, as revision item 37 records.
• Carrying a backup with a 13-digit id (every 0.1.x backup) into the store format: refused by design, since the format's ids carry their creation instant; the way out is a restore and a new backup.
• A Cassandra wire leg (direct tests only).
• On Cassandra, a conversion stopped after its lightweight transaction and then a DeleteBackup of the migrated backup, whose index row is then removed, leaves the item buckets in the account's keyspace: nothing deletes them until PR 14 drops backup_items. The manual says to run migrate-legacy <arn> before deleting such a backup.

Breaking changes

None for a released build's data. backup sync skips a prefix whose ARN an in-catalog backup holds, in any status, where it could adopt it before; no release wrote such a prefix. ListBackups lists an ARN both readers hold once. On PostgreSQL, MongoDB and Cassandra an in-catalog DeleteBackup that read the backup before a replacement committed answers BackupNotFoundException, where it could have answered DELETED for a backup that lives on in the store. On PostgreSQL the second of two concurrent DeleteBackups of one in-catalog backup now answers BackupNotFoundException (the status is read under the row lock), where it answered DELETED. On Cassandra a legacy restore whose registration lands after a migration converted its backup answers BackupNotFoundException. A maintenance command (purge, migrate-legacy, the options of backup sync) whose heartbeat does not confirm the claim within two periods stops as for a lost claim, as #825 made it, counting a beat in flight only on SQLite, where it parks, and now there only when it started on schedule. On PostgreSQL a maintenance command (purge, remove-target, migrate-legacy, the writing forms of backup sync) started after another was killed, or after the other's catalog session ended, now fails at once, naming the holder, until the dead holder's heartbeat is older than [backup] stale_job_after_secs (300 s by default), as on MongoDB and Cassandra; before, it took the claim as soon as the advisory lock was free. A script that reruns a killed command at once on PostgreSQL must retry or wait for the window. SQLite is unchanged: under the serve lock the next command takes the claim at once.

DCO

Signed-off-by: Scott Robinson robinnsc@amazon.com

[LeeroyHannigan]

I built the head on SQLite, PostgreSQL, MongoDB and Cassandra, with and without the backup test hooks, and ran migrate-legacy in containers beside running servers. The stores were filesystem roots and an S3 root on moto. I killed it at each kill point. I sent it one and two signals, ran two commands and two --all at once, and froze a holder until another run took its claim. I also ended a holder's PostgreSQL catalog session, paused the MongoDB replica set under a beat and rewrote extenddb-store.json under a running command. Through a proxy in front of moto I slowed and dropped PutObject calls, refused connections and answered conditional deletes with 409.

I merged main at 0cfb0d9 into the head with no conflict. On the merge, clippy 1.97.0 with both CI feature sets, the 1.88.0 check and fmt are clean. The targeted engine, app, storage, core and SQLite tests pass on it. The 42 CI checks at this head pass. The 105 new Rust tests and 4 new wire cells all ran in CI on their backends. I ran 56 mutants of the engine, the app, the storage crate, the four backends and both stores against the suites. The suites killed 52, one of them only by an abort, and four survived. I compared DescribeBackup, ListBackups and RestoreTableFromBackup of migrated backups with Amazon DynamoDB in us-east-1, and sent no DeleteBackup to the service.

What matched the description:

• <arn> and --all copy a backup under its own ARN on the four backends and into the S3 root. DescribeBackup answers as before the migration except BackupSizeBytes, and TableSizeBytes on Cassandra and MongoDB, as the manual says. Restores held all their items, 36,000 of 36,000 from S3.
• After SIGKILL at after_data_file, before_manifest and after_manifest, the server serves and lists the legacy backup once, and a rerun converges with nothing lost, on the four backends.
• A second maintenance command fails within a second, naming the holder. Of two --all started together, one migrates and the other exits 1. While a command holds the claim, CreateBackup and RestoreTableFromBackup answer 400 LimitExceededException.
• On PostgreSQL the take refuses while a dead holder's heartbeat is younger than stale_job_after_secs, after a kill and after its catalog session ended.
• In takeovers before and after the manifest on PostgreSQL, MongoDB and Cassandra, the stale run wrote and deleted nothing more. The taker's objects kept their size, mtime and sha256, and the backup restored whole.
• A DeleteBackup during the read or after the manifest ends the backup skipped with its copy deleted, and DescribeBackup then answers BackupNotFoundException. A later --all deletes the in-catalog records of those backups.
• The command waits for a running legacy restore of the backup it replaces, and the restore held 36,000 of 36,000 items.
• A refused item deletes the copy and keeps the in-catalog backup, and --all goes on. An unreadable leftover is not deleted.
• ListBackups lists a backup once across the legacy and migrated readers, with the filters and with Limit 1 and 2. Per table with Limit=1, the service and the head both return pages of one in creation order with no empty page.
• The exit codes 0, 1, 2, 6 and 130 and the sample output in the manual and the admin guide match the binary. The refusal of a 13-digit id and its way out, a restore and a new backup, work as documented.

CodeQL has not run on this PR because its base is not main. It runs once you retarget the PR to main.

From #825, which this PR keeps: HeldClaim::lost reads last_confirmed and then in_flight under two separate locks. The timer records the confirmation and then clears the flight, and both writes can fall between those reads. That read then sees the old instant and no flight, and can count a parked SQLite beat as lost.

One Blocking item, five Major items and thirteen Minor items are inline. Items without an inline comment:

• The Breaking-changes sentence on an in-catalog DeleteBackup that read the backup before a replacement committed does not match the code. After the legacy not-found, backup.rs (lines 1262 to 1274) reads the catalog row again and deletes the migrated backup. The client then gets the answer of the store-backed DeleteBackup, where the sentence says BackupNotFoundException. Please fix the sentence.
• Question: which answer should a DeleteBackup get while another DeleteBackup deletes the same in-catalog backup? With no command running, I sent DeleteBackups of one in-catalog backup at once from separate clients. On PostgreSQL, of two, one answered 200 DELETED and one 400 BackupNotFoundException "Backup not found: ", and of three, one DELETED and two BackupNotFoundException. On MongoDB, of two, one DELETED and one BackupNotFoundException, and of three, DELETED three times. On Cassandra, of two, DELETED twice, and of three, one DELETED and two BackupNotFoundException. A DeleteBackup after the pair answered BackupNotFoundException on PostgreSQL and MongoDB. A second DeleteBackup of a migrated backup answered 400 BackupNotFoundException on the four backends. The description says PostgreSQL answered DELETED to the later caller before this PR. I did not send the pair to a build of the base or to SQLite. I have not measured the service's answers, which need a DeleteBackup on DynamoDB. If the service answers DELETED to the later caller, this PR moves PostgreSQL away from it. If it answers BackupNotFoundException, MongoDB and Cassandra answer by timing. Should all four backends give one answer, and should MongoDB and Cassandra follow PostgreSQL in this PR?
• Question: finish (migrate_legacy.rs line 644) deletes the in-catalog backup on an AVAILABLE catalog row of the configured store alone. The main path reads the copy back and compares the count before its replacement (line 1033). This path reads nothing under the prefix. When the row's objects are gone, finish still deletes the in-catalog backup, which is the only other copy. This PR's backup sync skips a prefix an in-catalog backup holds. The comment at backup.rs line 965 names the source of such a row: a #825 or #880 build's sync during a rolling upgrade. Should finish check the prefix as sync does, or at least read its manifest, before it deletes the in-catalog backup?

Once you fix the Blocking item and the five Major items, with their tests, and answer the two questions, I will approve. The Minor items can land in the same revision.

[LeeroyHannigan]

This re-check and the one at line 997 read only the claim, so neither the manifest's put nor the replacement checks the store identity. On the four backends I rewrote catalog_id in extenddb-store.json at a before_manifest pause, and the runs exited 0, printed "migrated" and deleted the in-catalog backup. On PostgreSQL, SQLite and Cassandra the server then refused RestoreTableFromBackup of that backup with ValidationException "The filesystem backup store configured on this server is unhealthy (owned by another catalog); the operation is refused until the store's check passes", and in-catalog backups under the same identity restored all their items. Please run configured.fence().check_catalog(storage) beside both re-checks, keep the copy and the in-catalog backup when it fails, and add a test whose replacement succeeds: a_copy_on_a_root_fenced_after_its_manifest_is_left_with_a_note sets NotAvailable.

[LeeroyHannigan]

Deleting a copy a stopped run left sets wrote (migrate_legacy.rs lines 892 and 1442), but audited ignores wrote for a refused, skipped or failed outcome. The admin guide, the manual and the module doc say the command logs that line, and an_audit_line_follows_a_write (line 392) fails once audited follows wrote. The Cassandra finish through <arn> also deletes item buckets with wrote = false. Please log by wrote alone, change the test, and count the Cassandra deletion as a write or list the unaudited deletions in the docs.

[LeeroyHannigan]

This record has size 0 and no ETag, so delete_written deletes the key on S3 without If-Match and never matches it on a filesystem root. With the second data file's PutObject answers dropped, the deletion sent DELETE .../data/000002.json.gz with no condition and DELETE .../data/000001.json.gz with If-Match. A run stopped before this delete and resumed after a takeover then deletes the taker's object at that key, against line 908 of the manual. Please head the key after a put whose outcome is unknown and record that metadata, or leave the key to the next run.

[LeeroyHannigan]

No test pins the 1.1-period window: with the bound changed to 2.5 periods, all 433 backup:: tests pass. The cells that test this rule resume the process three periods after the last confirmed beat, so a window under three periods passes them. Please add a paused-clock cell with a parked beat that starts 1.5 periods after the last confirmed beat, and assert that lost() is Some at 2.1 periods.

[LeeroyHannigan]

No test exercises this arm: with it replaced by std::future::pending(), all 433 backup:: tests pass. The doc above says a recorded loss stops the walk at its next deletion. Without the arm, a walk the timer marked lost keeps deleting objects until acting_deadline. Please add a cell whose store deletes one object per delete_prefix_before call and whose beat records a loss mid-walk, and assert no deletion after it.

[LeeroyHannigan]

This full-range read of backups_by_arn goes through query_rows_quorum, which sets no page size, so the coordinator returns the whole table at LOCAL_QUORUM in one response. The table keeps the converted rows too, and list_all drops the rows it does not want only after the read. On a catalog with many backups the read can pass the coordinator's range timeout, and --all then stops before its first backup. Please page the scan, or read only the rows list_all and list_deleted keep.

[LeeroyHannigan]

The ids this line refuses start with their creation instant: v0.1.14 formats them {epoch_millis}-{suffix:08x}, and the manual's sample 1793254932797-4116998c is 2026-10-29 in epoch milliseconds. The check refuses the 13-digit shape, which a store prefix does not carry. The upgrade manual, the admin guide, the differences document and the backup manual repeat the wording. Please write "its backup id N has the 13-digit shape a 0.1.x release issued" here, in the help (cmd_backup.rs line 348) and those documents.

[LeeroyHannigan]

For a prefix a run left before its manifest, backup sync reports "not a complete backup: no manifest" where this sentence quotes the in-catalog message. I saw that on PostgreSQL and MongoDB after a kill at before_manifest. The prefix is never adopted, and the next migrate-legacy deletes it. Please say the message applies once the manifest is present, or run the in-catalog check in adopt_prefix before verify_prefix.

[LeeroyHannigan]

This cell has no deadline of its own. Without the no-progress check at migrate_legacy.rs line 1336, the cell looped until the process exhausted its address space (109 s under an 8 GB limit). That aborted the whole test binary, where a deadline would fail only this cell. Please wrap the run in tokio::time::timeout.

[LeeroyHannigan]

No test checks the summary counts: with NotFound left out of refused, all 18 cmd_backup tests pass. An unknown ARN then exits 1 and prints "0 refused" and "0 backup(s) were not migrated". Please add a unit cell that builds a MigrateReport with all eight Outcome variants and checks the summary line.