[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework (merged) · PR 6 the catalog index · PR 7 configuration, the store identity file and backup purge · #700 the CreateBackup job (merged) · #740 the restore job, the cross-backend matrix and backup remove-target (merged) · #825 backup sync, --prune-missing, --adopt-store · #855 the CI filters, the open measurement cells and the backup and restore manual · this PR: backup sync --move-store, --forget-store, --forget-backup and the report of the rows of other stores · next: #821 migrate-legacy, #748 export and import through the store.
Closes #820 (part of #624).

Stacked on #855 (feat/backup-docs, head eb9e48de), itself on #825 (feat/backup-sync, 84bc2639) on main at 887b6e26 (#840); opened as a draft against feat/backup-docs. Rebased from #855's earlier head 695dc66b with git rebase --onto eb9e48de 695dc66b, after #825 moved onto main at 887b6e26 and took its changes for the store fence (a missing or unreadable identity file recorded as a store problem, the fence's own call slot, a fence in the backup job before its first put), the other region's spelling of an ARN (BackupArn::names_same_backup in the dead-owner probe), --adopt-store's check before it writes, the identity file read once more after every sync, and the sync summary's incomplete; #855 changed the manual and the admin guide in the same paragraphs. Eight files conflicted, each resolved with both sides kept: cmd_backup.rs (#825 moved sync_held's prune counts into PruneCounts and its summary word into run_word; this PR's other-stores exit, the landed-rows refusal and its helpers follow PruneCounts' check, in that order, before complete_run), reconciler.rs (probe_manifest keeps #825's region rule and doc, with this PR's pub(super) and its --move-store sentence), tests/test_backup_job.py's module doc (#825's list, with this PR's three options in the claim line), the design's backup sync paragraph, the differences doc's "Where backups live" row, the admin guide's identity, sync and sync-health paragraphs, the deployment guide's shared-root bullet and the manual's destroy paragraph (each a word-level merge with no overlapping change, or #855's sentences followed by this PR's). The design's items keep their numbers (36 is this PR's). The commit is 35 files, +15,683 −525. Before that, rebased from #855's earlier head 925509bd with git rebase --onto 695dc66b 925509bd, after #825 gave HeldClaim the job's two-period rule and HeldClaim::recheck(), and plain sync on SQLite stopped running the dead-owner resolutions. Five files conflicted: the engine's tests (both sides appended; #825's cells, then this PR's), cmd_backup.rs's sync (#825's removal of plain SQLite sync's resolution kept), the design's revision log (#855's item 35, then this PR's item 36), the admin guide's identity and sync paragraphs and the backup manual's destroy paragraph (each side's added sentences kept, word for word). On that base the move and the forget options now confirm the claim with a beat right before each step that cannot be undone (below), and the commit is 35 files, +15,054 −530. Before that, rebased from #855's head 88d445da with git rebase --onto 925509bd 88d445da. One file conflicted, the design's revision log, only because #855's rebase changed one phrase of its item 35 (its quickstart sample lines at migration 8): item 35 is #855's, and this PR's item 36 follows it unchanged, so no item is renumbered. The other files both sides changed merged without a conflict, and the commit is the same size as before (35 files, +14,027 −528). One commit. No migration, no schema change, no new dependency; Cargo.lock is unchanged. The design's PR 10 ships as three PRs (revision item 34): #825 is the catalog-loss recovery, this PR the options that rewrite or delete rows of other stores, #821 migrate-legacy. Every binary below is a debug build of the tree before the rebase (88d445da plus this diff), except the lines of the rebased tree.

What

Storage trait (crates/storage). Three BackupCatalog methods, with no default body. list_backup_rows_by_root(kind, root, status): every row of the kind and root whatever its store id, of one status or all, in list_backup_rows' order, the root matched byte for byte (an S3 root's ?endpoint= suffix included). rewrite_store(from_kind, from_root, keys, to) -> u64: each row of keys that is still of the old kind and root and not CREATING gets to's kind, root and id; the count of rows rewritten. The design's signature had no "from" arguments; with them a row another writer moved, or the CREATING row of a server still configured with the old root, is left rather than moved without its bytes. A partial rewrite (between PostgreSQL batches, MongoDB chunks or Cassandra keys) leaves rows under both roots, which the trait doc states: every step is idempotent and the command names what is left. summarize_store_roots() -> Vec<StoreRootCount>: the rows per store (kind, root, id) and status, in one order on every backend (StoreRootCount::sort), for sync's report; the design had no method that lists the roots. BackupStatus::ALL. remove_backup_row gains the store kind and root the caller read, and removes only a row still of them, on every backend (Cassandra's lightweight transaction IF backup_status = ? AND store_kind = ? AND store_root = ?, a NotMatched answering false), so a tombstone duty of a server on the old root does not remove a row the move rewrote after the duty read it. Two methods with default bodies: summary_counts_status_entries() (false; Cassandra's true, its summary counting status entries) and remove_orphan_status_entries(kind, root) -> StatusEntrySweep { removed, repaired } (zero; Cassandra deletes the root's entries whose row is gone or names another store, and writes again under the row's status, then deletes, an entry whose row is of the root under another status, each checked against the row at LOCAL_SERIAL). The docs of forget_available_backup_row, remove_backup_row, set_backup_deleted (the listing-order reason no duty meets the DELETED state) and remove_dead_creating_backup_row name their new callers.

Store (crates/backup-store). BackupStore::delete_prefix_before(prefix, deadline) -> PrefixDeletion { deleted, finished }: the filesystem walk checks the deadline beside its cancel flag before each removal, the S3 loop before each ListObjectsV2, DeleteObjects, ListMultipartUploads and AbortMultipartUpload; a call that reaches it returns what it removed with finished false. The default body checks the deadline once and runs delete_prefix (a one-step store); the engine's bounded-calls adapter keeps that default (an override it had has no caller, the duties calling the store directly, and is removed). delete_prefix is unchanged for its other callers.

PostgreSQL, SQLite, MongoDB, Cassandra. PostgreSQL: the by-root listing is list_backup_rows' statement without the id (the (store_kind, store_root) prefix of idx_backup_index_store_status); rewrite_store in batches of REWRITE_BATCH = 5,000 keys, each one UPDATE ... WHERE (account_id, backup_id) IN (SELECT * FROM unnest($4, $5)) AND store_kind = $6 AND store_root = $7 AND status <> 'CREATING', whose row lock orders it against a restore's FOR UPDATE and DeleteBackup's transaction, run in a transaction of its own that the client commits (a batch that fails leaves the batches before it committed) under SET LOCAL lock_timeout = '3s' and statement_timeout = '30s', with up to three attempts on a lock timeout (pauses of 0.5 s and 1 s): the statement of a command killed while it waits on a row lock commits nothing, where an autocommit statement on the pool committed once the lock was granted, after the claim was gone; the summary one GROUP BY. SQLite: the same statements, rewrite_store one BEGIN IMMEDIATE transaction under the write lock, all rows or none (a cell with a trigger that fails the last key shows no row moved). MongoDB: find on kind, root and status; update_many in chunks of REWRITE_CHUNK_KEYS = 1,000 keys, filtered on the old kind and root and status: {$ne: "CREATING"}, that also $inc lock_seq, a convention: the update and a transaction that writes the document (claim_restore_tx, start_backup_delete_tx) are serialized (when the update commits after the transaction's snapshot and before its write, the transaction re-runs and reads the row as the update left it; when the transaction wrote first, the update waits for it and applies its filter to the committed document), and a registration keeps the store its server is configured with; a $group for the summary. Cassandra: the by-root listing reads the five (root, status) partitions of backup_index_by_status whatever status it is asked for, filters entries on the kind, dedupes them by (account, ARN), re-reads each backups_by_arn row once and keeps it when its kind and root match (and its status, when one is asked for), whatever partition its entry is in (root_index_arns, refactored from status_index_arns): a row whose only entry is under an earlier status (a status change whose entry batch failed, which is logged and not sent again, or whose writer died between its transaction and its batch) is listed under its own status. rewrite_store, per key: a LOCAL_SERIAL read, skipped unless of the old kind and root and not CREATING; the new root's status entry written; the lightweight transaction UPDATE backups_by_arn SET store_kind, store_root, store_id ... IF store_kind = ? AND store_root = ? AND backup_status = <read>; the old root's entry deleted (a failure logged). An unknown outcome is settled by a serial read (moved only when the row now carries to whole, its kind, root and store id; a row of to's root under another store id is another store's, left and not counted, as the trait doc now says); NotMatched reads the row at LOCAL_SERIAL first and keeps the new entry when the row already names to (a lost answer completed by the retry's own prepare), and otherwise removes it and reads again, up to three attempts, after which the key is left uncounted. start_backup_delete's and set_backup_deleted's lightweight transactions now carry the store kind and root they read, so a DeleteBackup of a server on the old root that read the row before the rewrite does not apply, reads the row again and moves its status entry under the root the row now names. The module doc gives the one exception this makes to its row-first rule: a stop after the new entry leaves only an entry every reader skips, while the row first would leave a moved row that no listing of the configured store returns. The summary counts the status entries, unverified, as the trait doc says. Test hooks fail_rewrite_writes, stop_rewrite (RewriteStop::{AfterNewEntry, AfterRow}), pause_before_rewrite_write and pause_before_deleted_write. The rewrite_store trait doc says what a client that dies in the call leaves on each backend, and set_backup_deleted's why no deletion duty meets the DELETED state between it and the removal.

Engine (crates/engine/src/backup/). New store_maintenance.rs, written against BackupCatalog and the configured store:

• locate_root(storage, root, configured_root): the kind of the rows under a root (both kinds are tried through read_store_ids; an S3 label starts with s3:// and a filesystem label is an absolute path) and the store ids the catalog records for it; RootRefusal::ConfiguredRoot, and NoRows with the summary, so the command can print the stores the catalog knows.
• move_store: reads the old root's rows (each row's source_table_details_json dropped as it is read) and the configured root's (account, backup id) pairs, both before anything changes, so a failed read stops the move with nothing changed; the two-rows check runs first: the configured root's rows and the old root's are grouped by (account, backup id), and every old-root row of a group of more than one stops the move and is checked no further (on Cassandra two spellings of an ARN share one prefix, which a finished deletion would empty); then it checks each other row against the configured store and names every row that stops the move in one pass. A CREATING row goes through reconciler::resolve_dead_creating_row, split out of resolve_dead_creating_rows so both callers share one body (CreatingOutcome), with the configured store's calls and OnIncomplete::Leave: it is written AVAILABLE when the copy holds its complete backup, and any other probe result (no manifest, a data file missing, short or long, no sidecar, a manifest of another ARN) stops the move with no status written; a row not handed over has a live owner. A DELETING row with a verified manifest under its copied prefix stops the move; otherwise finish_backup_delete deletes what the copy brought behind the fence and sets DELETED. A DELETED or FAILED row with a verified manifest there stops the move. The stop's text is per status: for a DELETING or DELETED row it says the backup was deleted and to remove that prefix from the configured root, and names removing the row with --forget-backup only as what would have a sync adopt the prefix and bring back AVAILABLE a backup a client deleted; for a FAILED row it offers the prefix's removal or --forget-backup and an adopting sync. A store error during any of these checks stops the move with the row named ("the configured store could not be read"). An AVAILABLE row is probed with probe_manifest (now pub(super)) and its manifest's size and item total must be the row's. A row that would give an account a second row of one backup id (against the configured root's rows, and among the moved rows, which only Cassandra can hold) stops the move, as said. With none stopping, StoreFence::check_catalog re-reads the identity file once, rewrite_store runs in slices of REWRITE_SLICE = 500 keys with HeldClaim::recheck() before each (a claim lost between slices stops the move, MoveReport::stopped naming the rows it came before, with the earlier slices moved and the rest left under the old root, and the command's text says how many moved), and the old root is listed again: a row still there is reported as left (a CreateBackup of a server still on the old root, a row rewrite_store did not count, the slices after a lost claim); a key the old root no longer lists counts as moved only when its row, read by its ARN (MOVED_ROW_READS = 16 at a time, in place of the listing of the configured store), names the configured store, and one that names no such store (removed, or moved to a third store) is reported as gone (MoveReport::gone); when the old root's re-list fails after a rewrite that counted every key, every key is taken as moved, and when a key's read fails, that key and the rest are. The AVAILABLE resolutions and finished deletions are kept when the move stops. Audit lines (extenddb::audit, backup sync --move-store) for each row moved, resolved or finished.
• forget_store and forget_backups: rows removed outright with the removals that exist (forget_row): AVAILABLE forget_available_backup_row, DELETED and FAILED remove_backup_row, CREATING remove_dead_creating_backup_row on the catalog's clock, DELETING set_backup_deleted then remove_backup_row (no duty meets the DELETED state between them: one that listed rows before the claim listed this one DELETING, and one that lists it DELETED did so after the claim and reads it held). A DELETING row of the configured store is finished first when the store's check passed (finish_then_remove: the wait for the duties, so a stop leaves the row DELETING and untouched, then finish_backup_delete behind the fence, then the removal, Forgotten::RemovedAfterDeletion, "removed: the row is gone, and this command deleted its objects first ..."), and refused otherwise (Forgotten::DeletionPending); when the opening's check failed on IdMismatch or MultipleRowIds and the row carries one of those row ids (StoreProblem::mismatch_row_ids), the refusal is Forgotten::DeletionBehindMismatch, which names the way out (the same command with [backup.store] kind = "disabled", a sync that adopts the backup, DeleteBackup again); a DELETED row whose deletion the command's own SQLite pass finished is reported the same way (ForgetRun::deletions_finished). Every row of a configured root whose opening found no identity file under rows is refused (Forgotten::RootWithoutFile), and so is every row of a defaulted root found missing with rows for it (ForgetRun::unmounted_root). A removal that meets another status re-reads once and applies that status's removal (a DELETING row of the configured store that a server's reconciler finishes during the wait is removed as the DELETED row it then is). A FAILED row whose heartbeat is within the owner window is left (Forgotten::LiveOwner, whose text for a FAILED row gives the condition to run again on, the heartbeat older than stale_job_after_secs, since its job ended with the FAILED write; FailedOwners, one list_backup_rows_older_than per store; a failed read of that listing leaves the row, Forgotten::Failed). --forget-backup reads the ARN as given, then under its other spelling (BackupArnSpellings), then through LegacyBackupReader::describe; an ARN that none of them finds is NoRow, printed "no row carries this ARN; nothing to remove" and counted as done, and the run stops at the first ARN it does not remove; check_arns checks every ARN's form for the command before the claim. Both run under a ForgetRun (the owner window, the instant before which no FAILED or DELETED row is removed, the callback that prints the wait, the stop, the configured store as ForgetConfigured { store, healthy, mismatch_row_ids, file_missing_with_rows }, the missing defaulted root, the deletions the command's own pass finished): beside servers the first FAILED or DELETED removal waits until FORGET_WAIT (60 s) after the claim was taken, asking the stop each second. After --forget-store's removals the root is listed again and the rows it still holds are kept by name (ForgetStoreReport::rows_left), remove_orphan_status_entries runs for the root and both counts are reported (entries_swept), then the root's store ids are read again and reported. The in-use text names extenddb backup remove-target --account <id> <table>.
• other_stores and group_counts: the summary grouped per store other than the configured one, and each CREATING and DELETING row named through list_backup_rows(Some(status), <that store>); when the backend's summary counts status entries, the store's rows are read from list_backup_rows_by_root of its root kept to its id (so a row whose entry is under an earlier status counts under its own status and is named), each count is checked against them, and what no row of its status stands behind goes to OtherStore::entries.

maintenance::wait_for_quiet and busy_jobs take &[StoreIdentity]. reconciler.rs: Duty::fenced re-reads the row after the fence passes and skips one that is gone or whose status or store changed, then reads the maintenance claim again and starts nothing while it is held (each logged), for the FAILED-prefix, DELETED-prefix and tombstone duties; each deletion or removal that read the claim free must end by DUTY_DELETION_BOUND (30 s, S3_CHECK_TIMEOUT) after that read was sent: fenced answers Fenced::OutOfTime when the read was answered at or after the deadline (a slow catalog), the prefix deletion goes through delete_prefix_before with the deadline (an unfinished answer stops the duty), and the call is also dropped at the deadline; each of these stops its duty for the cycle (Acted::OutOfTime). The tombstone removal passes the store the duty listed the row with. ReconcilerState::for_maintenance_holder_without_store_check runs a cycle with no store check, for the forget options' own pass on SQLite. The module doc, DUTY_DELETION_BOUND and FORGET_WAIT say what remains (the one removal or S3 request in flight at the deadline) and the assumption every lease shares (a process paused past the window between its last check and a call already sent is outside it), name the DELETING exception and word the OTHER_STORE line for the move. job.rs reads the maintenance claim again immediately before claim_backup, and restore.rs immediately before claim_restore (refused LimitExceededException, nothing inserted); backup.rs's comment on the check and job.rs's state the window that remains, from that read to the insert, and what reports a row that lands in it. restore.rs: OTHER_STORE reads "...; configure that store in [backup.store], or copy its tree under the configured root and move the backup's rows to the configured store with extenddb backup sync --move-store <root>, and restore again", as design 5.3 step 7 says. runtime.rs: check_configured_store_read_only, the store's check that writes nothing (the identity read without the create and without validate, the id against the one the command opened the store with, the manifest probe). sync.rs: the module doc says that sync lists and changes no row of another store and that the command reports them. store_maintenance.rs: MoveTarget::claim and ForgetRun::claim carry the held claim (None in the engine cells that send no beat), and each step that cannot be undone is preceded by HeldClaim::recheck() (#825's), as sync's insert is: the move's resolution of a dead CREATING row, its finished deletion of a DELETING row and each slice of its rewrite; each removal of the forget options, a DELETING row's set_backup_deleted or finished deletion and the removal of the DELETED row it leaves; and --forget-store's sweep of the status entries. A loss stops the run there (MoveReport::stopped, Forgotten::Stopped), naming the step and what it left; the module doc states the bound that remains, a call already sent when the process stopped, as #825 states it for sync and purge.

App (crates/app/src/cmd_backup.rs). backup sync --move-store <old root>, --forget-store <root>, --forget-backup <arn> (repeatable), each with clap help, conflicting with each other, with --adopt-store and with --prune-missing (exit 2). --move-store needs Identity::Known, locates the root before the claim, takes the claim on every backend (on SQLite too, beside the serve lock, as the other options do), waits over the configured identity and the old root's ids, runs the move, prints its sections (Resolved, Finished, Stopping the move, Moved, Gone during the move, Left under the old root) and the summary line (which counts the rows gone), exits 1 unless every row moved, and only then syncs the configured root; a row of the old root in that sync's closing report of other stores is named under "Rows that landed under the old root during the move" and exits 1. The forget options open in Mode::Forget: the catalog as a writing command opens it (the serve lock on SQLite), the store described as a read leaves it (Caller::ReadOnlyCommand: no default root created, no validate probe; a defaulted root that does not exist is looked up in the catalog under the path and under each path met while every symbolic link along it is replaced by its target, component by component, whether or not the target exists, up to 40 links (a chain, a relative target, a directory link in the path), each also with its longest existing ancestor canonicalized, and with rows there the server crate answers NotConfigured::DefaultRootMissingWithRows, logged at WARN and printed in the header, whose rows the forget refuses and whose label --forget-store refuses as the configured root), the identity file read and never created, and, when it agrees with the rows, check_configured_store_read_only run once and recorded, so the store is healthy for a DELETING row's finished deletion only when that check passed; their own pass on SQLite runs without the store check. They keep the catalog open when no store is configured (open returns Opening::CatalogOnly for them), print the store's state in the header, locate the root (--forget-store), take the claim on every backend, wait (over the configured identity when known, otherwise the configured root's ids from read_store_ids, whose failed read refuses the command before the claim, and the root's ids), remove, and print each row's outcome and the summary line; --forget-backup checks every ARN's form before the claim and refuses at once, with nothing changed; beside servers the forget options print the wait before their first FAILED or DELETED removal. After --forget-backup, when a store is configured, check_configured_store_read_only runs once and is printed as "The configured store's check passes (store id ...)" or the problem, which for ManifestMissing names the next ARN and the command for it; when the opening found no identity file (Identity::Empty, a fresh root; or FileMissingWithRows, whose rows the removals refused) the check is not run and the command says so (ForgetCheck): for a fresh root that a server's first check creates the file, and otherwise to mount the volume before the root's last row is gone. Plain sync's ManifestMissing refusal says to check first that the volume is mounted and the root shared, then prints --forget-backup with the ARN. Every sync that listed the root to its end prints "Rows of other stores": per store the counts per status (on Cassandra, what no row of its status stands behind as "status entries with no row of that status behind them, which readers skip; with no row of the root left, --forget-store removes them"), the CREATING and DELETING rows, and the line naming --move-store and --forget-store, printed only when a root other than the configured one is listed (a store of the configured root under another store id, which both options refuse, names --forget-backup <arn> on its own line); a failed read of them is printed and exits 1. --forget-store prints the counts of status entries it removed and repaired, names under "Rows that landed under the root during the run" each row the listing after the removals holds that the first did not, counts those as left, and its summary says "incomplete" (not "stopped") for a run that read every row and left some. The forget header prints a store whose opening check gave no answer as "check pending" (its DELETING rows are refused), and the note that the check after the removals reads the store only when that check will run. A move whose rows were partly removed by another writer prints "No row is left under " with both counts; a move stopped between two slices says how many rows moved and to run it again; after a completed move, the advice of a failed listing, a stop and a failed report of other stores is plain backup sync, which the option would otherwise refuse as a root with no rows. The clap help of --forget-backup names the DELETING exception and the skipped check, says a later sync adopts the backup again where its bytes are, as --forget-store's does, and both forget options' help says they wait for running jobs as purge does and leave a row whose job writes a heartbeat after the wait. The left-row and landed-row exits of the move say to copy the row's prefix under the new root or --forget-backup it. The module doc gains the three options. take_claim, adopt_wait_identity and root_refusal are factored out; the line and exit decisions are pure functions with unit tests, and so are the decisions taken from the backend and the located root (owner_window, removals_from, forget_wait_identities, forget_check, others_outcome, manifest_missing_hint, old_root_listed). A move whose re-list fails says to run plain sync rather than the move again.

Harness. tests/test_backup_job.py: thirteen new cells and five extended (below); helpers _set_root, _set_store_disabled, _copy_root, _copy_tree_into, _tree_without_identity, _section, _age_row. tests/test_backup_store_config.py: the serve-lock cell runs the three options beside a SQLite server. The cell of a CreateBackup during the move's wait gives each skip its own reason (SQLite's serve lock; on MongoDB, the cell's SQL catalog edits). The server crate's cell of a loop of links runs the walk on a thread and fails by name when it has not ended within 5 s.

Docs and texts. The backup manual: the restored-catalog paragraph now gives the --forget-backup <arn> procedure in place of "This release has no command that clears it"; the claim paragraph names the three options among those that wait; the configuration section points a root that has to change at the move; a new section, "Moving a store to a new root" (reconfigure every server first, copy whole with the modes and without the old root's extenddb-store.json, what to do when it was copied, finish the copy, the root label byte for byte, the checks per status, a CREATING row whose copy is incomplete and its two ways out, the stop and its way out, the summary line, the re-list, plain sync after an interrupted move, what to check before deleting the old root, OTHER_STORE, memory, and --forget-store to drop a root's rows, with the wait beside servers and what a paused server can still do, a DELETING row of another store keeping its bytes, the status entries a Cassandra report line may count, a row landed during the move's sync, and --forget-backup's DELETING row and unmounted-root refusal); the restored-catalog paragraph says to check that the volume is mounted before forgetting rows. The admin guide: the identity paragraph (the move, the copy without the identity file), the synopsis, the exit-1 list, the sample output with the other stores' section, the newest-manifest sentence (--forget-backup), the full report in place of "reported in part", and two paragraphs on the three options with the measured memory of both reads, the rewrite's slices and PostgreSQL's batches and bounds with the measured statement times, the wait (any running job holds the command, up to twice stale_job_after_secs, 600 s at the default) and a sample output (the CREATING rule, a key counted as moved, the jobs' claim read before the insert, the duties' claim read, the deadline in the store and the forget's wait, with what remains and the lease assumption, the DELETING row finished by the forget, the opening that writes nothing and the unmounted-root refusal, --forget-store's sweep of Cassandra's status entries, the check after --forget-backup that writes nothing, a rerun of an ARN list, the per-row memory figure). The deployment guide's shared-root bullet says how to change the root later, the copy leaving the identity file out; the differences doc's "Where backups live" row says a moved root keeps its ARNs. Design: revision item 36; the trait block's remove_backup_row signature, the PR 10 section's split line, interfaces block (rewrite_store's arguments, summarize_store_roots, remove_orphan_status_entries) and sync contract, and section 7's memory paragraph, edited with it.

Why

Design 4(c), 4(g) and the PR 10 section. #825 left three gaps that this PR closes. A store root could not change without losing its rows: a tree copied to a new root has no rows there, and sync skips every prefix whose backup id a row of the old root holds, so the backups were reachable only by adopting them again under the new root's store, and a failed restore of another store had no remedy but configuring that store. Rows of a root that is gone could not be dropped without a tombstone, and a tombstone is what a deletion duty acts on. And a catalog restored from a database backup whose newest AVAILABLE backup was deleted after the catalog's backup stays unhealthy ("manifest probe failed"), which --prune-missing refuses, with the prefix put back as the only way out. Plain sync also reported the rows of other roots only in part.

Choices to weigh

• A third trait method, summarize_store_roots. The design's interfaces gave list_backup_rows_by_root, which takes a root; nothing listed the roots the report has to name. The alternative was one list_account_backup_rows per account, which reads every row of the catalog whole (about 22.5 KiB per row of a five-GSI table) to count them. The method is one grouping query on three backends; on Cassandra it counts status entries without re-reading the rows, so the report checks each of that backend's counts against the root's listing kept to the store's id (summary_counts_status_entries) and prints what no row of its status stands behind as status entries, which --forget-store of that root then removes. That listing reads all five status partitions of the root, so a row whose entry batch failed counts as a row: the move, the forget options and the report all see it, which the closing check before deleting an old root needs. The CREATING and DELETING rows it names are read through list_backup_rows of each store, which re-reads every row.
• rewrite_store carries the old kind and root and excludes CREATING. Between the move's checks and its rewrite a server still on the old root can write (its CreateBackup after the wait, its DeleteBackup, which the claim does not exclude). Moving a CREATING row whose job writes to the old root would give an AVAILABLE row of the new root with no bytes there. A row another writer moved to DELETING may still move; the configured store's reconciler then finishes that deletion under the new root, which is what the DeleteBackup asked for. The rows not counted are found by the re-list and reported.
• Cassandra writes the new root's status entry before the row. The module doc's rule is "authoritative row first". With the row first, a stop before the new entry would leave a moved row invisible to every listing of the configured store: to the reconciler, to sync's row read (sync would then insert, answer false and report a row it cannot list) and to the manifest probe. With the entry first, a stop leaves an entry whose row still names the old root, which every reader skips. read_store_ids of the new root then counts that entry; its id is the configured id, so no identity mismatch follows. A DeleteBackup of a server on the old root that read a row before the rewrite would have moved the status entry under the old root and left the moved row listed under no root; its lightweight transactions now carry the store it read, so it does not apply, reads the row again and moves the entry under the root the row names.
• A deletion duty and a forgotten row's bytes: Duty::fenced's two reads, a deadline inside the store, and a wait. A reconciler cycle reads the maintenance claim once, at its start, and its FAILED-prefix, DELETED-prefix and tombstone duties act on rows listed when each began. A cycle that started just before a forget's claim could delete the prefix of a FAILED row the operator forgot precisely to keep it (measured through the binary on PostgreSQL before this change: up to 101 of 3,000 prefixes deleted after their rows' removal). fenced re-reads the row (gone, another status, another store: skipped) and then the claim (held: nothing starts), two catalog reads per deletion; a deletion or removal that read the claim free must end by DUTY_DELETION_BOUND, 30 s after that read was sent (the S3 store's per-call bound, under the SDK's 60 s attempt timeout): a read answered after the deadline starts nothing (a claim read answered two minutes late deleted the whole prefix of a row a forget had removed before this change, in an engine cell), and the deadline is checked inside the store, before each removal of the filesystem walk and each request of the S3 loop, since a timer outside it fires only once a paused process's runtime runs again (a server SIGSTOPped mid-deletion and resumed after the forget removed 150 and 21 links in two runs through the binary before this change; 0 and 1 after it, the one being the removal in flight). Beside servers the forget options wait FORGET_WAIT, the bound plus one 30 s store call, 60 s from the claim, before their first FAILED or DELETED removal, printing the wait (on SQLite the serve lock keeps every server out and nothing waits). What remains is the one removal, or one S3 request (up to 1,000 keys), in flight at the deadline, and a process paused past the wait between its last check and a call already sent; the bound shares that assumption with every lease of the design, and the texts state it. The alternative of marking the row before each deletion, which the removal would be conditioned on, would close it at a write per deletion and a fourth removal condition on every backend.
• The forget options run on any store state and with no store, and write nothing to the root. --forget-backup is the way out of ManifestMissing, so it cannot require a healthy store, and a deployment that turned kind = "disabled" must still be able to drop an old root's rows; the removals delete no object (but for the DELETING exception below) and read no store, the opening reads the identity file without creating it and runs the check that writes nothing, and the store's check after --forget-backup writes nothing either. An empty mount point under the rows (FileMissingWithRows) is the case that needs more: a running server's check rewrites a missing file once the root has no rows and reports the store healthy, and the documented rerun opened the store with the writing check (through the binary before this change: a new identity file on the empty mount point, and a CreateBackup landing there). So every row of that root is refused, naming the unmounted volume, and nothing changes until the volume is mounted or [backup.store] points at the root that holds the backups. SQLite's defaulted root, moved away or behind a link to a volume that is gone, left the store unconfigured for a read-only command and so was not refused (the rows were removed, and the next server wrote a new identity file at the empty path); the server crate now reads the catalog's ids for it, following every symbolic link along the path (a chain of links, a relative target, a directory link whose target is gone, up to 40 links), and the forget refuses its rows the same way.
• A DELETING row of the configured store is finished by the forget. Its DeleteBackup asked for the bytes to go; removing the row alone left a complete prefix that the next sync adopted again (through the binary on PostgreSQL before this change), and on SQLite the command's own pass finished it while the output said no object was deleted. The forget now runs finish_backup_delete behind the fence before the removal, which needs the store's check to have passed, and refuses the row otherwise ("let a server finish the DeleteBackup, or run again once the store's check passes"); the per-row line says when the command deleted the objects. The wait for the duties comes before the deletion, so a stop during it leaves the row DELETING and its objects untouched, and "not reached" is then true. When the check failed on an identity mismatch that the row's own store id takes part in (the manual's copied identity file), the refusal says so and names the way out, the same command with [backup.store] kind = "disabled", whose generic DELETING path removes the row and keeps its bytes; run through the SQLite binary: the refusal, the removal with the bytes kept, the move's sync adopting the backup AVAILABLE, and a server's DeleteBackup emptying its prefix. A DELETING row of another store keeps its bytes unless a server configured with that store finishes the deletion (the one its DeleteBackup asked for): the command cannot reach that root, and a sync of it after its return with the bytes adopts the backup again, which the texts say. --prune-missing keeps refusing on ManifestMissing, which is also the host-local-root detector (4(c)): pruning while it holds would remove rows whose bytes are on another host. The store's check after --forget-backup is printed and does not change the exit: the rows asked for were removed, and the check names the next ARN to forget when another newest manifest is gone.
• No fourth removal method; remove_backup_row names its store. DELETING is removed as set_backup_deleted then remove_backup_row. The DELETED state between them is never acted on, by the order of the listings: a duty that lists rows after the claim reads the claim before it acts, and one that listed them before saw the row DELETING. remove_backup_row takes the store the caller read, so a tombstone duty of a server still on the old root, past its fence just before the move took the claim, leaves a row the rewrite moved (the copy's leftovers would otherwise have no row and no duty).
• A dead CREATING row of the old root resolves only to AVAILABLE. The job writes its manifest last, but a copy's order is arbitrary, so a copy without the manifest or a data file says nothing about the old root's prefix. Writing FAILED there would have the configured store's reconciler delete the copy and a server still on the old root delete the original (reproduced through the binary before this change). The move stops on such a row instead, naming the two ways out: a reconciler on the old root resolves the row there (a server, or on SQLite plain sync, configured with it), or --forget-backup removes it.
• A CreateBackup admitted before the claim. The request reads the claim before its permit; the store-backed job now reads it again immediately before claim_backup, and the restore immediately before its registration, so an open held on a lock that returns while a command holds the claim inserts nothing (through the binary on PostgreSQL: refused LimitExceededException, no row). The window from that read to the insert remains, however long the insert takes: a row that lands during the move's wait is listed by it, one before the rewrite stays CREATING under the old root (the rewrite skips CREATING) and the re-list reports it, one during the move's sync is named by that sync's closing report of other stores and the move exits 1 (through the binary on PostgreSQL: a CreateBackup held at claim_backup past the re-list and released during the sync, then "a row landed under the old root during the move", exit 1), and one after the report only a later plain sync reports; the backup manual says to check that report before deleting the old root.
• The FAILED live-owner rule applies to both forget options. A FAILED row whose heartbeat is within the owner window may still have its job writing under the prefix (the job has not yet seen its claim lost); the wait, as purge's, holds on such a row and refuses the command if it is still live at the wait's end, and the removal leaves one whose heartbeat was written after the wait, or one of a store the wait does not cover, rather than removing a row whose prefix is still changing. The rule applies to --forget-store and to --forget-backup alike.
• The move procedure copies the tree without extenddb-store.json. The servers restarted in step 1 give the new root its own file; copying the old one over it turns every running server's store unhealthy (an identity mismatch), and with a backup taken under the new root in between the move refuses too. The manual gives the copy commands with the exclusion and the way out when the file was copied (a restart; with a backup taken meanwhile, --forget-backup of it before the move, whose sync adopts it again), both run through the binary.
• The move claims and waits on SQLite too, and a stopped move runs no sync. On SQLite the serve lock already excludes every other writer; the claim is taken as the other waiting options take it, so the stop closure and the claim-lost path are one code path on every backend. A move that stops exits 1 before the sync, so its report is the move's; the resolutions it made are kept and the next run sees them.
• Memory is the backends' reads of both roots' rows. The move keeps each row of the old root without its parsed description, plus the configured root's (account, backup id) pairs for the two-rows check, and reads one manifest and one prefix listing at a time; the backends still collect each listing whole first, as for sync and backup list, so both the old root's read and the configured root's count (the sync after the move reads the configured root again). Measured at 20,000 rows moved into an empty store (below), within 12 % of backup list over the same rows: about 23 KiB per row on PostgreSQL and 27 on SQLite above about 50 MiB, at 1,452 bytes of table description per row; the figure grows with the description. A 50-row move on the PostgreSQL build peaked at 49 MiB into an empty store and 231 MiB into one of 8,000 rows, the same 23 KiB per row of the configured root. After the rewrite the move reads each moved row by its ARN rather than listing the configured store a second time. Streaming the listings out of the backends, or reading the configured root's (account, backup id) pairs alone, is not done here; the texts state both reads.
• The rewrite in slices, and PostgreSQL's in batches. On MongoDB and Cassandra the rewrite writes chunk by chunk or key by key, so a command stopped past stale_job_after_secs whose claim another command took finished its rewrite on resuming. The move now calls rewrite_store 500 keys at a time with a beat of the claim before each, so such a move stops within one slice; what remains is the slice in flight. On PostgreSQL one statement over every key outgrew the 30 s statement timeout well inside the sizes a catalog can reach (in plain SQL over a 400,000-row catalog with 0006's indexes and 1,449-byte descriptions, 200,000 keys in one statement took 16.4 and 16.9 s; batches of 5,000 0.23 to 0.29 s each, 20,000 0.9 to 1.1 s), so rewrite_store runs batches of 5,000 keys, each committed on its own; scaling the timeout with the key count instead would keep a long transaction holding row locks against DeleteBackup and restores. Each slice of the move is one batch.

Testing done

On the tree rebased onto eb9e48de (#825 at 84bc2639, main at 887b6e26), debug builds into a shared target, every tracked file touched before each build: cargo fmt --all -- --check and cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, also with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; engine --lib 757 passed, app 61, server 376, core 1087, storage 241; SQLite --lib backup 69; PostgreSQL --test backup_restore 64 (--test-threads=2); Cassandra direct, this PR's 13 cells, on a node of its own; the move and forget cells of tests/test_backup_job.py on the SQLite build 9 passed, 3 skipped (-n 2), and, with the second-command cell, on the PostgreSQL build 12 passed, 2 skipped (-n 2).

On this tree (head 37fde534 with the changes for the rewrite's slices, PostgreSQL's batches, the per-key read after the rewrite and the texts), debug builds into a shared target, every tracked file touched before each build: cargo fmt --all -- --check and cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, also with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; engine --lib 536 passed, app 60, server backup_store 11, storage 237; PostgreSQL --test backup_restore 64 (--test-threads=4); Cassandra direct, the nine rewrite cells, on a node of its own; the move and forget cells of tests/test_backup_job.py on the SQLite build 9 passed, 3 skipped (-n 4), and on the PostgreSQL build 10 passed, 2 skipped (-n 2). New cells: engine move_store_rechecks_the_claim_before_each_slice_of_the_rewrite (501 rows: a claim lost before the second slice leaves 500 moved and the last left and listed, after one rewrite_store call; a second run moves it; a claim kept throughout makes two calls, of 500 keys and 1) and a_failed_row_left_for_its_heartbeat_names_the_heartbeats_age; a_key_counts_as_moved_only_when_its_row_names_the_configured_store gains a row moved to a third store before the rewrite (gone, not moved) and a failed read of a moved key's row (every key taken as moved, the move incomplete); PostgreSQL a_rewrite_of_more_keys_than_one_batch_commits_each_batch_on_its_own (5,001 keys with the last row locked past every attempt: an error, 5,000 rows moved, then the last alone on a second call); Cassandra test_a_lost_rewrite_answer_does_not_count_the_row_under_anothers_store_id (a lost answer whose serial read finds the new root under another store id: counted 0, the other store kept and listed). Mutants, each a targeted edit reverted by a second edit: the beat only before the first slice; PostgreSQL's batches as one statement (chunks(keys.len())); Cassandra's lost-answer comparison on store_root alone; the link walk's hop limit removed (the server cell fails by name after 5 s); the per-key read counting any row as moved; a failed read counted as gone; the configured-root hint, the landed rows and the advice after a move each disabled: 9 of 9 killed. Through the PostgreSQL build: a move of 3,001 rows under a trigger that sleeps 12 ms per row the rewrite updates (about 36 s for one statement over every key, past the 30 s bound, and about 6 s per slice) completed in 44.3 s, exit 0; peak RSS of a 50-row move, 49 MiB into an empty store and 231 MiB into a store of 8,000 rows (1.8 and 5.8 s). A move stopped with SIGSTOP at its claim line on Cassandra or MongoDB, past the stale window, was not run through a binary here; the engine cell covers the check between slices.

On the tree rebased onto 695dc66b (#825 at b42dbfa9, main at b8ad9cd5), debug builds, every tracked file touched: cargo fmt --all -- --check and cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, also with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; engine --lib 534 passed, app 60, server 344, storage 237; SQLite --lib backup 69; PostgreSQL backup_restore 63 (--test-threads=2); MongoDB live, this PR's backup-method cells and the insert cell, 7 passed; Cassandra direct on a node of its own, this PR's cells and the insert cell, 21 passed; the wire suites tests/test_backup_store_config.py tests/test_backup_job.py on the SQLite build 46 passed, 13 skipped (-n 4) and on the PostgreSQL build 39 passed, 20 skipped (-n 2). New engine cells for the claim's beat before each step that cannot be undone: forget_store_rechecks_the_claim_before_each_step_that_cannot_be_undone (each status's removal, another store's DELETING row's DELETED write and removal, the sweep), forget_backup_rechecks_the_claim_before_a_deleting_rows_deletion_and_its_removal and move_store_rechecks_the_claim_before_each_resolution_deletion_and_the_rewrite, with a claim that confirms N beats and then answers Lost, exact beat counts and the row and objects asserted at each stop. Mutants, each re-check removed in turn with the file written back and checked by hash: 11, all killed, each at its own case.

On the tree rebased onto 925509bd, before that (main at b8ad9cd5), debug builds, every tracked file touched: cargo fmt --all -- --check and cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, also with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; engine --lib 518 passed, app 59, server 343, storage 237; SQLite --lib backup 68; PostgreSQL backup_restore 63 (--test-threads=2); the wire suites tests/test_backup_store_config.py tests/test_backup_job.py on the SQLite build 44 passed, 13 skipped (-n 4).

Then, on the same tree, for the store errors during the move's checks, the guards of the forget options and the report, the move's stop text per status, the missing default root's lookup through every link of its path, the configured root's ids read before the forget's wait, and PostgreSQL's rewrite bounds: cargo fmt --all -- --check and the three clippy lines above clean; engine --lib 522 passed, app 60, server 344, storage 237; PostgreSQL's six catalog_index:: cells of the backup methods passed (--test-threads=2). New engine cells: a_store_error_during_the_moves_checks_stops_it (a store read failing under the prefix of a DELETED, a FAILED, a DELETING and an AVAILABLE row over a complete copy: the row named, nothing rewritten, no object deleted, the row unchanged), a_failed_read_of_the_failed_rows_owners_removes_nothing (through the Fake's new older_than_error), a_deleting_row_a_server_finishes_during_the_wait_is_removed_deleted and the_report_of_other_stores_keeps_two_ids_of_one_root_apart (Cassandra's report path, two ids under one root); move_store_names_every_row_that_stops_it_and_moves_none now checks the text per status, and forget_backup_refuses_the_rows_of_a_missing_root_with_rows a row of a root whose label extends the missing one. New server cell a_missing_default_root_is_looked_up_through_every_link_of_its_path (an absolute chain, a directory link, a relative chain, a database directory reached through a link, a .. path, a loop of links), against a StubCatalog that answers for one root; app cell a_failed_read_of_the_configured_roots_ids_refuses_the_forget. PostgreSQL's held-lock cell now reads SHOW lock_timeout and statement_timeout inside the rewrite's transaction (3s, 30s) as well as after it. The dead Unsupported arm of --forget-backup's legacy read is removed: no backend's describe returns it. Mutants, each killed by its cell: the three store errors taken as success (DELETED and FAILED, DELETING, AVAILABLE), the owner read's failure taken as dead, a finish that found the row DELETED reported gone, Cassandra's report counting every id of the root, the unmounted root matched by prefix, every status given the adoption remedy, one link followed, a relative target used as it is, no canonical-ancestor label, the failed read of the configured root's ids taken as none, and PostgreSQL's rewrite without its statement_timeout: 13 of 13. Through the SQLite build: the five link shapes of a missing default root (one link, a relative link, a chain, a directory link in the path, a relative chain, each with the volume renamed away) and the database path through /home/robinnsc -> /local/home/robinnsc all refuse --forget-backup of the row (exit 1, the row kept; the three chained shapes removed it with exit 0 before); test_move_store_moves_a_copied_tree_and_stops_on_an_incomplete_copy, which now also stops on a DELETED row over a complete copy with its text, and test_forget_refuses_the_rows_of_a_missing_default_root passed on the SQLite build, the first also on the PostgreSQL build.

Measured on the tree before the rebase, debug builds, a 16-core Xeon Platinum 8124M host (30 GiB, Linux 5.10), the services local.

• cargo fmt --all -- --check clean; cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, and with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks, with --features extenddb-engine/backup-test-hooks, and on -p extenddb-storage-cassandra --all-targets --features test-hooks; cargo +1.88.0 check --locked --workspace --all-targets clean, no warning.
• Engine --lib: 516 passed, 50 of them new: 13 (move_store_resolves_checks_and_moves_each_status_of_the_old_root, move_store_names_every_row_that_stops_it_and_moves_none, rows_the_rewrite_does_not_move_are_listed_as_left, move_store_on_a_root_another_catalog_adopted_moves_nothing, move_store_refuses_a_second_row_of_one_backup_id, the_wait_covers_every_store_it_is_given, locate_root_finds_the_kind_and_ids_and_refuses_the_rest, forget_store_removes_the_roots_rows_outright_and_reports_what_it_left, forget_backup_removes_each_row_in_order_and_stops_at_the_first_refusal, a_stop_before_the_first_row_forgets_nothing, forget_backup_of_the_newest_rows_deleted_backup_heals_the_store, a_duty_whose_row_is_forgotten_or_moved_after_its_listing_acts_on_nothing, the_report_of_other_stores_counts_and_names_their_rows_and_nothing_acts_on_them) and 16 (a_dead_creating_row_whose_copy_is_incomplete_stops_the_move_unwritten over six copy shapes, the_moves_owner_window_is_the_one_the_takeover_reads with a Fake whose takeover honours stale_after, an_available_row_whose_manifest_counts_other_items_stops_the_move, two_rows_of_one_backup_id_among_the_moved_stop_the_move, a_stop_during_the_checks_leaves_the_later_rows_unchecked, a_key_counts_as_moved_only_when_its_row_names_the_configured_store, a_removal_that_meets_another_status_or_store_reads_the_row_again, forget_backup_goes_past_an_arn_no_row_carries, the_arns_are_checked_before_the_claim, a_duty_that_finds_the_claim_held_after_its_reread_acts_on_nothing, a_duty_keeps_the_prefix_of_a_row_that_turned_available_after_its_listing, a_duty_deletion_that_outlasts_the_bound_stops_the_duty_for_the_cycle, the_deleted_prefix_and_tombstone_calls_stop_their_duty_at_the_bound, the_forget_options_wait_out_the_duties_before_a_failed_removal, a_claim_taken_after_the_requests_check_refuses_the_create_before_its_row, the_read_only_check_writes_nothing_to_the_root) and 14 (a_claim_read_answered_after_the_bound_starts_no_deletion, a claim read answered 120 s late on the paused clock, for a prefix deletion and a tombstone removal; a_deletion_the_store_stopped_at_its_deadline_stops_the_duty; the_two_rows_check_comes_before_any_row_of_the_old_root_is_checked, with the configured root's failed read changing nothing; a_stop_after_the_last_check_rewrites_nothing; the_forget_options_wait_out_the_duties_before_a_deleted_removal; a_duty_whose_claim_read_or_reread_fails_deletes_nothing; forget_backup_reads_a_legacy_backup_under_the_arn_as_written; forget_backup_tells_dead_failed_rows_of_two_stores_apart; forget_store_removes_the_roots_entries_with_no_row_and_reports_the_rest, through a Fake with orphan and pending status entries; forget_backup_finishes_a_deleting_row_of_the_configured_store_first; forget_backup_refuses_the_rows_of_a_configured_root_without_its_file; the_report_of_other_stores_names_status_entries_with_no_row; the_tombstone_duty_leaves_a_row_moved_after_its_fence; a_cycle_without_the_store_check_writes_nothing_to_the_root) and 7 (every_row_of_one_backup_id_among_the_moved_is_checked_no_further, a DELETING and an AVAILABLE row of one id over a copy missing a data file: neither checked, the prefix whole; the_failed_prefix_duty_keeps_a_row_moved_after_its_listing, the store half of the duty's re-read; move_store_checks_a_row_whose_status_entry_is_under_an_earlier_status, through the Fake's new stale_entries, stopping on a complete copy and otherwise finished and moved; the_report_of_other_stores_counts_a_row_whose_entry_is_under_an_earlier_status; a_stop_during_the_wait_before_a_deleting_rows_finish_deletes_nothing; a_deleting_row_behind_an_identity_mismatch_is_refused_naming_the_way_out; forget_backup_refuses_the_rows_of_a_missing_root_with_rows); forget_store_removes_the_roots_entries_with_no_row_and_reports_the_rest now also covers a removed row's stale entry and a left row's entry repaired, and two_rows_of_one_backup_id_among_the_moved_stop_the_move names both rows; the existing move cell no longer resolves a CREATING row FAILED, the restore claim cell reads the claim before and after the registration, and three cells count the extra claim reads. App --lib 59 passed (10 new: the landed row of the old root and the status-entries line; the conflicts and the repeat, the other-stores lines, the exit and line functions of the move, the forgets and the store check, the check's plan after --forget-backup, the window, wait and stores taken from the backend, and the other stores' exit with the refusal's ARN). Server --lib backup 19 passed (StubCatalog, and the sentence of DefaultRootMissingWithRows). Storage --lib 233 passed (the_root_tally_counts_each_store_and_status_apart_in_one_order).
• Backends, against the live services: SQLite --lib backup 67 passed, PostgreSQL --test backup_restore 61 passed (--test-threads=2), each with list_backup_rows_by_root_returns_every_id_and_status_of_the_root_and_nothing_else, rewrite_store_moves_only_the_old_roots_rows_that_are_not_creating and summarize_store_roots_counts_the_rows_of_each_store_and_status; MongoDB --lib -- --ignored --test-threads=1 live_ 71 passed, the same three as live_ cells plus live_rewrite_store_moves_every_chunk_of_keys (1,001 keys); Cassandra, on a single 4.1 node of its own, the three plus test_a_stopped_rewrite_leaves_the_row_listed_under_one_root (each RewriteStop) and test_an_unknown_rewrite_outcome_is_settled_by_a_serial_read, with test_read_store_ids_lists_the_distinct_ids_of_a_root: 6 passed (--features test-hooks --test direct_integration -- --exact). Added with the deleters' store condition and the rewrite's transaction: on Cassandra test_a_delete_that_read_the_row_before_a_rewrite_moves_its_entry_under_the_new_root, test_a_rewrite_whose_row_changes_before_its_transaction_reads_the_row_again, test_list_backup_rows_by_root_skips_an_entry_whose_row_has_another_status_or_kind and test_the_summary_counts_the_old_entry_a_stopped_rewrite_leaves (17 passed with the 13 existing cells on the changed methods); on PostgreSQL a_rewrite_whose_client_dies_mid_statement_rewrites_nothing and a_rewrite_that_meets_a_held_row_lock_runs_again_a_bounded_number_of_times (catalog_index:: 25 passed); on SQLite a_rewrite_that_fails_on_a_later_key_moves_no_row (4 passed with the PR's three); MongoDB's four cells passed again. Added with the deadline, the store-conditioned removal and the sweep: the store crate's delete_prefix_before_honours_its_deadline (filesystem and moto), delete_prefix_before_stops_at_its_deadline (filesystem, a walk held past its deadline right before a removal: the removal in flight and no other) and delete_prefix_before_sends_no_request_after_its_deadline (moto, a deletion held at its first ListObjectsV2 and at its first DeleteObjects: that request and no other), with the conformance suite 127 passed; on every backend the removal cell of start_backup_delete_checks_the_status_and_the_in_use_rule_under_the_lock now rewrites the DELETED row to another store and finds it left by a removal naming the store read (SQLite --lib backup 68 passed, PostgreSQL --test backup_restore 63 passed, MongoDB's five live_ cells passed); PostgreSQL's held-lock cell bounds the three attempts from both sides and reads SHOW lock_timeout and statement_timeout on the rewrite's own connection after it (the session's defaults); on Cassandra test_the_entries_deleters_leave_behind_a_rewrite_are_swept_from_the_old_root (the interleavings, through three new pause hooks: two orphan entries under the old root, swept to none, the moved rows still listed, a tombstone removal that read before the rewrite leaving the moved row), with 22 existing cells on the changed methods by --exact, all passed. Added with the all-partition by-root listing and the sweep's repair: on Cassandra test_a_row_whose_entry_is_under_an_earlier_status_is_listed_moved_and_repaired (a DeleteBackup stopped after its DELETING transaction, a job's AVAILABLE transaction applied with its entry left under CREATING, and a row with an extra DELETED entry: the status-scoped listing lists neither of the first two, the by-root listing lists each row once under its own status, rewrite_store of the DELETING row moves it, and the sweep deletes its stale entry and repairs the other two), passed with the PR's 11 other cells and test_start_backup_delete_checks_... by --exact; on every backend the removal cell also rewrites the DELETED row to the same root under the other kind and finds it left by a removal naming the kind read, and the summary cell holds a CREATING and an AVAILABLE row in one store (PostgreSQL 4, SQLite 4 and MongoDB 5 changed cells passed); the store crate's delete_prefix_before_aborts_no_upload_after_its_deadline (moto: held at the last DeleteObjects of a page with an upload open, no ListMultipartUploads sent and the upload left; held at the first AbortMultipartUpload of two, the second not sent) and a one-object prefix past its deadline in delete_prefix_before_honours_its_deadline (filesystem and moto), the 18 delete_prefix conformance cells passed.
• Wire, Python 3.10, moto variables: tests/test_backup_store_config.py tests/test_backup_job.py on the SQLite build 44 passed, 13 skipped (the cells beside servers or on PostgreSQL only; -n 4), on the PostgreSQL build 39 passed, 18 skipped (the SQLite-only cells; -n 2); the move and forget cells of test_backup_job.py on the MongoDB build 8 passed, 5 skipped (the SQLite-only, PostgreSQL-only and catalog-editing ones), on a replica set of their own. Added: test_forget_refuses_the_rows_of_a_missing_default_root (SQLite, the default root renamed away and behind a dangling link: the row refused as an unmounted root's, the header naming the missing root with rows, --forget-store of it refused as the configured root, nothing created at the path). Added: test_forget_backup_beside_a_server_keeps_the_rows_of_an_unmounted_root (PostgreSQL, MongoDB: the refusal, no identity file written, the server still unhealthy), test_forget_backup_finishes_a_deleting_row_of_the_configured_store (SQLite, PostgreSQL: the prefix deleted, the line saying so, nothing adopted by the next sync) and test_the_forget_options_write_nothing_to_a_fresh_root (all three, run alone after the full runs); test_forget_backup_checks_its_arns_first_... now ends with the unmounted root's last row refused twice, nothing written. The new cells: test_the_manuals_move_procedure_beside_a_running_server and test_a_copied_identity_file_is_undone_by_the_manuals_remedy (PostgreSQL, MongoDB), test_forget_backup_checks_its_arns_first_goes_past_removed_ones_and_writes_no_store (all three), test_a_maintenance_command_waits_out_a_deleting_row_and_a_dead_restore (PostgreSQL); the other-roots cell now copies the other root's tree under the configured root before the reconciler's cycles and checks it unchanged, and checks the wait line beside servers. No PostgreSQL database left (the list equal to the one taken before).
• Binary-level checks before the wire cells, on the SQLite and PostgreSQL builds against a filesystem root: each option beside a SQLite server (exit 6), a clap conflict (exit 2), a copy with a data file missing, --forget-backup and the move finishing, a server on the new root describing and restoring a moved backup, --forget-store and the root's return, and the forget options with kind = "disabled".
• Mutants, each a targeted one-occurrence edit, the named tests run, the original bytes written back and checked by hash: 26 in the four backends and the storage crate (the old-root condition dropped, CREATING not excluded, the by-root listing matching a root prefix or dropping the kind, MongoDB's chunks cut to one, a status or an id folded in the summary or its tally, the tally's sort reversed; on Cassandra also the old entry not deleted, the new entry written after the lightweight transaction, and the by-root listing trusting the entry's root); 14 in the engine (the tombstone refusal, for DELETING too, the data-file check, the fence before the rewrite, the live-owner stop, the duty's re-read, the two-rows check, the CREATING and DELETING rows not named in the report, the wait reading only the first store, a live FAILED row removed, --forget-backup going past a refusal, the left rows not reported, locate_root accepting the configured root, the configured store in the report); 4 in the CLI's pure functions; 5 in the CLI's wiring against the wire cells, each with the binary rebuilt (the other stores' section always empty, no check after --forget-backup, a stopped move's exit ignored, --forget-store not knowing the configured root, and on PostgreSQL the move's wait over the configured store only). 49 of 49 killed, each on its test's assertion. For the cells added with the CREATING rule, the duties' claim read and bound, the forget's wait, the jobs' claim read, the read-only check and the deleters' store condition, each the same way: 13 in the backends (Cassandra 8: the store condition dropped from either deleter, NotMatched without its serial re-read or keeping the new entry, the by-root listing without its status or kind re-check, the rewrite's transaction not conditioned on the status, the old entry removed before the after-row point; PostgreSQL 4: the rewrite on the pool, one attempt, unbounded attempts, no lock timeout; SQLite 1: no transaction), all killed; 24 in the engine (the move writing FAILED for an incomplete copy, the duty's claim read, each of the three time bounds, each duty's stop at the bound, the forget's wait skipped or shortened, the status half of the re-read, the item total, the two-rows check among the moved rows, the removal's re-read and its store check, the move's owner window, the re-list failure branch, a key counted moved without its row under the configured store, the per-row stop, NoRow not done, the ARN check, the job's and the restore's claim reads before the insert, the read-only check's id comparison and its write), all killed; 6 in the CLI's new pure functions, all killed; 4 in the CLI's wiring against the wire cells with the binary rebuilt (the check after --forget-backup run on an empty root with the writing check, the ARNs checked after the claim, the forget's wait for jobs skipped: killed; the writing check put back only where the opening found a file: survives, since no wire cell reaches that path with a root that loses its file after the opening, and the engine's the_read_only_check_writes_nothing_to_the_root pins the function); and 2 of the procedure cells (the identity file copied, the remedy's --forget-backup left out): killed. For the deadline, the store-conditioned removal, the sweep, the forget's configured store and the move's order: 3 in the store crate (the walk's deadline check, the S3 check before ListObjectsV2, before DeleteObjects); 10 in the backends (the removal's store condition dropped on PostgreSQL, SQLite and MongoDB, and on Cassandra from the read and the transaction or from the transaction alone; the sweep removing nothing, or rows of the root too; PostgreSQL's fourth attempt, no pause, session-level settings); 19 in the engine (the late claim read acted on, the deadline taken after the read, an unfinished deletion taken as done, a failed claim read or re-read taken as free, the SQLite pass's store check, the two-rows row checked anyway, the stop before the fence, the wait for FAILED rows only, the legacy read under the ARN as written, one dead set for every store, the ids after --forget-store, the sweep skipped, the report's counts unverified, the DELETING row removed unfinished or finished on an unhealthy store, the unmounted root's rows removed, the SQLite pass's deletions unreported); 3 in the CLI's wiring with the binary rebuilt (the unmounted root's refusal, the DELETING finish, the forget opening with the writing check: the last survived the existing cells and is killed by the fresh-root cell added for it). 35 of 35 killed, each on its test's assertion. For the all-partition by-root listing, the sweep's repair, the wait before the DELETING finish, the two-rows grouping, the mismatch refusal and the missing default root: 4 in the store crate (no check before ListMultipartUploads, none before AbortMultipartUpload, the abort sweep's unfinished answer ignored, the walk's one-file branch without its check); 11 in the backends (remove_backup_row without the kind on PostgreSQL, SQLite, MongoDB and Cassandra; the summary's sort dropped on PostgreSQL, SQLite and MongoDB; on Cassandra the by-root listing requiring the entry's status, the by-root listing without its dedupe, the sweep leaving a stale entry, the sweep's repair without its write); 8 in the engine (the report of other stores through the store-scoped listing, the sweep's counts dropped, the finish before the wait, the first row of a group checked, the mismatch refusal dropped, the missing root's rows removed, the re-read without its store half, and the Fake's by-root listing skipping a stale-entry row as the old Cassandra listing did); 3 in the wiring with the SQLite binary rebuilt (the catalog's rows for a missing default root not read, a dangling link's target not tried, the missing root not passed to the forget). 26 of 26 killed, each on its test's assertion. The landed-row exit has its pure function's unit test and the binary run below; no wire cell holds a CreateBackup inside the move's sync.
• Through the binary on PostgreSQL, two race scenarios that failed before these changes, run again on this tree: 3,000 aged FAILED rows of a root with prefixes and a server running its FAILED-prefix duty, --forget-store of that root from a configuration naming another root: in three runs the duty deleted 249, 258 and 259 prefixes before the claim and none after it (its re-read of the claim refused the other 3,000 times), the command waited 58 s and removed the 3,001 rows, and 0 prefixes were deleted after their row's removal (101 before this change). A CreateBackup whose snapshot open waits on a table lock while --move-store runs, released once the move has exited: the move exits 0 and the CreateBackup still lands AVAILABLE under the old root, its re-read of the claim coming after the claim's release; plain sync on the new root reports it among the rows of other stores, which the manual now says to check. The same CreateBackup released while a command holds the claim (waiting on a live job): LimitExceededException, no row. Run again after the deadline moved into the store and the landed-row exit: a server SIGSTOPped in the middle of a FAILED-prefix deletion of 200,000 hard links, --forget-store of its root waiting 59 s and removing the row, then SIGCONT: 0 and 1 links removed after the SIGCONT in two runs (150 and 21 before), the server logging "reached its 30 s bound after 513 object(s) and stopped"; a CreateBackup held at claim_backup (after its claim read) until the move's sync, then released: the move prints the row under "Rows that landed under the old root during the move" and exits 1 (it exited 0 before).
• Memory and time of --move-store (design section 7 and the admin guide state it): a root of 20,000 AVAILABLE rows of a five-GSI table, each prefix complete under the copy (one CreateBackup and 19,999 copies with their manifests rewritten to their own ARN, adopted by backup sync; the copy made with hard links): 569 MiB peak RSS and 37.1 s on the SQLite build, 500 MiB and 36.9 s on the PostgreSQL build (51 and 49 MiB at 50 rows); backup list over the same rows 508 and 472 MiB, and the plain sync of the new root after the move 554 and 495 MiB. Filesystem roots on a local NVMe disk, PostgreSQL on localhost: an S3 store pays a request per manifest, sidecar and prefix listing, and a network database the read's round trips, neither measured.

Acceptance lines of #820 (the design's PR 10 acceptance list) and where each is tested:

Line	Tests
rows for another store root are reported and untouched, and the reconciler deletes none of their prefixes	engine the_report_of_other_stores_...; wire test_another_roots_rows_are_left_alone_until_forget_store_drops_them (a FAILED and a DELETED row dated 2000 and a DELETING row of another root keep their rows and bytes through three cycles of a server on the configured root, which holds a copy of that root's tree, unchanged too, and the report counts and names them) and the move cell's report
after a store tree is copied to a new root, --move-store re-points its rows and Describe and Restore work against the new root	engine move_store_resolves_...; wire test_move_store_moves_a_copied_tree_and_stops_on_an_incomplete_copy (SQLite, PostgreSQL, MongoDB)
--move-store resolves the old root's CREATING and DELETING rows, stops on a row whose data files were not all copied, and refuses a DELETING, DELETED or FAILED row whose copied prefix holds a verified manifest	engine move_store_resolves_..., move_store_names_every_row_... (each status and each incomplete shape) and a_dead_creating_row_whose_copy_is_incomplete_stops_the_move_unwritten (a CREATING row resolves only to AVAILABLE; each incomplete shape stops the move with no status written); wire, the move cell (a FAILED row over a complete copy, a missing data file)
--forget-backup refuses a CREATING row with a live owner and a backup from which a restore target is CREATING, and says so and does nothing for a legacy ARN	engine forget_backup_removes_each_row_...; the backends' existing forget_available_backup_row in-use cells (PostgreSQL, SQLite, MongoDB, Cassandra), which the option calls; wire test_forget_backup_leaves_a_live_creating_row_and_an_in_catalog_backup (the live CREATING row on PostgreSQL; on SQLite the serve lock makes every owner dead)
a second maintenance command while one holds the claim fails at once	wire test_a_second_maintenance_command_fails_at_once with the three options (PostgreSQL)
on SQLite every writing command refuses to run beside a server	wire, the sync cell and test_backup_list_and_purge_on_an_empty_store_and_the_serve_lock with the three options (exit 6)
with the volume unmounted the store is unhealthy, sync without the option that resolves it refuses, and no row changes	wire test_prune_missing_and_the_unhealthy_store_refusals with --move-store on each shape; the forget options run on an unhealthy store by design (the clears cell)
after --forget-store and the store's return, sync adopts its prefixes	engine forget_store_removes_...; wire, the another-roots cell
a CreateBackup started on another instance during --move-store either finishes before the rewrite or is refused (or, landing during the move's sync, makes the move exit 1)	engine rows_the_rewrite_does_not_move_are_listed_as_left (a CREATING row landing after the checks is left and reported), the_wait_covers_every_store_it_is_given and a_claim_taken_after_the_requests_check_refuses_the_create_before_its_row; wire test_a_create_backup_during_the_moves_wait_is_refused (PostgreSQL). One admitted before the claim whose open returns after the move has exited still lands under the old root (through the binary, above): the line holds for a CreateBackup that reaches its insert while the move runs, and the manual says to check plain sync's report of other stores before deleting the old root
after --move-store A stops on a row with one data file missing and --forget-backup lets the move finish, sync reports that prefix as incomplete and does not adopt it, and adopts it once the file is copied	wire, the move cell
a maintenance command started while a DELETING row and a RUNNING restore with a dead owner exist proceeds once the reconciler has finished the one and failed the other	wire test_a_maintenance_command_waits_out_a_deleting_row_and_a_dead_restore (PostgreSQL: the two edited at the catalog, --forget-backup of the restored backup waiting, a server started during the wait finishing the one and failing the other under the claim, then the row removed); the engine's the_reconciler_reports_a_held_maintenance_claim_and_still_resolves_dead_owners covers the cycle under the claim
4(g): --forget-store deletes that root's rows outright, leaving no DELETED tombstone	backends' by-root listing after the removals; engine forget_store_removes_... (a cycle afterwards deletes nothing); wire, the another-roots cell (backup list empty, the tree byte-identical)
--move-store and --forget-store match store_root exactly as stored, the ?endpoint= suffix included	backends' by-root listing (a trailing /, a prefix); engine locate_root_... (the S3 label shape); wire, the move cell (a trailing /)
the reconciler cleans the leftovers of the moved tombstones	engine move_store_resolves_... (the next cycle deletes a moved FAILED row's prefix and a DELETED row's objects and removes the tombstones)
the restored catalog whose newest AVAILABLE backup was deleted: --forget-backup is the way out	engine forget_backup_of_the_newest_rows_deleted_backup_heals_the_store; wire test_forget_backup_clears_a_store_whose_newest_manifests_are_gone (SQLite, PostgreSQL, MongoDB; beside servers the running server's /health turns healthy) and test_forget_backup_checks_its_arns_first_goes_past_removed_ones_and_writes_no_store (the check after it writes nothing to an empty root)
the backup manual's move procedure, as written	wire test_the_manuals_move_procedure_beside_a_running_server and test_a_copied_identity_file_is_undone_by_the_manuals_remedy (PostgreSQL, MongoDB)

Cassandra has no wire leg (its CreateBackup stays in-catalog, item 19); its share is the direct tests above.

Not in this PR

• Backup/restore: backup migrate-legacy #821: extenddb backup migrate-legacy <arn>|--all, with its acceptance cells.
• A sweep of Cassandra's orphan backup_index_by_status entries beyond the root --forget-store forgets: an orphan under the configured root, or under a root no one forgets, stays (every reader skips it; item 34 says the same of the status-entry orphans).
• A fencing marker per deletion: what a server paused past the forget's wait can still do is the one removal or S3 request it had sent (above).
• On Cassandra, the same all-partition read for the store-scoped list_backup_rows (from before this PR), which the reconciler reads: a status change whose entry batch failed (logged, not sent again) leaves the row out of that listing until its entry is repaired, so a DELETING row of the configured store whose DELETING batch failed is not finished by the reconciler; --forget-backup of its ARN, which reads the row by its ARN, finishes the deletion and removes the row, and --forget-store's sweep repairs such entries on the roots it may forget, not the configured one.
• Detecting a server left configured with the old root after a move: its later rows appear in the next sync's report of other stores, and the manual's procedure reconfigures every instance first.

Breaking changes

None on the wire. backup sync's report gains a section, and a sync that cannot read the rows of other stores now exits 1. OTHER_STORE's text names the move; tests assert the constant by identity. Each FAILED-prefix, DELETED-prefix and tombstone deletion of the reconciler reads its row and the maintenance claim once more, and ends by 30 s after that read before the duty stops for the cycle. The store-backed CreateBackup and restore read the maintenance claim once more before their insert. BackupStore gains delete_prefix_before, with a default body; BackupCatalog::remove_backup_row takes the store kind and root (every implementation in the tree is updated), and gains two methods with default bodies (remove_orphan_status_entries answering StatusEntrySweep). --forget-backup of a DELETING row of the configured store now deletes its objects first, and refuses it while the store's check fails; it refuses the rows of a configured root that has no identity file under them, and on SQLite those of the default root found missing with rows for it; the read-only commands (backup list, purge's dry run) print that missing root with rows in place of "no server has run with this database file".

DCO

Signed-off-by: Scott Robinson robinnsc@amazon.com

[LeeroyHannigan]

I built the head on SQLite, PostgreSQL, MongoDB and Cassandra, with and without the backup test hooks. The servers and the three options ran in containers as uid 1000, against filesystem roots and an S3 root on moto. Beside running servers, I stopped, killed and signalled the move and the forget options at their claim, inside the rewrite and during their waits.

On the merge of main at 0cfb0d9 (no conflict), clippy 1.97.0 with both CI feature sets and the 1.88.0 check are clean. The PR's targeted engine, app, storage, SQLite, PostgreSQL and server tests pass on it. All 36 CI jobs at this head pass. Each of the 100 new Rust tests and 13 new wire cells ran in CI on at least one backend.

I ran 111 mutants of the engine, the four catalogs, the store, the server and cmd_backup.rs against the suites. The suites killed 108, one of them only by a timeout I put around it. Of the three that survived, two are equivalent and one is inline.

I compared the answers after each option with Amazon DynamoDB in us-east-1 for the requests the service has. I also timed the PostgreSQL rewrite statement alone at 100,000 to 400,000 rows.

What matched the description:

• The three options conflict with each other and with --adopt-store and --prune-missing (exit 2). A SQLite server's serve lock refuses them (exit 6), and the refusals the docs list exit 1.
• After the move, DescribeBackup and ListBackups answer as they did before it on SQLite, PostgreSQL and MongoDB, across 13 ListBackups shapes on PostgreSQL. Moved backups restore with all their items on the four backends and from the S3 root.
• The move refuses an incomplete copy, a tombstone over a complete copy, an unreadable copy and a rewritten identity file, and then moves no row.
• The move resolves a dead CREATING row whose copy is complete to AVAILABLE, and its restore reaches ACTIVE with all 25 items.
• A PostgreSQL move killed while its rewrite waits on a row lock commits nothing. A MongoDB move killed during its rewrite leaves a state that a re-run settles, and the catalog then holds no duplicate ARN.
• A second maintenance command fails at once and names the holder. On MongoDB and Cassandra a later command takes a stale claim from its holder by heartbeat age.
• While a command holds the claim, CreateBackup and RestoreTableFromBackup answer 400 LimitExceededException and land no row. A CreateBackup that waited on a table lock while a command took the claim answered 400 at this head and 200 CREATING at the base.
• --forget-store of another root removes its rows, deletes no object and refuses the configured root. Beside servers the forget options wait 59 s and print the wait. A FAILED-prefix deletion paused past its 30 s bound deleted nothing more on a filesystem root or on S3.
• After either forget option, DescribeBackup, DeleteBackup and RestoreTableFromBackup of the ARN answer 400 BackupNotFoundException "Backup not found: ". The service answers DescribeBackup and RestoreTableFromBackup of an unknown ARN with the same text. A ListBackups cursor that names the removed backup resumes after its instant, as the service's does.
• --forget-backup clears the unhealthy store of a restored catalog whose newest manifest is gone, and finishes the deletion of a DELETING row of the configured store.
• SIGTERM during a forget option's wait exits 1 and releases the claim.
• After --forget-store, configuring the root again lets the next plain sync adopt its backups.
• The sample output in the admin guide and the manual matches the binary line for line.
• The new and changed catalog methods behave alike on the four backends, except for the partial rewrites and Cassandra's entry counts the trait docs allow.

CodeQL has not run on this PR, because its base is not main. It runs once you retarget the PR to main.

Each item is inline. Four are Major:

1. The claim goes unchecked during the Cassandra and MongoDB rewrite.
2. The move fails on each run for a PostgreSQL root whose rewrite statement outlasts 30 s.
3. The docs' memory statement for the move omits the configured store's rows.
4. No test checks that Cassandra's rewrite compares the whole store after a lost answer, and the trait doc names only the root.

Eleven are Minor. Once you fix the four Major items, with their tests, I will approve. The Minor items can land in the same revision.

[LeeroyHannigan]

On Cassandra this call writes key by key, and on MongoDB in chunks of 1,000 keys, with no claim check between them. Twice, I sent SIGSTOP to a 10-key Cassandra move when it printed its claim line, and it stopped after rewriting 2 keys. Past stale_job_after_secs a plain sync took the claim, and on SIGCONT the move finished its rewrite and printed "complete: 10 row(s) moved". Please call rewrite_store in slices with recheck() before each, count the rest as left, and test a claim lost after the first slice.

[LeeroyHannigan]

The rewrite is one UPDATE over all keys of the old root under this 30 s bound, and line 730 retries only a lock timeout. I ran the statement in plain SQL on a shared PostgreSQL 16 host, with migration 0006's indexes and 1,452-byte descriptions. 200,000 keys took 13.7 and 14.8 s, against the doc's 3.6 s, and 400,000 keys took 29.1 and 44.0 s. With a trigger slowing the statement to about 36 s, two consecutive moves exited 1 after 31 and 32 s, moving 0 of 3,001 rows. Please rewrite in bounded batches of keys or scale the timeout with the key count, and state the bound in the admin guide.

[LeeroyHannigan]

This listing and the one at line 1389 read every row of the configured store whole, since each backend collects the rows first (fetch_all, try_collect). The admin guide (line 203), the manual (line 700) and the design (lines 172 and 613) base the peak on the old root's rows alone. A 50-row move on a PostgreSQL debug build peaked (memory.peak) at 6.8 MiB into an empty store. Into a store of 8,000 rows with 1.4 KB descriptions, it peaked at 60.7 MiB. Please state both reads in those four places, or read one row per moved key here, which removes the second read.

[LeeroyHannigan]

After a lost answer, this doc counts a row carrying to's root as rewritten, while rewrite_store_row requires to's whole store, row.store == *to at line 1012 of crates/storage-cassandra/src/backup_catalog.rs. No test tells the two apart: with that comparison changed to row.store.store_root == to.store_root, the four Cassandra rewrite cells pass, including test_an_unknown_rewrite_outcome_is_settled_by_a_serial_read. The move decides what moved from its re-list, so today only the returned count would differ. Please name the whole store here, and add a direct cell where a lost answer's re-read finds to's root under another store id.

[LeeroyHannigan]

This hint names --move-store <root> and --forget-store <root> for each listed store, and other_stores lists the configured root under another store id too. locate_root refuses that root for both options (RootRefusal::ConfiguredRoot), and the refusal points at purge and --prune-missing. No printed report reaches this case today, since a healthy store has one row id for its root and sync refuses an unhealthy store. Please omit that store from the hint or name --forget-backup <arn> for it, or drop the case from other_stores and design item 36.

[LeeroyHannigan]

After a completed --move-store, the same command refuses the old root as having no rows (RootRefusal::NoRows). So when moving is set, this advice leads to that refusal. Please name plain backup sync there, as the re-list failure's text does. The listing failure at line 1831 and the stop at line 1843 give the same advice after a move and need the same change.

[LeeroyHannigan]

This gives --forget-backup as the way out for all three tombstone states, while check_row's stop text and the manual offer --forget-backup for FAILED only. For a DELETING or DELETED row they say to remove the prefix from the configured root. They name --forget-backup there only as what would let a sync adopt the deleted backup again. Please split the sentence as item 36 does.

[LeeroyHannigan]

With a restore running from backup b1 and stale_job_after_secs = 45, --forget-backup b2, a row no job reads, waited and exited 1 after 90.9 s. --forget-store and --move-store from a configuration on another root did the same. This paragraph and the admin guide's (line 205) give as examples only the forgotten rows' own jobs, and state no length for the wait. Please say that any running job holds the command up to 600 s at the default (twice stale_job_after_secs), or scope --forget-backup's wait to its ARNs.

[LeeroyHannigan]

This cell skips every backend but PostgreSQL with a reason about SQLite's serve lock, and the MongoDB CI job reports that reason. On MongoDB the cause is that _set_status and _catalog edit the catalog with SQL (line 1714). I ran the scenario by hand on MongoDB, and CreateBackup during the move's wait answered 400 LimitExceededException twice and wrote no row. Please give each skip its own reason, or give the cell a MongoDB catalog edit so it runs in mongodb-backup-store.

[LeeroyHannigan]

This assertion runs only after link_chain returns. With the hop limit at line 854 removed (for _ in 0..usize::MAX), the cell hung until a 300 s timeout I put around it. In CI that regression would show as the job's timeout after 40 to 55 minutes, in place of this cell's name. Please run the loop case on a thread with std::thread::spawn and fail when mpsc::Receiver::recv_timeout sees no result within a few seconds.