[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework (merged) · PR 6 the catalog index · PR 7 configuration, the store identity file and backup purge · #700 the CreateBackup job (merged) · #740 the restore job, the cross-backend matrix and backup remove-target (merged) · #825 backup sync, --prune-missing, --adopt-store · this PR: the CI filters, the open measurement cells and the backup and restore manual · next: #820 the store-maintenance options, #821 migrate-legacy, #748 export and import through the store.
Closes #747 (part of #624).

Stacked on #825 (feat/backup-sync, head 84bc2639), itself on main at 887b6e26 (#840), opened as a draft against it. Rebased from #825's earlier head b42dbfa9 with git rebase --onto 84bc2639 b42dbfa9, after #825 moved onto main at 887b6e26 and took its changes for the store fence, the other region's spelling of an ARN and --adopt-store's check before it writes. Three files conflicted: .github/workflows/integration.yml (both pytest lines take main's --terraform without the filter, as this PR's change intends), the design's revision log (#825's item 34, then this PR's item 35, no renumbering) and the admin guide's store-identity paragraph (#825's text with this PR's destroy and init --overwrite sentences, word for word). The change set is otherwise the same (25 files, +1,240 −55); doc_version_literals passes and every link and anchor in the changed Markdown files resolves. Before that, rebased from #825's earlier head be84a7eb with git rebase --onto b42dbfa9 be84a7eb: one conflict, in the design's revision log, where #825 rewrote its item 34 and this PR appends item 35 after it (this PR's change set is otherwise unchanged; none of its sentences states a rule #825 changed there). Before that, rebased from #825's head 358ff8bd with git rebase --onto be84a7eb 358ff8bd. Two files conflicted, both over main's migration number (#740 made restore_progress migration 8): the quickstart's sample lines, which this PR had moved from 6 to 7 and main now shows at 8, are main's, so this PR no longer changes them; the upgrade manual's backup paragraph keeps this PR's sentences with main's "Migration 8" twice. Revision item 35 says the sample lines show migration 8. No revision item is renumbered (35 follows #825's 34). The diff is one workflow file, documentation, and one source change: the help text of init --data-db and --catalog-db (crates/app/src/cmd_init.rs), which named the old defaults. The figures and most of the checks below used release builds of the tree before the rebase (358ff8bd plus this diff). Builds with and without the help-text line serve the same requests. The base's code did change in the rebase (#740's final heartbeat rule, under which a beat in flight counts on SQLite only, and restore_progress as migration 8); the checks named as run on this tree ran on release builds of the rebased tree.

What

CI filters (.github/workflows/integration.yml). The PostgreSQL job's and the SQLite job's "Run integration tests" steps drop --filter "not import_export". History: the filter came with the PostgreSQL job's first version (687e46f, "ci: add integration test workflow with PostgreSQL") and was copied into the SQLite job (556c501); neither commit gives a reason. A foreground server wrote a PID file in every tree up to e0d6a0a (crates/bin/src/cmd_serve.rs at 687e46f, crates/server/src/serve.rs at e0d6a0a^), so stop could signal the job's server. e0d6a0a made foreground mode write no PID file by default and added --write-pid-file to the job's start in the same commit. 556c501 came after e0d6a0a (git merge-base --is-ancestor e0d6a0af 556c501e), and its SQLite job passes the flag. So the restart devtools/run-tests makes for the import and export tests could stop the job's server in every tree, and the filter was not working around it. The MongoDB and Cassandra jobs already run tests/test_import_export.py unfiltered (--pytest --comprehensive --parallel). What the expression selects out (collect-only, Python 3.10, the job's own ignore list): the 10 tests of tests/test_import_export.py and nothing else (0 of 502 comprehensive, 0 of the 52 CLI lifecycle tests, 0 of the 18 backup-store step tests). Run the job's way (release build, init, serve --foreground --write-pid-file, the job's environment, run-tests --extenddb --pytest --parallel --filter import_export): the main pytest step passed 10 of 10 on SQLite and on PostgreSQL. That command exits 1, because its later steps select nothing under the filter (on SQLite the backup-store step's 18 tests are deselected, on PostgreSQL the CLI lifecycle list's 52) and pytest exits 5 there. No test fails, so both filters go and no narrower filter is kept. Both lines carry --terraform, which main's #474 (2671ee1) added to them after this PR's base, so each reads devtools/run-tests --extenddb --pytest --comprehensive --terraform --parallel; the timeout-minutes values main raised there are not copied and come with the rebase onto main. A three-way merge with main still meets both lines (both sides change them), with this PR's line as the resolution and no other conflict in the file.

The open measurement cells (docs/design/15-backup-restore-bench-pr11.md, new; row 15d of docs/design/README.md beside 15b and 15c). In the PR 9 record's method: release builds, a filesystem store, [backup] max_concurrent_jobs = 2, the table loaded through the wire from four processes of eight threads, the script's control_plane_delay_seconds = 0.05 and index_propagation_delay_ms = 0, the server restarted before each phase, VmHWM read one second after the start and one second after the phase and sampled every 200 ms. The plain items' 200-byte pad is one repeated character, which the record states. Host: 16 cores (Xeon Platinum 8124M), 30 GiB, Linux 5.10, glibc 2.26; PostgreSQL 16.15 with pgvector 0.8.7, MongoDB 7.0.43, Cassandra 4.1.12, each in a container on loopback; one run each, load average beside each figure.

Cell	Duration	Peak over idle
MongoDB restore, 1M × 200 B (own backup, unordered)	33.81 s	51.8 MiB
Cassandra restore, 1M × 200 B (SQLite backup adopted with backup sync)	48.99 s	33.1 MiB
PostgreSQL vector table, 1M × 128 dimensions: backup / restore	243.17 s / 246.41 s	104.2 / 111.1 MiB
SQLite vector table, 1M × 128 dimensions: backup / restore	209.72 s / 214.23 s	137.6 / 552.1 MiB
the same under MALLOC_ARENA_MAX=1	222.48 s / 282.86 s	57.8 / 203.9 MiB

The 100k cells for each shape are in the record. From 100k to 1M the restore's peak moved +16.7 MiB on MongoDB, +13.2 on Cassandra, +10.5 on PostgreSQL with a vector index, and +198 MiB on SQLite with a vector index under the default arenas (the 1M run levelled off after about 150 s; no other size was measured) against +8.6 MiB under one arena. One arena cost that restore 32% in duration. A whole-table SearchVectors over the 1M restored vectors answered in 0.24 s on PostgreSQL and 14.7 s on SQLite (an exact scan; 5.5 to 15.8 s over the seven runs the record gives). MongoDB's restore becomes ACTIVE at the backfill worker's next cycle after the load (GSI_BACKFILL_INTERVAL, 5 s, crates/storage-mongodb/src/ttl_worker.rs), so its duration is the load's plus up to 5 s: 3.9 to 8.9 s at 100k over five runs, and 33.8 s at 1M in one run, which another may exceed by up to one cycle. No cell is left open: the 1M Cassandra restore finished. It ran on a Cassandra 4.1 node of its own with the Cassandra integration workflow's service settings (PasswordAuthenticator, MAX_HEAP_SIZE=1024M), under the Cassandra lock, removed afterwards: the shared node on this host runs AllowAllAuthenticator, where extenddb init cannot create its role, and it was not reconfigured. The record says the node was used by these runs alone.

The record's "Later runs on the same host" section holds a second set of runs on release builds of 30553b3 (this tree's sources before the help-text line) and, for two confirming runs, of this tree, on the same host, with a separate harness and the server's default settings. They reproduce every cell in kind (within 12% on the peak, 20% on the durations; the SQLite 1M vector restore 522.8 MiB, 218.7 s) and add:

• SQLite restores of 100k items with a 128-number list and no vector index: 407.5 MiB over idle (184.0 MiB under one arena), against 373.3 and 376.5 MiB with the index, so the index is not what the memory belongs to. 20,000 items of 1,536 numbers with a vector index: 668.6 MiB, and under one arena 301.0 MiB (load 1.6) and 315.9 MiB (load 2.8, the run on this tree that confirms it), 48.9 and 48.4 s; their backup 520.9 MiB. On SQLite the vector index is about 20% of the restore's duration (22.7 to 23.3 against 18.3 s).
• The PostgreSQL vector index's share of the 100k restore, from five pairs of runs with and without it whose two runs share a harness: the later runs' pair (112.5 against 60.0 MiB in 29.7 against 15.6 s), two pairs with a third harness (99.9 against 66.5 MiB in 24.84 against 16.26 s, 98.5 against 71.4 MiB in 25.23 against 15.72 s) and two in the record's last section (104.5 against 63.9 MiB in 26.99 against 18.05 s, 99.8 against 69.8 MiB in 28.30 against 17.34 s): 27 to 53 MiB and 33 to 47% of the duration. The record's cell against the later runs' no-index run (100.6 against 60.0 MiB) crosses harnesses and is left out.
• Two 100k vector restores at once: 24.9 s and 696.7 MiB under the default arenas, 57.5 s and 278.9 MiB under one arena.
• The SQLite 100k vector restore's peak over nine runs, 344.6 to 409.0 MiB (two of them pinned to two cores, idle and beside four busy loops), and the PostgreSQL 100k no-index restore's over five, 60.0 to 71.4 MiB: the peaks varied by up to about 20% between runs of one cell, idle or loaded.

The record's new "Runs on this commit's Rust sources" section holds a fourth harness's runs on release builds of this commit's sources, which reproduce the cells in kind, and a control: 100k SQLite items whose 128-number list is replaced by a random 2,290-character string (2,569 bytes of line text per item against 2,503) held 253.4 and 252.5 MiB over idle in two runs, 129.6 and 129.8 MiB under one arena, against 38 MiB for 100k items of 200 bytes (the PR 9 record). So SQLite's restore memory is its write batch (up to RESTORE_BATCH_ITEMS, 10,000 items, or RESTORE_BATCH_BYTES, 32 MiB of item text) and the decode chunks in front of it, which grow with the item's size up to the batch bound whatever the attribute types; lists of numbers decode larger at the same text size (344.6 to 424.5 MiB and 184.0 to 195.3 MiB under one arena at 100k items of 128 numbers, with or without the index). The one-arena 1M restore measured 203.9 to 208.0 MiB over three runs, flat from 100k within about 25 MiB over the measured runs. The record, the manual, section 7, the testing summary, item 35 and the admin guide's row say so, and the MALLOC_ARENA_MAX=1 advice and the sizing are keyed to items of more than a few hundred bytes.

The manual (docs/manuals/14-backup-and-restore.md, new), for operators and derived from the design, which remains the record. Its first sentence: a filesystem store keeps the backups in the host's failure domain, so S3 is the production recommendation. Sections: strategy (what a table backup holds and does not: the catalog, scheduling with a crontab line that escapes % and names AWS_CA_BUNDLE, retention, PITR; Cassandra's in-catalog create); RPO and RTO per store kind (RPO equals the backup interval, no PITR; the restore and backup durations and the memory figures from the three records and the later runs, with the host class and what was not measured; the condition for a key-ordered PostgreSQL backup, which also needs the text key columns at collation C or POSIX, with the warning the server logs when it streams an older table unordered, and the record's three paired SQLite sets with the loaded one left out of the index's share, and why; MALLOC_ARENA_MAX=1 recommended on a SQLite host that restores tables of items of more than a few hundred bytes, with the measured points to size from and its duration cost for one job and for two); configuration (filesystem and S3 examples, the root that must exist, the spool's mode rule, the root certificates an S3 store needs (ca-certificates or SSL_CERT_FILE, with an http:// endpoint too), the split-root identity mismatch that no command in this release clears (backup purge --confirm, plain backup sync and --prune-missing refuse while the store is unhealthy, and --adopt-store takes only a root another catalog owns) with the two remedies that exist: a catalog copy from before the split, or one directory's rows deleted from the catalog by hand and its backups copied into the kept directory for backup sync to adopt; pointing at the admin guide's [backup] and [backup.store] rows and crates/backup-store/README.md); securing the store (including that the check does not probe kms:Decrypt); what a running backup costs each database; a restore drill (every aws command with --endpoint-url, and the copy modes an off-host drill needs, its account and table directories created 0700); catalog-loss recovery with backup sync (a catalog restored from a dump, a re-created catalog with --adopt-store in both SQLite shapes, the config file copied aside before init --overwrite, which regenerates the whole file with only the storage connection settings carried over (a port, bind_addr, TLS path or limit edited in the default sections reverts too, and the new server listens on 18443), and the copy put back whole apart from the storage settings before the server starts, the management API as the way to import a key pair without its secret in the process list, sync's exit codes, the in-memory startup sync and the accounts it needs re-created); legacy backups (they stay restorable; migrate-legacy is #821, so the manual gives today's path: restore, back up again with a store, check, delete, and restore into the original name only once that table is gone, since deleting a live table loses its writes since the legacy backup); what a restore preserves (the table the differences document carries, with its one gap); and moving data to and from Amazon DynamoDB (the format is the S3 export layout; the service's ImportTable parameters and that it creates no LSI; ExtendDB's ImportTable reads one uncompressed local file named by its canonical path; the export's FilePath; a store-prefix import is #748, not in this release).

Each statement of the design's PR 11 acceptance is in the manual: the first-sentence failure domain and S3 recommendation; encryption at rest is the store's job; a copy tool runs as the server's user or as root, and a root under a storage administrator's directory, an NFS mount owned by nobody or a group-writable parent without the sticky bit is refused; a bind mount or a filesystem without Unix permission semantics is outside what the store observes, and a root mounted with a mask that reports 0700 enforces nothing; AbortIncompleteMultipartUpload, and for a versioned bucket NoncurrentVersionExpiration and ExpiredObjectDeleteMarker; the least-privilege IAM policy (pointed at); a PostgreSQL snapshot retains every dead row version and the SQLite WAL grows during a backup, with figures measured for the manual (below); the claim connection's keepalive bound (30 s idle, three probes 10 s apart: about a minute) and pg_terminate_backend on the claim's session, with the query that finds it by its lock key in the catalog database; a transaction-mode pooler is unsupported; the SDK's trace logs carry the session token; write access to an account's prefix is backup-administrator authority for that account, with per-account IAM prefix policies; MongoDB's minSnapshotHistoryWindowInSeconds.

Figures measured for the manual (the record's new "Figures measured for the manual" section gives their method, build and load, and their reproductions on this tree), same host class, release builds, 100k-item vector tables as the backup's source: on PostgreSQL a 25.2 s backup beside 12,156 PutItem overwrites of 1,000 200-byte items in another table grew that table from 0.4 to 4.4 MiB, and a VACUUM (VERBOSE) 10 s into the backup reported 4,948 row versions "dead but not yet removable"; the same writer for the same time with no backup left the table at 0.5 MiB, and the first VACUUM after the backup removed every dead version. On SQLite the WAL grows by the pages each write changes, which depends on the table's indexes. A 22.8 s backup beside 15,472 such writes to a table with no secondary index grew the -wal file from 4.0 to 61.1 MiB, about 3.9 KB per write. The same writer on a table with two GSIs grew it from 4.1 to 1,294.1 MiB over 13,996 writes in a 25.6 s backup (96.6 KB per write), 58.7 to 1,277.6 MiB over 13,953 writes in a second run on a WAL already grown (91.6 KB), and 4.1 to 1,261.8 MiB over 13,891 writes in 26.0 s in the run on this tree (94.9 KB, load 3.7); a fourth run on this tree grew it from 4.2 to 1,117.7 MiB over 14,212 writes in 25.9 s (82.2 KB, load 3.9). The reproductions of the other two on this tree: a 28.1 s PostgreSQL backup beside 12,623 overwrites grew the table from 0.49 to 4.91 MiB, "3952 are dead but not yet removable" 10 s in; a 22.5 s SQLite backup beside 15,267 writes grew the -wal file from 4.1 to 63.0 MiB. With no backup the writers left it at 4.0 and 4.1 MiB, and the file kept its size after the backup and after the server stopped. The manual's sizing rule names the indexes: about one 4 KiB page per write for a table with no index and about 24 for a table with two GSIs (20 to 24 over the four runs), since an asynchronous GSI write goes through gsi_pending and is applied in later transactions.

Binary checks of the manual's process-level statements (release builds of this tree, SQLite and PostgreSQL unless named):

• A root under a 0775 parent: startup error, "has mode 0775 and is writable by group or others without the sticky bit"; a 0755 root: "chmod it to 0700 (on a filesystem without Unix permissions, mount it with a mask that yields 0700)"; a 0700 root under a 1777 parent starts healthy; an absent configured root stops startup, "backup root is not usable: No such file or directory (os error 2)".
• The drill on SQLite and on PostgreSQL: CREATING from both requests, AVAILABLE, ACTIVE, the count equal; backup list beside a running server. An off-host copy with 0755/0644 modes is refused by backup sync (exit 1, "has mode 0755 and is accessible to other users; chmod it to 0700"), a restore of such a backup fails, and only a copy that is the newest backup makes the next check mark the store unhealthy; with the modes kept, or after chmod -R u=rwX,go= and sync again, the copy adopts and restores.
• Catalog loss on SQLite with [backup.store] configured explicitly: the database removed and init run again; /health reports "reason":"owned by another catalog", degraded; DescribeBackup is BackupNotFoundException; plain backup sync exits 1; --adopt-store adopts (its output quoted in the manual), the store is healthy and the backup restores. Plain sync on an owned root reports the prefix as having a row.
• Catalog loss on SQLite with the default root (no [backup.store] section): the server logs "the default root ... failed its check (extenddb-store.json names catalog ; this catalog is ); the catalog has no backup rows for it, so the store is unconfigured and CreateBackup writes in-catalog backups", /health is {"components":{},"status":"healthy"}, and a CreateBackup answers AVAILABLE with the in-catalog warning. The account re-created with manage create-account --account-id 123456789012, its user and policy, and the old key pair imported with manage import-access-key ... --yes: the old key pair signs requests again. --adopt-store beside the running server exits 6 ("another extenddb process is already using ..."); with it stopped, plain sync exits 1 and --adopt-store exits 0; restarted, backup_store is healthy, the old backup and the in-catalog one are AVAILABLE, and the old one restores with 300 of 300 items.
• The in-memory server: the startup sync's log line, DescribeBackup AVAILABLE in the new catalog once its account exists, the restore, backup list refused with exit 1. On the sqlite-memory,dev-mode build: a backup taken in the first start's default account (995720651603); the second start's default account was 823675310910, ListBackups listed nothing and DescribeBackup answered AccessDeniedException after the line "startup sync of the in-memory catalog: 1 backup(s) adopted"; after manage create-account --account-id 995720651603, a user, a policy and a key, DescribeBackup answered AVAILABLE and the restore reached ACTIVE with 300 of 300 items.
• kind = "disabled": the in-catalog create answers AVAILABLE and logs the deprecation warning; with the store back, ListBackups lists both kinds, the legacy backup restores, the restored table backs up to the store, backup list lists only store rows; backup migrate-legacy is an unrecognized subcommand.
• An S3 store at moto (a bucket of its own, removed): healthy, the objects under <prefix><account>/<table>/<id>/ beside extenddb-store.json, a restore through backup-spool/ beside the database file.
• backup sync beside a running SQLite server exits 6 ("another extenddb process is already using ...").
• ImportTable of a backup's data file: the gzip file is refused, "I/O error reading import file at line 1"; decompressed, 1,000 of 1,000 items imported. A FileSource.Path through a symbolic link (/home to /local/home on this host) is refused, "Path must resolve under one of the configured allowed paths", whether the [import] paths entry is written through the link or not; the canonical path imports with either entry. Export writes one uncompressed {"Item": ...} file under the account's directory, at FilePath, or at S3Prefix when FilePath is absent; S3Bucket alone is refused, "Either FilePath or S3Prefix must be provided".
• manage create-account --help names --account-id; manage import-access-key --help names --account-id, --user-name, --access-key-id, --secret-access-key and --yes; init --help gives the two new defaults.
• PostgreSQL init --catalog-db <name>_catalog creates the data database <name> and records data_database_name. The claim session is found by hashtextextended('backup:<backup id>', 4539474) in pg_locks (and a restore's by <account id>/<table>), distinct from the extenddb serve registration session; with two catalogs on one cluster holding the same key, the query without a database predicate returned both sessions, and with l.database limited to the current database (as the manual gives it) only its own. Terminating the claim stopped the job 13 s later at its heartbeat ("stopped with no further store or catalog access"), and with stale_job_after_secs = 45 the reconciler failed the row and deleted its partial objects 47 s after the termination. Terminating the registration session as well (a first run that ended every advisory holder) makes the server answer ServiceUnavailable while it registers again, which the manual warns against.
• On PostgreSQL, terminating the extenddb serve registration session during a backup failed that backup at the registration's next renewal ("FAILED (shutdown); its objects were deleted"), and the server registered again. backup sync --prune-missing then waited 3 min 16 s on that backup ("the FAILED backup ... has a live owner (heartbeat within stale_job_after_secs)"), as purge waits.
• A PostgreSQL catalog database restored alone beside a data database that survived: a table deleted after the dump was listed and answered ResourceNotFoundException until deleted again; a table created after it was absent, and creating it again gave an empty table; sync then listed the backups taken since.
• The PostgreSQL pair on a pgvector/pg16 container: both dumps with the server stopped under umask 077 (files 0600; the catalog dump carries encryption_key and data_database_connection_string); psql -f over the existing databases printed 110 and 33 ERROR lines, exited 0 and restored nothing (-v ON_ERROR_STOP=1: exit 3); into databases created with plain createdb (owner postgres) the restore succeeded and CreateTable failed, "permission denied for schema public"; into databases created with createdb -O extenddb it succeeded with 0 errors, the table and its GSI held the dump's 50 items, a table created after the dump was gone, and CreateTable and PutItem of a new table worked.
• The admin guide's PostgreSQL procedure as it now stands, end to end on a pgvector/pg16 container of its own (PostgreSQL 16.15, default names): the host's pg_dump 15.18 refused, "aborting because of server version mismatch", exit 1, leaving a 0-byte file; the container's pg_dump under umask 077 wrote both files (exit 0, 41,258 and 18,413 bytes, 0600); after a table created since, the old databases renamed aside with ALTER DATABASE ... RENAME TO ..._before_restore, createdb -O extenddb, psql -v ON_ERROR_STOP=1 -f (exit 0, no ERROR line), the restored server listed the dump's table only, scanned its 50 items and its GSI's, and CreateTable and PutItem worked; then DROP DATABASE of the two old ones. On a second, empty container: createdb -O extenddb failed, role "extenddb" does not exist, until CREATE USER extenddb WITH PASSWORD with the dump's password; an empty file restored with exit 0 and the server then refused, "the catalog ... is not initialized"; after the restore the server, whose catalog's data_database_connection_string still named the first container, put a new table's items into the first container's data database while it ran and failed to start once it was removed ("connect to the data database extenddb: pool timed out"); with the row set to the new address by UPDATE settings it served the restored table and CreateTable and PutItem landed on the new server, and verify passed. Both containers removed with -v.
• A key pair imported with curl --netrc-file ... --data @key.json to POST /management/accounts/<id>/users/<user>/access-keys/import: 201 "Access key imported", and the pair signs requests. init beside an existing extenddb.toml exits 255, "Config file "extenddb.toml" already exists. Use --overwrite to delete and regenerate it."; with --overwrite the PostgreSQL bootstrapper prints "Loading defaults from extenddb.toml" (the Cassandra one prints the same line; the SQLite one does not, and the manual quotes neither). version prints a fifth line, test-hooks disabled.
• backup sync and the other backup commands exit with serve's code on a catalog it refuses: 3 behind, 4 ahead with a breaking migration (both observed on this tree's sources), 5 needing attention (CatalogStatus::exit_code).
• FileSource.Path naming a file that is itself a symbolic link: ValidationException "Symbolic links are not allowed in import/export paths". DescribeBackup of an on-demand table reports each GSI's throughput as 0 and 0, which botocore's parameter validation refuses in TableCreationParameters.
• A SQLite catalog restored from a copy taken before its newest backup was deleted: /health "reason":"manifest probe failed", CreateBackup and the restore of an older backup refused, plain sync and --prune-missing exit 1 ("sync changes no row while the store is unhealthy"), still unhealthy after a restart. The manual says this release has no command that clears it without a copy of the prefix, and to take the catalog dump after deletions.
• A split-root identity mismatch on PostgreSQL (debug build of this tree): two empty directories started in turn at one root path, each given its own extenddb-store.json, two backups taken with the second in place, then the first put back. /health "reason":"identity mismatch", degraded; backup list (exit 0) prints the problem in its header, "extenddb-store.json holds store id ; the catalog's rows for the root carry <rows' id>", and every row with its store_id; backup purge without --confirm prints the plan and "Dry run: nothing was deleted. The store is unhealthy (identity mismatch); --confirm refuses until the store's check passes."; backup purge --confirm exits 1, "the filesystem backup store is unhealthy: ...; fix that before purging its backups", with both rows and all 11 files of the root kept; plain backup sync and backup sync --prune-missing exit 1, "...; sync changes no row while the store is unhealthy". With one row's store_id set to the other directory's id (the row a backup on the other instance writes): "the catalog's rows for the root carry more than one store id: , ", the same three refusals, and --adopt-store refuses the same way. The manual's by-hand remedy: with the kept directory in place and the other id's row deleted from backup_index, the server started healthy and backup sync beside it exited 0, adopting the backup whose row had been deleted from its prefix under the kept root.
• On release builds of this tree (the rebased one), SQLite and PostgreSQL: a catalog lost and re-created. destroy leaves extenddb.toml; init beside it exits 255; init --overwrite regenerates it from defaults with only the storage settings carried over (on PostgreSQL the connection string, on SQLite the database path), dropping [backup], [backup.store] and [import]. On PostgreSQL (debug build of this tree) a config with port, bind_addr, cert_path, run_dir, [logging] level and [limits] max_item_size_bytes edited and [backup] and [backup.store] added came back after destroy and init --overwrite as [server] [server.tls] [storage] [storage.postgres] [auth] [auth.cache] [logging] [limits] with every edited value at its default (no port line) and the three added sections gone; the regenerated connection string names the new catalog. Started that way, the PostgreSQL server's /health was {"components":{},"status":"healthy"}, it logged "backup store: none configured", and --adopt-store exited 1, "no backup store is configured"; the SQLite server opened a new, empty backups/ beside the database, and --adopt-store adopted 0. With the sections put back from a copy taken before init: "owned by another catalog", plain sync exit 1, --adopt-store (beside the PostgreSQL server; with the SQLite server stopped) adopting the backup, the store healthy, and the restore giving 2,000 of 2,000 and 300 of 300 items.
• The off-host copy on SQLite, three ways: account and table directories made with install -d -m 0700 and the prefix copied with cp -rp, and the account directory copied whole with cp -rp, each adopted by backup sync and restored (300 of 300); directories made with mkdir -p under a 022 umask are 0755, and sync exits 1, "the store's listing failed: permission denied: .../123456789012 has mode 0755 ... 0 backup(s) listed before the failure were adopted", until chmod -R u=rwX,go= on the account directory, after which it adopts and the backup restores.
• The upgrade manual's step 2 and rollback on the shared PostgreSQL 16 server: with the host's pg_dump 15.18 the dump command exits 1, leaves a 0-byte catalog file and does not print "both dumps written"; with the server's pg_dump 16.15 it prints it (both files 0600, non-empty). After a table created since the dumps, the rollback as written (both databases renamed to ..._before_rollback, created with OWNER the application role, restored with -v ON_ERROR_STOP=1, each exit 0 with no ERROR line) gave a server that listed only the dumped table, scanned its 50 items and created and filled a new table; the renamed databases were then dropped. A failing restore with ON_ERROR_STOP exits 3 and an empty file restores with exit 0; dropping the new database and renaming the old one back restored the previous state.
• init --catalog-db with a name lacking the suffix exits 1, leaves that database and writes no config file; init again with --data-db fails on the leftover until it is dropped, then succeeds. manage without --password or EXTENDDB_PASSWORD exits 1, "No password provided. Use --password VALUE or set EXTENDDB_PASSWORD."; with EXTENDDB_PASSWORD it creates the account. botocore's parameter validation refuses a table-level ProvisionedThroughput of 0 and 0 in ImportTable's TableCreationParameters and accepts the parameters without it.

Other documentation. The admin guide's Data Recovery entry named the PostgreSQL data database extenddb_catalog_data; init derives it from the catalog's name without _catalog (extenddb by default) and records it in settings as data_database_name. The entry now has a table of each backend's databases, says that a catalog name without the suffix needs --data-db and that an init that failed that way leaves the catalog database to drop first, and treats the two PostgreSQL databases as one copy: both pg_dumps with the server stopped (each takes its own snapshot of one database, so two dumps beside a running server disagree when a table is created or deleted between them), or a base backup of the PostgreSQL server holding both, with umask 077 before the dumps, since the catalog dump holds the encryption key and the application role's password. The restore goes into new, empty databases created with createdb -O extenddb by the admin role and psql -v ON_ERROR_STOP=1 -f: a restore over the existing databases restores nothing, and one into databases the admin role owns leaves the server unable to create tables. The old databases are renamed aside (ALTER DATABASE <name> RENAME TO <name>_before_restore) and dropped only once the restored server has read every table; the dumps use the pg_dump of the server's major version and go on only when both exited 0 and are not empty; on a new PostgreSQL server the application role is created first with the password the catalog dump's data_database_connection_string carries, and at another address that row is set to the new address before the server starts. The SQLite row says how to restore: stop the server, remove the current -wal and -shm, and put the copied files in place. The security model's block and the deployment guide's PostgreSQL checklist say the same (the security model now says the entry restores the dumps, and a base backup, which covers both databases only when they are on one server, is restored with PostgreSQL's own procedure), the upgrade manual's three pg_dump lines take umask 077, its step 2 dumps are checked (each exit code and test -s), and its rollback renames both databases aside, restores each with psql -v ON_ERROR_STOP=1, drops the renamed copies once the previous binary has read every table, and points at the manual for a catalog restored alone, and the manual's catalog-loss section restores the pair and says what a catalog database restored alone gives. The same wrong name is corrected in the architecture, design and security-model guides; the quickstart's --data-db row gave <catalog>_data and now gives the derived name, and its --pg-user and --pg-password rows (extenddb, extenddb-local-dev) are --pg-user (the admin role, $USER then postgres), --pg-pass, --extenddb-user and --extenddb-pass with their defaults (bootstrapper.rs), its sample lines being at migration 8 as main has them; AGENTS.md, the troubleshooting guide's and the extenddb skill's pg_stat_activity query, the technical-debt register and the storage-extension guide named the PostgreSQL data database extenddb_data, MongoDB's name, and now say extenddb (the last two also named the catalog extenddb, now extenddb_catalog, and the register's A-1 entry cites functions in place of line numbers that had moved); the design guide's settings key was data_database_url and is data_database_connection_string (and data_database_name); init --help said "Data database name (default: extenddb)" and "Catalog database name (default: _catalog)", and now says "the catalog's name without its _catalog suffix" and "extenddb_catalog, or the config file's" (bootstrapper.rs: the catalog never derives from --data-db). git grep '<catalog>_data\|<data-db>_catalog\|data_database_url' finds nothing. The admin guide's [backup] section and the upgrade manual's backup paragraph point at the manual; the upgrade manual's shared-root and re-created-catalog sentences move to the manual, which states both. The admin guide's max_concurrent_jobs row says that on SQLite a restore of items of more than a few hundred bytes, of any attribute types, holds more than its page-cache term (its write batch and decode chunks) and points at the manual's figures, and its backup commands' paragraph gives the exit codes 3, 4 and 5 on a catalog serve would refuse. The deployment guide's PostgreSQL checklist has "Configure [backup.store] (S3 for production) per the Backup and Restore manual", its backup line limits a base backup or an instance snapshot to both databases on one PostgreSQL instance, its overview says where table backups go on each backend, and its backup mentions and service-unit comment point at the manual (the comment keyed to SQLite restores of items larger than a few hundred bytes). The differences document's restore row gives the 1M Cassandra (49.0 s) and MongoDB (33.8 s, which another run may exceed by up to the backfill worker's 5 s cycle) restores with a pointer to the PR 11 record. ADR-0019 and the Cassandra RFC's "Backup and restore" section are superseded in part, since CreateBackup on Cassandra still writes their in-catalog format, before it returns. docs/adr/README.md allows no edit of an accepted ADR but its Status, so ADR-0019 changes only its Status line (and its index row), "Superseded in part by the backup and restore design (section 10)", and design section 10's closing paragraph states what that covers: what still holds, what changed, and that the reconciler now removes an in-catalog backup whose CreateBackup did not finish, so the ADR's Negative consequence about a future reconciliation worker no longer holds for the create. docs/adr/README.md's process now names that form, "Superseded in part by [document]", for a decision a later design document replaces in part. docs/rfcs/ has no such rule, so the RFC's section carries a note at its top pointing at the manual and the design, and its section 4 line on synchronous data movement and its Known Gaps lines on synchronous backup and restore and on restored indexes carry "Superseded in part" notes (the last for a backup that recorded its index definitions, since one an earlier release took may not have). docs/build-docs.py lists the manual as document 19; the full build ran on this host (Python 3.10, WeasyPrint 70.0): 19 HTML, 19 PDF, 0 failed. The README's and AGENTS.md's documentation tables have a row for it, and AGENTS.md's Extending Storage row names docs/manuals/12-extending-extenddb-storage.md in place of a file that does not exist. The quickstart's version sample has the binary's fifth line, and the quickstart and the extenddb skill's init reference say that init refuses an existing extenddb.toml without --overwrite. The deployment guide's overview sentence is reworded to parse.

Design (docs/design/15-backup-restore.md). The testing summary's "still to be measured" sentence now states the measured cells, what the vector restore's memory belongs to and the vector index's share of the restore's duration (earlier revision items keep their wording); section 7's memory paragraph says that secondary indexes add only the page cache, what a SQLite restore of items larger than a few hundred bytes holds (the write batch and decode chunks, growing with the item's size up to the batch bound whatever the attribute types), and, in a sentence of its own, what a PostgreSQL vector index adds (15-backup-restore-bench-pr11.md); the PR 11 section's line says what shipped, and its "Documentation it writes" line says the ADR and the RFC section are superseded in part; section 10's row says "superseded in part" and its closing paragraph says what the ADR's Status and the RFC's notes cover and what the two manuals now carry; revision item 35 records the filter history, the measurements, the manual's place and checks (the catalog-loss config file, the off-host copy's directories and the upgrade manual's rollback among them), and the database-name correction, and names its builds as versions of this change's sources (04c2915 has this commit's Rust sources; 30553b3's differ only in the two init help lines), since neither hash stays in the history. The design's revision items 32 to 34 already exist, so this PR's item is 35. The bench record's Durations section gives the one-arena SQLite backup (222 s) and restore (283 s) separately.

Cross-references with #737

#737 (open, another author's) touches four files this PR also edits: 05-admin-guide.md, 07-upgrade-manual.md, 10-security-model.md and 11-deployment-guide.md. It is not on this branch and nothing it adds is edited here. The overlaps:

• The admin guide's "Data Recovery" entry: docs: back up and restore SQLite and PostgreSQL with their own tools #737 replaces its body with a pointer to its new section, and this PR rewrites the body (the databases table, the derived data-database name, the pg_dump lines with the server stopped or a base backup). The manual links to that entry.
• The security model's "Backup and Recovery" block: docs: back up and restore SQLite and PostgreSQL with their own tools #737 replaces it with a pointer to its admin guide section, and this PR rewrites the same lines.
• The deployment guide's PostgreSQL checklist: docs: back up and restore SQLite and PostgreSQL with their own tools #737 rewrites the "Set up automated backups (pg_dump, ...)" line, and this PR rewrites the same line and adds the [backup.store] line directly below it.
• docs: back up and restore SQLite and PostgreSQL with their own tools #737's new section opens with "CreateBackup stores each table backup in the deployment's own database, in the catalog's backups and backup_items tables", which is false of this series once a backup store is configured.

Whichever of the two lands second reconciles these: the entry's anchor and body, the security model's block, the checklist's lines, its CreateBackup sentence, and the manual's links.

Verification

• After the documentation changes for the split-root sentence, the workflow lines, the collation condition, the SQLite sets, the CA store and init --overwrite, with every tracked file touched, a debug PostgreSQL build from this tree: cargo fmt --all -- --check clean; doc_version_literals 3 passed; every relative link and anchor in the changed Markdown files resolves; the split-root and init --overwrite checks above.
• After the latest documentation changes, on the rebased tree with every tracked file touched: cargo fmt --all -- --check clean; cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, also with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; doc_version_literals 3 passed; every relative link and anchor in the 22 changed Markdown files resolves; release builds (SQLite and PostgreSQL) for the catalog-loss, off-host copy, rollback, init and manage checks listed above as run on this tree.
• After the rebase onto b42dbfa9, every tracked file touched, debug builds: cargo fmt --all -- --check clean; cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, also with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; engine --lib 477 passed, app 49, server 343, storage 236; SQLite --lib backup 65; PostgreSQL backup_restore 58 (--test-threads=2); doc_version_literals 3.
• After the rebase onto be84a7eb, every tracked file touched, debug builds: cargo fmt --all -- --check clean; cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, also with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; engine --lib 468 passed, app 49, server 343, storage 236; SQLite --lib backup 64; PostgreSQL backup_restore 58 (--test-threads=2); doc_version_literals 3. The lines below are from the tree before the rebase.
• cargo fmt --all -- --check: clean.
• cargo build -p extenddb-app and cargo clippy -p extenddb-app --all-targets --no-deps -- -D warnings: clean. No test or snapshot holds the init --help text.
• cargo test -p extenddb-storage-postgres --test doc_version_literals: 3 passed (the manual carries no binary version literal; the record's service versions are not flagged).
• No other workflow runs a documentation check. actionlint 1.7.7 on integration.yml: only the pre-existing custom runner-label findings.
• docs/build-docs.py on an exported copy of the tree (Python 3.10, WeasyPrint 70.0): 19 HTML, 19 PDF, 0 failed; the manual's HTML has one heading per Markdown heading and none made from a wrapped line (no line of the changed manuals or design documents begins with # and a digit).
• Every relative link and anchor in the changed Markdown files resolves.
• The CI-way runs and the binary checks above. In CI the workflows that run on a stacked pull request run on this PR: 36 checks at the previous head (49 on feat(backup): backup sync, --adopt-store and --prune-missing [backup/restore 10] #825). CodeQL runs only on pull requests against main, and the fuzz and scripts workflows run only for their paths, which this PR does not touch.
• git grep '<catalog>_data\|<data-db>_catalog\|data_database_url': no hit; extenddb_catalog_data only in revision item 35, as history; extenddb_data in the documentation only for MongoDB; and no lists of numbers as the restore memory's cause, 16%, five runs for the spread, 14.7 s as the scan's bound, 0.2 to 0.7, 340 bytes, within 9 MiB, 27 to 41, 35 to 38 or dropdb in the documentation.
• No non-inclusive term in the diff.

Not here

• migrate-legacy (Backup/restore: backup migrate-legacy #821), the store-maintenance options (Backup/restore: backup sync --move-store, --forget-store, --forget-backup #820), export and import through the store (Backup/restore: export and import through the store #748).
• A live check of an ExtendDB backup imported into Amazon DynamoDB, and of a DynamoDB export imported into ExtendDB from the service: the manual states that the import into the service has not been run.
• RestoreTableFromBackup of a backup whose prefix has loose modes answers InternalServerError (HTTP 500), with the log line naming the directory's mode; the base's existing answer, not this diff's; the manual says such a restore is refused with InternalServerError and that the server log names the directory.
• An S3 store's upload and download durations, and a database on another host: not measured, and the manual says so.
• init --catalog-db with a name that lacks the _catalog suffix derives the catalog's own name for the data database, creates the database once, fails on the second create with "Run 'destroy' first", and leaves an empty database behind with no config file for destroy to read. An existing init defect; the admin guide now says such a name needs --data-db and that the leftover database is dropped first.
• init --overwrite keeping only the storage settings and regenerating every other section from defaults, and extenddb destroy's printed message, which says a re-created catalog finds the store owned by another catalog without saying that the regenerated config file has no [backup.store]: the base's behaviour and text; the manual and the admin guide say to copy the file aside and put the sections back.
• MongoDB's backfill worker is not woken when a restore's load ends, so the table becomes ACTIVE only at the worker's next 5 s cycle; waking it at the load's end is a possible later change.
• A catalog restored from a dump whose newest AVAILABLE backup was deleted after the dump keeps its store unhealthy (every store operation refused, across restarts), and this release has no remedy without a copy of that backup's prefix: --prune-missing refuses while the store is unhealthy. The manual states it and says to take the dump after deletions.

[LeeroyHannigan]

I built the head on SQLite, PostgreSQL and MongoDB and ran the manual's and the guides' procedures as written. That covered the PostgreSQL dump and restore pair, the upgrade manual's rollback with its fallback, and the restore drill. I ran catalog loss and --adopt-store on SQLite with an explicit store, with the default root, and on PostgreSQL. I covered the off-host copy three ways with its refusals, the in-memory restart, kind = "disabled" with a legacy backup, and an S3 store at moto. The filesystem root and spool rules, the claim-session query with pg_terminate_backend, and ImportTable and export ran as written. I also started a server on a catalog restored from a copy taken before its newest backup's deletion. The builds were dev builds. I checked the manual's commands, options, exit codes, log lines, defaults and limits against the code. I checked the 159 relative links and anchors in the 22 changed Markdown files. I recomputed the record's arithmetic. I located the 66 figures the manual takes from the three records. On Amazon DynamoDB in us-east-1 I measured DescribeBackup of an on-demand table, RestoreTableFromBackup, and reads of the target while it was CREATING. I merged main to check the workflow change. All 36 checks at this head pass. The PostgreSQL and SQLite jobs ran the 10 import_export tests and passed them.

What matched the description:

• The procedures above behaved as the manual and the guides say, and the quoted messages matched the output word for word.
• The quoted refusals and log lines match the code. So do the [backup] defaults, the permit wait, the restore batch bounds and the limits. The lock key seed 4539474, the keepalives, exit codes 3 to 6 and the import route match too.
• The CI filter's history (687e46f, then 556c501 after e0d6a0a) and the 10 tests the expression selects out. The MongoDB and Cassandra jobs run the file unfiltered.
• In DescribeBackup the service reports an on-demand table's and its GSI's ProvisionedThroughput as 0 and 0 beside OnDemandThroughput. botocore 1.43.81 refuses 0 and 0 in TableCreationParameters, which has no member for local secondary indexes. GetItem, Scan and DescribeTimeToLive on the CREATING target answer ResourceNotFoundException. DeleteTable on a CREATING target answers ResourceInUseException on the head.
• The record's arithmetic, the peak-minus-idle and growth figures included. The manual's figures are in the records with the shape the manual gives them.
• init --help, the admin guide's database names, the derived data database name and the init refusal with exit 255 against the code. The design's section 7 and item 35, the ADR's status line and the RFC's four notes against the code and the record.
• The control cell in kind on a dev build: 311.6 MiB over idle for 100k SQLite items of a 2,290-character string. Under one arena it was 143.3 MiB.

Smaller items without an inline comment:

• The description says every workflow runs on this PR in CI. 36 checks ran here against 49 on #825. CodeQL does not run, because the base is not main. Fuzz and Scripts test run only for their paths, and this PR touches none of them.
• The description says init --overwrite prints "Loading defaults from extenddb.toml". The PostgreSQL and Cassandra bootstrappers print that line. The SQLite bootstrapper does not, and the manual does not quote it.

Two items are Major and inline. The manual names backup purge as the command that clears a split-root identity mismatch, and purge refuses an unhealthy store at this head. The two workflow lines this PR edits conflict with main, where #474 added --terraform to both of them after this PR's base. Four inline items are Minor. Once you fix the two Major items, I will approve. The Minor items can land in the same revision.

[LeeroyHannigan]

Major: backup purge --confirm does not clear this state. It opens the store in write mode (crates/app/src/cmd_backup.rs line 331) and runs the store check. On any problem it refuses with "the {kind} backup store {root} is unhealthy: {problem}; fix that before purging its backups" (line 821). An identity mismatch is such a problem (crates/engine/src/backup/store_identity.rs lines 217 to 219). Plain sync and --prune-missing refuse the same way (line 1217), and --adopt-store handles a foreign catalog only (line 1188). So at this head no command clears a split-root identity mismatch, and purge would delete the root's backups if it ran. Say that purge refuses while the store is unhealthy. Name a remedy that exists in this release: a catalog copy taken before the split, or one directory's identity file and rows deleted by hand. Or move the sentence to the PR that adds --forget-store. The deployment guide's line 298, which this PR edits for other words, says the same ("only backup purge resolves").

[LeeroyHannigan]

Major: this line and line 335 conflict with main. #474 (2671ee1) added --terraform to both after this PR's base, so main reads devtools/run-tests --extenddb --pytest --comprehensive --terraform --parallel --filter "not import_export". git merge-tree --write-tree <head> origin/main reports CONFLICT (content): Merge conflict in .github/workflows/integration.yml with these two hunks. The same merge of #825's head alone merges the file without conflict, so the conflict is this PR's. When this PR moves to main as its base, keep --terraform and drop the filter on both lines: devtools/run-tests --extenddb --pytest --comprehensive --terraform --parallel. Main also raised the two jobs' timeout-minutes to 70 and 60, which merges without conflict.

[LeeroyHannigan]

On PostgreSQL the stream is key-ordered only when the key schema is orderable and the text key columns have collation C or POSIX. That is key_ordered = orderable && collate_c at crates/storage-postgres/src/snapshot_engine.rs lines 623 to 634. For a table whose key columns predate TEXT COLLATE "C", the server streams the table unordered and logs a warning. The RTO table's PostgreSQL rows differ by that condition: 9.5 s in key order against 17.5 s unordered at 1M items. Add the collation condition here and say that the server logs a warning when it streams such a table unordered.

[LeeroyHannigan]

The record has three paired sets on SQLite. This sentence reports two of them without saying that it leaves the third out, or why. The record's table at docs/design/15-backup-restore-bench-pr11.md line 337 pairs 20.53 and 22.36 s without the index against 22.26 and 36.58 s with it. Those runs were at load 7.0 to 13.1. With them the no-index duration spans 18.0 to 22.4 s over the record. Either give all three sets and say why the share leaves the loaded set out. Or write "two sets of runs on a lightly loaded host".

[LeeroyHannigan]

A server with kind = "s3" needs a system CA store or SSL_CERT_FILE, with an http:// endpoint too. In a bare Ubuntu 24.04 image with the endpoint and force_path_style = true this paragraph gives, the server stopped at startup. It printed "TrustStore configured to enable native roots but no valid root certificates parsed!" (aws-smithy-http-client 1.1.9, rustls_provider.rs line 116). With SSL_CERT_FILE naming a bundle it started, and /health went from checking to healthy in 3 s. Add a sentence naming the ca-certificates package the deployment guide's image installs, or SSL_CERT_FILE.

[LeeroyHannigan]

init --overwrite regenerates [server], [limits] and [logging] from defaults as well. On SQLite and on PostgreSQL the new file had [server] [server.tls] [storage] [storage.<backend>] [auth] [auth.cache] [logging] [limits] and no port line. A port, bind_addr, TLS path or limit set there is gone, and the new server starts on 18443. An operator who edited only values inside the default sections reads this sentence as nothing to restore. Say that edited values in the default sections revert too, and that the copy goes back whole apart from the regenerated storage settings.

[LeeroyHannigan]

Round 2, at eb9e48d. I checked the 8 items of my round-1 review against this head and #825's head 84bc263, and this head fixes all 8. The split-root text in the manual and the deployment guide now quotes the purge and sync refusals. backup list and backup sync behave at 84bc263 as its two remedies say. Both integration.yml lines carry --terraform, and git merge-tree with main reports no conflict.

The two failing checks do not come from this PR. cassandra-pytest (1) fails test_all_increments_apply in tests/python/test_rfc0003_concurrency.py with TransactionConflictException, which is #878. cassandra-integration is the job that reports that failure. This PR changes the PostgreSQL and SQLite jobs and no Cassandra code.

The changes since round 1 bring five new Minor items, all in text that this push changed. They are not conditions for my approval, and they can land in the same revision.

1. docs/manuals/14-backup-and-restore.md lines 247 to 252, the by-hand remedy.

   • "delete from the catalog database the backup rows that carry the other directory's store id" names no table. The rows are in the backup_index table on SQLite and PostgreSQL and in the backup_index collection on MongoDB. On Cassandra they are in two tables, backups_by_arn (the rows with store_kind set) and backup_index_by_status. All of them carry store_kind, store_root and store_id. Fix: name the table or tables for each backend, and the store_id column.
   • The copy step gives no copy rules, and the two directories are on separate hosts. The restore drill's copy rules (lines 440 to 452) say that backup sync exits 1 on a copy that lands 0755. A copy by tar, by rsync without permissions or by aws s3 cp --recursive lands 0755 under a 022 umask. Fix: point the copy step at those rules.

2. Same file, lines 245 to 247, the catalog-copy remedy. Backups that the other instance took after the catalog copy are in neither the restored catalog nor the kept root. So sync does not adopt them, and they stay in the other directory, unknown to the catalog. Also, the restored copy can hold rows for the root. Those rows must then carry the kept directory's store id. If not, the next check reports the mismatch again (compare, crates/engine/src/backup/store_identity.rs lines 833 to 852). Fix: say to copy the other directory's account directories into the kept root before backup sync, as the by-hand remedy does. Also say to keep the directory whose store id the copy's rows carry.

3. Same file, line 515. The list of regenerated sections leaves out [auth.cache], which the template has (crates/app/src/init_helpers.rs line 265). A ttl_seconds or enabled = false set there reverts too. Fix: add it to the list.

4. Same file, lines 223 to 225. "needs root certificates the SDK can parse, with an http:// endpoint too" holds only for a debug build. The startup stop is a debug_assert! in aws-smithy-http-client 1.1.9 (src/client/tls/rustls_provider.rs line 116), and a release build compiles it out. A release server with an http:// endpoint sends no TLS request, so it runs without roots. The next sentence already limits the stop to a debug build. Fix: drop "with an http:// endpoint too", or limit it to a debug build.

5. The "CI filters" paragraph of the PR description, and the commit message. Main has not changed .github/workflows/integration.yml since the merge base 887b6e2. So "both sides change them" does not hold: the merge takes this PR's lines with no conflict. #474 (2671ee1) is in this PR's base 84bc263, so it did not add --terraform "after this PR's base". Fix: say that main's lines carry --terraform since #474, which the base includes, and that the merge with main has no conflict.