Skip to content

fix(storage): reconcile native and automatic library bindings - #38

Merged
opendev9 merged 3 commits into
devfrom
fix/storage-reconcile-20261020
Oct 20, 2026
Merged

opendev9 merged 3 commits into
devfrom
fix/storage-reconcile-20261020

Conversation

@opendev9

@opendev9 opendev9 commented Oct 20, 2026 •

Copy link
Copy Markdown

Problem and behavior

Existing automatic libraries have native RepoStorageId=auto-local:<key> routes but no cf_library_storage record, so V2 refuses access. Add a shared reconciliation path for CLI and Hub runtime using native Repo/RepoTrash/GarbageRepos, VirtualRepo, RepoStorageId, Seafile configuration, and physical local directories. No bytes move, no UUID changes, and the legacy storage-key table is neither read nor changed.

Baseline: origin/dev at ee732e74ce02f184586e1313b489e2198dbc15fe.
Head: 2a546363723e3c4aa24ddaa675d2d6c4213aa3c5.

  • storage-reconcile defaults to strictly read-only dry-run: only SELECT/SHOW, no normal or temporary DDL. Missing registry reports independent SCHEMA_MISSING and continues classifying against an empty registry. Only explicit apply may create the exact reviewed table and insert registrations.
  • Verify schema with information_schema plus pure SHOW CREATE parsing. Verify native source tables before any DDL. The command works with SELECT-only credentials and an explicit READ ONLY transaction.
  • Apply uses one transaction per library and INSERT ... SELECT to recheck native source facts and byte-exact routing. Key/owner/route conflicts remain BLOCKED. Unique-key races cannot overwrite records; explicit default-class normalization rolls back both route deletion and registration on failure.
  • Existing reserved with expected active is compatible and unchanged. Existing active/retired lifecycle disagreement is independent STATE_DRIFT, reports actual and expected state, and never blocks or changes that state. Newly inserted records derive active from Repo/Trash, retired from Garbage-only.
  • Managed storage roots reject symlink ancestors and internal key/storage/type/repo aliases. Native default-class paths may resolve CE's standard seafile-data symlink to the same real native tree. Default classes remain pinned unless explicitly normalized; runtime never normalizes.
  • ORPHAN uses all native routes, irrespective of batch or selected library. Legacy install/reconcile detects existing Seafile environment/layout configuration; storage mode flags cannot be silently ignored by older mutating commands.

CLI arguments, single-JSON reporting, schema status, lifecycle compatibility, and operational semantics are documented in docs/storage-reconcile.md. Exit 2 requires business BLOCKED; SCHEMA_MISSING and STATE_DRIFT alone return 0; execution/schema errors return 1.

Validation

Only dedicated disposable local database containers and synthetic schemas were used. Exact SELECT VERSION() results:

11.4.13-MariaDB-ubu2404
10.11.19-MariaDB-ubu2204

Deployment-recipe evidence: Docker origin/dev a7e8c2bf1e4e7630dfa2e54427c1c03d98ad667e, deploy/compose/docker-compose.yml:27, pins mariadb:11.4. The 11.4 test matches that recipe series; 10.11 is additional compatibility evidence. The actual remote DEV/production running version is not verified because its device is disconnected.

CF_STORAGE_TEST_MYSQL_PORT=13319 python3 -m unittest \
  cloudfile_ext.v2.tests.test_storage_reconcile cloudfile_ext.v2.tests.test_native_discovery -v
Ran 39 tests in 8.903s
OK

CF_STORAGE_TEST_MYSQL_PORT=13320 python3 -m unittest \
  cloudfile_ext.v2.tests.test_storage_reconcile cloudfile_ext.v2.tests.test_native_discovery -v
Ran 39 tests in 20.440s
OK

Those runs include all classifications, SELECT-only/read-only transaction with existing and missing registry, absent-table dry-run followed by explicit install/apply, reserved compatibility, unchanged lifecycle drift, CE native symlinks, managed-root symlink refusal, route rechecks, concurrent same-library writers, uniqueness, atomic normalization rollback, all lifecycle tables, VirtualRepo skip, global ORPHAN, and legacy sentinel preservation.

Two final focused tests were then verified on both versions:

CF_STORAGE_TEST_MYSQL_PORT=13319 python3 -m unittest \
 cloudfile_ext.v2.tests.test_storage_reconcile.StorageSQLTests.test_reserved_active_compatibility_and_state_drift_do_not_write_or_block -v
Ran 1 test in 0.205s
OK
CF_STORAGE_TEST_MYSQL_PORT=13320 python3 -m unittest \
 cloudfile_ext.v2.tests.test_storage_reconcile.StorageSQLTests.test_reserved_active_compatibility_and_state_drift_do_not_write_or_block -v
Ran 1 test in 0.218s
OK

CF_STORAGE_TEST_MYSQL_PORT=13319 python3 -m unittest \
 cloudfile_ext.v2.tests.test_storage_reconcile.StorageSQLTests.test_read_only_schema_verifier_rejects_changed_contract_without_altering_it -v
Ran 1 test in 0.443s
OK
CF_STORAGE_TEST_MYSQL_PORT=13320 python3 -m unittest \
 cloudfile_ext.v2.tests.test_storage_reconcile.StorageSQLTests.test_read_only_schema_verifier_rejects_changed_contract_without_altering_it -v
Ran 1 test in 0.505s
OK

The lifecycle case includes the exact native-retired-but-still-in-Trash example. The added schema case rejects changed CHECK, uniqueness, collation, and column default without altering the existing table. During implementation, initial metadata verification failed on index sort ordering and MariaDB hiding CHECK_CONSTRAINTS from SELECT-only users; deterministic index sorting and SHOW CREATE parsing corrected both, and the final runs above pass.

Independent CLI acceptance against 11 synthetic libraries + 1 orphan, batch_size=2, passed on both versions:

{"schema_missing_dry_run":{"AUTO_LOCAL":3,"BLOCKED":2,"NATIVE":4,"NATIVE_CLASS_PINNED":1,"ORPHAN_DIRECTORY":1,"SCHEMA_MISSING":1,"VIRTUAL_REPO":1},"dry_run":{"AUTO_LOCAL":3,"BLOCKED":3,"NATIVE":3,"NATIVE_CLASS_PINNED":1,"ORPHAN_DIRECTORY":1,"VIRTUAL_REPO":1},"apply":{"BLOCKED":3,"ORPHAN_DIRECTORY":1,"REGISTERED":7,"VIRTUAL_REPO":1},"replay":{"BLOCKED":3,"ORPHAN_DIRECTORY":1,"REGISTERED":7,"VIRTUAL_REPO":1},"result":"PASS"}

The missing registry remained absent after dry-run. An intentionally conflicting registration was then introduced in the fixture; the three expected business BLOCKED cases are S3, invalid key, and registration conflict. Apply added seven correct records, replay made no writes, Trash+Garbage overlap remained active, and the legacy sentinel stayed unchanged.

PYTHONPYCACHEPREFIX=/tmp/cf-storage-pycache python3 -m py_compile \
 cloudfile_ext/v2/storage_reconcile.py cloudfile_ext/v2/native_discovery.py cloudfile_ext/v2/__main__.py
(exit 0)
git diff --check
(exit 0)

Not covered

Actual remote database version, real Server upload/download/history/restore, GC/fsck, complete three-repository image acceptance, DEV deployment, or production writes. The real historic 1,407 automatic / 63 default library comparison is post-release acceptance per the user's updated instruction and is not replaced by synthetic counts. The separate effective-permission adaptation is outside this PR. Do not merge before the broader integration requirements are satisfied.

@opendev9
opendev9 merged commit 52f6400 into dev Oct 20, 2026
1 check passed
@opendev9
opendev9 deleted the fix/storage-reconcile-20261020 branch October 21, 2026 00:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant