Skip to content

fix: docker check never modifies the project; atomic catalog cache save (#951) - #956

Merged
ako merged 3 commits into
mainfrom
fix/951-check-readonly-cache-race
Oct 3, 2026
Merged

ako merged 3 commits into
mainfrom
fix/951-check-readonly-cache-race

Conversation

@ako

@ako ako commented Oct 3, 2026

Copy link
Copy Markdown
Owner

Closes #951.

1. docker check never modifies the project

Measured before the fix (11.13 project, whole-tree sha256 + mtime diff before/after):

  • MPRv1, default: App.mpr rewritten permanently, theme-cache/web/theme.compiled.css(.map) rewritten, deployment/sass/main.scss created.
  • MPRv2, default: every mprcontents/*.mxunit rewritten (restored by the snapshot, new mtimes), plus theme-cache/deployment as above.
  • Also with --no-update-widgets, on v1 and v2: mx check itself rewrites theme-cache/ and creates deployment/sass/. So the copy is needed for both modes, not only for update-widgets.

Fix (cmd/mxcli/docker/check.go, new check_copy.go): Check copies the project to $TMPDIR/mxcli-check-* and runs mx update-widgets (unless --no-update-widgets) and mx check there. It skips deployment/, releases/, theme-cache/, .mendix-cache/, .mxcli/, .docker/ at the root, and .git/.svn/.hg/node_modules at any depth. A symlinked file is copied as a file, so nothing can write through it. mx output is rewritten line by line from the copy's path to the project's path. The copy is removed afterwards. A missing project file fails early. With $TMPDIR inside the project, the walk skips its own copy. The output says a temporary copy is checked. With update-widgets it also prints a note: widgets were normalised on the copy, so a CE0463 the stored project still has is not reported; --no-update-widgets checks the project as stored; mxcli fix widgets applies the normalisation. This is the honesty #568/#646 ask for; those issues stay open for their other items. --help, docs-site/src/guides/marketplace.md and the custom-widgets / migrate-design-prototype skills (they said docker check clears CE0463) are updated.

Performance: on this ~37 MB project, the check took 12 s (v1) and 15 s (v2) with update-widgets, and 4–6 s without. The copy is the model plus the widgets/theme/source folders.

Other docker subcommands: lint/report don't run mx. status/logs/up/down/shell don't touch the project. docker build still uses runUpdateWidgets (v2 snapshot) and is left alone, as agreed. See follow-ups.

2. Catalog cache save race

Repro (/home/vscode/t/jts/race copy, 16 parallel mxcli lint -p on a fresh copy): before the fix, 4× failed to create table catalog_meta: table catalog_meta already exists and 3× database is locked. After: 0 errors in 3×16 + 5×8 runs. integrity_check is ok and no temp files are left.

Fix:

  • Catalog.SaveToFile now does VACUUM INTO (or the manual-copy fallback) into os.CreateTemp in the cache's directory, then os.Rename over the cache.
  • buildCatalog and refresh catalog communities no longer os.Remove the cache first.
  • A rename alone moves the failure to readers. SQLite refuses a write on a file renamed under an open connection (attempt to write a readonly database, 1032), and NewFromFile always wrote (createTables plus the schema-version row). So opening a cache already at the current schema version is now read-only.
  • File-backed connections get busy_timeout(10000) in the DSN, which covers the remaining writes when an old-version cache is upgraded.

Test plan

  • TestSaveToFile_ConcurrentWritersAndReaders (mdl/catalog): 8 goroutine writers over an existing cache, plus 4 reader loops. It asserts no write or read error, a final cache in the new mode, and no leftover files. Revert check: on the unfixed code it fails with catalog_meta already exists ×8 and readers' database is locked. With only the rename, it fails with attempt to write a readonly database (which is why the read-only open is needed).
  • TestCheck_DoesNotModifyProject (v1, v2, each with and without --no-update-widgets): stub tools do to their target what the real ones do (rewrite the .mpr, drop mprcontents, write theme-cache and deployment). The test asserts the tree is identical, mx never points into the project, the copy's path does not leak into output, the "normalised on a temporary copy" note appears, and the temp dir is cleaned up. Revert check: with mx pointed back at the project it fails with changed App.mpr, changed theme-cache/..., added deployment/... and removed mprcontents/....
  • TestCopyProjectForCheck_SkipsOutputs, _TempDirInsideProject (revert check: without the skip, the copy recursed until mkdir failed on path length), _MissingProject.
  • Integration TestCheck_LeavesProjectUntouched (-tags integration, real mx 11.x, mx create-project, v2 plus v1 converted via update-widgets, both flag settings): PASS. Revert check: v1 fails with "docker check modified the project". TestCheck_PreservesMPRv2StorageFormat still passes.
  • E2E with real mx 11.13 on v1 and v2 copies of a real project: tree hash and mtime identical before/after, default and --no-update-widgets. Control: a microflow with declare $n Integer = 1 + 'a' is still reported ([CE0117], exit 1) on both formats, with the tree unchanged.
  • go test ./mdl/catalog/ ./mdl/executor/ ./cmd/mxcli/docker/ ./cmd/mxcli/, make build, make lint, make check-conformance, make check-findings, make check-skill-mdl, make sync-skills.
  • Findings appended (mdl-other.jsonl, cmd-mxcli.jsonl). CHANGELOG entries are under Unreleased → Fixed.

Follow-ups (not in this PR)

  • docker build still rewrites an MPRv1 .mpr via update-widgets, and mx check there writes theme-cache. build writes deployment/ by design, but the model rewrite on v1 is the same defect. It could use the same copy for its pre-check, though MxBuild then needs the normalised model.
  • The TUI checker (cmd/mxcli/tui/checker.go) and evalrunner (cmd/mxcli/evalrunner/checks.go) run plain mx check on the project, which writes theme-cache/ and deployment/sass/. They are not docker subcommands, so they were left alone.
  • A file-backed cache that a process has loaded and then writes to (refresh catalog communities on a loaded cache via AddGraphAnalysis) can still hit readonly database if another process renames a fresh cache over it mid-run. This is rare, and it surfaces as an error rather than corruption.

🤖 Generated with Claude Code

ako and others added 3 commits October 3, 2026 14:39
…race (#951)

SaveToFile now writes to a temp file in the cache's directory (VACUUM INTO,
manual-copy fallback into the same temp file) and renames it over the cache.
buildCatalog and refresh catalog communities no longer remove the cache first.
Opening a cache at the current schema version no longer writes to it: a write
on a file renamed underneath an open connection fails with
SQLITE_READONLY_DBMOVED. File-backed connections get a busy_timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docker check ran mx update-widgets on the user's project with only the MPRv2
storage snapshotted, so an MPRv1 .mpr was rewritten permanently, and mx check
itself rewrote theme-cache/ and created deployment/sass/ on both formats, with
or without --no-update-widgets. Both mx steps now run on a temporary copy
(build output, caches and VCS folders skipped), mx output is rewritten to the
project's own paths, and the output says that widgets were normalised on a
copy and what that hides (#568, #646). docker build keeps its snapshot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nto itself (#951)

Check copies the project's directory, so a missing .mpr (or a path in /tmp)
would copy an unrelated directory: fail early instead, and give the existing
fake-mx tests a project file of their own. With TMPDIR inside the project the
walk met its own copy and recursed until the path was too long; skip it.
Trim the custom-widgets skill back under the 700-line limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docker check rewrites an MPRv1 project; catalog cache save races between parallel processes

1 participant