Skip to content

fix(deno): stop replacing cache databases that are still in use - #747

Merged
johnstonmatt merged 2 commits into
mainfrom
fix/cache-db-sigbus
Sep 25, 2026
Merged

johnstonmatt merged 2 commits into
mainfrom
fix/cache-db-sigbus

Conversation

@jgoux

@jgoux jgoux commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

Fixes #746.

What is wrong

The process can die with SIGBUS (exit 135) while user workers boot at the same time on a fresh DENO_DIR.

  1. EmitterFactory::caches() builds a new Caches on every call, and each one opens dep_analysis_cache_v2 and node_analysis_cache_v2. The main service and every user worker have their own factory. The result is several connections initializing the same two SQLite files at once, in the same process.
  2. Some of these initializations fail with SQLITE_BUSY, or with SQLITE_CANTOPEN while the parent directory is still being created. After two failures, open_connection deletes the database and creates it again.
  3. The new file has a new inode, so SQLite treats the existing -shm file as unused. POSIX locks held by the same process are invisible to F_GETLK, so SQLite truncates the file. Connections that still have the old wal-index mapped then raise SIGBUS on their next access past the first page.

denoland/deno#34873 stops the deletion on lock contention, but it only checks the first error, so a CANTOPEN followed by BUSY still deletes the file.

Changes

  • deno/cache/cache_db.rs: every failed attempt is classified, not just the first. BUSY and LOCKED are retried with bounded exponential backoff (about 1.3 s in total). The file is deleted only for SQLITE_CORRUPT and SQLITE_NOTADB. Other failures fall back to the configured on_failure mode as before. That includes a path that still can't be opened after its parent directory is created: it falls back immediately, without deleting anything. The helpers follow fix(cache): retry locked cache database instead of deleting it denoland/deno#34873 (retry_open_with_backoff, log_failure_mode, handle_failure_mode), and the init functions return rusqlite::Error as they do upstream, which should make future syncs easier.

  • crates/deno_facade/emitter.rs: caches() is memoized in a Deferred, like the other factory accessors, so each factory opens each cache once.

  • The vendored cache_db.rs gets a test module with regression tests:

    • a database locked by another connection is never replaced (checked by inode);
    • CANTOPEN followed by BUSY does not delete the file;
    • contention that never clears falls back without deleting;
    • a path that can't be opened falls back immediately without deleting;
    • a corrupt file is still recreated on disk;
    • concurrent initialization of a fresh cache ends with every connection on the same file.

    Without the fix, the four contention and open-error tests fail every time. The concurrent test is timing-dependent: it failed in 4 of 30 runs, for example with SQLITE_IOERR_SHORT_READ ("file truncated?").

Notes

  • In a reproduction that bursts requests at freshly started containers, each creating new user workers, exit 135 went from 31/150 runs to 0/150 with timing-amplified tracing, and from 3/100 to 0/100 without it. The patched runs never deleted a cache database and never fell back to in-memory caches.
  • rusqlite already sets a 5 s busy timeout when it opens a connection. The failures above happen in the cases where SQLite returns SQLITE_BUSY without calling the busy handler, so the fix retries with a new connection and leaves the busy timeout as it is.
  • Separate factories in one process still open separate connections to the same files. That is safe once nothing replaces a file that is in use. Replacing a corrupt file while another connection in the same process has it open can still cause the same truncation, but that path now needs real corruption. Sharing one Caches per DENO_DIR across the process would remove that case too, and could be done as a follow-up.

🤖 Generated with Claude Code

jgoux and others added 2 commits September 25, 2026 08:36
When a cache database fails to initialize twice, `open_connection` deletes
the file and recreates it. Contention between connections in the same
process (`SQLITE_BUSY`, including after a `SQLITE_CANTOPEN` while the parent
directory is being created) reaches that path. The recreated file has a new
inode, so SQLite treats the existing `-shm` file as unused and truncates it
while the other connections still have it mapped, and their next wal-index
access raises SIGBUS.

Retry lock contention (`SQLITE_BUSY`, `SQLITE_LOCKED`) with bounded
exponential backoff on every attempt, and delete the file only for
`SQLITE_CORRUPT` and `SQLITE_NOTADB`. Any other failure, including a path
that still cannot be opened once its parent directory exists, falls back to
the configured failure mode without deleting anything. The shape follows
denoland/deno#34873, which only classifies the first error.

Refs #746

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`caches()` built a new `Caches` on every call, so one factory opened
several connections to the same dependency and node analysis databases.
Keep it in a `Deferred` like the other factory accessors.

Refs #746

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@johnstonmatt
johnstonmatt merged commit 457b435 into main Sep 25, 2026
5 checks passed
@johnstonmatt
johnstonmatt deleted the fix/cache-db-sigbus branch September 25, 2026 19:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cold-cache bundle intermittently SIGBUS in SQLite after duplicate cache initialization

2 participants