Cold-cache bundle intermittently SIGBUS in SQLite after duplicate cache initialization

[kvz]

Summary

edge-runtime bundle intermittently terminates with SIGBUS (Docker exit 135) on
a fresh cache while bundling a local CommonJS import. No Supabase project,
credentials, network imports, or third-party npm package is required.

The reduced control below failed 6/90 times across three GitHub-hosted x64
runners. A separate SDK consumer produced three native crash dumps, all pointing
to SQLite WAL shared-memory access during Deno cache initialization.

Environment

• Image: public.ecr.aws/supabase/edge-runtime:v1.76.2@sha256:edd22bef4477b900d5c300e287ce9b18bff9b81a0291bee14ee0b7c7b71a2899
• Embedded Deno: 2.1.4.
• GitHub-hosted Ubuntu 24.04 x64; kernel 6.17.0-1022-azure.
• Observed on AMD EPYC 9V74 and 7763 hosts across the diagnostic runs.
• Docker records OOMKilled: false; ample host RAM and disk were available.
• Enlarging /dev/shm did not prevent the crash. A local ARM control did not
  reproduce it; that is not proof the bug is x64-only.

Reduced reproduction

Save as repro.ts, then run node repro.ts with Node 24 and local Docker.
It uses fresh containers and deletes only its own temporary directory/containers.
Repeat on a native Linux x64 host; the failure is timing-sensitive.

import { execFileSync, spawnSync } from 'node:child_process'
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'

const image = 'public.ecr.aws/supabase/edge-runtime:v1.76.2@sha256:edd22bef4477b900d5c300e287ce9b18bff9b81a0291bee14ee0b7c7b71a2899'
const root = mkdtempSync(join(tmpdir(), 'edge-cache-repro-'))
let failures = 0

try {
  mkdirSync(join(root, 'node_modules/control'), { recursive: true })
  writeFileSync(join(root, 'package.json'), '{"private":true,"type":"module"}\n')
  writeFileSync(join(root, 'deno.json'), '{"nodeModulesDir":"manual","lock":false}\n')
  writeFileSync(join(root, 'node_modules/control/package.json'), '{"name":"control","main":"index.cjs"}\n')
  writeFileSync(join(root, 'node_modules/control/index.cjs'), "exports.value = 'control'\n")
  writeFileSync(join(root, 'index.ts'), "import { value } from './node_modules/control/index.cjs'\nDeno.serve(() => new Response(value))\n")

  execFileSync('docker', ['pull', image], { stdio: 'inherit' })
  for (let attempt = 1; attempt <= 30; attempt++) {
    const id = execFileSync('docker', [
      'create', '--workdir', '/work',
      '--env', 'RUST_LOG=deno::cache::cache_db=trace',
      image, 'bundle', '--entrypoint', '/work/index.ts',
      '--output', '/work/function.eszip', '--timeout', '120',
    ], { encoding: 'utf8' }).trim()
    try {
      execFileSync('docker', ['cp', `${root}/.`, `${id}:/work`])
      const result = spawnSync('docker', ['start', '--attach', id], {
        encoding: 'utf8', timeout: 150_000,
      })
      console.log({ attempt, exitCode: result.status })
      if (result.status !== 0) {
        failures++
        console.log(result.stdout, result.stderr)
        console.log(execFileSync('docker', ['inspect', '--format', '{{json .State}}', id], { encoding: 'utf8' }))
      }
    } finally {
      execFileSync('docker', ['rm', '--force', id])
    }
  }
} finally {
  rmSync(root, { recursive: true, force: true })
}
console.log({ failures })
process.exitCode = failures === 0 ? 0 : 1

Trace and native evidence

All six reduced-control crashes log duplicate openings of both analysis caches,
then an initialization retry and a database is locked error before deletion:

DEBUG Opening cache /root/.cache/deno/dep_analysis_cache_v2...
DEBUG Opening cache /root/.cache/deno/node_analysis_cache_v2...
DEBUG Opening cache /root/.cache/deno/dep_analysis_cache_v2...
DEBUG Opening cache /root/.cache/deno/node_analysis_cache_v2...
Could not initialize cache database '.../node_analysis_cache_v2', retrying...
  Error code 14: Unable to open the database file
Could not initialize cache database '.../node_analysis_cache_v2', deleting and retrying...
  Error code 5: The database file is locked

Representative native stack from an uninstrumented SDK bundle crash:

Program terminated with signal SIGBUS, Bus error.
#0  walIndexAppend at sqlite3/sqlite3.c:66237
#1  walFrames
#2  sqlite3WalFrames
#3  pagerWalFrames
...
#14 rusqlite::raw_statement::RawStatement::step
#16 rusqlite::Connection::execute_batch
#17 deno::cache::cache_db::CacheDB::initialize_connection at deno/cache/cache_db.rs:237
#18 CacheDB::open_connection_and_init at deno/cache/cache_db.rs:270
#20 deno::cache::cache_db::open_connection at deno/cache/cache_db.rs:419
...
#31 CacheDB::spawn_eager_init_thread

The signal is SIGBUS / BUS_ADRERR; its address falls inside the mapped
dep_analysis_cache_v2-shm or node_analysis_cache_v2-shm file. Line 419 is the
retry after removing the database. A syscall trace from another run shows one
thread unlinking the DB while another still uses it, followed by both opening
and truncating the same -shm pathname. That traced run itself completed; the
three crashes were captured as bounded core dumps and analyzed offline.

Suspected mechanism

EmitterFactory::caches()
creates a fresh Caches on every call and eagerly initializes both databases.
module_graph_builder()
calls it and then calls module_info_cache(), which calls it again. Their mutexes
are independent but their file paths are identical.

open_connection
retries after creating the parent directory, then deletes the database after
another initialization error, including SQLITE_BUSY. Replacing an open SQLite
database lets different DB inodes share WAL/shm paths, an unsafe pattern also
documented in SQLite's guidance.

Sharing the factory's cache instance and not treating SQLITE_BUSY as a reason to
unlink an in-use database appear to be the appropriate fixes. These fixes have
not been implemented or tested here.

Counterfactuals, not a claimed upstream fix

Across three x64 runners, the same SDK graph/image passed 90/90 times with
--tmpfs /root/.cache/deno:size=268435456. This also pre-creates the directory, so
it does not distinguish filesystem behavior from initialization timing. Twelve
runs still logged busy errors; none logged deletion. Merely preheating a disk
cache also gave 90/90 passes, but 39 runs still deleted/recreated a database, so
that does not appear to be a safe solution.

The reduced test contains no application secrets and all failures were retained;
these were not retries used to turn a failing CI job green.

[jgoux]

We hit this in the Supabase CLI's CI and traced it to the root cause. It isn't limited to bundle: start crashes the same way when user workers cold-boot. It also reproduces on native linux/arm64.

Mechanism (v1.76.2, unchanged in v1.77.0)

1. EmitterFactory::caches() is not memoized, unlike its sibling accessors, so one factory builds more than one Caches: directly and through module_info_cache(). Each caches() call primes both databases, and every CacheDB::from_path starts initializing eagerly on a blocking thread. The main service and every user worker each get their own EmitterFactory, so concurrent boots add more connections to dep_analysis_cache_v2 and node_analysis_cache_v2 in the same process.
2. Initialization switches to WAL and writes, and concurrent initializations fail immediately. On a fresh cache the first failure is SQLITE_CANTOPEN (the parent directory doesn't exist yet), then SQLITE_BUSY. rusqlite's default 5 s busy timeout doesn't help: SQLite returns these SQLITE_BUSY errors without calling the busy handler, e.g. SQLITE_BUSY_SNAPSHOT, or a lock upgrade it won't wait on because waiting could deadlock.
3. After the second failure, open_connection removes the main database file and reopens it. The -shm file stays in place.
4. The recreated file has a new inode, so SQLite's unix VFS creates a new shm node for the same -shm path. F_GETLK never reports locks held by the calling process, so unixLockSharedMemory() concludes it is the first connection and truncates -shm to 3 bytes.
5. The earlier connections in the process still have the 32 KiB wal-index mapped. Their next hash-table access past the first page raises SIGBUS (walIndexAppend in the backtrace above).

This needs two connections in the same process. Across processes, F_GETLK sees the other lock and the truncation doesn't happen, which is why the non-memoized caches() matters so much here.

Evidence

• Reproduction setup: native linux/arm64 VM, docker.io/supabase/edge-runtime:v1.76.2 running start with a main service that creates a user worker per request, and bursts of concurrent first requests on fresh containers.

  • 2–4% of containers died with exit 135.
  • With a narrow in-container strace --seccomp-bpf -e trace=unlinkat,ftruncate, which shifts timing, 20–26% did.

• Every crash had the same sequence, and no crash happened without the unlink:

  unlinkat(AT_FDCWD</work>, "/.cache/deno/dep_analysis_cache_v2", 0) = 0
  ftruncate(133</.cache/deno/dep_analysis_cache_v2-shm>, 3 <unfinished ...>
  --- SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRERR, si_addr=0xffff7411d2fe} ---
  

  SIGBUS arrived 0.1–8 ms after the truncation. In this run, si_addr is at offset 0x42fe of the 32 KiB dep_analysis_cache_v2-shm mapping. In all 39 cores we captured, the fault address is inside a *_cache_v2-shm mapping at offset ≥ 0x1000; both caches appear.

• Pure-SQLite control: a small C program (SQLite 3.40.1) with one thread writing while another unlinks and reopens the same database in the same process crashed with SIGBUS in 18 of 20 runs, faulting inside db-shm. The same program without the unlink crashed in 0 of 20.

• Workaround check: creating directories at the two database paths before the runtime starts forces the CacheFailure::InMemory fallback and removed the crash: 0 of 250 runs, against 32 of 250 without it.

• CI occurrences:

  • https://github.com/supabase/cli/actions/runs/36035204276/job/107753440643: the first /functions/v1/hello request after the stack started a fresh container.
  • https://github.com/supabase/cli/actions/runs/36054774240/job/107818874167: three concurrent first requests against public.ecr.aws/supabase/edge-runtime:v1.76.2.

Fix direction

• In open_connection, don't delete the database on contention errors (SQLITE_BUSY, SQLITE_LOCKED, or SQLITE_CANTOPEN from the parent-directory race). Retry those, and delete only on corruption. fix(cache): retry locked cache database instead of deleting it denoland/deno#34873 handles part of this, but it only inspects the first error, so the CANTOPEN → BUSY sequence above still deletes.
• Memoize EmitterFactory::caches().

Fix with regression tests: #747.