fix(browser-db-sqlite-persistence): fairly schedule cold hydrations

.changeset/fix-shared-sqlite-hydration-fairness.md

@@ -0,0 +1,7 @@
+---
+'@tanstack/db-sqlite-persistence-core': patch
+'@tanstack/browser-db-sqlite-persistence': patch
+'@tanstack/electron-db-sqlite-persistence': patch
+---
+
+Schedule complete SQLite hydration units fairly without holding coordinator work inside the local hydration scope. Preserve per-Collection leader adapter routing, mutation results across transport retries, terminal coordinator disposal, real-adapter restart order, and promise-discovered shared scheduling.

.github/workflows/e2e-tests.yml

@@ -74,6 +74,11 @@ jobs:
           cd examples/react/start-ssr-e2e
           pnpm exec playwright install --with-deps chromium
 
+      - name: Run Browser SQLite OPFS fairness E2E tests
+        run: |
+          cd packages/browser-db-sqlite-persistence
+          pnpm test:opfs-fairness
+
       - name: Run React Start SSR E2E tests
         run: |
           cd examples/react/start-ssr-e2e

docs/contributing/oracle-coverage.md

@@ -110,7 +110,7 @@ comment and the current API/architecture contract before extending its model.
 | Opaque backend pagination | [window oracle](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/cursor-pagination.oracle.test.ts), [cache histories](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/cursor-pagination.cache-oracle.test.ts), [cache publication](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/cursor-pagination.publication-oracle.test.ts), [browser acquisition boundaries](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/cursor-pagination.boundary-oracle.test.ts), [QueryCollection integration](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/cursor-pagination.integration.test.ts) | Full filter/sort/slice reference, opaque token transport, actual Query cache expiry/invalidation/GC, forced refresh during growth, protocol failure publication/recovery, bounded slice work, nested cancellation/replacement, reader abort, browser retry defaults, manual-write cache isolation, and production window publications. Stable backend sequences; not snapshot guarantees for changing endpoints. Peek-ahead remains enabled. |
 | Electric and TrailBase | [Electric histories](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/tests/electric-oracle.property.test.ts), [PostgreSQL semantics](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/e2e/sql-predicate-semantics.e2e.test.ts), [TrailBase contract](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/tests/ORACLE.md) | Installed SDK delivery/framing, independent predicates, exact subscription arguments and late errors. SDK fixtures and a real service test earn different credit. |
 | PowerSync | [tests](https://github.com/TanStack/db/tree/main/packages/powersync-db-collection/tests), `tests/correctness-oracle.test.ts` | Applied receipt positions crossed with held peers, native SQLite/SDK and cleanup evidence. Run the focused owner with the package's `test:oracles` command. A timeout mutant proves a progress failure, not every value assertion. |
-| SQLite persistence and native hosts | [persisted histories](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/persisted.test.ts), [driver contracts](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/contracts/sqlite-driver-contract.ts), [browser OPFS lifecycle](https://github.com/TanStack/db/blob/main/packages/browser-db-sqlite-persistence/tests/opfs-page-lifecycle-oracle.test.ts), [worker diagnostics](https://github.com/TanStack/db/blob/main/packages/browser-db-sqlite-persistence/tests/opfs-worker-diagnostics-oracle.test.ts), [113-law manifest](https://github.com/TanStack/db/blob/main/packages/db-collection-e2e/src/fixtures/persisted-conformance-manifest.ts) | Cache/remote rejection/peer/reopen histories, exact driver results, controlled page/worker ownership, and diagnostic-cause retention. Fake workers and synthetic page events do not prove native handle release or real bfcache admission. The manifest excludes progressive and move suites; registration and shim runs are not device execution. |
+| SQLite persistence and native hosts | [persisted histories](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/persisted.test.ts), [driver contracts](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/contracts/sqlite-driver-contract.ts), [shared-driver fairness](https://github.com/TanStack/db/blob/main/packages/browser-db-sqlite-persistence/tests/shared-driver-fairness-oracle.test.ts), [browser OPFS lifecycle](https://github.com/TanStack/db/blob/main/packages/browser-db-sqlite-persistence/tests/opfs-page-lifecycle-oracle.test.ts), [worker diagnostics](https://github.com/TanStack/db/blob/main/packages/browser-db-sqlite-persistence/tests/opfs-worker-diagnostics-oracle.test.ts), [113-law manifest](https://github.com/TanStack/db/blob/main/packages/db-collection-e2e/src/fixtures/persisted-conformance-manifest.ts) | Cache/remote rejection/peer/reopen histories, exact driver results, K=1 complete-logical-hydrate fairness over one shared Browser SQLite driver, controlled page/worker ownership, and diagnostic-cause retention. The fairness owner runs one identical generated property in retained fixed-seed and seedless-random lanes, or only a requested checked seed+shrink-path replay. Its grammar controls reconstruct the known witness, exercise bounded marginals, reject an invalid storm boundary, and ablate tail order; a named persist-first FIFO wrong answer calibrates the checker. It does not establish elapsed-time latency, unbounded eventuality, multi-process coordination, or a browser matrix; Chromium OPFS separately refines the provider boundary. Fake workers and synthetic page events do not prove native handle release or real bfcache admission. The manifest excludes progressive and move suites; registration and shim runs are not device execution. |
 | Offline execution | [scheduler](https://github.com/TanStack/db/blob/main/packages/offline-transactions/tests/KeyScheduler.property.test.ts), [leadership](https://github.com/TanStack/db/blob/main/packages/offline-transactions/tests/leadership-replay.property.test.ts), [settlement](https://github.com/TanStack/db/blob/main/packages/offline-transactions/tests/transaction-settlement.property.test.ts), [serialization](https://github.com/TanStack/db/blob/main/packages/offline-transactions/tests/transaction-serializer.property.test.ts) | Declarative FIFO eligibility, per-transaction outcomes, durable state and typed wire trees. Issued work may finish after ownership loss, but new work must not start. Exactly-once network execution is not promised. |
 | Frameworks | [React conformance](https://github.com/TanStack/db/blob/main/packages/react-db/tests/conformance.test.tsx), [React pagination](https://github.com/TanStack/db/blob/main/packages/react-db/tests/infinite-query-conformance.test.tsx), [shared suites](https://github.com/TanStack/db/tree/main/packages/db-collection-e2e/src/suites) | Exact exposed rows/pages and each framework's own lifecycle cuts. A React witness does not prove Vue/Solid/Angular/Svelte scheduling. Preserve their receiving registrations. |
 | Structural values and ordered primitives | [hash values](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash.property.test.ts), [hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-graph.property.test.ts), [mixed hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-mixed-graph.property.test.ts), [hash retry](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-failure-retry.property.test.ts), [comparison](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison.property.test.ts), [deep equality](https://github.com/TanStack/db/blob/main/packages/db/tests/utils.property.test.ts), [cursor](https://github.com/TanStack/db/blob/main/packages/db/tests/cursor.property.test.ts), [indexes](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update.property.test.ts), [query identity](https://github.com/TanStack/db/blob/main/packages/db/tests/query/identity-output-shape-oracle.test.ts) | Independent flat values, graph topology, algebraic laws, Map/group/sort recomputation, expression denotation, and compiled output bags. Hash collision freedom is not promised. Unsupported composite cursors reject. |

packages/browser-db-sqlite-persistence/e2e/shared-driver-fairness.opfs.html

@@ -0,0 +1,13 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <link rel="icon" href="data:," />
+    <title>Shared driver OPFS fairness oracle</title>
+  </head>
+  <body>
+    <output id="oracle-status">running</output>
+    <script type="module" src="./shared-driver-fairness.opfs.ts"></script>
+  </body>
+</html>

packages/browser-db-sqlite-persistence/e2e/shared-driver-fairness.opfs.spec.ts

@@ -0,0 +1,102 @@
+/**
+ * Browser checkpoint assertions for the OPFS refinement. Expected public rows
+ * are rebuilt from the scenario IDs rather than from production output. The
+ * neutral case proves cold-query reach; the storm case requires no K=1
+ * violation. Failure-before-checkpoint, semantic mismatch, driver cleanup, and
+ * OPFS cleanup remain distinct outcomes so setup or teardown cannot satisfy the
+ * scheduling law.
+ */
+import { expect, test } from '@playwright/test'
+import type { Page } from '@playwright/test'
+import type { OPFSOracleResult } from './shared-driver-fairness.opfs'
+
+async function readOracleResult(
+  page: Page,
+  mode: `neutral` | `storm`,
+): Promise<OPFSOracleResult> {
+  let rejectPageError!: (error: Error) => void
+  const pageError = new Promise<never>((_resolve, reject) => {
+    rejectPageError = reject
+  })
+  void pageError.catch(() => undefined)
+  const onPageError = (error: Error) => {
+    rejectPageError(
+      new Error(
+        `OPFS fairness page failed before publishing a result: ${error.message}`,
+      ),
+    )
+  }
+  page.on(`pageerror`, onPageError)
+
+  try {
+    await page.goto(`/e2e/shared-driver-fairness.opfs.html?mode=${mode}`)
+    await Promise.race([
+      page.waitForFunction(
+        () => window.__tanstackDriverFairnessOracle !== undefined,
+      ),
+      pageError,
+    ])
+    return page.evaluate(() => window.__tanstackDriverFairnessOracle!)
+  } finally {
+    page.off(`pageerror`, onPageError)
+  }
+}
+
+function expectedHydratedCollections(scenarioId: string, count: number) {
+  return Array.from({ length: count }, (_, index) => ({
+    collectionId: `${scenarioId}-hydrate-${index}`,
+    rows: [
+      { id: `row-${index}-0`, value: index * 10 },
+      { id: `row-${index}-1`, value: index * 10 + 1 },
+    ],
+  }))
+}
+
+test(`real Chromium OPFS fixture reaches and cleans up cold hydration`, async ({
+  page,
+}) => {
+  const result = await readOracleResult(page, `neutral`)
+
+  if (result.status !== `complete`) throw new Error(result.primaryFailure)
+  expect(result.provider).toBe(`Chromium OPFSCoopSyncVFS worker`)
+  expect(result.observation.admittedHydrateIds).toHaveLength(2)
+  // These are actual public Collection rows captured after preload, compared
+  // with seed values built independently by this browser assertion.
+  expect(result.observation.hydratedCollections).toEqual(
+    expectedHydratedCollections(`opfs-neutral-reach`, 2),
+  )
+  expect(
+    result.observation.rawDequeues.some((entry) =>
+      entry.sql.startsWith(`SELECT key, value, metadata, row_version FROM`),
+    ),
+  ).toBe(true)
+  expect(result.observation.cleanupFailures).toEqual([])
+  expect(result.opfsCleanupFailures).toEqual([])
+})
+
+test(`real Chromium OPFS fixture bounds pending cold hydration behind persists`, async ({
+  page,
+}) => {
+  const result = await readOracleResult(page, `storm`)
+
+  if (result.status !== `complete`) throw new Error(result.primaryFailure)
+  expect(result.provider).toBe(`Chromium OPFSCoopSyncVFS worker`)
+  expect(result.observation.admittedHydrateIds).toHaveLength(4)
+  expect(result.observation.hydratedCollections).toEqual(
+    expectedHydratedCollections(`opfs-fixed-persist-storm`, 4),
+  )
+  // This is the semantic RED checkpoint. Setup, wall time, and cleanup are
+  // reported independently and cannot satisfy this assertion.
+  if (result.violation !== undefined) {
+    throw new Error(
+      `real OPFS fairness mismatch: ${JSON.stringify(result.violation)}; ` +
+        `logical completion order: ${JSON.stringify(result.observation.logicalCompletionOrder)}; ` +
+        `driver admissions: ${result.observation.driverAdmissions.length}; ` +
+        `raw dequeues: ${result.observation.rawDequeues.length}; ` +
+        `driver cleanup diagnostics: ${JSON.stringify(result.observation.cleanupFailures)}; ` +
+        `OPFS cleanup diagnostics: ${JSON.stringify(result.opfsCleanupFailures)}`,
+    )
+  }
+  expect(result.observation.cleanupFailures).toEqual([])
+  expect(result.opfsCleanupFailures).toEqual([])
+})

packages/browser-db-sqlite-persistence/e2e/shared-driver-fairness.opfs.ts

@@ -0,0 +1,160 @@
+/**
+ * Real-provider refinement of the shared-driver fairness oracle. The page runs
+ * the same legal neutral and fixed-storm histories through public `preload()`,
+ * the browser/core adapter boundary, and Chromium's OPFSCoopSyncVFS worker. It
+ * freezes logical completions, public rows, raw dequeue reach, and the K=1
+ * violation result before cleanup, then reports provider cleanup separately.
+ * This fixture adds real OPFS/worker evidence; it does not claim multi-tab,
+ * multi-process, non-Chromium, latency, or unbounded-eventuality coverage.
+ */
+import { openBrowserWASQLiteOPFSDatabase } from '../src/index'
+import {
+  findSharedDriverFairnessViolation,
+  observeSharedDriverFairness,
+} from '../tests/shared-driver-fairness-oracle'
+import type {
+  SharedDriverFairnessObservation,
+  SharedDriverFairnessScenario,
+  SharedDriverFairnessViolation,
+} from '../tests/shared-driver-fairness-oracle'
+
+export type OPFSOracleResult =
+  | {
+      status: `complete`
+      provider: `Chromium OPFSCoopSyncVFS worker`
+      observation: SharedDriverFairnessObservation
+      violation: SharedDriverFairnessViolation | undefined
+      opfsCleanupFailures: ReadonlyArray<string>
+    }
+  | {
+      status: `failed-before-checkpoint`
+      provider: `Chromium OPFSCoopSyncVFS worker`
+      primaryFailure: string
+      opfsCleanupFailures: ReadonlyArray<string>
+    }
+
+declare global {
+  interface Window {
+    __tanstackDriverFairnessOracle?: OPFSOracleResult
+  }
+}
+
+async function removeOPFSArtifacts(
+  databaseName: string,
+): Promise<ReadonlyArray<string>> {
+  const failures: Array<string> = []
+  const root = await navigator.storage.getDirectory()
+  for (const suffix of [``, `-journal`, `-wal`]) {
+    try {
+      await root.removeEntry(`${databaseName}${suffix}`)
+    } catch (error) {
+      if (!(error instanceof DOMException && error.name === `NotFoundError`)) {
+        failures.push(
+          `${databaseName}${suffix}: ${error instanceof Error ? error.message : String(error)}`,
+        )
+      }
+    }
+  }
+
+  // OPFSCoopSyncVFS creates one private temporary access-handle directory per
+  // worker. This page owns its isolated origin for the fixture run.
+  const iterableRoot = root as FileSystemDirectoryHandle & {
+    entries: () => AsyncIterableIterator<[string, FileSystemHandle]>
+  }
+  for await (const [name, handle] of iterableRoot.entries()) {
+    if (handle.kind !== `directory` || !name.startsWith(`.ahp-`)) continue
+    try {
+      await root.removeEntry(name, { recursive: true })
+    } catch (error) {
+      failures.push(
+        `${name}: ${error instanceof Error ? error.message : String(error)}`,
+      )
+    }
+  }
+  return failures
+}
+
+function scenarioFromLocation(): SharedDriverFairnessScenario {
+  const mode = new URL(location.href).searchParams.get(`mode`)
+  if (mode === `neutral`) {
+    return {
+      id: `opfs-neutral-reach`,
+      work: [0, 1].map((index) => ({
+        kind: `hydrate` as const,
+        id: `hydrate-${index}`,
+        seededRows: [
+          { id: `row-${index}-0`, value: index * 10 },
+          { id: `row-${index}-1`, value: index * 10 + 1 },
+        ],
+      })),
+    }
+  }
+  return {
+    id: `opfs-fixed-persist-storm`,
+    work: [
+      ...Array.from({ length: 5 }, (_, index) => ({
+        kind: `persist` as const,
+        id: `persist-${index}`,
+        mutationsPerPersist: 2,
+      })),
+      ...Array.from({ length: 4 }, (_, index) => ({
+        kind: `hydrate` as const,
+        id: `hydrate-${index}`,
+        seededRows: [
+          { id: `row-${index}-0`, value: index * 10 },
+          { id: `row-${index}-1`, value: index * 10 + 1 },
+        ],
+      })),
+    ],
+  }
+}
+
+async function run(): Promise<void> {
+  const status = document.querySelector<HTMLOutputElement>(`#oracle-status`)
+  const databaseName = `ws5b-${crypto.randomUUID()}.sqlite`
+  let observation: SharedDriverFairnessObservation | undefined
+  let violation: SharedDriverFairnessViolation | undefined
+  let primaryFailure: string | undefined
+  let opfsCleanupFailures: ReadonlyArray<string> = []
+  try {
+    observation = await observeSharedDriverFairness(
+      () => openBrowserWASQLiteOPFSDatabase({ databaseName }),
+      scenarioFromLocation(),
+    )
+    // Freeze the reached semantic checkpoint before attempting OPFS cleanup.
+    violation = findSharedDriverFairnessViolation(observation)
+  } catch (error) {
+    primaryFailure = error instanceof Error ? error.message : String(error)
+  }
+
+  try {
+    opfsCleanupFailures = await removeOPFSArtifacts(databaseName)
+  } catch (cleanupError) {
+    opfsCleanupFailures = [
+      cleanupError instanceof Error
+        ? cleanupError.message
+        : String(cleanupError),
+    ]
+  }
+
+  if (observation) {
+    window.__tanstackDriverFairnessOracle = {
+      status: `complete`,
+      provider: `Chromium OPFSCoopSyncVFS worker`,
+      observation,
+      violation,
+      opfsCleanupFailures,
+    }
+    if (status) status.value = `complete`
+  } else {
+    window.__tanstackDriverFairnessOracle = {
+      status: `failed-before-checkpoint`,
+      provider: `Chromium OPFSCoopSyncVFS worker`,
+      primaryFailure: primaryFailure ?? `unknown failure before checkpoint`,
+      opfsCleanupFailures,
+    }
+    if (status) status.value = `failed-before-checkpoint`
+  }
+}
+
+void run()

packages/browser-db-sqlite-persistence/package.json

@@ -23,6 +23,8 @@
     "dev": "vite build --watch",
     "lint": "eslint . --fix",
     "test": "vitest --run",
+    "test:oracles": "vitest --run tests/shared-driver-fairness-oracle.test.ts",
+    "test:opfs-fairness": "playwright test --config playwright.opfs.config.ts",
     "test:e2e": "pnpm --filter @tanstack/db-ivm build && pnpm --filter @tanstack/db build && pnpm --filter @tanstack/db-sqlite-persistence-core build && pnpm --filter @tanstack/browser-db-sqlite-persistence build && vitest --config vitest.e2e.config.ts --run"
   },
   "type": "module",
@@ -56,6 +58,7 @@
   },
   "devDependencies": {
     "@journeyapps/wa-sqlite": "^1.4.1",
+    "@playwright/test": "^1.60.0",
     "@types/better-sqlite3": "^7.6.13",
     "@vitest/coverage-istanbul": "^3.2.4",
     "better-sqlite3": "^12.6.2"

packages/browser-db-sqlite-persistence/playwright.opfs.config.ts

@@ -0,0 +1,25 @@
+import { defineConfig } from '@playwright/test'
+
+const baseURL = `http://127.0.0.1:4185`
+const browserChannel =
+  process.env.PLAYWRIGHT_CHANNEL ?? (process.env.CI ? undefined : `chrome`)
+
+export default defineConfig({
+  testDir: `./e2e`,
+  testMatch: `shared-driver-fairness.opfs.spec.ts`,
+  timeout: 60_000,
+  fullyParallel: false,
+  workers: 1,
+  use: {
+    baseURL,
+    ...(browserChannel ? { channel: browserChannel } : {}),
+    headless: true,
+    trace: `retain-on-failure`,
+  },
+  webServer: {
+    command: `vite --config vite.opfs.config.ts --host 127.0.0.1 --port 4185`,
+    reuseExistingServer: false,
+    timeout: 120_000,
+    url: `${baseURL}/e2e/shared-driver-fairness.opfs.html`,
+  },
+})