anvil

A multi-tenant job service whose results can be trusted after crashes and audited after the fact.

Problem and users

Teams that run work on behalf of many tenants hit three problems:

• Crashes cause lost or doubled work. A worker crash, a worker that resumes after its lease expired (a "zombie"), or a relay crash between publishing an event and recording it can lose a job or apply it twice.
• Tenants leak into each other. Shared connections, cached query plans or a guessable ID can show one tenant another tenant's data, or reveal that it exists.
• Results can't be checked. Nobody can prove afterwards what a job did, or detect that its record was altered.

ANVIL is for:

• Tenant clients that submit documents and jobs over HTTP (X-Tenant-Id, Idempotency-Key), read their documents, and read or replay their jobs' audit chains.
• Operators who run the service, watch its health and saturation signals, inspect a job's state, chain and replay result in a browser, and replay or verify a job's chain from the command line.

Guarantees

Each guarantee links to the tests that prove it. Tests that touch the database run against real PostgreSQL in containers, restored from a backup before the suite starts; the console has unit/component tests plus a browser acceptance test against the same stack.

1. No lost work, no double effect. Execution is at-least-once with fenced, idempotent effects; it is never exactly-once.
   • A duplicate submit returns the same IDs and creates one job: SubmitIdempotencyTest.
   • Two workers never claim the same job. A stale or zombie worker affects 0 rows. Terminal jobs are never re-run: JobServiceTest, DurableV1SchemaTest.
   • A relay crash between publish and mark-sent re-publishes the same event, which applies once: OutboxRelayTest.
   • Content survives a crash during write or garbage collection, and a torn file is never served: ContentStoreTest.
2. No tenant can read or change another tenant's data, or learn that it exists.
   • The database enforces isolation even with pooled connections and cached plans: RlsLeakTest.
   • A 28-case matrix over direct IDs, search, content addresses and presigned URLs shows 0 leaks: ReadIsolationTest. The same holds for audit reads: AuditReadIsolationTest.
   • Revoking a capability mid-run blocks the commit (0 rows, 403 NEEDS_APPROVAL): CapRevokeMidRunTest.
   • Jailed execution runs in a kernel sandbox. Untrusted native code in it can't read or write another tenant's workspace or the shared /tmp, and has no network route. This holds on the host and inside a container: ExecJailTest, JailContainerEscapeTest. Why: record 0019.
   • Brokered egress is HTTPS-only with verified certificates, denies an 80-plus-case SSRF matrix, re-checks DNS at connect time, never follows redirects, and takes the same time (within 1 ms) for an existing and a missing target: ExecJailTest.
3. Any job can be replayed bit-for-bit from its audit chain, and tampering is detected.
   • Every effect is recorded in an append-only, hash-linked chain in the same transaction: AuditChainTest.
   • GET /audit/replay returns {input_sha, output_sha, match: true} for an intact chain. A corrupted, dropped or re-linked event gives a mismatch. Checkpoint replay gives the same result: ReplayTest.
   • The operator console shows a job's current state (GET /api/jobs/{id}), its audit chain and its replay result. It uses the API's tenant scoping for every request, and a missing or another tenant's job takes the same generic not-found path: OperatorConsoleTest.
   • A long chain replays from its latest checkpoint within a bound. Replay reads the checkpoint and the events after it, not the prefix, and a declared 100,000-event chain replays at p99 8 ms against a 500 ms bound on containerized Postgres. A prefix rewrite that also rehashes through the boundary is caught there; one that leaves the boundary event unchanged, like a privileged partial edit, is caught by replayFull or the offline verifier. Why: record 0027; measurement.
   • Without the running API. Why: record 0026.
     • anvil replay reads the database directly and prints the API's result for the same job, byte for byte.
     • anvil verify checks an exported chain with no database connection. It gives the API's result for an intact export, and on a tampered one it reports the first bad sequence number. Editing the export's job state or input root can't turn the verdict, because both are checked against the chain. A file can't prove which tenant it belongs to, or that nothing after the terminal commit was cut, and a chain rewritten whole still verifies. Compare head with a value kept elsewhere (record 0026).
     • Both scope by tenant as the API does: another tenant's job is not found, like a missing one. The CLI reads as the runtime role under row-level security, read-only.
     • Proven by ReplayCliTest.
4. Fencing holds under real faults.
   • A deterministic simulator catches each seeded fencing break in under 500 runs: FenceSimTest.
   • The fencing model keeps a single holder and no double effect: FenceModelTest. It mirrors docs/faults/fencing.tla.
   • TLC model-checks docs/faults/fencing.tla on every gate run, with TLC pinned to 1.7.4 and a SHA-256 check. The job is tla in gate.yml and runs task check:tla. Its runs are on the gate workflow page.
   • Process suspend, kill and restart, and duplicate delivery each produce a single effect: FaultProofTest.
   • The real-Postgres fault matrix has nine cases: a crash before and after the effect, a lost ack, duplicate and out-of-order delivery, a stuck lease, a GIN pending-list flush, autovacuum and connection-pool exhaustion. Each case shows a single effect and no lost work: FaultMatrixTest.
   • The faults workflow runs the simulator 100 times nightly and the real-Postgres fault suite 5 times weekly, each from a fresh seed. A failed run opens an issue with its seed and trace: FaultScheduleTest, FenceSimTest. Its runs are on the faults workflow page.
   • A simulated point-in-time restore that rewinds the epoch lets a stale owner affect 0 rows with no double effect: PitrTimelineCheckTest. Why: record 0013.
   • When the database timeline is unknown or rewound, claims, heartbeats and commits are refused (503 TIMELINE_UNCONFIRMED) until leases expire and capabilities are re-issued: PitrTimelineCheckTest. Why: record 0016.
   • A backup → clean → point-in-time restore to a declared target reports its measured recovery time and data loss. Afterwards, tokens and grants from either side of the target affect 0 rows, a re-published event applies once, and isolation and capability checks hold: PitrRestoreDrillTest. The whole suite runs against a restored database.
5. One tenant's flood can't starve the others.
   • A claim never takes a tenant past its quota of running jobs, even with 64 claims at once. Jobs are claimed in the order of a penalty (the tenant's running count), then next_run_at on the database clock, then arrival: FairClaimTest. Why: record 0020.
   • While one tenant floods 8 workers, a quiet tenant's jobs are each claimed within 16 flood claims of their submit. This holds whether the flood's quota is below the pool or above it: FairClaimTest.
6. Extracted facts cite their source, stay in their tenant, and poison never reaches the index. Why: record 0022.
   • There is one extraction path, and it cuts a document to 500 characters in code before any model call: ExtractorTest.
   • A fact is indexed only when its citation is an exact substring of the text the model saw and holds its value. Anything else is quarantined: ExtractorTest, ExtractHarnessTest.
   • Forged, malformed and other tenants' document IDs get the same 404 before any model call, and identical content in two tenants is extracted separately, with no shared cache or dedup: ExtractionIsolationTest.
   • Poison input and rejected facts go to a dead-letter table with a reason, and no route serves it: ExtractionIsolationTest.
   • Model calls are cached per tenant and capped, and can be stopped. Why: record 0023.
     • A repeat of the same text, prompt and model in the same tenant makes no model call. Another tenant never hits that cache. Quarantined, failing or revoked entries are never served: ExtractionCostTest, ExtractionFallbackTest.
     • Each call takes one from the tenant's budget, even across processes. With no budget left or the kill-switch off, the frozen rules answer with zero model calls, and the result is marked as fallback: ExtractionCostTest.
     • Cost and latency are published in the cost report: about $0.0001 to $0.0003 per document, with model-call p95 under 1.5 s.
   • Quality and injection resistance are measured at full scale, every night. Why: record 0024.
     • The full set has 660 documents with 3,367 gold facts and 72 injection cases in thirteen languages, 20 of them base64-encoded. It is generated from a seed and frozen, and the gate scores it with 10,000-resample intervals and McNemar's test against the baseline: FullSetTest, EvalStatsTest.
     • The live model scores F1 0.996 [0.995, 0.998] against the baseline's 0.632, and 0 of 72 attacks succeed, none in any language or in base64: report.
     • The eval-nightly workflow runs the full set against the live model every night. It uses DeepSeek V4.1 Flash through OpenCode Go. It fails, and files one issue, when the model falls below the baseline or drifts below the committed run: EvalNightlyTest, EvalGateTest. Its runs are on the eval-nightly workflow page.
   • Load and the full-text index are measured before they're claimed. Why: record 0025.
     • A GIN index on fact values was measured over one soak day against a budget set beforehand, on a fresh Postgres server per variant. Neither variant met it, so the schema has none, and no soak numbers are claimed. Dropping such an index is its kill-switch, and extraction works without it: GinIndexTest, GIN report.
     • One process tops out near 170 extractions a second, because each model call holds a request thread. So 140 requests per second is at its edge: p95 2.2 s to 9.2 s, and 0.7% to 19% not started, depending on how many clients wait. A 10x flood is queued, not refused, and recovers at once. The tables grow about 4 MiB per 1,000 documents, and autovacuum keeps them under 8% wasted. No soak numbers are claimed, because the index is over budget: load summary, LoadRigTest.
7. Saturation is visible. Queue age, claim latency and pool wait are exported without tenant data. The burn alerts fire on induced burn and stay silent when healthy: SaturationSignalTest.

Not yet guaranteed:

• Jobs don't run in the service. The running app serves submit, reads, the audit and replay reads, extraction, the console, the saturation signals and the health probes. The claim, commit, capability, relay, reconcile, content-store and jail code exists and is tested through its Java APIs, but the app starts no worker loop and has no route to it. So the claims about those paths, in guarantees 1, 2, 4 and 5, hold for that code and not for a job submitted to a running app. There, a job stays QUEUED, its replay answers match: false, claim latency stays 0, and QueueBurn fires once a job has waited 60 s. Tracked in #144.
• X-Tenant-Id selects a tenant; it doesn't authenticate one. Anyone who can reach the API and knows a tenant's ID acts as that tenant, so guarantee 2 holds between callers who are honest about their tenant. Put real authentication in front of the API before exposing it. See the threat model.
• Row-level security needs the runtime role. The Compose stack logs in as a superuser, which bypasses it, so isolation under Compose rests on the queries' own tenant filters (record 0018). The default test context logs in as a superuser too, so it proves those filters. RlsLeakTest and the other tests that build their own pool log in as anvil_app, and they prove row-level security.
• A target for recovery time or data loss (the drill measures them; it doesn't promise them), or detecting a restore that keeps the database timeline, such as a storage snapshot rolled back in place. See record 0016.

Success

Success is measured by the guarantees above. Each measure names the test that shows it.

• A duplicate submit returns the same IDs and creates one job: SubmitIdempotencyTest.
• Two workers never claim the same job, and a stale or zombie worker affects 0 rows: JobServiceTest.
• A relay crash between publish and mark re-publishes an event that applies once: OutboxRelayTest.
• A crash during a content write or garbage collection never serves a torn file: ContentStoreTest.
• The database enforces tenant isolation with pooled connections and cached plans, with 0 leaks: RlsLeakTest.
• The 28-case read matrix and the audit reads show 0 cross-tenant leaks: ReadIsolationTest, AuditReadIsolationTest.
• A capability revoked mid-run blocks the commit (0 rows, 403 NEEDS_APPROVAL): CapRevokeMidRunTest.
• Untrusted native code in the jail can't reach another tenant's workspace, the shared /tmp or the network, on the host and in a container, and every SSRF matrix case is denied: ExecJailTest, JailContainerEscapeTest.
• A job replays bit-for-bit from its audit chain, and a corrupted, dropped or re-linked event is detected: AuditChainTest, ReplayTest.
• A declared 100,000-event chain replays from its latest checkpoint at p99 8 ms, under the 500 ms bound, with the same output_sha as a full replay: ReplayTest, measurement.
• The CLI prints the API's replay result for the same job, and an exported chain verifies with no database, reporting the first bad sequence number when tampered, with the API's tenant scoping: ReplayCliTest.
• The browser console shows a job's state, audit chain and replay result, and a missing job and another tenant's job look the same: OperatorConsoleTest.
• A seeded fencing break is caught in under 500 simulator runs: FenceSimTest.
• The fencing model keeps a single holder and no double effect: FenceModelTest.
• Process suspend, kill and restart, and duplicate delivery each produce a single effect: FaultProofTest.
• A tenant never exceeds its quota of running jobs, and a quiet tenant's jobs keep moving while another tenant floods the workers: FairClaimTest.
• Extraction sends a model at most 500 characters, indexes only facts with an exact citation, and dead-letters the rest: ExtractorTest.
• Extraction shows 0 cross-tenant reads, results or dedup hits, and never serves a dead letter: ExtractionIsolationTest.
• A cache hit, an exhausted budget and the kill-switch each make 0 model calls, and the budget is never overspent: ExtractionCostTest.
• The full set of 732 documents reports precision, recall and F1 with 10,000-resample intervals and McNemar against the baseline, and its 72 injection cases report attack success by encoding and language: FullSetTest.
• The full set runs nightly against the live model, fails when it drifts below the committed run, and files one issue: EvalNightlyTest, FileIssueScriptTest.
• The full-text index was measured against its budget before any soak number, and over budget it stays out of the schema; dropping it leaves extraction working: GinIndexTest.
• Load at 140 rps and a 10x flood report p95, error rate and table growth, and the soak refuses to run while the full-text index is over budget: LoadRigTest.
• Each of the nine fault-matrix cases shows a single effect and no lost work, and a failed scheduled fault run opens an issue with its seed and trace: FaultMatrixTest, FaultScheduleTest.
• Queue age, claim latency and pool wait are exported without tenant data, and the burn alerts fire on induced burn and stay silent when healthy: SaturationSignalTest.
• An unconfirmed or rewound database timeline refuses claims, heartbeats and commits: PitrTimelineCheckTest.
• A backup, clean and point-in-time restore reports its measured recovery time and data loss, with no target claimed: PitrRestoreDrillTest.

How to run

You need Java 25, Docker, Task, Node 22.22.1 and bubblewrap (bwrap, with unprivileged user namespaces allowed), which the exec jail runs every child under. Node is only needed to work on the operator console; the committed bundle is served by the Java app.

task dev       # run locally; starts no Postgres, so point it at one (below)
task test      # full suite against Testcontainers Postgres restored from a backup (also compiles with lint)
task frontend:check # lint, test and rebuild the TypeScript console; fails if the committed bundle is stale
task frontend:dev   # Vite dev server for the console, proxying /api and /audit to localhost:8080
task verify    # everything the gate runs, except its PR description check; needs ports 5432 and 8080 free
task faults:sim   # the nightly simulator runs (FAULT_SEED, FAULT_SIM_RUNS, FAULT_SIM_BREAK)
task faults:pg    # one weekly real-Postgres fault run (FAULT_SEED)
task eval         # eval gate on the frozen set (EVAL_CANDIDATE, default the committed extraction run)
task eval:extract # harness run: the extraction path over a frozen set (EVAL_SET; needs ANVIL_MODEL_API_KEY)
task eval:full    # eval gate on the full set (EVAL_CANDIDATE, default eval/runs/extract-v1-deepseek-flash-full-v1)
task eval:nightly # what the nightly workflow runs: the full set against the live model, then its gate
task load:gin     # measure the full-text index against record 0025's budget
task load:steady  # 140 extraction requests per second for 5 minutes (also load:flood; load:soak needs a within-budget GIN verdict)
task -x audit:replay -- --tenant T --job J               # replay a job from the database, without the API
task -x audit:export -- --tenant T --job J --out F.json  # write the job's chain to a file (never overwrites)
task -x audit:verify -- --tenant T --job J F.json        # verify that file with no database

The load runs start the app with a stub model that has the live model's recorded latency, containerized Postgres and k6 on this host. They need Docker and write to load/runs/. A committed run is never overwritten. The soak refuses to run until task load:gin says the budget is met, which it doesn't today (record 0025). LOAD_OUT=<dir> and arguments after --, such as PRE_VUS=300, write a run elsewhere with changed k6 settings.

A failed fault run writes its seed, trace and rerun command to target/fault-runs/.

The replay CLI (AnvilCli, record 0026) starts no Spring context, so it runs nothing but the command.

• replay and export connect with SPRING_DATASOURCE_URL, SPRING_DATASOURCE_USERNAME and SPRING_DATASOURCE_PASSWORD, the variables the service reads. The session is read-only, and a login that bypasses row-level security is switched to anvil_app first, or refused if it can't be.
• replay prints {input_sha, output_sha, match}. verify adds first_bad_seq, problem and head, the last event's hash. Compare head with one you kept, because a chain rewritten whole with every hash recomputed still verifies.
• Exit statuses: 0 match, 1 no match, 2 usage or malformed input, 3 not found, 4 unknown tenant, 5 database error, 6 failure (the output could not be written, or an unexpected error; never 1).
• Run them with task -x. Without -x, task replaces any non-zero status with 201, so the status above is lost.
• In the built image: docker compose exec app java -cp app.jar -Dloader.main=com.g1do.anvil.cli.AnvilCli org.springframework.boot.loader.launch.PropertiesLauncher replay --tenant T --job J.

To run the stack with real Postgres:

docker compose up --build
curl -s localhost:8080/healthz    # liveness
curl -s localhost:8080/readyz     # readiness (checks the database)

Submit a document as the seeded tenant t_single. The response holds the job_id to look up. The job stays QUEUED, because the app runs no workers yet (see "Not yet guaranteed").

curl -s localhost:8080/api/documents -H 'X-Tenant-Id: t_single' -H 'Idempotency-Key: demo-1' \
  -H 'Content-Type: application/json' -d '{"title":"demo","mime_type":"text/plain","body":"hello"}'

To run from source instead, start only the database and point task dev at it:

docker compose up -d db
SPRING_DATASOURCE_URL=jdbc:postgresql://localhost:5432/anvil SPRING_DATASOURCE_USERNAME=anvil SPRING_DATASOURCE_PASSWORD="${POSTGRES_PASSWORD:-anvil}" task dev

Open http://localhost:8080/console/ to inspect a job. The operator console is a TypeScript single-page app (React 19, Vite 8) built from frontend/ into the app's static resources. It sends X-Tenant-Id to the same API routes as any other client, shows the job's state, chain and replay result, and shows the same "Job not found" for a missing job and another tenant's job.

Compose reads POSTGRES_PASSWORD from your environment or an untracked .env; the names are in .env.example. With nothing set it uses a local-only default.

Extraction calls the model configured under anvil.extraction in application.yaml (DeepSeek's chat completions API, deepseek-flash) with the key in ANVIL_MODEL_API_KEY. ANVIL_MODEL and ANVIL_MODEL_BASE_URL override the model and URL for the app and task eval:extract alike. The nightly live eval sets those overrides to OpenCode Go's deepseek-v4.1-flash and uses ANVIL_GO_API_KEY; the app's default stays DeepSeek. Without a key the app still starts, and extraction answers 503 without calling anything or spending budget (a cached result is still served, and with the kill-switch off the rules still answer). A tenant also needs a call budget, which an operator grants in extraction_budgets. Without one, or with the kill-switch in extraction_switch off, the rules answer and the result is marked as fallback. The statements are in record 0023.

To run one test class: ./mvnw -B -ntp -Dtest=JobServiceTest test.

HTTP API (spec in contracts/openapi.yaml):

Route	What it does
POST /api/documents	Submit a document
GET /api/documents?query=&limit=&cursor=	Title-prefix search
POST /api/documents/{id}/extract	Extract cited facts from a document, once per version, prompt and model; from the tenant's cache, the model, or the rules fallback
GET /api/documents/{id}/facts	The document's indexed facts, marked when they come from the fallback
GET /api/jobs/{id}	A job's current state, scoped like the other reads
GET /audit/events?job_id=	A job's audit chain
GET /audit/replay?job_id=	Replay a job
GET /observe/saturation, GET /metrics	Saturation signals
GET /healthz, GET /readyz	Health probes
GET /console/	Operator console page (browser; calls the routes above with X-Tenant-Id)

Working on it

Work starts from an issue and lands through a PR that fills in the template. The gate must pass, and PRs merge with a merge commit, so every commit stays on main. Run ./mvnw clean before task test or task verify when an IDE has compiled the sources: the VS Code Java build can write stub classes into target/ that fail with "Unresolved compilation problem", which is a stale build, not a bug in the code.

Where things are

What	Where
Code	src/main/java/com/g1do/anvil/
Tests	src/test/java/com/g1do/anvil/
Migrations (schema)	src/main/resources/db/migration/
API contract	contracts/openapi.yaml
Intentional breaks	contracts/oasdiff-ignore.txt (API), contracts/schema-breaks.txt (schema)
Exact dependency versions	pom.xml: Spring Boot BOM plus explicit versions, no ranges
Architecture diagrams	docs/architecture/
Decisions	docs/decisions/
Threat model	docs/threat-model.md
Frozen eval sets and gate fixtures	eval/sets/pilot-v1/ (record 0021), eval/sets/full-v1/ (record 0024; built by src/test/java/com/g1do/anvil/eval/FullSetGenerator.java), never edited; eval/fixtures/
Extraction harness runs and the cost report	eval/runs/ (records 0022, 0023, 0024)
Load script, GIN budget and load reports	load/extract.js, load/runs/ (record 0025)
Operator console	frontend/ (React 19 + TypeScript + Vite source, tests and lockfile); the committed bundle is src/main/resources/static/console/, served by src/main/java/com/g1do/anvil/console/ConsoleController.java (record 0028)
Replay CLI, export format and offline verifier	src/main/java/com/g1do/anvil/cli/AnvilCli.java; replay/ChainExport.java (anvil-chain/1) and replay/OfflineVerifier.java (record 0026)
Replay from the latest checkpoint and its latency report	src/main/java/com/g1do/anvil/replay/ReplayService.java; load/runs/replay-latency/ (record 0027)
Filing a failed scheduled run	scripts/file-issue.sh (one issue per title while it's open)
Fencing model (TLA+ spec)	docs/faults/fencing.tla
Gate	.github/workflows/gate.yml, which runs the same task commands as Taskfile.yml
Scheduled fault runs	.github/workflows/faults.yml (task faults:sim, task faults:pg, task faults:issue)
Nightly live eval	.github/workflows/eval-nightly.yml (task eval:nightly, task eval:issue; needs the ANVIL_GO_API_KEY repository secret)

The breaking-change check (task test:contract) watches contracts/openapi.yaml, src/main/resources/db/migration/ and eval/sets/.

task eval scores a candidate's predictions (one JSON line per document) against the frozen eval set and its baseline. It prints precision, recall and F1 with intervals, the paired F1 difference, attack success and the citation check, and writes the report to target/eval/report.md. It fails on a difference interval wholly below 0, a rise in attack success, or a citation that isn't an exact substring of its document. By default it scores the committed extraction run, eval/runs/extract-v1-deepseek-flash (record 0022). task eval:full does the same on the full set, with McNemar's test and attack success by encoding and language (record 0024).