[kentare]

What and why

Kudos is too large for today's pgvector setup: every run rebuilds a whole generation in tables shared with every dataset on the platform database, and nothing between the source and the index is kept. ADR 0042 sets out how genki keeps its own knowledge data instead. Nothing in the code changes yet. It runs on the job queue that ADR 0041 proposes (#179, merged).

• The server: its own flexible server in Norway East, starting at Standard_D2ds_v5 with no standby and bound to its platform server. The API and the worker log in with their own identities; the API reads only through SECURITY DEFINER functions and has no rights on the job tables there.
• Records and runs: document records owned by pipelines and updated in place; one open run per pipeline, with queued and waiting beside today's states; a listed source deletes only after two complete listings; a change brake on large runs.
• Jobs, on ADR 0041: run, activation and drop jobs live on the platform server and commit with the record they follow; conversion, index and build jobs live on the knowledge server. A dataset is dropped only after a catalog version that deletes it, never because its catalog entry is missing. In lockdown, a run's admission lasts 24 hours.
• Erasure: a withdrawal removes a document from every generation and the caches at once, and from the Blob copy that night. Every tombstone keeps only the id's HMAC under a key per dataset, and the date; neither database holds the key. A numbered log on the platform server keeps the document out of later runs and makes a restored server refuse searches until it has caught up. Backups are kept 7 days, and the Marker client asks the host to keep nothing.
• Generations: a partition per generation, with the catalog naming the active one; a new generation only when the recipe changes; heavy work only in a night window; the workaround for the pgvector 0.8.2 VACUUM bug.
• Left to Decide what moving Kudos into the knowledge database commits us to #175: moving the full Kudos corpus, removing Typesense (Remove Typesense from genki's code and infra #150), where Kudos may be processed (Declare Kudos's residency global, for now #131), and the server size and standby Kudos would need, each with its cost.
• Pointers in ADR 0004, 0011, 0017, 0022, 0025 and 0033, a Knowledge section in GLOSSARY.md, and the README row.

The evidence is at the tags evidence/knowledge-spikes, evidence/kudos-pilot and evidence/ingestion-benchmark. GLOSSARY.md also comes with #100; this adds its Knowledge section. ADR numbers are settled at merge, since #100 and #117 both claim 0040.

Part of digdir/kunnskapsassistenten#151. The build is in #127 to #151.

How it was tested

• Checked the record against the code it names and the records it touches, through three reviews, the change request on this PR, and a review of the choices made in answer to it. Every finding is fixed or decided in the text.

• Spikes in pgvector/pgvector:0.8.2-pg17 containers: attach 18.5 ms and detach 7.3 ms under full search load; recall 0.995 after 20 rounds of 1 % in-place changes; reads through definer functions take 0.09 to 0.12 ms on 301 partitions.

• Read pg-boss 12.38.0's stately policy in its source: one job per state per key, so a second send in the same state returns nothing. The run queue therefore lives in the run records, not in pg-boss.

• Costs in Decide what moving Kudos into the knowledge database commits us to #175 from the Azure Retail Prices API.

• npx prettier --check docs/adr GLOSSARY.md.

• npm run check passes (docs only)

Checklist

• Every new rule has a test whose name states it
• User-facing text is in the i18n catalogues, in bokmål and English, with no em-dashes
• Vendor SDKs stay in packages/providers-*
• ADRs, docs and docs/architecture/modules are updated if a contract, boundary or module changed
• No secrets, personal data or local paths

[nlundee]

Solid work, and good that it rests on the spikes. I agree with the direction, but it's not ready to approve yet. These need to be settled first:

1. Deletion shouldn't follow absence (0039:28). A missing catalog entry triggers knowledge.drop, so a wrong PGHOST or an old restore could delete every dataset. Delete from an explicit deletion record that the handler re-checks.
2. Erasure (0040:73-82, :154). Marker gets cache=true and keeps every PDF; ids with paths end up in tombstones and jobs; restores bring withdrawn documents back; no pg-boss retention; ADR 0011 needs amending. The tombstone title is also described two ways.
3. Run lifecycle (0039:31, :52). Admission never expires on runs that can stay open for nights, and the needed statuses (queued, waiting, cancelled) don't exist yet. Our current 6-hour stale-run sweep would also kill these runs.
4. #108 solves the same issue (#107) with different ownership rules: there a disabled pipeline loses its documents, deletion happens as a side effect of a sibling's run, and the first run after the upgrade drops unattributed documents. @bdb-dd, could #108 follow 0040's rules as an interim step on today's tables (disabled keeps documents, explicit removal run, refuse instead of dropping)? Then 0040 can reference it.
5. What it commits us to: full Kudos corpus, removing the shared Typesense (#150), Kudos global (#131), and a 64 GiB server with standby. I'd like these as explicit decisions with cost, not implied by issues.

Smaller: the reader role's grants on pgboss; that #123/#120 rules apply in the worker; residency for server, backups and Blob; evidence links pointing at tags; and ADR numbers (0039/0040 collide with #100, #109 and #117).

Could 0039 (pg-boss) go in on its own first? It's the smallest piece and fixes real problems we have today.

[kentare]

Thanks, all fair. I've split it as you suggested, and fixed the rest in the knowledge record.

Split. Durable jobs is now its own record and PR: ADR 0041 in #179, with the JobQueue port, providers-pgboss and the rules every queue follows, but no queue of its own. The Langfuse proposal (#100) no longer needs a queue: its score wait stays in the run's lease, and trace deletes are pending rows the nightly purge works through. So 0041 changes no behaviour by itself; it settles the queue so this PR's review can be about the knowledge database alone. The knowledge queues, the reconciler, run states and admission moved into this record, now ADR 0042, and this branch is based on #179. #179 can merge first, so could you take that one first?

1. Deletion follows a record, not absence. A dataset is deleted by a catalog version that says so (with who, when and why), and knowledge.drop is sent in the same transaction. The handler reads that version again before it deletes anything. Data for a dataset the catalog doesn't know is reported every night and never dropped. The knowledge server is also bound to its platform server's id, so a worker pointed at another environment does nothing.

2. Erasure.

   • Marker cache: our client asks Marker to keep every PDF and its result (cache=true), so a withdrawn document survives on the GPU host. 0042 switches it to cache=false, since genki keeps its own conversions, and lists this as a defect on main today, together with the 300-second timeout.
   • Jobs name a document by its record's number, never its source id.
   • Every tombstone keeps only a hash of the id and the date, so the two descriptions no longer clash, and a citation can't tell an erasure from an ordinary delete.
   • A withdrawal removes the document's unshared conversions and vectors at once, and their Blob versions that night.
   • Withdrawals are numbered and logged on the platform server. A later listing skips a withdrawn document, so a source that still lists it doesn't bring it back, and a restored server answers "not ready" until it has applied the log.
   • The knowledge server's backups are kept 7 days.
   • Job retention is set in code: 7 days after a job ends, 14 for one nobody takes.
   • ADR 0011 points to 0042.

3. Run lifecycle.

   • Runs get queued and waiting beside running, succeeded, failed and cancelled. At most one run per pipeline is queued, and the end of a run starts the queued one in the same transaction.
   • In lockdown, admission lasts 24 hours, so a run that stays open for nights stops and is started again.
   • Runs that run as jobs are marked on their record, and failStaleRuns leaves them alone; the reconciler ends them by their jobs.
   • Side note on Erase conversation task content on delete and close stale ingestion runs #159: the exception list is only the starting instance's own runs. With two API replicas, a second instance that starts marks a run older than 6 hours as failed while the first is still working on it. Filed as Fail an ingestion run only when no instance is still working on it #176.

4. Keep the other pipelines' documents when a pipeline runs #108. Agreed. 0042's rules are: a disabled pipeline keeps its documents, deletion only through a removal run, and refuse rather than drop when ownership is unknown. Keep the other pipelines' documents when a pipeline runs #108 would be a good interim step on today's tables if it followed them. I've put the three differences on Keep the other pipelines' documents when a pipeline runs #108, with the lines in its code, plus a fourth: with several pipelines, a run that lists no documents now deletes all of its pipeline's documents.

5. Commitments. 0042 now says under Left: that it doesn't decide moving the full Kudos corpus, removing Typesense (Remove Typesense from genki's code and infra #150), where Kudos may be processed (Declare Kudos's residency global, for now #131), or the 64 GiB server and standby. The server starts at the platform server's size (Standard_D2ds_v5) without a standby. They are decided in Decide what moving Kudos into the knowledge database commits us to #175, with their costs. These are Azure list prices for Norway East, from the Azure Retail Prices API on 2026-10-29, at 730 hours a month:

   • Standard_D2ds_v5, where the server starts: about $185 a month (compute $0.239/h, storage $0.151 per GiB a month, 64 GiB).
   • Standard_E8ds_v5 with 128 GiB of storage, what Kudos would need: about $990 a month (compute $1.328/h, same storage price). A standby would roughly double that.
   • Embedding all of Kudos with text-embedding-3-large: about $270 one-off on a global deployment, or $330 on a regional one in Norway East ($0.13 and $0.158 per million tokens). This is scaled from the pilot's 13.5 million tokens for 299 documents (timings.md, costs.csv).
   • Marker GPU time and the Typesense cluster's cost: not measured.

   These are list prices for a size estimated in a container. Measure a Kudos-sized index build on an Azure knowledge server #132 measures a Kudos-sized build on a real Azure server and prices the size it supports, standby and backups included, for Kostnader og nødvendige rammer kunnskapsassistenten#30.

Smaller:

• The API sends jobs only to the platform server, so the reader role has no rights on pgboss. knowledge.ingest, knowledge.activate and knowledge.drop now live there and commit with the run, activation or deletion they follow. That also removes the crossing step for starting a run. Cancels and send-backs from the API go through the worker.
• The worker fetches through the same fetch guard (Pin ingestion connections to validated addresses #120) and uses tenant secrets only for their hosts (Bind tenant secrets to the hosts they are set for #123, ADR 0015).
• Residency: server, backups and Blob copy all stay in Norway East.
• Evidence links point at tags (evidence/knowledge-spikes, evidence/kudos-pilot, evidence/ingestion-benchmark).
• ADR numbers: these two are now 0041 and 0042, clear of the others. Propose evaluating with Langfuse #100 and Serve tool-using OpenAI deployments through the stateless Responses API #117 both claim 0040, so whichever merges first keeps it.

[nlundee]

Every point from the last review is answered, and the record is much stronger for it: deletion only from a deletion version that the drop reads again, the server bound to its platform server, withdrawals logged where a restore can't undo them, queued and waiting runs with admission that runs out, ownership per pipeline, and the commitments moved to #175 with real prices. Fine to merge as a proposed record once #179 is in.

The record also finds three defects on main today. They shouldn't wait for the build, and they already have their own issues: #128 (autovacuum beside HNSW writes on pgvector 0.8.2), #127 (the Marker client's cache=true and five-minute timeout) and #129 (the reranker's residency). Worth doing those first.

One suggestion: the tombstones and the withdrawal log keep a plain sha256 of the id. For guessable ids, such as folder paths with a name in them, anyone with the table can confirm a guess, and the same document gives the same hash in every dataset. An HMAC with a key per dataset costs nothing extra and closes both.

Approving.

[kentare]

Thanks. The HMAC is in 2058e37: tombstones and knowledge_withdrawals hold HMAC-SHA256 under a key per dataset, derived from one Key Vault secret (KNOWLEDGE_ID_KEY) that neither database holds, and the API passes kn_resolve_citation the keyed hash.

It isn't quite free: the key can never be rotated, since the ids are gone and old entries can't be hashed again, and it must outlive every dataset, or withdrawn documents come back on the next run. A check value on the platform server turns a lost or replaced key into a readiness failure for the worker and a refusal for withdrawals. Chunk ids stay plain hashes, since the answers that keep them also keep the title and excerpts.

The branch is rebased onto main now that #179 is merged. #186, #187 and #188 were already tracked as #128 (draft PR #183), #127 and #129, so I closed them as duplicates and moved the one new check from #186, rebuilding indexes already damaged on test and production, into #128.

[nlundee]

Makes sense. A leaked key only takes us back to the plain-hash baseline, so not rotating it is fine, and purge protection on the vault covers losing it. Approved as is.