[kentare]

What and why

Teams already judge answers in Langfuse and want those judges' scores in genki, while genki keeps the run record and the publishing decision. This PR proposes how, as ADR 0040, and amends the three records it touches. Part of #93.

• ADR 0040 Evaluating with Langfuse (proposed).
  • genki starts, records and gates every eval run. A run may list Langfuse judges beside genki's own. genki then sends it to the tenant's Langfuse as one experiment, without holdout cases and with only the fields the judges read, and collects the scores for at most 5 minutes.
  • Langfuse judges are advisory: their scores and their cost, as Langfuse reports it, are shown on the run and never block a publish. "Langfuse judges that block (proposed separately)" lists what blocking needs; the catalog check has no answer yet.
  • Each tenant connection has a residency covering where Langfuse stores data and where the judges' models run, so a norway service cannot use a connection outside Norway.
  • On Digdir's instance, operators create each tenant's organisation and invite the owner the tenant names, who manages members in Langfuse. Tenant connections carry eval runs only, never live answers.
  • A Langfuse judge's identity names its tenant and connection: langfuse:<tenant>/<connection>/<name>@v<version>+<model>.
  • evaluator@2 replaces evaluator@1, so a judge may answer later.
  • The score wait runs inside the eval run's background lease, and trace deletes are pending rows the nightly purge works through, so no durable jobs are needed.
  • Labels from Langfuse annotation queues count once someone with platform.judge.label accepts them. A Langfuse dataset can be imported once as a reviewed test set, never into holdout.
• ADR 0003, 0011 and 0024 (amended, not built). Each keeps its built decision and gets a "Proposed with ADR 0040 (not built)" section:
  • 0003: one runner; every model-based judge blocks only with a Cohen's kappa of at least 0.6 over at least 30 labels and a model from another family than the answer's; labels kept per judge identity; expected.criteria; holdout never leaves genki.
  • 0011: when a trace sink honours erasure, only what judges read is sent, and pending sink deletes.
  • 0024: a skill's pass rate blocks only once its criteria judge is calibrated.
• docs/adr/README.md: a row for 0040, and 0003, 0011 and 0024 marked "amendment proposed in 0040".
• GLOSSARY.md (new): eval run, experiment, split, holdout, judge, calibration label, advisory and feedback.
• .gitignore: ignores .scratch/, where local planning notes live.

Open before agency data goes to langfuse.digdir.cloud: whose model key the agencies' judges use, and so who pays; where the instance stores data; and the items in #93 (the 7-day lifecycle rule, the judge model deployments and a data-protection assessment).

Not in this PR: building it, in #94 to #99.

How it was tested

• Read Langfuse 4.4.0's source (Prisma schema, ClickHouse migrations, fern/ API definitions, the worker's evaluation and ingestion code) and compared API changes up to 4.67.0.

• Read-only calls to langfuse.digdir.cloud: version 4.4.0, one organisation, two projects, 17 judges, no enterprise licence. In one project the last 100 judge traces all carry tokens and a Langfuse list-price cost (claude-sonnet-5, about USD 0.008 per call), and each judge trace's metadata names target_trace_id and job_configuration_id.

• A throwaway round trip against a local Langfuse 4.4.0 with genki's eval runner on six made-up cases: genki's own scores were unchanged, the judge's scores came back in 3 to 9 seconds with its identity, labels were traced back to the case and run, and nothing was sent with content off.

• The same round trip showed what deleting misses: deleting a case's trace leaves the judge's trace and the raw files. Deleting the judge's trace, found from job_execution_id on its score, removed every ClickHouse row with the text within 5 seconds.

• One call to each of five judge models on the judges' Azure resource answered from Sweden Central.

• Filtering traces by target_trace_id through Langfuse's API is still unverified, as ADR 0040 says.

• npx prettier --check docs/adr GLOSSARY.md.

• npm run check passes (docs only; with main merged, 4900 tests pass, 12 skipped)

Checklist

• Every new rule has a test whose name states it
• User-facing text is in the i18n catalogues, in bokmål and English, with no em-dashes
• Vendor SDKs stay in packages/providers-*
• ADRs, docs and docs/architecture/modules are updated if a contract, boundary or module changed
• No secrets, personal data or local paths

Nothing is built, so no rule has a test yet; ADR 0040's Consequences require one per rule when it is built, and modules change with the code.

[kentare]

@nlundee kan du komme med feedback her?

[nlundee]

It fits well with ADR 0003 and keeps the gates in genki, which is what we want. Thumbs up for the part that only changes genki (deferred judges, one runner, calibration, judge identity with version and model, expected.criteria), so Nathalie can start on that. The part where an organisation brings its own Langfuse judge needs a few things before we accept it:

1. Spend and models. Langfuse judges run on Langfuse's own model connection, so their cost skips the tenant quota and their models aren't checked against the catalog (candidate models included).
2. Tenant in judge identity and labels. Two organisations with a judge of the same name would share a calibration pool, so one tenant's labels could decide whether another's judge may block publishing.
3. Residency. Nothing ties the Langfuse connection or the judge model to the service's residency. The instance's judges answered from Sweden Central, so a norway service could be judged there.
4. Access on Digdir's instance. Who creates organisations and users there, how that maps to Altinn rights, and whether a tenant connection carries live traces or only eval traces.

Two practical things: the number 0039 collides with #109 and #125, and 0040 with #117 and #125. I'd suggest 0039 for this one and that it merges before #125 (they conflict on GLOSSARY.md and the ADR index). And #116 changes the criteria judge, so it should go in first.

Would it work to mark the Langfuse part as proposed and accept the rest now?

[kentare]

Thanks, this was useful. I've taken the split with one change: instead of keeping the whole Langfuse part proposed, Langfuse judges are now advisory, and only judges that block stay proposed, in their own section of the ADR. Their scores can show in genki without deciding anything. The four points:

1. Spend and models. A Langfuse judge never blocks now, so its model is not part of any publish decision. Its cost is visible: every judge call writes a trace with tokens and Langfuse's list-price cost. On langfuse.digdir.cloud the last 100 judge calls were all priced (claude-sonnet-5, about USD 0.008 each). genki stores it on the run as reported by Langfuse, not against the quota. Blocking needs that cost counted against the quota and the judge's model in the catalog; the catalog part has no answer yet. Who pays for judge calls on Digdir's instance is listed as open, to settle before agency data goes there.
2. Tenant in identity and labels. The identity is now langfuse:<tenant>/<connection>/<name>@v<version>+<model>, and in ADR 0003 a new judge version takes over labels only from the same tenant, connection and name. judge_labels is keyed by the judge string alone today, so this keeps the pools apart, also for two projects in one tenant.
3. Residency. Each connection has a residency (norway, eu or global) covering where its host stores data and where its judges' models run. It starts as global until an operator sets it, and a run uses a connection only when the service's residency allows it. The judges on Digdir's instance answered from Sweden Central, so a norway service cannot use them. Where the instance stores data is listed as to confirm.
4. Access on Digdir's instance. Digdir operators create each tenant's organisation and invite the owner the tenant names; the owner manages members in Langfuse. There is no Altinn mapping, which is why Langfuse scores never block and Langfuse labels count only after someone with platform.judge.label accepts them in genki. A tenant connection carries eval runs only, never live answers.

One more change: the score wait and the trace deletes no longer need durable jobs. The wait runs inside the eval run's background lease, as the cases do, and deletes are pending rows the nightly purge retries. So #97 and #98 no longer wait for #155 or #125.

Numbering: #109 merged as 0039 shortly after your review, so this is now 0040, with main merged in. Agreed that it goes before #125, which moves to 0041 and 0042, and #117 needs a new number too. Fine with #116 going first. One overlap you didn't list: #106 also edits ADR 0024.

With the judges advisory, would you accept ADR 0040 now, leaving only "Langfuse judges that block" proposed? That keeps #94 and #96 going as you suggested and lets #97 and #98 start.

[nlundee]

Advisory first works for me, and the four points are answered: the tenant and connection in the judge identity, residency per connection starting at global, operators creating organisations, and eval runs only on a tenant's connection. Yes to accepting 0040 now with "Langfuse judges that block" left proposed. Three things before merge:

1. Erasure when a connection goes away. A delete that keeps failing because the connection was removed or its secret rotated is only reported to operators, and the traces with content stay in Langfuse. A connection with pending deletes shouldn't be removable, or removing it should need the tenant's confirmation that the traces are gone, recorded in the audit log.
2. The erasure check rests on unverified internals. Finding judge traces uses the undocumented sha256(job_execution_id) rule, and the check that nothing is left filters on target_trace_id, which the record says isn't verified yet. Please add verifying that filter to "Before agency data goes to Digdir's instance", so a content-receiving connection isn't opened before we can show the deletes are complete.
3. Status line. Once accepted, the status should say it's accepted and not built, except the section on blocking judges, which stays proposed. The same goes for the README row, so nobody reads the blocking part as decided.

Smaller: the blocking section still points at #125 for durable jobs. That's now ADR 0041 in #179. #117 still uses 0040 too, so it needs a new number before either merges.

[kentare]

Thanks. All three are in (status, removal blocked while deletes are pending, target_trace_id verification before agency data), plus the ADR 0041 pointer. #117 needs its own number.