[kentare]

Stacked on #197.

What and why

Agencies that judge answers in Langfuse want those judges' scores on genki's test runs, as advice that never decides a publish (ADR 0040). Part of #98; judge cost (98b) and labels (98c) follow.

• Each active experiment judge in a connection's Langfuse project is a judge building block. The agency takes it in use with a reason. A judge that genki cannot name or that would leave cases unscored (default model, a field genki does not send, sampling or a filter, a connection without answer text) is refused.
• A run scored by such a judge goes to Langfuse with only the fields the judge maps. Scores are read back per case trace (experimentId returns nothing on 4.4.0), marked judge changed when the judge changed, and not applicable for holdout cases and missing fields. The judge's reason is kept only on the case, so retention purges it.
• Langfuse scores stay out of the run's metrics and never weaken a gate. A gate on a Langfuse metric is refused on save in Kvalitetskrav and at publish.
• Portal: judge blocks in Byggeklosser, a locked "Send til Langfuse" with a residency check before start (GET .../eval-runs/sending), and a separate advisory "Langfuse-dommere" part on a run, with per-case verdicts and a link to the experiment. The link needs Langfuse's "V4 Preview".
• ADR 0040, the providers-langfuse module and OpenAPI are updated.

Not in this PR: judge cost, Langfuse labels, and naming a Langfuse judge directly in an API eval run.

:camera_with_flash: Screenshots

Byggeklosser: a Langfuse judge with its model, version, answer type, fields and what it gets access to

Kjør testsettet: sending to Langfuse is ticked and locked, naming the judge

Resultat: the Langfuse judge apart from genki's metrics, marked advisory

Per testtilfelle: the judge's verdict and reason

How it was tested

• npm run check: 471 files, 5133 tests pass.
• 29 new tests, each seen failing first or with its code switched off (names below).
• Once against a local Langfuse 4.4.0 with a real judge and made-up data: the run was scored, with a reason in Norwegian, and the criteria reached the judge.
• In the portal, on a separate stack (API on 8090, portal on 5175, mock login as Kari Testansatt) with the local Langfuse: the judge block, taking it in use, the locked checkbox, a run of the Klarspråk test set scored by the real judge, and the refusal in Kvalitetskrav. This found that cases with only a reference answer were not judged (fixed).

New tests

• "scores a run's cases with a Langfuse judge under the identity langfuse://@v+"
• "keeps two agencies', or two connections', judges of the same name apart, scores and labels alike"
• "takes only scores from the snapshotted rule, never a same-named score from another rule"
• "marks scores judge changed when the rule, its mapping, the evaluator version or its model changed since the start"
• "refuses a judge on the project's or organisation's default model"
• "refuses a judge whose mapping reads a field genki does not send, such as tool calls"
• "is not applicable on a case without the fields its mapping reads, on a connection without content, and on a holdout case"
• "reads a boolean as 1 or 0, scales a number by its config's bounds, and keeps a category as it came"
• "refuses a service definition with a gate on a Langfuse judge's metric, naming the rule"
• "purges a judge's reason with the case, as its retention says"
• "lists each experiment rule of a connection's project as a judge building block, and scores a run only with the ones the agency added"
• "refuses a run whose adopted Langfuse judge is refused or gone when it starts, and never adopts a refused one"
• "checks a gate's calibration only against the judges that scored its metric"
• "leaves an advisory judge out of every gate, so a judge taken in use since the baseline never fails a publish"
• "keeps an advisory judge's scores on its cases but out of the run's metrics"
• "refuses a judge that judges only some items, by sampling or a filter, or that is not an LLM judge"
• "refuses a run with a reason when the connection's Langfuse cannot be read as it starts"
• "reads a criteria case's reference answer as its criteria when it names none, as genki's criteria judge does"
• "sends a criteria case's reference answer as its criteria when it names none, as genki's criteria judge reads it"
• "refuses a Langfuse judge on a connection that gets no answer text, when it is taken in use and when a run starts"
• "shows that a Langfuse judge on a connection without answer text cannot be taken in use"
• "says before a run starts that the Langfuse judges' connection gets no answer text, and does not start it"
• "shows each active experiment rule of a connection's project as a judge block with its model, evaluator version, data type and mapped fields, and adds it only with Ta i bruk and a reason"
• "shows why a Langfuse judge block is refused: the project's default model, or a field genki does not send"
• "ticks and locks Send til Langfuse when the service uses a Langfuse judge, naming the judges"
• "says before a run starts that the service's residency forbids sending to a Langfuse judge's connection, and does not start it" (portal and API)
• "shows a Langfuse judge's scores apart from genki's metrics, marked advisory, with judge changed, not-applicable counts, missing scores and a link to the experiment"
• "shows each Langfuse judge's verdict and reason per case"
• "refuses a Langfuse judge's metric in Kvalitetskrav, naming the rule" (portal and API)

• npm run check passes

Checklist

• Every new rule has a test whose name states it
• User-facing text is in the i18n catalogues, in bokmål and English, with no em-dashes
• Vendor SDKs stay in packages/providers-*
• ADRs, docs and docs/architecture/modules are updated if a contract, boundary or module changed
• No secrets, personal data or local paths

[kentare]

Ready for review. A blind review and a claim-checking review are done and CI passes; Copilot did not review (no quota). Open: judge cost and Langfuse labels come in 98b and 98c; the experiment link works only for Langfuse users with "V4 Preview" on; Langfuse also judges cases genki marks not applicable, which costs a call each (98b); naming a Langfuse judge directly in an API eval run is not built; no screenshots yet.

[kentare]

Update: the three commits after the last ready comment (7df1d59, 575d837 and the judge block's source and access rows) have now had a claim-checking review. It found two gaps, fixed in 8133242: a test that a reference answer reaches Langfuse as criteria, and ADR 0040 and 0011 saying so. CI passes on the head. Still open as before: cost and labels come in 98b (#207) and 98c; the experiment link needs Langfuse's "V4 Preview"; naming a Langfuse judge directly in an API eval run is not built.