Run the portal's test runs as eval runs, with judges, usage and a lease by kentare · Pull Request #194 · digdir/genki

kentare · 2026-10-29T16:23:25Z

Stacked on #193.

What and why

"Kjør test" had a runner of its own: it graded each case pass or fail, recorded no judge or provenance, and stored a ServiceRun that no publishing gate could read. It now starts an eval run (ADR 0003 as amended by ADR 0040). Closes #94 with #192 and #193.

The draft or version is registered once as a capability (drafts.prepare), and platform.eval.run invokes it for each case. The run is judged by contract, by retrieval when cases name sources and by judge:criteria when there are criteria cases; the judge is picked outside every model the service may answer with (avoidAsJudge).
The portal's EvalRun is stored when it starts and keeps its state (running, waiting-for-scores, completed, failed, interrupted), with progress and the usage so far, under the same background lease as other portal work. A run left by a stopped instance reads as interrupted on every eval run endpoint, ended at its last heartbeat; a recovery only replaces a run that is still as it was read (EvalRunStore.replace), and late notices never reopen a finished run.
It is stored under the spec it tested, without the risk confirmation publish stamps, so a publishing gate finds a draft's run. Gates, comparisons, the console, building-block scores, the optimiser and the model matrix read only completed runs.
A case passes on its type's own metric, the criteria judge before the exact text before the refusal. A case that reached no model, such as a refusal before it, is scored; a run that reached none is refused. A draft that cannot run answers 422.
The run history shows the run's state, its metrics in bokmål with confidence intervals, its judges, which judge left cases unscored and why a run stopped. Test runs from before are no longer listed.
A case's output in a run is the answer's text, or its fields as JSON when it has none (answerOfParts), so a service that answers with a JSON part shows what it answered rather than nothing.
No eval run endpoint shows the lease, and usage shows only with view-usage. A run names the service it tested in provenance.services, so two runs of the same version compare, and the eval task counts as testing, not service work.
definitionSubject moves to apps/api/src/definition-subject.ts. Docs: docs/api/work.md, docs/api/README.md (API changes), ADR 0003 (amendment partly built), ADR 0041, OpenAPI (ServiceEvalRun, ServiceEvalRunSummary, lease omitted). i18n: apps/api/src/i18n/{en,nb}/errors.ts, portal operate-quality.

Not in this PR: score reasons from evaluators and judges are English, as the judges write them.

Screenshots

Before: a test run graded each case pass or fail, with the reason for a failure

After: the run shows its metrics with 95 % confidence intervals; a failed case has no reason yet

How it was tested

CI on this head passes (static, size and the four test shards). The top of the stack (Delete what genki sent to an agency's Langfuse when it purges the case #197) passes npm run check: 470 files, 5103 tests.
New tests: "runs as an eval run whose metrics have confidence intervals and whose judge is named with its model", "feeds a publishing gate: the run is found under the spec it tested, as the gate looks it up", "finds a draft's run under the spec the publish check computes for that draft", "finds a risk-confirmed draft's run under the spec publish checks, though publish stamps the confirmation", "never judges with a model the service may answer with, its fallback too", "reads only completed runs: a newer run still running, failed or interrupted never hides the last completed one", "marks test runs and proposals left running or waiting before the restart as interrupted or failed, ended at their last heartbeat, but not a live run on another instance", "replaces a run only while it is still as it was read, so a late recovery never overwrites a finished run", "never shows the lease of a run, owner and heartbeat, in the list or the run", "shows the cost and tokens of an eval run only with view-usage", "compares two runs of the same version, since a run names the service it tested, not its one-off draft", "compares only completed runs, and finds a run only under its own tenant", "shows no cost for a run whose model has no price, rather than a cost of nothing", "keeps a test run's own eval task out of Kjøringer, as its answers are", "answers 422 with the reasons instead of starting", "counts every case in the summary, so one without a verdict shows as not passed, as in the case list", "decides a criteria case by its criteria, also when it must refuse as well", "never moves a finished run back to waiting or to running progress", "scores a case no model answered, such as a refusal before the model, and reports usage so far with its progress", "picks the judge again once a case has writers, if the first case reached no model", "scores a block by the services' latest completed runs, never a newer run still running or failed", "says in bokmål why a run stopped and which judge left cases unscored", "shows a fields-only answer as the case's output".
The review broke 17 rules on purpose; the four checks no test caught now have the tests above.
In the portal after npm start: open a service's quality page as Kari Testansatt with the mock login, run its test set, open the result and see the metrics table with confidence intervals.
npm run check passes

Checklist

Every new rule has a test whose name states it
User-facing text is in the i18n catalogues, in bokmål and English, with no em-dashes
Vendor SDKs stay in packages/providers-*
ADRs, docs and docs/architecture/modules are updated if a contract, boundary or module changed
No secrets, personal data or local paths

"Kjør test" had a runner of its own: it graded each case pass or fail, recorded no judge or provenance, and stored a ServiceRun that no publishing gate could read. It now starts an eval run (ADR 0003 as amended by ADR 0040; closes #94 with the two commits before it). - The draft or version is registered once as a capability, and platform.eval.run invokes it for each case. The run is judged by contract, by retrieval when cases name sources and by judge:criteria when there are criteria cases. - The EvalRun is stored when it starts and keeps its state: running, waiting-for-scores, completed, failed or interrupted, with progress, under the same background lease as other work. A run left by an instance that stopped reads as interrupted. - It is stored under the spec it tested, computed as the publish check computes it, so a publishing gate reads a draft's run. Gates, comparisons and the console overview read only completed runs. - The run records the tokens and cost of the answers and of the judge calls, priced from the price list. - The run history shows the run's state, its metrics with confidence intervals, its judges and any incomplete judging. A case passes on its type's own metric; its answer is read from its task. - The optimiser and the model matrix read the latest completed eval run. Test runs from before are no longer listed.

Copilot

🟡 Changes recommended

Unresolved authorization leaks expose restricted usage and lease data, while race conditions can overwrite completed runs and valid model-free refusals can fail evaluation entirely.

8 open issues

What changed in this PR

This PR moves the portal’s test runs onto the shared evaluation mechanism, connecting draft and version testing to publishing gates, judge provenance, and usage reporting.

Changes:

Replaces service-run records with eval runs that track lifecycle state, progress, leases, and answer/judge usage.
Updates gates, comparisons, the optimiser, and model matrix to use completed eval runs.
Adds metrics, confidence intervals, judge identities, and new run states to the portal and API documentation.

File	Description
`packages/eval/src/strategy.ts`	Passes run IDs and answer metering into evaluation.
`packages/eval/src/service.ts`	Rejects comparisons involving unfinished runs.
`packages/eval/src/schema.ts`	Adds lifecycle, usage, lease, and tested-version fields.
`packages/eval/src/runner.ts`	Records completion state and answer/judge usage.
`packages/eval/src/platform.test.ts`	Tests gate selection of completed runs.
`packages/eval/src/gates.ts`	Selects the latest completed run per test set.
`packages/eval/src/evaluators.ts`	Adds evaluator usage reporting.
`packages/eval/src/cases.ts`	Exposes criteria-judge usage.
`docs/architecture/system-atlas.html`	Updates the generated eval configuration schema.
`docs/api/work.md`	Documents unified service evaluation behavior.
`apps/portal/src/screens/services/setup-facts.test.ts`	Updates fixtures to the completed state.
`apps/portal/src/screens/services/live-builder.test.tsx`	Updates builder run fixtures.
`apps/portal/src/screens/quality/RunHistory.tsx`	Displays run states, metrics, intervals, and judges.
`apps/portal/src/screens/quality/live-quality.test.tsx`	Tests metric and judge presentation.
`apps/portal/src/mock/quality.ts`	Adds eval states and metrics to mock runs.
`apps/portal/src/i18n/nb/operate-quality.ts`	Adds Norwegian run-state and metric labels.
`apps/portal/src/i18n/en/operate-quality.ts`	Adds English run-state and metric labels.
`apps/portal/src/api/quality.ts`	Updates run types and completed-run selection.
`apps/api/src/service-eval.ts`	Starts leased eval runs and builds service-facing results.
`apps/api/src/routes/console-routes.ts`	Limits console evaluation summaries to completed runs.
`apps/api/src/restart.test.ts`	Tests interrupted-run recovery and live leases.
`apps/api/src/quality.ts`	Reads shared eval records and task-backed answers.
`apps/api/src/quality.test.ts`	Tests provenance, metrics, and publishing-gate lookup.
`apps/api/src/platform-modules.ts`	Exposes eval capability and application version dependencies.
`apps/api/src/openapi/work.ts`	Updates the detailed service eval response schema.
`apps/api/src/openapi/services.ts`	Updates the service eval summary schema.
`apps/api/src/model-matrix.ts`	Estimates answer usage from completed eval runs.
`apps/api/src/main.ts`	Supplies the application version for provenance.
`apps/api/src/improvements.ts`	Reads optimiser evidence from completed eval runs.
`apps/api/src/eval-judges.ts`	Meters criteria-judge calls per run.
`apps/api/src/eval-judges.test.ts`	Checks judge usage reporting.
`apps/api/src/draft-runner.ts`	Adds reusable draft preparation and usage summaries.
`apps/api/src/definition-subject.ts`	Extracts shared spec hashing for gate lookup.
`apps/api/src/composition.ts`	Wires evaluation metering, model families, and dependencies.
`apps/api/src/background-lease.ts`	Aligns optional lease-field types with eval records.

_{Review effort: Balanced}

…one-eval-runner # Conflicts: # apps/api/src/composition.ts # packages/eval/src/evaluators.ts # packages/eval/src/runner.ts # packages/eval/src/strategy.ts

…y right - No eval run endpoint shows a run's lease, and usage shows only with view-usage, also in GET /tenants/{tenant}/eval-runs and GET /eval-runs/{id}. - A run left by a stopped instance is marked interrupted only while it is still as it was read (EvalRunStore.replace), so a late recovery never overwrites a finished run. - Progress and waiting notices never move a finished run back. - A case that reached no model, such as a refusal before it, is scored; only a run that reached no model at all is refused. - Progress carries the usage so far, so a failed run keeps what it spent. - The summary counts every case, as the case list does. - A run names the service it tested instead of its one-off draft capability, so two runs of one configuration can be compared. - The eval task of a test run counts as testing, so it stays out of Kjøringer and the service's usage.

…one-eval-runner

…dge off the service's models A portal test run passes the service's own models, fallback included, as avoidAsJudge, so its criteria judge never shares a family with a model the service may answer with.

One kind of run, evaluators that answer later, expected.criteria (upgraded on read) and the criteria judge's identity are built; the rest of the amendment is listed as not built yet.

…ywhere - A run is looked up by its spec without the risk confirmation, which publish stamps with its time, so a risk-confirmed service's gate finds its portal run. - Building-block scores read only completed runs. - A draft that cannot run answers 422 with the reasons. - A criteria case that must also refuse is decided by its criteria. - A run left by a stopped instance reads as interrupted on every eval run endpoint, ended at its last heartbeat, also while waiting for scores. - The run history labels metrics in bokmål, says which judge left cases unscored from the counts, and shows why a run stopped. - The judge is picked again once a case has writers, OpenAPI hides the lease, and the docs say what changed.

kentare · 2026-10-30T08:11:59Z

Ready for review. Copilot and a claim-checking review are done (no blind review) and CI passes except one test this PR does not touch, apps/console/src/english-blocks-search.test.tsx ("previews a new version in English and asks for a reason"), which passes locally and needs a rerun. Open: score reasons from evaluators and judges are English, as the judges write them; a draft's run is version 1, so runs compare only within one version.

kentare · 2026-10-30T08:48:11Z

Found while testing the stack in the portal: a run of a service with structured output shows no answer.

A Klarspråk run (output kind: fields) returns "output": "" for every case from GET /tenants/{t}/eval-runs/{id}. answersOf reads text parts only (textOfParts, apps/api/src/service-eval.ts:124), and Klarspråk answers with one JSON part.
In the portal, the text shown under each case is the reference, so the run looks fine while nothing was answered as text. Same cause as the judge passing it on Judge case types as evaluators, with the judge's model in its identity #193.

Fix: show the JSON part when there is no text part, for example its text field or the JSON itself. A test: a fields-only answer appears as the case's output.

A run of a service that answers with a JSON part stored an empty output for every case, so the portal showed only the reference. The output is now the answer's text, or its fields as JSON when it has none, as the judge reads it.

… the judge pick test An empty answer now scores 0 without picking a judge, so the test's first case needs an answer to reach the pick.

kentare · 2026-10-30T09:29:18Z

Re #194 (comment)

Fixed in f955c19: a case's output is the answer's text, or its fields as JSON when it has none (answerOfParts, the same reading the judge uses since #193). Checked by "shows a fields-only answer as the case's output" and npm run check on the top of the stack (470 files, 5103 tests).

kentare mentioned this pull request Oct 29, 2026

Let an agency connect its services to a Langfuse #195

Draft

6 tasks done

kentare requested a balanced review from Copilot October 30, 2026 04:20

Copilot started reviewing on behalf of kentare October 30, 2026 04:20

View session

Copilot AI reviewed Oct 30, 2026

View reviewed changes

kentare added this pull request to stack #199 October 30, 2026 04:36

kentare added 10 commits October 30, 2026 05:44

Merge remote-tracking branch 'origin/feat/case-evaluators' into feat/…

5368b3f

…one-eval-runner # Conflicts: # apps/api/src/composition.ts # packages/eval/src/evaluators.ts # packages/eval/src/runner.ts # packages/eval/src/strategy.ts

Merge remote-tracking branch 'origin/feat/case-evaluators' into feat/…

442c555

…one-eval-runner

Merge feat/case-evaluators into feat/one-eval-runner, and keep the ju…

0766d8a

…dge off the service's models A portal test run passes the service's own models, fallback included, as avoidAsJudge, so its criteria judge never shares a family with a model the service may answer with.

Merge branch 'feat/case-evaluators' into feat/one-eval-runner

8b02944

Mark what the stack builds of ADR 0003's amendment

014cabf

One kind of run, evaluators that answer later, expected.criteria (upgraded on read) and the criteria judge's identity are built; the rest of the amendment is listed as not built yet.

Merge branch 'feat/case-evaluators' into feat/one-eval-runner

2c3d8c6

Give the run history test only the fields a case result has

5ad928b

Give the 422 title for a draft that cannot run its catalogue entries

1246b8a

kentare mentioned this pull request Oct 30, 2026

Let judges deliver their scores later #191

Draft

6 tasks done

kentare added 3 commits October 30, 2026 10:03

Merge branch 'feat/case-evaluators' into feat/one-eval-runner

6b9d5e7

Show a structured answer as the case's output in a run

f955c19

A run of a service that answers with a JSON part stored an empty output for every case, so the portal showed only the reference. The output is now the answer's text, or its fields as JSON when it has none, as the judge reads it.

Let a guardrail's refusal stand for the case that reached no model in…

f369b32

… the judge pick test An empty answer now scores 0 without picking a judge, so the test's first case needs an answer to reach the pick.

kentare mentioned this pull request Oct 30, 2026

Judge case types as evaluators, with the judge's model in its identity #193

Draft

6 tasks done

Navigation Menu

Repository navigation

Run the portal's test runs as eval runs, with judges, usage and a lease - #194

Run the portal's test runs as eval runs, with judges, usage and a lease#194
kentare wants to merge 14 commits into
feat/case-evaluatorsfrom
feat/one-eval-runner

kentare commented Oct 29, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kentare commented Oct 30, 2026

Uh oh!

kentare commented Oct 30, 2026

Uh oh!

kentare commented Oct 30, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

kentare commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What and why

How it was tested

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kentare commented Oct 30, 2026

Uh oh!

kentare commented Oct 30, 2026

Uh oh!

kentare commented Oct 30, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kentare commented Oct 29, 2026 •

edited

Loading