[kentare]

Stacked on #192.

What and why

The portal grades exact, must_reject and criteria cases in a runner of its own. Eval runs now have evaluators for the same case types, graded as the portal graded them, so one runner can grade them. Part of #94.

• The contract checks (contract v2) score an exact case's text (exact-text, 0 when the case has no text) and count an empty answer as a refusal on a must_reject case. Only an application/json part is a structured answer, so an empty RAG answer, which still carries its sources and retrieval parts, is a refusal.
• judge:criteria scores criteria cases on expected.criteria, or on the reference answer when a case has none. Its model is picked once per run for the run's tenant, from another family than the models that wrote the answer and than avoidAsJudge, and its calls are charged to that tenant (apps/api/src/eval-judges.ts). A failed pick or judge call gives a failed score with the reason.
• judge:criteria reads a structured answer's fields as JSON when it has no text, and scores an empty answer 0 without asking its model (answerText). A service that answers with a JSON part, such as Klarspråk, was judged on an empty answer, and the judge passed it on the criteria alone.
• The run's provenance names the judge judge:criteria@<version>+<model>; the label queue and gate calibration read it as the evaluator judge:criteria (evaluatorOfJudge).
• The family check compares a judge only with the models that wrote the answer; an embedder that only searched no longer counts.
• Each eval run gets its own judge:criteria, so a model picked for one run is never reused by the next. The API's eval runs get the model catalogue's family resolver.
• Docs: docs/api/work.md, docs/api/README.md (API changes).

How it was tested

• CI on this head: static, size and three test shards pass; shard 4 failed on the console test "previews a new version in English and asks for a reason", which this PR does not touch and which passes locally (the same test failed once on Run the portal's test runs as eval runs, with judges, usage and a lease #194 before). The top of the stack (Delete what genki sent to an agency's Langfuse when it purges the case #197) passes npm run check: 470 files, 5103 tests.

• New tests: "checks the case types a person writes: exact text, which fails without its text, and a refusal that may be no answer at all, but never a structured answer", "checks a judge against the models that wrote the answer, never an embedder that only searched", "scores a criteria case on its criteria under its own identity, with its model in the version", "is not applicable to exact and must_reject cases, or to a case without criteria", "records a failed score when no judge model is available, picking one fails, or the judge call fails", "puts its identity with version and model in the provenance of the run", "gives each run its own evaluators, so a judge picked for one run is never reused by the next", "picks the criteria judge's model for the run's tenant, from another family than the writers and the service's own models, and charges it there", "gives a principal without a tenant no criteria judge", "reads the calibration of a judge that names its version and model from its evaluator", "queues the verdicts of a judge that names its version and model under its evaluator", "judges criteria cases with a model from another family, named in provenance, and falls back to the reference answer", "judges a fields-only answer on its fields, and scores an empty answer 0 without asking the model".

• The last test runs platform/eval through createPlatform, and fails when the composition's family resolver is removed. The review broke 33 rules on purpose and 7 stayed green; the ones it named, the composition's forRun and modelFamily and the reference-answer fallback, are covered by that test, and the reranker exclusion it found dead is gone.

• npm run check passes

Checklist

• Every new rule has a test whose name states it
• User-facing text is in the i18n catalogues, in bokmål and English, with no em-dashes
• Vendor SDKs stay in packages/providers-*
• ADRs, docs and docs/architecture/modules are updated if a contract, boundary or module changed
• No secrets, personal data or local paths

[Copilot]

🟡 Changes recommended

Unresolved evaluation defects prevent judge calls, allow model-selection failures to abort entire runs, and misclassify structured answers as refusals.

3 open issues

• Pass model family resolver to runEval · New
• Handle model selection failures within evaluate try block · New
• Recognize structured answers before marking refusal · New

What changed in this PR

This PR moves portal case grading toward the shared eval runner for #94, building on #192’s criteria storage changes. It adds a run-local criteria judge whose model is recorded in its identity.

Changes:

• Extends contract checks for exact text and empty-answer refusals.
• Adds tenant-scoped criteria judges with model selection, failure scores and provenance.
• Separates answer-writing models from search-only models for judge-family checks.

File	Description
packages/​eval/​src/​strategy.ts	Adds per-run evaluator creation.
packages/​eval/​src/​runner.ts	Uses answer writers for judge-family checks while preserving all served models.
packages/​eval/​src/​evaluators.ts	Updates contract scoring for exact and must-reject cases.
packages/​eval/​src/​eval.test.ts	Tests case grading, writer filtering, judge identity and run isolation.
packages/​eval/​src/​cases.ts	Implements the criteria evaluator and failed-score handling.
docs/​api/​work.md	Documents evaluator choices and judge provenance.
apps/​api/​src/​eval-judges.ts	Creates tenant-scoped judges with tenant-charged calls.
apps/​api/​src/​eval-judges.test.ts	Tests judge selection and tenant context.
apps/​api/​src/​composition.ts	Registers run-local judges with the eval strategy.

_{Review effort: Balanced}

[kentare]

Ready for review. Copilot and a claim-checking review are done (no blind review) and CI passes. Open: the criteria judge's calls count under the builder origin, so /usage leaves them out; for discovered Azure deployments a model's family is its name, so a judge can share a vendor with the writer (this was so before); the guard for a principal without a tenant stays, though the kernel refuses one first.

[kentare]

Found while testing the stack in the portal: judge.criteria passes an empty answer.

• Klarspråk answers with a JSON part ({ text, changes }), so defaultObserve gives answer: '' (packages/eval/src/observe.ts:43 takes text parts only).
• non-empty-answer scores 0 on all 6 cases, but judge.criteria scores 100 %, with reasons such as "Svaret bekrefter at søkeren får 150 000 kroner". The judge prompt gets answer: o.answer (empty) and the reference text (packages/eval/src/evaluators.ts:266), so it is grading the reference.

Fix: fail judge.criteria without calling the model when the answer and fields are both empty, and give the judge the fields when there is no text (for example the text field, or the JSON). A test: a fields-only answer is judged on its fields, and an empty one fails.

[kentare]

Re #193 (comment)

Fixed in 9c6f6cc: judge:criteria reads a structured answer's fields as JSON when it has no text (answerText), and an empty answer scores 0 without a model call. Checked by "judges a fields-only answer on its fields, and scores an empty answer 0 without asking the model" and npm run check on the top of the stack (470 files, 5103 tests).