[kentare]

Stacked on #191.

What and why

A criteria case kept its criteria in referenceAnswer, so a reference answer and the criteria a judge checks were the same field. The nynorsk test set, for example, held "Svaret er skrive på nynorsk ..." there. ADR 0003 as amended by ADR 0040 moves them to expected.criteria, named fields such as criteria or supporting_quotes. Part of #94.

• A case reads its criteria from expected.criteria, and referenceAnswer holds only a reference answer. A new criteria case from the quality page, an import or an accepted candidate writes expected.criteria.
• Test sets are upgraded on read: a type: criteria case with its criteria in referenceAnswer is read with them in expected.criteria. Both test set stores parse what they read, so stored versions keep their numbers. packages/eval/testsets/nynorsk-sporsmal.yaml is rewritten in the new shape.
• POST /tenants/{tenant}/test-sets/{setId}/cases takes expected as a text, as before, or as { referenceAnswer, criteria }, kept as they are. Given as fields, a criteria case needs criteria and an exact case referenceAnswer, and an expected that names neither is refused.
• An import keeps every named criterion and the reference answer beside them, and refuses a row whose criteria name judges do not read.
• The case editor saves the stored case with only the fields it edits changed, since a save replaces the whole case. It picks its field by the case's type, labels it "Kriterier" for a criteria case and asks for criteria rather than saving one without them.
• The criteria judge reads expected.criteria.criteria, and the reference answer when a case has none, so the golden sets are judged as before.
• An evaluator plug-in receives the case's criteria beside its reference answer (docs/bring-your-own-logic.md).
• Docs: docs/api/work.md, OpenAPI. i18n: apps/api/src/i18n/{en,nb}/errors.ts, portal data-config and operate-quality.

How it was tested

• CI on this head: check and the three test shards pass. The top of the stack (Delete what genki sent to an agency's Langfuse when it purges the case #197) with main merged passes npm run check: 470 files, 5092 tests.

• New tests: "keeps the criteria of a new criteria case in expected.criteria, and an exact text as the reference", "keeps a reference answer and named criteria given together, as an import brings them", "reads a criteria case written before expected.criteria with its criteria moved there", "judges a criteria case on its criteria, and on its reference answer when it has none", "names criteria fields as judges read them", for both stores "reads a test set stored before expected.criteria with its criteria moved there", "adds a case with a reference answer and named criteria as they are, and refuses a shape its type cannot be judged on", "keeps every named criterion and a reference answer beside them, so none is dropped on import", "refuses a row whose criteria name is not one judges read, rather than importing half the file", "keeps what the form does not edit when it saves a case: the reference answer beside the criteria, its type and tags", "saves an unchanged case as it is stored, and asks for criteria rather than saving a criteria case without them", "shows a criteria case by its criteria, and by its reference answer when it has none", "sends a case's reference answer and named criteria to the judge, and leaves out what the case lacks".

• Rules broken on purpose: the review broke 21; the five that no test stated now have the tests above. Two are equivalent mutants, since every parse upgrades a case.

• npm run check passes

Checklist

• Every new rule has a test whose name states it
• User-facing text is in the i18n catalogues, in bokmål and English, with no em-dashes
• Vendor SDKs stay in packages/providers-*
• ADRs, docs and docs/architecture/modules are updated if a contract, boundary or module changed
• No secrets, personal data or local paths

[Copilot]

🟡 Changes recommended

Saving or importing cases can silently discard reference answers and supporting quotes, while imports with only supporting quotes are incorrectly rejected.

2 open issues

• Preserve reference answer when saving cases with criteria · New
• Preserve structured expectations during case import · New

What changed in this PR

This PR separates evaluation criteria from reference answers by introducing expected.criteria, advancing the test-case migration in #94 on top of #191.

Changes:

• Adds named criteria fields and upgrades legacy criteria cases when test sets are read.
• Updates case creation, judging, and HTTP evaluator payloads to use the separate criteria.
• Updates portal handling, the Nynorsk test set, tests, and documentation.

File	Description
packages/​providers-postgres/​src/​eval.ts	Parses stored test sets to migrate legacy criteria on read.
packages/​eval/​testsets/​nynorsk-sporsmal.yaml	Moves Nynorsk criteria into the new structure.
packages/​eval/​src/​testsets.test.ts	Checks the migrated Nynorsk expectations.
packages/​eval/​src/​store.ts	Applies read-time migration in the memory store.
packages/​eval/​src/​schema.ts	Defines named criteria, legacy migration, and expectation creation.
packages/​eval/​src/​feedback.ts	Writes accepted candidates’ criteria separately.
packages/​eval/​src/​cases.ts	Updates case creation, display, and grading.
packages/​eval/​src/​cases.test.ts	Tests criteria storage, migration, validation, and judging.
docs/​bring-your-own-logic.md	Documents criteria in evaluator requests.
docs/​architecture/​system-atlas.html	Updates the generated expectation schema.
apps/​portal/​src/​types.ts	Adds criteria to portal test-case types.
apps/​portal/​src/​screens/​quality/​TestSetsTab.tsx	Displays criteria from the new field.
apps/​portal/​src/​screens/​quality/​import-cases.ts	Reads criteria text during imports.
apps/​portal/​src/​screens/​config/​TestSetEditor.tsx	Adds criteria handling to editor drafts and saves.
apps/​api/​src/​stores.contract.test.ts	Tests legacy migration across both stores.
apps/​api/​src/​plugin-contracts.ts	Sends named criteria to HTTP evaluators.

_{Review effort: Balanced}

[kentare]

Ready for review. Copilot and a claim-checking review are done (no blind review) and CI passes. Open: test sets are upgraded on read rather than migrated, so stored versions keep their numbers; the mock's flattening of a structured expected has no test of its own.