Repository navigation
Conversation
A Langfuse judge scores a trace a few seconds after it arrives and cannot be asked on request, while evaluator@1 only allowed an answer within one request. evaluator@2 replaces it (ADR 0040, issue #96). - An evaluator answers at once or pending, and the runner asks again with collect every 5 seconds, for at most 5 minutes from when the last case was sent. What has not arrived then is recorded as missing, and the run's incomplete names the judge and the count ("no score from judge:x for 20 of 84 cases"). - A score has a state: scored, not-applicable, pending, failed or missing. Only a scored score has a value, and only a not-applicable one does not apply. - A gate whose metric has a missing or failed score in its split fails with that count, and never averages over what arrived. - GET /eval-tasks/{task} answers waiting-for-scores with how many are pending, on the instance running the eval. #94 stores the run's state under a lease, so a restart during the wait is handled there. - An HTTP judge answers pending with a ticket and is collected in batches of 50 (docs/bring-your-own-logic.md). One contract suite covers immediate and pending answers. - Registering a plug-in under evaluator@1 is refused, and one stored under it judges nothing until it is registered again under evaluator@2.
6 tasks done
There was a problem hiding this comment.
🟡 Changes recommended
The no-regression gate can pass with an incomplete baseline—even without any comparison—leaving a critical correctness issue unresolved.
7 open issues
What changed in this PR
This PR introduces evaluator@2 so evaluation runs can collect scores asynchronously, supporting judges such as Langfuse. It lays the groundwork for the unified eval runner and tracing integrations in #94 and #97.
Changes:
- Adds explicit score states, deferred collection, incomplete-run summaries, and waiting status.
- Updates publishing gates to reject missing or failed candidate scores.
- Adds ticket-based HTTP collection, rejects obsolete evaluator contracts, and updates tests and documentation.
| File | Description |
|---|---|
packages/jobs/src/evaluators.ts |
Adds explicit states to job evaluator scores. |
packages/eval/src/strategy.ts |
Emits progress and waiting notices. |
packages/eval/src/service.ts |
Exposes waiting status and pending counts. |
packages/eval/src/schema.ts |
Defines score states and incomplete-run summaries. |
packages/eval/src/runner.ts |
Collects deferred scores and records missing results. |
packages/eval/src/platform.test.ts |
Tests waiting status through the eval service. |
packages/eval/src/language-gates.test.ts |
Updates fixtures for the score schema. |
packages/eval/src/gates.ts |
Checks gated metrics for unscored results. |
packages/eval/src/evaluators.ts |
Extends the evaluator interface with collection. |
packages/eval/src/eval.test.ts |
Tests deferred scoring, deadlines, notices, and gates. |
docs/bring-your-own-logic.md |
Documents the HTTP judge protocol. |
docs/architecture/system-atlas.html |
Updates evaluator descriptions and score schema. |
docs/architecture/modules/eval.yaml |
Documents deferred evaluator support. |
docs/api/work.md |
Explains waiting status and incomplete scores. |
apps/api/src/stores.contract.test.ts |
Updates stored-run fixtures. |
apps/api/src/routes/console-routes.test.ts |
Updates console score and run fixtures. |
apps/api/src/quality.test.ts |
Updates the eval-run fixture. |
apps/api/src/plugins.ts |
Rejects incompatible plug-in contracts. |
apps/api/src/plugin-routing.test.ts |
Tests deferred HTTP judges and contract rejection. |
apps/api/src/plugin-contracts.ts |
Implements evaluator@2 checks and batched collection. |
apps/api/src/plugin-contracts.test.ts |
Tests immediate and deferred contract responses. |
apps/api/src/openapi/work.ts |
Documents the waiting-status response. |
apps/api/src/i18n/nb/errors.ts |
Adds the Bokmål outdated-contract message. |
apps/api/src/i18n/en/errors.ts |
Adds the English outdated-contract message. |
apps/api/src/external-plugins.ts |
Uses the new contract and excludes obsolete evaluators. |
apps/api/src/external-plugins.test.ts |
Tests re-registration of stored obsolete evaluators. |
Review effort: Balanced
- The runner stops asking at the deadline, bounds each collect by the time left and takes nothing that arrives after it. A collect call still open at the deadline is cancelled. - Cancelling the eval task stops the wait: the task's signal reaches the runner, its pauses and the judge's requests. - Only the collect call is caught, so a collected score that cannot be recorded on its case fails the run, as an immediate score does. - A no-regression gate fails when its baseline has missing or failed scores in the gated split, naming the count. - The contract check collects the ticket the pending answer gave, instead of asking again. - An HTTP judge's score needs a value or a state, and a failed collect batch no longer discards the batches that arrived.
- A malformed score in a collect answer fails its own case with the reason; the judge's other scores are kept. - Cancelling an eval run stops it before the next case and the next evaluator, and an HTTP judge's requests get the run's signal when it judges and when it collects, also in the registration check. - A score stored before scores had a state reads with the state its value implies, so older run files still parse. - Tests now state the rules that had none: a pause cut short by a cancel, a collect cancelled at the deadline, a score answered pending again, a cancelled eval task, a pending score without a ticket, at most 50 tickets per call, and the judge sender's signal. - docs/api/README.md lists the evaluator@2 changes.
Contributor
Author
|
Ready for review. Copilot and a claim-checking review are done (no blind review) and CI passes. Open: a run's |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
A Langfuse judge scores a trace a few seconds after it arrives and cannot be called on request, but
evaluator@1only allowed an answer within one request. ADR 0040 replaces it withevaluator@2. Closes #96. First of a stack: #94 and #97 follow on top.pending, and the runner asks again withcollectevery 5 seconds, for at most 5 minutes from when the last case was sent (packages/eval/src/runner.ts). Each collect gets a signal that aborts at the deadline, and nothing that arrives after it is taken. What has not arrived is recorded asmissing, and the run'sincompletenames the judge and the count ("no score from judge:x for 20 of 84 cases").state:scored,not-applicable,pending,failedormissing(packages/eval/src/schema.ts). A score stored before reads with the state its value implies.GET /eval-tasks/{task}answerswaiting-for-scoreswith how many are pending, on the instance running the eval.docs/bring-your-own-logic.mddocuments the protocol. One contract suite covers immediate and pending answers, and it collects the ticket it was given.evaluator@1is refused, and one stored under it judges nothing until it is registered again.docs/api/work.md,docs/api/README.md(API changes),docs/bring-your-own-logic.md,docs/architecture/modules/eval.yaml. i18n:apps/api/src/i18n/{en,nb}/errors.ts.Not in this PR: a restart during the wait marking the run interrupted. That needs the lease on
EvalRun, which #194 adds.How it was tested
CI on this head:
checkand the three test shards pass. The top of the stack (Delete what genki sent to an agency's Langfuse when it purges the case #197) with main merged passesnpm run check: 470 files, 5092 tests.New tests: "lands a pending judge's scores in the run and its metrics once they are collected", "records a score still missing 5 minutes after the last case was sent as missing, naming the judge and the count", "takes no score that arrives after the deadline, and bounds each collect by the time left", "cancels a collect call still open at the deadline", "stops running cases once the eval run is cancelled", "cuts a pause for scores short when the eval run is cancelled", "stops asking the judge once the eval task is cancelled", "fails the run when a collected score cannot be recorded on its case, as for an immediate score", "records nothing for a score the judge answers pending again", "reads a run stored before scores had a state, or before incomplete existed", "fails a gate whose metric has a missing or failed score in its split, in the run or its baseline, naming the count", "passes an evaluator that answers pending and delivers when collected", "fails when the ticket it was given is never collected, even if asking again answers at once", "refuses a score with neither a value nor a state, instead of reading it as not applicable", "refuses a pending score without a ticket", "keeps the scores of the batches that arrived when a later batch fails", "records a malformed collected score as failed with the reason, and keeps the rest", "passes the signal on to every request, when it judges and when it collects", "asks for at most 50 tickets per collect call", "passes the signal on to the egress request", "judges nothing with a stored evaluator@1 plug-in until it is registered again under evaluator@2", "refuses an evaluator plug-in that claims evaluator@1", "answers that a run is waiting for scores, and for how many, while its judge has not delivered".
Rules broken on purpose: each turns a test red. The review found seven that no test stated (a cancel cutting the pause short, a collect cancelled at the deadline, a score answered pending again, a cancelled eval task, a pending score without a ticket, the request signal, a run stored before
incomplete); each now has the test above that fails when it is broken.npm run checkpassesChecklist
packages/providers-*docs/architecture/modulesare updated if a contract, boundary or module changed