[kentare]

Stacked on #220.

What and why

Langfuse 4.31.0 removed the unstable evaluator API genki reads its judges through, and genki read the 404 as "no rules". On 4.72.0 Byggeklosser showed no judges, and a ticked run went out to a project whose rules would judge and bill it. Closes #221.

• v2 API. packages/providers-langfuse/src/judges.ts reads /api/public/v2/evaluation-rules and /api/public/v2/evaluators, every page by cursor, and by id for "Dommeren er endret". A judge is an enabled rule with one active LLM-as-a-judge evaluator and the filter isExperimentItemRootSpan = true. Any other enabled rule with an active evaluator, or with an evaluator the listing lacks, is a rule genki cannot read. A rule with more than one evaluator is refused, and so are rules that share a name, which Langfuse allows. A paused evaluator is neither listed nor counted, but it refuses a run that uses it with Langfuse's pausedMessage.
• Unreadable is never empty. A 401, 404 or 5xx refuses a sent run. The start dialog says so before Start (judges-unreadable in GET …/eval-runs/sending), and Byggeklosser shows one refused block per connection (sink-judges:<connection>). 4.4.0 has no v2 API, so it ends up here.
• Labels follow the score name. 4.72.0 names scores after the evaluator. Each judge score records details.scoreName; older scores match the rule name. When two rules write scores under one name, neither gets labels: POST /operator/quality/sink-labels/fetch answers shared, and the console warns why. The 35-character text is gone from the labels help.
• ADR 0040 is amended: v2 routes, the item-root rule, the unreadable refusal, labels by score name, and what an upgrade from 4.4.0 does. That was measured on a throwaway 4.4.0 upgraded to 4.72.0 and checked against Langfuse's backfill migration. A version-2 judge stays at version 2. Its evaluator takes the score name, and rules with the same template, mapping, score name and pause state share one evaluator. A dataset filter stays beside the item-root filter, so that rule is refused as filtered.
• OpenAPI, the API, portal and console i18n catalogues (nb, en) are updated. The live tests default to the local 4.72.0 keys; the Langfuse compose stays outside the repo.

Not in this PR: reading dataset, environment and name filters; the judge set (#218).

How it was tested

• npm run check passes after merging main, which fixed the month-name tests.

• New tests, each seen red first:

  • Provider: "reads every page of rules and evaluators through the v2 API, and lists an enabled rule with one active LLM judge and the item-root filter as a judge", "takes the rule's own variable mapping over its evaluator's, and the evaluator's when the rule has none", "counts an enabled rule without the item-root filter, or with a non-LLM evaluator, as a rule genki cannot read", "neither lists nor counts a rule whose evaluator is paused, and refuses it by Langfuse's message when read for a run", "refuses a rule with more than one evaluator, or with a filter beside the item-root filter", "throws when the rules or evaluators cannot be read (404, 401 or 5xx), never reading them as no rules", "names each score after the judge's evaluator, as Langfuse names it".
  • API: "shows a connection whose Langfuse judges cannot be read as one refused judge block, which cannot be taken in use", "says before a run starts that a connection's Langfuse judges cannot be read, and refuses the run saying so", "refuses a sent run whose judge in use has a paused evaluator, with Langfuse's message", "matches an annotation to a judge by its evaluator's name and type, not the rule's name", "makes no label for rules that share an evaluator, and names them in the fetch result".
  • After the reviews: "shows that a Langfuse judge whose rule shares its name with another cannot be taken in use" (portal), "counts a rule as unreadable when one of its several evaluators is still active, or its evaluator is not listed", "refuses judges whose rules share a name, which genki cannot tell apart", "names a score name that two rules write as shared, also when their evaluators differ", "sends a run beside a judge whose evaluator is paused and not taken in use, and counts no cost for it", "matches a label by the judge's rule name on a run whose scores did not record their name, as 4.4.0 named them", plus the paused and lost-filter cases in "marks scores judge changed".
  • Portal and console: "shows a connection whose Langfuse judges cannot be read as a refused block saying why", "says in the start dialog that a connection's Langfuse judges cannot be read", "says after a fetch which evaluators got no labels because rules share them".

• The fakes in judges.test.ts and quality.test.ts now answer the v2 routes, with shapes measured on a local 4.72.0.

• Live tests ran with LANGFUSE_LOCAL_URL against a local 4.72.0: "lists an item-root rule of a live Langfuse 4.72.0 as a judge, and counts a rule without the filter as unreadable", the connection check, and a traced run that lands as an experiment and is purged. The judge-trace deletion test now waits up to 4 minutes. Langfuse's worker deletes on a cycle of about 2 minutes, and the old 60 s wait failed in about half the runs.

• npm run check passes

Checklist

• Every new rule has a test whose name states it
• User-facing text is in the i18n catalogues, in bokmål and English, with no em-dashes
• Vendor SDKs stay in packages/providers-*
• ADRs, docs and docs/architecture/modules are updated if a contract, boundary or module changed
• No secrets, personal data or local paths

[kentare]

Ready for review. Blind review, a claim-checking review and a check of the fixes are done and CI passes.

Open:

• After the upgrade from 4.4.0, an experiment rule keeps its dataset filter beside the item-root filter, so genki refuses it as filtered. Reading dataset filters is proposed separately, so such a rule needs its dataset filter removed in Langfuse before it can be taken in use.
• The issue says an upgraded judge's version restarts at 1. Measured on an upgraded copy, and in Langfuse's backfill migration, it keeps its version. Its evaluator id changes, so a run that spans the upgrade shows "Dommeren er endret". ADR 0040 says so.
• The local Langfuse compose stays outside the repo. The live tests default to the local 4.72.0 keys and run only with LANGFUSE_LOCAL_URL.