Bring Langfuse annotations back as calibration labels an operator accepts by kentare · Pull Request #220 · digdir/genki

kentare · 2026-11-02T09:18:53Z

Stacked on #207.

What and why

Agencies label answers in Langfuse, and those labels should calibrate their Langfuse judges in genki, but only once someone with the labelling right has accepted them (ADR 0040). Part of #98.

A tracing connection gets a labels switch (labels). In the console's new "Fra Langfuse" tab, an operator fetches a connection's annotations (POST /operator/quality/sink-labels/fetch). An ANNOTATION whose score config has a judge's exact name and type, on a case trace a run sent, becomes a pending label paired with the judge's own score on that case; one without a judge score is unpaired. Annotations on other traces are ignored.
An operator accepts or rejects a batch (one connection and judge) with a reason: platform.judge.label.accept and .reject, in the audit log. Only the labels shown are decided. An edit in Langfuse comes back pending, a delete stops counting, and a Langfuse that cannot be read (502) changes nothing.
Labels are kept under langfuse:<tenant>/<connection>/<rule>, so a new judge version takes them over and another agency's never count. The labeller is an HMAC pseudonym with a key per agency.
A boolean label is good or bad, a number is split at 0.5 within its score config's range (no range, no label), and a judge's own number without a range is split with the label's. A case sent with an empty answer can be labelled, as unpaired.
The run result shows each Langfuse judge's agreement: κ, share alike, labels (and how many another version scored), unpaired.
New append-only sink_labels and sink_label_keys tables with tenant isolation; OpenAPI, i18n catalogues and ADR 0040 and 0003 updated.

Langfuse 4.4.0 takes no score config name over 35 characters, so a judge with a longer rule name gets no labels; the switch's help text says so.

:camera_with_flash: Screenshots

Tracing: the labels switch on a connection

Console, Fra Langfuse: three labels made in a local Langfuse, waiting

Run result: the judge's agreement once the batch is accepted

How it was tested

npm run check: everything passes except apps/console/src/app.test.tsx and apps/workspace/src/projects.test.tsx, which expect October month names and fail on every branch since 1 November, main included.
New tests, each seen failing first: "turns an ANNOTATION with a judge's exact name and type into a pending label for that judge and case", "counts a label only after its batch is accepted with a reason, in the audit log" (API and console), "keeps the labeller as a pseudonym per agency", "counts a label without a judge score as unpaired", "ignores annotations on traces genki did not send", "lets a new judge version take over labels only from the same tenant, connection and name", "shows a Langfuse judge's agreement with its accepted labels", "turns down a batch with a reason, in the audit log, and does not bring its labels back unless they are edited", "brings an annotation edited in Langfuse back as pending, and stops counting one deleted there", "brings labels back from a tracing connection only while its labels switch is on" (API and portal), "keeps each state a sink label goes through, lists the latest per tenant, and keeps one pseudonym key per tenant" (memory and Postgres).
Against a local Langfuse 4.4.0: an annotation made in the UI was read through the provider with its trace, author and label; editing it kept updatedAt (so a revision includes the label), and deleting it removed it from the API at once.
End to end on a separate stack (API 8090, portal 5175, console 5174, mock login) against the local Langfuse with its real judge, renamed to svar_oppfyller_kriteriene for the check, since the 39-character name gets no labels: a Klarspråk run sent to Langfuse, three annotations made in Langfuse's UI, fetched and accepted in the console as Ola Drift, and the run result showed "Samsvar med godkjente etiketter: κ 0,40 · 67 % like · 3 etiketter".
To try it: npm start, Kari Testansatt in the portal (Integrasjoner, Sporing: connect a Langfuse with "Hent etiketter" on; take its judge in use in Byggeklosser; run a test set with "Send kjøringen til Langfuse"), annotate in Langfuse, then Ola Drift in the console (Kvalitetsmatrise og dommere, "Fra Langfuse": Hent nye etiketter, Godta…).

Checklist

npm run check passes
Every new rule has a test whose name states it
User-facing text is in the i18n catalogues, in bokmål and English, with no em-dashes
Vendor SDKs stay in packages/providers-*
ADRs, docs and docs/architecture/modules are updated if a contract, boundary or module changed
No secrets, personal data or local paths

npm run check fails only on the two month-name tests named above. One rule has no test, by choice: a case sent with an empty answer keeps its label, as unpaired (details.sent on the judge's score).

…epts An annotation whose score config has a Langfuse judge's exact name and type, on a case trace a run sent, becomes a pending label for that judge and case. An operator fetches them per connection in the console and accepts or rejects each batch with a reason, in the audit log. Edits come back pending, deletes stop counting, labellers are pseudonyms per agency, and the run result shows each judge's agreement with its accepted labels. Labels are kept under langfuse:<tenant>/<connection>/<rule>, so a new judge version takes them over and no other tenant's do. ADR 0040 and 0003 amended.

…e labels shown A Langfuse judge's number without a score config is split with the annotation's range. Every Langfuse judge score carries its data type, so labels pair before the judge has scored any case. Accepting or rejecting takes the labels the operator was shown, so one fetched since is not decided unseen. The console lists connections as a table.

Copilot

🟡 Changes recommended

Concurrent fetch and decision updates can overwrite newer states, causing deleted labels to count again or accepted labels to stop counting before their edits are accepted.

4 open issues

What changed in this PR

Adds operator-approved Langfuse annotations to genki’s judge calibration workflow, extending the judge scoring and cost reporting from #207 toward #98.

Changes:

Imports annotations as pending labels, with audited acceptance or rejection and handling for edits and deletions.
Adds tenant-isolated label storage, pseudonymous labellers, and agreement statistics shared across judge versions.
Adds console batch review, a portal connection switch and agreement display, plus tests, translations, OpenAPI definitions and ADR updates.

File	Description
`packages/providers-postgres/src/schema.ts`	Adds label tables, isolation and append-only enforcement.
`packages/providers-postgres/src/isolation.test.ts`	Covers isolation of the new tables.
`packages/providers-postgres/src/index.ts`	Exposes and constructs the label store.
`packages/providers-postgres/src/eval.ts`	Implements persistent labels and pseudonym keys.
`packages/providers-langfuse/src/judges.ts`	Reads annotations and preserves judge type metadata.
`packages/providers-langfuse/src/index.ts`	Exports the annotation type.
`packages/eval/src/calibration.ts`	Defines sink labels, memory storage and agreement calculations.
`docs/adr/0040-evaluating-with-langfuse.md`	Documents label import and operator approval.
`docs/adr/0003-evaluation-is-part-of-the-platform.md`	Clarifies label reuse across Langfuse judge versions.
`apps/portal/src/screens/quality/RunHistory.tsx`	Displays accepted-label agreement.
`apps/portal/src/screens/quality/live-quality.test.tsx`	Tests agreement display variants.
`apps/portal/src/screens/integrations/LiveTracing.tsx`	Adds the connection’s labels switch.
`apps/portal/src/screens/integrations-tracing.test.tsx`	Tests enabling label import.
`apps/portal/src/screens/changes/changes.ts`	Maps label decision audit actions.
`apps/portal/src/i18n/nb/operate-quality.ts`	Adds Norwegian agreement text.
`apps/portal/src/i18n/nb/operate-integrations.ts`	Adds Norwegian label-switch guidance.
`apps/portal/src/i18n/nb/core-screens.ts`	Adds Norwegian decision action names.
`apps/portal/src/i18n/en/operate-quality.ts`	Adds English agreement text.
`apps/portal/src/i18n/en/operate-integrations.ts`	Adds English label-switch guidance.
`apps/portal/src/i18n/en/core-screens.ts`	Adds English decision action names.
`apps/portal/src/api/tracing.ts`	Adds the connection labels flag.
`apps/portal/src/api/quality.ts`	Defines agreement response types.
`apps/console/src/types.ts`	Defines label batches and connection listings.
`apps/console/src/screens/SinkLabels.tsx`	Adds fetching and batch decision controls.
`apps/console/src/screens/QualityMatrix.tsx`	Adds the Langfuse labels tab.
`apps/console/src/screens/Audit.tsx`	Maps acceptance and rejection audit actions.
`apps/console/src/screens.test.tsx`	Tests reason-required batch acceptance.
`apps/console/src/mock.ts`	Mocks label fetching and decisions.
`apps/console/src/i18n/nb/qualityMatrix.ts`	Adds Norwegian label-review text.
`apps/console/src/i18n/nb/audit.ts`	Adds Norwegian audit action names.
`apps/console/src/i18n/en/qualityMatrix.ts`	Adds English label-review text.
`apps/console/src/i18n/en/audit.ts`	Adds English audit action names.
`apps/console/src/example-data.ts`	Supplies example pending label batches.
`apps/console/src/api.ts`	Adds fetch and decision API calls.
`apps/api/src/trace-connections.ts`	Extends connection and sink contracts for labels.
`apps/api/src/trace-connections.test.ts`	Checks the default labels flag.
`apps/api/src/stores.contract.test.ts`	Tests label storage, keys and append-only enforcement.
`apps/api/src/sink-labels.ts`	Implements reconciliation, decisions and agreement lookup.
`apps/api/src/sink-labels.test.ts`	Tests pseudonyms and judge identity scoping.
`apps/api/src/service-eval.ts`	Includes agreement in run summaries and details.
`apps/api/src/routes/console-routes.ts`	Adds operator label endpoints and decision handlers.
`apps/api/src/rights.ts`	Restricts label decisions to operators.
`apps/api/src/quality.ts`	Loads agreement for run responses.
`apps/api/src/quality.test.ts`	Tests annotation import and approval lifecycle.
`apps/api/src/platform-modules.ts`	Adds the label service dependency.
`apps/api/src/openapi/work.ts`	Documents operator label endpoints.
`apps/api/src/openapi/services.ts`	Documents agreement response fields.
`apps/api/src/openapi/access.ts`	Documents the connection labels flag.
`apps/api/src/main.ts`	Wires persistent label storage into startup.
`apps/api/src/i18n/nb/errors.ts`	Adds Norwegian label-operation errors.
`apps/api/src/i18n/en/errors.ts`	Adds English label-operation errors.
`apps/api/src/composition.ts`	Wires providers, storage and label services together.
`apps/api/src/change-reason.ts`	Exempts fetching from the change-reason requirement.
`apps/api/src/admin-tasks.test.ts`	Includes label decisions in admin-task coverage.

_{Review effort: Balanced}

…swers A missing annotation page or score config stops the fetch (502) instead of reading as no labels. Every annotation still in Langfuse counts as seen. A number whose config has no range is no label. A case sent with an empty answer is marked sent on its score, so its label comes in as unpaired. Authorless annotations get one pseudonym each, labels that are not a list are refused, and sink_label_keys is append-only.

kentare · 2026-11-02T09:58:28Z

Ready for review. Blind review and a claim-checking review are done. Copilot did not review: its monthly quota was used up. CI fails only on apps/console/src/app.test.tsx and apps/workspace/src/projects.test.tsx, which expect October month names and fail on main too (#219 fixes them).

Open:

A judge whose rule name is over 35 characters gets no labels, since Langfuse 4.4.0 takes no longer score config name; the switch's help text says so. Byggeklosser should say it on such a judge (Take a project's Langfuse judges in use as one judge set #218).
A case sent with an empty answer keeps its label as unpaired; that rule has no test, by choice.
apps/api/src/main.ts conflicts with Connect the API and worker to a knowledge server through their own roles #202 and Store each generation as its own partition on the knowledge server #208 by one line each; keep both sides.

… with the live label A fetch or decision on a connection's labels runs alone under an advisory lock (in memory, a queue), so a stale read never overwrites a newer state. An annotation edited back to an earlier value is compared with the label as it stands, so it comes back pending. A judge score that lost its data type, such as a timed-out one, takes it from the judge's other scores in the run. Labels are kept per connection, so two connections to one project do not overwrite each other.

kentare · 2026-11-02T10:09:28Z

Correction to the comment above: Copilot did review this PR. Its four points (concurrent fetch and decision, a timed-out score losing its data type, an edit back to an earlier value, the same annotation through two connections) are fixed in 3788d70, answered in their threads and resolved. CI is unchanged: only the two month-name tests fail.

kentare · 2026-11-02T10:15:45Z

Checked against a local Langfuse 4.72.0 (#221), with a test rule, one experiment item and an annotation made in the UI:

Score names come from the evaluator, not the rule. A rule named regel_98c_navnesjekk with the evaluator svar_oppfyller_kriteriene wrote a score named svar_oppfyller_kriteriene (same job_configuration_id as the rule). This PR matches an annotation to a judge by the rule's name, so on 4.72.0 no annotation matches unless rule and evaluator share a name. Two rules with one evaluator write the same score name, so an annotation cannot tell them apart.
The 35-character limit is gone. 4.72.0 accepts a score config named svar_oppfyller_kriteriene-eksperimenter (39 characters); 4.4.0 refused it. The labels switch's help text and ADR 0040 state the limit.
Annotation shape: an annotation made on the trace sits on the root observation: subject: {kind: 'observation', id, traceId}. The provider reads subject.traceId, so this works. authorUserId, configId, dataType and value are as on 4.4.0.
Edits change updatedAt on 4.72.0 (4.4.0 kept it). The revision still includes the label, so both versions work.
Clearing an annotation removes it from v3/scores?source=ANNOTATION at once, as on 4.4.0.

Fixes needed: match an annotation to the judge by its evaluator's name (refusing labels when two rules in the project share an evaluator), and drop the 35-character limit from the help text and ADR 0040 once the instance runs 4.72.0.

kentare · 2026-11-02T10:17:13Z

The label fix for 4.72.0 (match by the evaluator's name, refuse labels when two rules share an evaluator, drop the 35-character text) is in #221, point 7.

kentare added 2 commits November 2, 2026 09:58

kentare requested a balanced review from Copilot November 2, 2026 09:19

Copilot started reviewing on behalf of kentare November 2, 2026 09:19

View session

Copilot AI reviewed Nov 2, 2026

View reviewed changes

Comment thread apps/api/src/sink-labels.ts Resolved

Comment thread apps/api/src/sink-labels.ts Resolved

Comment thread packages/providers-langfuse/src/judges.ts Resolved

Comment thread packages/providers-postgres/src/eval.ts Resolved

kentare mentioned this pull request Nov 2, 2026

Read Langfuse judges through the v2 evaluation API (Langfuse 4.72.0) #221

Open

10 tasks

Merge feat/langfuse-judge-cost into feat/langfuse-labels

dd1e322

kentare added this pull request to stack #199 November 2, 2026 10:40

This was referenced Nov 2, 2026

Read Langfuse judges through the v2 evaluation API #226

Draft

Give each test set a Langfuse dataset and mark sent runs with it #227

Draft

kentare added 3 commits November 2, 2026 13:00

Merge feat/langfuse-judge-cost into feat/langfuse-labels

800e01b

Merge feat/langfuse-judge-cost into feat/langfuse-labels

c4bae5d

Merge feat/langfuse-judge-cost into feat/langfuse-labels

147f1b4

kentare removed this pull request from stack #199 November 2, 2026 12:48

kentare added this pull request to stack #231 November 2, 2026 12:48

kentare added 2 commits November 2, 2026 19:00

Merge feat/langfuse-judge-cost into feat/langfuse-labels

cc3d5cf

Merge feat/langfuse-judge-cost into feat/langfuse-labels

3fb1479

Navigation Menu

Repository navigation

Bring Langfuse annotations back as calibration labels an operator accepts - #220

Bring Langfuse annotations back as calibration labels an operator accepts#220
kentare wants to merge 10 commits into
feat/langfuse-judge-costfrom
feat/langfuse-labels

kentare commented Nov 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kentare commented Nov 2, 2026

Uh oh!

kentare commented Nov 2, 2026

Uh oh!

kentare commented Nov 2, 2026

Uh oh!

kentare commented Nov 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

kentare commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What and why

How it was tested

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kentare commented Nov 2, 2026

Uh oh!

kentare commented Nov 2, 2026

Uh oh!

kentare commented Nov 2, 2026

Uh oh!

kentare commented Nov 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kentare commented Nov 2, 2026 •

edited

Loading