Repository navigation
Conversation
A run started from the portal can now go to the connection that covers the service, as ADR 0040 describes (part of #97). - With trace: true, each case except the holdout is sent as its own trace, one item of the run's experiment (item <case>#<repeat>). The trace starts at the case's task, with the tasks under it, although that task is a sub-task of the run's. - The run, case, repeat and split are recorded beside the case task's input as an audit field (evaluation), never in it: the strategy does not receive it, and the timeline and trace input leave it out. - Each run's cases go only to the exporter opened for that run, with the connection's keys resolved for its host and every call through the egress path. The sink is flushed when the run ends. - A traced run is refused (409) when no connection covers the service or the service's residency does not allow the connection's. Until judges map fields, the traces carry no text. - A draft or version that cannot run answers 422 with the reasons, instead of 500. - The portal offers to send a run, unticked by default, when a connection covers the service. Tested against a local Langfuse 4.4.0: one trace per case and repeat, no holdout case, no input or output.
This was referenced Oct 29, 2026
There was a problem hiding this comment.
🟡 Changes recommended
Critical issues allow DNS rebinding to bypass egress protections and omitted draft residency to permit trace export outside the service’s effective residency.
2 open issues
What changed in this PR
Adds opt-in export of service evaluation runs to the agency’s Langfuse connection, building on connection management from #195 toward #97.
Changes:
- Routes each non-holdout case and repeat into one run-level experiment, keeping evaluation metadata separate from strategy input.
- Adds connection and residency checks, exporter lifecycle handling, and 422 responses for definitions that cannot run.
- Adds the portal checkbox, translations, API documentation, and tests.
| File | Description |
|---|---|
packages/providers-langfuse/src/spans.ts |
Adds experiment and item attributes to spans. |
packages/providers-langfuse/src/index.ts |
Exports the experiment router. |
packages/providers-langfuse/src/exporter.ts |
Tracks experiment context and case-rooted traces. |
packages/providers-langfuse/src/experiment.ts |
Routes run events to dedicated exporters. |
packages/providers-langfuse/src/experiment.test.ts |
Tests trace structure and sink isolation. |
packages/kernel/src/spi.ts |
Adds evaluation context to invocation options. |
packages/kernel/src/kernel.ts |
Records evaluation context beside audited input. |
packages/kernel/src/kernel.test.ts |
Tests separation of evaluation context from strategy input. |
packages/eval/src/strategy.ts |
Attaches tracing context to non-holdout cases. |
packages/eval/src/schema.ts |
Records the run’s trace connection. |
packages/eval/src/runner.ts |
Passes case and repeat details to targets. |
packages/eval/src/eval.test.ts |
Tests case context and holdout exclusion. |
packages/contracts/src/index.ts |
Defines the shared evaluation context. |
docs/architecture/system-atlas.html |
Adds the evaluation sink configuration field. |
docs/api/work.md |
Documents traced runs and refusal responses. |
apps/portal/src/screens/quality/RunHistory.tsx |
Adds the opt-in Langfuse checkbox. |
apps/portal/src/screens/quality/live-quality.test.tsx |
Tests checkbox visibility and trace requests. |
apps/portal/src/i18n/nb/operate-quality.ts |
Adds the Norwegian tracing label. |
apps/portal/src/i18n/en/operate-quality.ts |
Adds the English tracing label. |
apps/portal/src/api/quality.ts |
Accepts the optional trace request flag. |
apps/api/src/trace-connections.ts |
Adds service connection lookup and sink lifecycle types. |
apps/api/src/service-eval.ts |
Checks tracing eligibility and manages run sinks. |
apps/api/src/quality.ts |
Handles tracing requests and refusal responses. |
apps/api/src/quality.test.ts |
Tests tracing, refusals, and local Langfuse export. |
apps/api/src/platform-modules.ts |
Exposes trace sinks to platform modules. |
apps/api/src/openapi/services.ts |
Documents the trace flag and error responses. |
apps/api/src/i18n/nb/errors.ts |
Adds Norwegian run refusal titles. |
apps/api/src/i18n/en/errors.ts |
Adds English run refusal titles. |
apps/api/src/composition.ts |
Wires per-run exporters, secrets, and egress. |
Review effort: Balanced
…feat/langfuse-export # Conflicts: # apps/api/src/quality.test.ts # apps/api/src/quality.ts
- The residency check uses the definition as it runs, so a draft that leaves its residency out is checked against the residency filled from its model; a refused run releases its prepared draft. - A service that two connections cover is sent to neither: the run answers 409 naming both, instead of the first one getting it. - The exporter's requests go through the egress guard's request, pinned to the address it checked and within its time and size limits, instead of a second lookup by native fetch.
# Conflicts: # apps/api/src/service-eval.ts # packages/eval/src/strategy.ts
# Conflicts: # docs/architecture/system-atlas.html
# Conflicts: # apps/api/src/quality.ts # apps/api/src/service-eval.ts # apps/portal/src/screens/quality/live-quality.test.tsx
- Only a run's own exporter exports its cases as experiment items; the platform-wide exporter leaves them as ordinary traces, so no tenant's run lands in Digdir's project. - A run's sink takes cases only from the tenant it was opened for. - The portal's "send to Langfuse" box is unticked for every new run. - A traced run of only the locked cases records no sink, since none of them is sent. - The refusals of a traced run read in bokmål, the label says "Låste testtilfeller", and the local Langfuse test checks the experiment's item count. providers-langfuse lists experiment.ts.
# Conflicts: # apps/api/src/i18n/en/errors.ts # apps/api/src/i18n/nb/errors.ts
Contributor
Author
|
Ready for review. Copilot and a claim-checking review are done (no blind review) and CI passes. Open: the portal shows a refused run's title but not its reasons; until judges map fields (#98) the traces carry no text, so Langfuse's judges score empty traces. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #195.
What and why
A run started from the portal can now go to the Langfuse connection that covers the service, as one experiment, as ADR 0040 describes. Part of #97.
trace: trueonPOST /tenants/{tenant}/services/{id}/eval-runs, each case except the locked ones is sent as its own trace, one item of the run's experiment (item<case>#<repeat>). The trace starts at the case's task, with the tasks under it. A run of only the locked cases sends nothing and records no sink.evaluation), never in it: the strategy does not receive it, and the timeline and the trace's input leave it out.docs/api/work.md, OpenAPI,docs/architecture/modules/providers-langfuse.yaml. i18n:apps/api/src/i18n/{en,nb}/errors.ts, portaloperate-quality.Screenshots
Before: starting a run offers no way to send it anywhere
After: with a connection covering the service, the run can be sent to Langfuse, unticked by default
How it was tested
CI on this head:
checkand the three test shards pass. The top of the stack (Delete what genki sent to an agency's Langfuse when it purges the case #197) with main merged passesnpm run check: 470 files, 5092 tests.New tests: "sends a traced run as one experiment to the connection covering the service, never its holdout cases", "refuses a traced run when no connection covers the service, or its residency forbids the connection", "checks the residency the draft runs with, filled from its model when the draft leaves it out", "sends a run nowhere when two connections cover its service, rather than picking one", "records no sink for a traced run of only the locked cases, since none of them is sent", "tells each case but the holdout which run, case and repeat it answers when the run goes to a sink", "records the evaluation context beside a sub-task input, never in it, and never gives it to the strategy", "exports each case as one item of the run, a trace of its own rooted at the case with its sub-tasks", "sends each run only to its own sink and tenant, so one tenant's run never reaches another's", "leaves an eval run as ordinary traces in an exporter that does not export experiments", "go through the guard's pinned request, with the method, headers, body and signal", "offers to send a run to Langfuse only when a connection covers the service, unticked by default", "sends a run to Langfuse when the person ticks it for that run, for a service a connection covers".
"lands as one experiment with a trace per case and repeat, and no holdout case" runs against a local Langfuse 4.4.0 with a made-up test set, and is skipped without one:
GET /api/public/experimentslists the run with 2 items, with one trace per case and no input or output.In the portal after
npm start, with a connection from Let an agency connect its services to a Langfuse #195 covering a service: open the service's quality page and its test set. When you start a run, there is a "Send kjøringen til Langfuse" checkbox, unticked. On a service no connection covers, it is not shown.npm run checkpassesChecklist
packages/providers-*docs/architecture/modulesare updated if a contract, boundary or module changed