Skip to content

Let an agency connect its services to a Langfuse - #195

Draft
kentare wants to merge 13 commits into
feat/one-eval-runnerfrom
feat/langfuse-connections
Draft

kentare wants to merge 13 commits into
feat/one-eval-runnerfrom
feat/langfuse-connections

Conversation

@kentare

@kentare kentare commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #194.

What and why

genki has one trace exporter for the whole platform, set from the environment, and the portal's Tracing tab said it was not available. ADR 0040 gives each agency its own connections to a Langfuse. Part of #97.

  • A tenant saves a connection with PUT /tenants/{tenant}/tracing/connections/{id} (manage-services, change reason, audited): a host, a public key, the secret key as a secret://tenant/... reference set for that host, the services it covers, a residency and whether it receives content. GET lists them without the secret, DELETE removes one and answers 404 for one that does not exist.
  • The connection and its secret are checked before the admin task starts, so a key typed in clear, or in place of a secret's name, never reaches a task's input. The routes act with the principal enterTenant returns.
  • The host is an HTTPS address only, without a path, credentials or a trailing dot, never a private or link-local address, and every call to it goes through the egress guard. In development mode a Langfuse on loopback may use HTTP.
  • A service is covered by at most one connection.
  • On Digdir's Langfuse (LANGFUSE_BASE_URL) a new connection starts as global, only an operator sets its residency (PUT /operator/tracing/connections/{tenant}/{id}/residency), and saving it again without a residency keeps the operator's. Content is refused there until LANGFUSE_INSTANCE_RECEIVES_CONTENT=true, on an agency's own Langfuse until the agency declares erasure, and outside development mode while LANGFUSE_BASE_URL is unset.
  • POST .../{id}/check opens the connection's project with its keys. TRACES_CONTENT is honoured only in development mode, in the API and the worker (tracesContent in @genai/composition).
  • The portal's tab "Sporing og evaluering" lists, adds, edits, checks and removes connections, and asks for another id rather than replacing a connection. The form takes the secret key and stores it as a tenant secret (langfuse-<id>) set for the connection's address alone, then saves the connection with a reference to it; an empty key keeps the stored one, and a new address needs the key again. The id field says it takes small letters, digits and hyphens, and refuses anything else in the portal's language. The check says that the secret key opens the project, since Langfuse accepts any public key with the right secret key.
  • Docs: docs/api/access.md, .env.example, OpenAPI. i18n: apps/api/src/i18n/{en,nb}/errors.ts, portal operate-integrations and the change-log labels in core-screens.

Not in this PR: sending a run to a connection (#196) and deleting what was sent (#197). Two saves at the same moment can both claim a service; #196 sends such a service nowhere.

Screenshots

Before: the Tracing tab says it is not available
Before: the Sporing og evaluering tab with a not available notice

After: a connection to a local Langfuse, checked against its project
After: the lokal connection with its service, residency, metadata only and the project its keys open

The form takes the secret key and says what an id may hold
The new connection form with the id error in bokmål and the secret key field

The check names the project the secret key opens, and says the public key is not checked
The lokal connection checked against the local Langfuse project

How it was tested

  • CI on this head passes (static, size and the four test shards). The top of the stack (Delete what genki sent to an agency's Langfuse when it purges the case #197) passes npm run check: 470 files, 5103 tests.

  • New tests: "saves a connection with a secret reference and lists it without the secret", "refuses a host that is not HTTPS, or resolves to a private or link-local address", "takes a host only as an address, without a path, credentials or a trailing dot", "refuses another tenant's secret, a secret not set for the host, and a key typed in clear", "covers a service with at most one connection", "starts a connection to Digdir's Langfuse as global, and only an operator sets its residency", "receives content only where erasure holds: Digdir's Langfuse once operators confirm it, an agency's own once it declares it", "refuses content outside development mode while Digdir's Langfuse is not configured", "removes a connection, and answers 404 for one that does not exist", "lets only a person with manage-services change connections, also an operator who is a member", "honours TRACES_CONTENT only in development mode", "shows each connection with its services, who set its residency, and whether it gets text", "stores the secret key for the connection's address alone, then saves the connection with only a reference to it", "asks for another name rather than replacing a connection that has it", "refuses an id with anything but small letters, digits and hyphens, in the language of the portal", "keeps the stored secret key when it is left empty, and asks for it again when the address changes", "says the check shows the secret key opens the project, and that Langfuse does not check the public key".

  • "checks a connection against its project through the egress path" runs against a local Langfuse 4.4.0 with made-up data, and is skipped without one.

  • In the portal after npm start:

    1. Open http://localhost:5173 and log in with the mock login as Kari Testansatt.
    2. Go to Integrasjoner, tab "Sporing og evaluering". On main it says the screen is not available.
    3. Click "Koble til Langfuse" and save with "Lokal Langfuse" as id: the id field says to use small letters, digits and hyphens.
    4. Fill in an id such as lokal, the host, the public key, the secret key, services and a reason, and save. Click "Sjekk koblingen": it says the secret key opens the project and that Langfuse does not check the public key.
    5. Click "Endre" and change the address without a key: it asks for the secret key again.
  • npm run check passes

Checklist

  • Every new rule has a test whose name states it
  • User-facing text is in the i18n catalogues, in bokmål and English, with no em-dashes
  • Vendor SDKs stay in packages/providers-*
  • ADRs, docs and docs/architecture/modules are updated if a contract, boundary or module changed
  • No secrets, personal data or local paths

genki has one trace exporter for the whole platform, configured from
the environment, and the portal's Tracing screen was a mock-up. ADR
0040 gives each agency its own connections (part of #97).

- A tenant saves a connection with PUT
  /tenants/{tenant}/tracing/connections/{id} (manage-services, change
  reason, audited): a host, a public key, the secret key as a
  secret://tenant/... reference set for that host, the services it
  covers, a residency and whether it receives content.
- The host must be HTTPS and never a private or link-local address,
  and every call to it goes through the egress guard. In development
  mode a Langfuse on loopback may use HTTP.
- A service is covered by at most one connection.
- On Digdir's Langfuse (LANGFUSE_BASE_URL) a new connection starts as
  global and only an operator sets its residency. Content is refused
  there until LANGFUSE_INSTANCE_RECEIVES_CONTENT=true, and on an
  agency's own Langfuse until the agency declares erasure.
- POST .../{id}/check opens the connection's project with its keys,
  tested against a local Langfuse 4.4.0.
- TRACES_CONTENT is honoured only in development mode.
- The portal's Tracing screen lists, adds, edits, checks and removes
  connections in live mode.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical issues leave secret keys in task logs, permit authorization bypass and loss of audit attribution, allow worker trace-content export outside development, and let concurrent saves create duplicate service coverage.

4 open issues
What changed in this PR

Adds tenant-specific Langfuse connection management, replacing the portal’s unavailable Tracing tab. This establishes connection configuration for the later run-export and deletion work in #196 and #197.

Changes:

  • Adds audited connection APIs with secret-reference validation, guarded outbound checks, service coverage rules, and operator-controlled residency.
  • Adds portal controls to list, create, edit, check, and remove connections.
  • Restricts API trace content to development mode and adds documentation, translations, and tests.
File Description
docs/​api/​access.md Documents connection endpoints and restrictions.
apps/​portal/​src/​screens/​integrations/​LiveTracing.tsx Implements connection management UI.
apps/​portal/​src/​screens/​Integrations.tsx Enables the live Tracing tab.
apps/​portal/​src/​screens/​integrations-tracing.test.tsx Tests connection display and creation.
apps/​portal/​src/​screens/​changes/​changes.ts Maps tracing audit actions to labels.
apps/​portal/​src/​i18n/​nb/​operate-integrations.ts Adds Bokmål connection text.
apps/​portal/​src/​i18n/​nb/​core-screens.ts Adds Bokmål change-log labels.
apps/​portal/​src/​i18n/​en/​operate-integrations.ts Adds English connection text.
apps/​portal/​src/​i18n/​en/​core-screens.ts Adds English change-log labels.
apps/​portal/​src/​api/​tracing.ts Adds the typed tracing API client.
apps/​portal/​src/​api.ts Registers the tracing client.
apps/​api/​src/​trace-connections.ts Implements storage, validation, checks, and routes.
apps/​api/​src/​trace-connections.test.ts Tests connection rules and access controls.
apps/​api/​src/​rights.ts Defines tracing administration permissions.
apps/​api/​src/​platform-modules.ts Adds tracing dependencies to modules.
apps/​api/​src/​openapi/​tags.ts Categorizes tenant tracing endpoints.
apps/​api/​src/​openapi/​access.ts Documents tracing schemas and routes.
apps/​api/​src/​main.ts Wires tracing settings and API content gating.
apps/​api/​src/​i18n/​nb/​errors.ts Adds Bokmål tracing errors.
apps/​api/​src/​i18n/​en/​errors.ts Adds English tracing errors.
apps/​api/​src/​composition.ts Registers and supplies the tracing module.
apps/​api/​src/​change-reason.ts Exempts connection checks from change reasons.
apps/​api/​src/​boot-guard.ts Adds the development-only trace-content predicate.
apps/​api/​src/​admin-tasks.test.ts Includes tracing writes in audit-route coverage.
.env.example Explains content gating and instance erasure confirmation.

Review effort: Balanced

Comment thread apps/api/src/main.ts Resolved
Comment on lines +183 to +185
for (const service of next.services) {
const covering = await coveringService(tenant, service);
if (covering && covering.id !== next.id)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept: the coverage check is not atomic across API instances, since config entries have a version check per id and no lock across ids; it would take a store-level lock. Saving a connection is a rare, audited admin action, and #196 makes the outcome safe: a service that two connections cover is sent to neither, so a race cannot send a run to the wrong Langfuse.

Comment thread apps/api/src/trace-connections.ts Resolved
Comment thread apps/api/src/trace-connections.ts Resolved
@kentare
kentare added this pull request to stack #199 October 30, 2026 04:36
…ghts

- PUT .../tracing/connections/{id} checks the connection before the
  admin task starts, so a key typed in clear is refused without ever
  entering a task's input.
- The connection routes act with the principal enterTenant returns, so
  an operator who is a member gets only their own rights there, and a
  support session keeps its actor.
- The worker's platform-wide exporter sends text only in development
  mode, as the API's does, through the shared tracesContent.
- The secret is checked against its host before the admin task starts,
  so a key typed in place of a secret's name never reaches the audit
  log; the portal field is now "Navn på hemmeligheten".
- A host is an address only, without a path, credentials or a trailing
  dot, so Digdir's Langfuse cannot be taken for an agency's own.
- Outside development mode no connection receives content while
  LANGFUSE_BASE_URL is unset, and .env.example names it.
- A connection to Digdir's Langfuse saved without a residency keeps the
  operator's, and deleting an unknown connection answers 404.
- The portal asks for another name instead of replacing a connection.
@kentare

kentare commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Ready for review. Copilot and a claim-checking review are done (no blind review) and CI passes. Open: the thread on two saves at once claiming a service (not atomic across instances; #196 sends such a service nowhere); the edit form for a connection to Digdir's Langfuse still offers residency and the declaration, removing has no confirm step, and egress errors read in English.

@kentare

kentare commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Found while testing the stack in the portal:

  1. "Navn på koblingen" is the connection's id. "Lokal Langfuse" gives the raw message Invalid string: must match pattern /^[a-z0-9][a-z0-9-]*$/, in English in the bokmål portal too. Fix: label it as an id with a hint (small letters, digits and hyphens), and map the error to a text in both catalogues.
  2. The form asks for "the name of the secret where you stored the secret key, set for the address above", but no screen stores a secret with allowed hosts: the MCP form's setSecret sends none. I stored it with PUT /tenants/{t}/secrets/{name}. Fix: a secret key field in the form that stores the secret for the connection's host, as the MCP form does for its own.
  3. "Sjekk koblingen" cannot catch a wrong public key: Langfuse 4.4.0 accepts any public key with the right secret key (pk-lf-feil opened the project). Fix: say in the help text that the check verifies the secret key and project, or check the public key some other way.

…ress alone

The form asked for the name of a stored secret, but no portal screen stores one for a host, so an agency could not finish the setup. The form now takes the secret key, stores it bound to the connection's address and saves the connection with a reference to it; an empty field keeps the stored key, and a new address needs the key again. The id field says what it accepts and refuses anything else in the portal's language, and the check says that Langfuse does not check the public key.
@kentare

kentare commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Re #195 (comment)

Fixed in df7dafd:

  1. The field is "Id for koblingen", with the hint "Små bokstaver, tall og bindestrek". Another id is refused in the form, in the portal's language, before anything is sent.
  2. The form takes the secret key and stores it as the tenant secret langfuse-<id>, set for the connection's address alone, then saves the connection with only a reference to it. An empty field keeps the stored key; a new address needs the key again.
  3. Langfuse has no way to check the public key with a project key, so the check now says that the secret key opens the project and that Langfuse does not check the public key.

Checked by "stores the secret key for the connection's address alone, then saves the connection with only a reference to it", "refuses an id with anything but small letters, digits and hyphens, in the language of the portal", "keeps the stored secret key when it is left empty, and asks for it again when the address changes" and "says the check shows the secret key opens the project, and that Langfuse does not check the public key"; npm run check on the top of the stack (470 files, 5103 tests); and in the browser on a local stack, connecting a local Langfuse 4.4.0 from the form alone.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants