Skip to content

Run a local Langfuse for development - #232

Merged
kentare merged 3 commits into
mainfrom
feat/local-langfuse
Nov 2, 2026
Merged

kentare merged 3 commits into
mainfrom
feat/local-langfuse

Conversation

@kentare

@kentare kentare commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

What and why

The Langfuse screens (judges, judge costs, labels, datasets) show something only with a Langfuse connection that has data in it, and the local stack starts no Langfuse, so a smoke walk cannot click through them before review. This adds one shared local Langfuse for development and walks:

  • compose.langfuse.yaml: Langfuse's own compose file at v4.72.0, the version production is moving to, with fixed local-only values. Compose project langfuse-genki, so every worktree shares one instance; only the web UI is published, on 127.0.0.1:34620; the start project has the keys pk-lf-genki-test / sk-lf-genki-test; no relative paths, so it also runs when read from origin/main into another folder.
  • npm run langfuse starts it or reuses the running one (-p langfuse-genki, so a COMPOSE_PROJECT_NAME in the shell starts no second one), and returns once /api/public/health answers (a health check on the web container). npm run setup and npm start leave it alone, and no workflow or deploy uses it. Renovate leaves the file alone too: it moves with production by hand.
  • scripts/langfuse-seed.ts (npm run langfuse:seed -- <project>), with only node: imports: deletes a project of the same name with its LLM connections, makes it again with its keys and prints the host, keys and ids as JSON. --judge <name> adds an LLM-as-a-judge evaluator and an enabled rule with isExperimentItemRootSpan = true; --llm-from <env file> adds a Foundry Claude LLM connection from that file's last FOUNDRY_CLAUDE_* values; --delete deletes the project. Langfuse without a licence makes no projects through its public API (403), so the seed logs in as the start user and calls the routes Langfuse's own UI calls for projects and keys; everything else goes through the public API.
  • docs/development.md: "A local Langfuse".

Closes #230.

Not in this PR, cut to what a walk needs: the annotation queue (--annotation-queue), the live Langfuse tests passing with only LANGFUSE_LOCAL_URL set (they live on the Langfuse PRs, and #226 defaults to other keys), and connecting the seeded project in the portal (no Langfuse connection on main yet; the first walk of a Langfuse PR does it).

How it was tested

By hand on macOS with OrbStack, made-up data only:

  • npm run langfuse from the worktree: healthy in 20 s; /api/public/health answers {"status":"OK","version":"4.72.0"} and the start keys answer 200. The same file copied to a folder outside the repository and started with docker compose -f <copy> up -d --wait reused the running containers (1.2 s, none recreated). With COMPOSE_PROJECT_NAME=genai-second set, npm run langfuse reused them too.

  • Seed run from that outside folder with --judge relevans --llm-from <main checkout>/.env.local: the project, its keys, the LLM connection and the judge were made, and the evaluator came out active. A second seed of the same name made a new project; the first project's keys then got 401 and its LLM connection was gone from Langfuse's database. A judge seeded without --llm-from comes out paused ("no default evaluation model is configured").

  • --delete: the keys get 401, and Langfuse's own database lists the project only as deleted, with no API keys and no LLM connection. A second --delete prints "deleted": null.

  • A Langfuse that does not answer prints "No Langfuse answers at …: npm run langfuse starts it."; an env file without FOUNDRY_CLAUDE_KEY stops the seed before it makes anything.

  • docker compose stop by project name, then npm run langfuse again: healthy.

  • npx tsc --strict on the script; npm run check passes.

  • npm run check passes

Checklist

  • Every new rule has a test whose name states it
  • User-facing text is in the i18n catalogues, in bokmål and English, with no em-dashes
  • Vendor SDKs stay in packages/providers-*
  • ADRs, docs and docs/architecture/modules are updated if a contract, boundary or module changed
  • No secrets, personal data or local paths

No tests: this is development tooling, checked by running it against a real Langfuse 4.72.0 as above.

The Langfuse screens show something only with a connection that has data,
and the local stack started no Langfuse. compose.langfuse.yaml runs 4.72.0,
production's version, as one instance every worktree shares, and
scripts/langfuse-seed.ts makes a project with keys and a judge through the
routes Langfuse's own UI calls, since its public API makes no projects
without a licence. Both run when read from origin/main, as walks do.
COMPOSE_PROJECT_NAME overrides the compose file's name, so npm run langfuse
passes -p langfuse-genki. A deleted Langfuse project keeps its LLM
connections, so the seed deletes them first. Renovate leaves the file
alone, since it moves with production by hand, and 4.72.0 is the version
production is moving to, not the one it runs.
@kentare

kentare commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Ready for review. Blind review, a claim-checking review and a check of the fixes are done and CI passes. Open: #226's live tests still default to the pk-lf-genki-472 keys, so they need LANGFUSE_LOCAL_PUBLIC_KEY and LANGFUSE_LOCAL_SECRET_KEY against this Langfuse.

@kentare
kentare marked this pull request as ready for review November 2, 2026 17:44
@kentare kentare changed the title Run a local Langfuse for development and smoke walks Run a local Langfuse for development Nov 2, 2026
@kentare
kentare merged commit 5d29c6c into main Nov 2, 2026
14 checks passed
@kentare
kentare deleted the feat/local-langfuse branch November 2, 2026 17:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run a local Langfuse for development and the live Langfuse tests

1 participant