Repository navigation
Conversation
…s daily rate For each score it takes, the run reads the judge's execution trace and adds up its calls' tokens and the cost Langfuse reports in USD. They are stored per judge on the run, outside the quota, and only those who see usage see them. A call Langfuse has no price for counts its tokens only. The worker reads Norges Bank's USD/NOK rate once a day through the egress guard and keeps it as a configuration entry; each run stores the rate and its day with the cost. The portal shows the cost in kroner with that rate, and in dollars until a rate is known.
Langfuse judges and bills every case it receives, and a case that no judge of the run can read (a must-reject case without criteria, say) was sent and judged, then shown as not applicable. A sink judge now tells the run before the case is answered whether it can read it, and a case none of them can read is left out of what is sent, as holdout cases are.
…e run does not use Langfuse runs every active experiment judge in the project on each case it gets. The run now follows every score on a case's trace to its judge call, counts each call once across the run's judges, and keeps the project's other judges' cost apart from the judges in use. A cost read that fails or comes late no longer drops the score: it is read again next pass, and an amount with unread calls is shown as at least. The run dialog names the project's other active judges before a run starts.
# Conflicts: # apps/api/src/quality.test.ts # docs/adr/0040-evaluating-with-langfuse.md
…cost is not lost A project judge that scored after the run's own judges was never counted. When the wait for scores ends, the run now reads every case it sent again: a late judge's calls are counted, and an active judge with no score on a case counts as not yet reported, so the amount shows as at least. The rate read at start runs on one worker only.
|
Ready for review. A blind review and a claim-checking review are done and CI passes; Copilot did not review (no quota). Open: a run sent with "Send til Langfuse" but no judge in use is billed by Langfuse but not counted, since it waits for no score (the dialog warns about the project's other active judges); kroner need the worker's daily Norges Bank rate, so a stack without a worker shows dollars. |
A run now goes to Langfuse only when it asks, and only while genki reads every active judge in the project, so the run dialog no longer warns about the project's other judges and the sending route drops otherJudges. Their cost bucket stays for a judge switched on during a run. A call still unread is now said to be unread when the run was saved.
# Conflicts: # packages/providers-langfuse/src/judges.ts
… sent The merge removed the only API test of the other judges' cost bucket. A new test switches a rule on as the first case reaches Langfuse and checks the cost apart from the judges in use, shown only with view-usage. The ADR now says a case is left out only when the run uses Langfuse judges, and that a judge switched on during a run without any is not counted.
|
Ready for review again. Since the last ready comment, #203's new rules are merged in: a run with judges genki does not read is refused before it starts, so the run dialog no longer warns about the project's other judges, and their cost line is kept only for a judge switched on during a run, with its own API test again. A claim-checking review of the merge is done and its findings are fixed in a8d37d8. CI fails only on |
Stacked on #203.
What and why
Langfuse bills every judge call, and an agency wants to see what its Langfuse judges cost on each test run (ADR 0040). Part of #98; labels (98c) follow.
usage.reported): per judge in use, and together for any other judge of the project, such as one switched on in Langfuse during the run. Each call counts once. It is outside the quota and shown only to those who see usage. A call Langfuse has no price for counts its tokens only. When the wait ends, every sent case is read once more, so a judge that scored late is counted; a call not reported yet, or an active judge with no score, makes the amount show as at least.usage.usdNok).Not in this PR: Langfuse labels (98c).
:camera_with_flash: Screenshot
Resultat: the cost under the Langfuse judge, in dollars on a stack with no rate read yet
How it was tested
npm run check: format, lint, types, boundaries and size pass; 5160 tests pass and 2 fail outside this PR:apps/console/src/app.test.tsxandapps/workspace/src/projects.test.tsxexpect the month names October and 1 November, so they fail on every branch since 1 November, main included.New tests, each seen failing first: "stores each Langfuse judge's tokens and reported cost on the run, outside the quota", "stores tokens only for a model Langfuse does not price", "shows the cost in kroner with the rate the run stored", "reads Norges Bank's USD/NOK rate once a day through the egress guard", "leaves a case out of what is sent when none of the run's Langfuse judges can read it, as holdout cases are", "counts every judge call Langfuse made on the run's cases once, its own judges' apart from the project's other judges", "keeps a judge's score when its cost cannot be read, and counts the cost when a later read finds it", "shows what the project's other judges cost apart from the judges taken in use, as at least while some costs are not yet reported", "counts what a judge switched on in Langfuse during a run costs apart from the judges in use, shown only to those who see usage", "shows the cost in dollars until a rate is known", "counts a project judge that scores after the run's own judges, and shows the amount as at least for one that never does", "gives a sink judge a last read once the wait for scores is over, and stores what it reports then".
In the portal on a separate stack (API 8090, portal 5175, mock login) against the local Langfuse with a real judge: a Klarspråk run shows "Kostnad rapportert av Langfuse: 0,0223 USD, 6 235 tokens" for four judge calls.
Measured on a local Langfuse 4.4.0: a judge call's trace has one
GENERATIONwithusageDetailsandcostDetails.totalin USD. Norges Bank's answer was read once to check its shape.npm run checkpassesChecklist
packages/providers-*docs/architecture/modulesare updated if a contract, boundary or module changednpm run checkfails only on the two month-name tests named above, which this PR does not touch.