Skip to content

Show what a run cost, and count test runs in usage and the quota alike - #190

Merged
nlundee merged 1 commit into
mainfrom
fix/run-cost
Oct 29, 2026
Merged

nlundee merged 1 commit into
mainfrom
fix/run-cost

Conversation

@nlundee

@nlundee nlundee commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

What and why

Every run on a Foundry model showed a cost of 0 in its timeline, whatever the price list said, and test runs were missing from usage while the monthly quota counted them. Fixes #189. Needed for the cost estimates in digdir/kunnskapsassistenten#30 and #197.

  • The price goes on the call. The model and embedding routers put the list price on each call's costMicroNok when it is metered (withListPrice in @genai/contracts). A cost the provider already reported wins, and a call on an unpriced model stays unpriced. The kernel already adds costMicroNok up, so the task, the quota and Langfuse spans get the cost without changes of their own, and a run keeps the price that held when it ran.
  • The timeline shows it. GET /tenants/{tenant}/tasks/{task}/timeline gains costMicroNok (null when the run used only unpriced models) and priced, and stats gains costMicroNok, unpricedTokens and a cost per model. costOre stays for compatibility. The total comes from the task's own sum, so a reader who cannot see the calls still gets it.
  • Test runs in usage. GET /tenants/{tenant}/usage gains testing (usage and byOrigin), and its totals include it, so usage and the monthly quota count the same runs. Builder-assistant turns stay out, as they do from the quota; they have their own daily budget. Test runs still never appear as a service or in Kjøringer.
  • The quota left assistant runs in. The Postgres quota, which genki-test and production use, excluded assistant runs by the prefix platform/builder-assistant, which no run has had since assistant and optimiser runs became drafts named platform/draft/assistant/<id>. Their tokens counted against the organisation's monthly quota, although the assistant has its own daily budget and the in-memory quota left them out. assistantCapabilityPrefix is now platform/draft/assistant/.
  • The portal shows a cost below one øre and "Ikke priset" from the new fields, and the text on Oversikt and Forbruk says that tests in the playground and the builder count, and that the assistant does not.
  • The AI Gateway (ADR 0038): a gateway that reports its charge per call wins over the price list without a change to the routers. ADR 0038 says so under Consequences. Currencies other than NOK stay the decision of step 1.

How it was tested

  • npm run check: 461 test files pass and 1 is skipped (4,966 tests pass, 8 skipped).

  • New tests:

    • "meters a call at its list price, keeps a reported cost and leaves an unpriced call unpriced" (model router);
    • "meters input tokens at the list price" (embedding router);
    • "shows what the run cost on its timeline, priced from the price list" (a draft run through the composed platform: 10 tokens in and 5 out at gpt-4o's price is 714 micro-NOK);
    • "adds up what each call cost, below one øre too", "marks a run on an unpriced model as unpriced instead of free", "keeps the priced part of a run that also used an unpriced model" and "shows the cost to a reader who cannot see the calls" (timeline);
    • "counts test runs on their own usage line and in the total, never as a service" and "playground runs count as testing in usage, never as a service run";
    • "shows a cost below one øre from the run in micro-NOK" (portal).
  • The end-to-end test failed first with a cost of 0, which is how the reader without access to the calls was found.

  • "names an assistant run so the monthly quota's assistant prefix leaves it out" fails with the old prefix.

  • Live against a local API on the shared dev database (Foundry models, about 0.02 kr): a published service run priced at gpt-4o-mini's list price to the micro-NOK (16,810 for 11,054 tokens in and 234 out), a playground test run on the testing line and in the totals but not under Kjøringer, a run on an unpriced model marked unpriced in the timeline and in usage, and usage for the month equal to the quota's count (62,197 tokens), which it was not before the prefix fix (80,531).

  • npm run check passes

Checklist

  • Every new rule has a test whose name states it
  • User-facing text is in the i18n catalogues, in bokmål and English, with no em-dashes
  • Vendor SDKs stay in packages/providers-*
  • ADRs, docs and docs/architecture/modules are updated if a contract, boundary or module changed
  • No secrets, personal data or local paths

The routers put the list price on each model and embedding call when it is
metered, unless the provider reported a cost, and leave unpriced calls
unpriced. The timeline returns the cost in micro-NOK and whether the run is
priced, and usage shows test runs on their own line. The Postgres quota now
leaves out assistant runs by the name they actually get, so usage and the
monthly quota count the same tokens.

Fixes #189.
@nlundee
nlundee merged commit e6fb278 into main Oct 29, 2026
13 checks passed
@nlundee
nlundee deleted the fix/run-cost branch October 29, 2026 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Show what a run cost: the run timeline says 0 for every model run

1 participant