Repository navigation
Show what a run cost, and count test runs in usage and the quota alike - #190
Merged
Merged
Conversation
The routers put the list price on each model and embedding call when it is metered, unless the provider reported a cost, and leave unpriced calls unpriced. The timeline returns the cost in micro-NOK and whether the run is priced, and usage shows test runs on their own line. The Postgres quota now leaves out assistant runs by the name they actually get, so usage and the monthly quota count the same tokens. Fixes #189.
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Every run on a Foundry model showed a cost of 0 in its timeline, whatever the price list said, and test runs were missing from usage while the monthly quota counted them. Fixes #189. Needed for the cost estimates in digdir/kunnskapsassistenten#30 and #197.
costMicroNokwhen it is metered (withListPricein@genai/contracts). A cost the provider already reported wins, and a call on an unpriced model stays unpriced. The kernel already addscostMicroNokup, so the task, the quota and Langfuse spans get the cost without changes of their own, and a run keeps the price that held when it ran.GET /tenants/{tenant}/tasks/{task}/timelinegainscostMicroNok(nullwhen the run used only unpriced models) andpriced, andstatsgainscostMicroNok,unpricedTokensand a cost per model.costOrestays for compatibility. The total comes from the task's own sum, so a reader who cannot see the calls still gets it.GET /tenants/{tenant}/usagegainstesting(usageandbyOrigin), and its totals include it, so usage and the monthly quota count the same runs. Builder-assistant turns stay out, as they do from the quota; they have their own daily budget. Test runs still never appear as a service or in Kjøringer.platform/builder-assistant, which no run has had since assistant and optimiser runs became drafts namedplatform/draft/assistant/<id>. Their tokens counted against the organisation's monthly quota, although the assistant has its own daily budget and the in-memory quota left them out.assistantCapabilityPrefixis nowplatform/draft/assistant/.How it was tested
npm run check: 461 test files pass and 1 is skipped (4,966 tests pass, 8 skipped).New tests:
The end-to-end test failed first with a cost of 0, which is how the reader without access to the calls was found.
"names an assistant run so the monthly quota's assistant prefix leaves it out" fails with the old prefix.
Live against a local API on the shared dev database (Foundry models, about 0.02 kr): a published service run priced at gpt-4o-mini's list price to the micro-NOK (16,810 for 11,054 tokens in and 234 out), a playground test run on the testing line and in the totals but not under Kjøringer, a run on an unpriced model marked unpriced in the timeline and in usage, and usage for the month equal to the quota's count (62,197 tokens), which it was not before the prefix fix (80,531).
npm run checkpassesChecklist
packages/providers-*docs/architecture/modulesare updated if a contract, boundary or module changed