Skip to content

feat(run): STRIX_RUN_RECORD_TIMINGS records every LLM request's and tool call's timing in run.json - #1539

Merged
0xallam merged 2 commits into
mainfrom
feat/run-record-llm-request-timings
Oct 20, 2026
Merged

0xallam merged 2 commits into
mainfrom
feat/run-record-llm-request-timings

Conversation

@ian-at-strix

@ian-at-strix ian-at-strix commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Per-request timings exist today only as DEBUG `llm_request` lines in `strix.log`, and tool timings only as event timestamps. Hosted runs and replays don't keep `strix.log`, so there's no way to measure a provider's latency, or where a run's time went, after the fact.

With `STRIX_RUN_RECORD_TIMINGS=true` (off by default), `run.json` gets:

  • `llm_requests`: one entry per provider attempt, including retries and failures:
    ```json
    {"started_at": "…", "agent_id": "…", "model": "…", "provider": "Together", "outcome": "success",
    "status": 200, "duration_ms": 4200, "ttft_ms": 900, "input_tokens": 90000, "cached_tokens": 85000,
    "output_tokens": 600, "retry_attempt": 0}
    ```
    `provider` is the upstream provider behind a gateway (e.g. OpenRouter's) when known, otherwise the route's provider.
  • `tool_calls`: one `{started_at, agent_id, tool, duration_ms}` entry per tool call, from the run hooks' `on_tool_start`/`on_tool_end`.

Both ride the existing `run.json` saves; on resume, `run_record.update` restores them.

Off by default because of size: about 370 bytes per request, so about 0.5 MB on a median scan (1,100 requests) and 1 MB on a large one. `run.json` is rewritten after every request. strix-app turns it on for PR replays only (usestrix/strix-app#1599).

@greptile-apps

greptile-apps Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge.

Summary

Adds per-attempt model request records to run.json under llm_requests.

  • Each provider request now leaves a record in the run file.

No actionable issues found. Checks were based on source review; tests were not run.

Reviews (1) · Last reviewed commit: "feat(llm): run.json records every provid..." · Reviewed by Greptile

@ian-at-strix ian-at-strix changed the title feat(llm): run.json records every provider request's timing, provider and tokens in llm_requests feat(run): STRIX_RUN_RECORD_TIMINGS records every LLM request's and tool call's timing in run.json Oct 20, 2026
@0xallam
0xallam merged commit aa2036f into main Oct 20, 2026
11 of 14 checks passed
@0xallam
0xallam deleted the feat/run-record-llm-request-timings branch October 20, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants