Skip to content

Support trace-replay datasets such as AIPerf AgentX in presets - #4350

Draft
r4victor wants to merge 2 commits into
masterfrom
pr_presets_aiperf_datasets
Draft

r4victor wants to merge 2 commits into
masterfrom
pr_presets_aiperf_datasets

Conversation

@r4victor

@r4victor r4victor commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Hasn't verified yet.

Presets could only benchmark datasets as independent requests, so a dataset of recorded agent sessions (e.g. the InferenceX AgentX traces) could not be used to optimize for real multi-turn traffic.

  • Add an aiperf profile row to the agent prompt's dataset options, with --public-dataset and the inferencex-agentx-mvp scenario of SemiAnalysis's AIPerf fork.
  • Require recorded multi-turn sessions to be replayed with their turn order and timing, with the same replay options in every trial and a shorter duration allowed for trials than for the final benchmark.
  • Allow a replay to lose requests up to the tool's own failure threshold instead of requiring every request to succeed.
  • For a replay, compute output_tok_per_s over the replay duration rather than the tool's total run time, and take per_user_tok_per_s from the tool's per-user metric, since idle sessions keep far fewer than concurrency requests in flight.

@r4victor
r4victor marked this pull request as draft October 8, 2026 07:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant