Skip to content

Head-to-head routing + end-to-end benchmark vs oh-my-openagent; cursor executor backend - #1

Merged
surlymochan merged 4 commits into
mainfrom
bench/head-to-head
Oct 17, 2026
Merged

surlymochan merged 4 commits into
mainfrom
bench/head-to-head

Conversation

@surlymochan

@surlymochan surlymochan commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner
  • Routing benchmark (bench/) against oh-my-openagent v5.1.35 model-core.
    • Setup: a synthetic 80-task trace × 7 quota scenarios. Both routers get the same providers. Costs are computed from price tables, not bills.
    • Results: openswarm makes 0% wasted first attempts vs OmO's 23.4%, and has 0% unrouted vs 19.8% (mostly the writing chain, which this provider set can't serve).
    • OmO leads on review independence (71.4% vs 54.8%) and provider coverage.
  • New opt-in cursor executor backend (--backend cursor). It records executor usage and wall time in result.json and the ledger, with tests that use a fake CLI.
  • End-to-end (bench/e2e/): 8 tasks with hidden tests, grok-4.7-high via cursor-agent.
    • baseline: 24/24, 55 s, 105k tokens.
    • openswarm: 24/24, 220 s, 207k tokens.
    • OmO arm: not runnable with the only subscription available. It is documented with a cost estimate (~$1.3 on metered DeepSeek).
  • The README now opens with these tables.

Dispatches a task to the local Cursor CLI (cursor-agent) in the run's
worktree, records executor token usage and wall time in result.json and
the ledger. Never auto-routed unless listed in policy routing.
@surlymochan
surlymochan marked this pull request as ready for review October 17, 2026 13:59
@surlymochan
surlymochan merged commit 3144eef into main Oct 17, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant