Skip to content

Head-to-head benchmark vs superpowers + gate hardening (0.2.1) - #3

Merged
surlymochan merged 6 commits into
mainfrom
bench/head-to-head
Oct 17, 2026
Merged

surlymochan merged 6 commits into
mainfrom
bench/head-to-head

Conversation

@surlymochan

@surlymochan surlymochan commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Adds a reproducible head-to-head benchmark against obra/superpowers v7.0.0 and hardens the TDD gates the benchmark found gaps in (released as 0.2.1).

Gate fixes (general checks, each with unit tests):

  • Verifiable proofs: I6a binds command, exit code, output hash, file hashes and HEAD, and snapshots the tree. I6b re-validates the binding and re-runs the command in the snapshot.
  • Red and green must target the same tests.
  • skip/todo tests don't count as red or green.
  • Changed tests must reference changed symbols, and type-only or existence-only assertions are flagged when they are the only ones.
  • Python assertions are now recognised.
  • J4b fails contradictory MUST/SHALL requirements.
  • New tdd_mode values refactor and docs, a legitimate exemption with misuse checks.

Benchmark (bench/):

  • 24 original and 12 held-out scenarios. The held-out ones were written by a separate author after the fixes, without access to the gate code.
  • Results: v0.2.0 caught 10/17 faulty scenarios with 3/7 false positives. v0.2.1 caught 16/17 original, 6/8 held-out, 0/11 false positives.
  • LLM judge arms with grok-4.7-high, one run per scenario: superpowers skills 25/25 with 1/11 false positives; no skills 25/25 with 0/11.

End-to-end (bench/e2e/): 8 tasks with hidden tests, same model for every arm.

Arm Success Median time Median tokens n
baseline 24/24 55 s 105k 24
superpowers 24/24 187 s 312k 24
OpenFlow 11/12 985 s 5.29M 12

The README now opens with these tables, losses included.

…tion and spec-contradiction checks, refactor/docs modes

- I6a records a digest binding (command, exit, output hash, tested-file hashes, HEAD) plus a tree snapshot; I6b re-validates and re-runs in the snapshot.
- Red and green must target the same tests (or green a superset); non-test red commands are rejected.
- Skipped/todo tests (node:test, pytest, unittest) no longer count as red or green.
- Changed tests must reference changed symbols; typeof/defined-only assertions are flagged as weak; Python assertions are recognised.
- J4b fails on contradictory MUST/SHALL requirements for the same subject.
- New tdd_mode: refactor (existing tests green before/after, no assertion removal) and docs (non-code diff only).
@surlymochan
surlymochan marked this pull request as ready for review October 17, 2026 13:59
@surlymochan
surlymochan merged commit 9708740 into main Oct 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant