[braboj]

Round 5d's efficacy report: hand and short at K = 15 beside round 5c's five arms, scored at the hidden suite's revision 1f07e42, judged, and archived as it stands.

File	What it is
2026-11-07-efficacy.md	the plain page: radar per arm, each arm's verdict, the six questions
2026-11-07-efficacy-appendix.md	what produced the results, the rows behind the radar, the verdict vector
2026-11-07-efficacy-record.md	the full record
2026-11-07-efficacy-radar-{dark,light}.svg	the radars

In short:

• reference, short, skills, hybrid and inline are better than no file; hand shows no clear gain; all cost more.
• Settled: inline and skills do as well as a person's file. Not settled: hand against none, hybrid, reference and short against hand.

Archive: braboj/tariff-runs, rounds/2026-11-04-round-5d and its scoring area, verified and stamped. Benchmark issue: solid-ai-dev/solid-ai-benchmark#89.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DpT5xQQ4hBtuBqvpuv9xGj

[braboj]

Codex review, round 1

One finding, fixed in the generator.

Severity	Finding	What became of it
P2	The reference section said "1 of 15 runs passed clearly fewer of the hidden tests, 2 of them because the app would not start", but the two builds that would not start (reference-8, reference-11) have no hidden-suite rate; the low run is reference-12.	Fixed in the report generator, solid-ai-dev/solid-ai-benchmark#140, with a check: a boot failure that is not a low run is counted apart ("2 others would not start after a clean install"). This report is regenerated here once #140 merges.

[braboj]

Codex review, round 2

Two findings: one by design, one fixed in the generator.

Severity	Finding	What became of it
P2	"Better than no rules file at: … data protection" for reference, skills, hybrid and inline, while the appendix marks both data-protection measures held, not better.	By design: the "better at" bullets restate the radar, whose axes group several measurements and read as a whole; the verdict reads only the measures chosen before the run, and the page says so above the verdicts. The axis moved past its margin; its two deciding rows did not.
P2	The record says every interval is over paired differences of a block, while seven contrasts cross rounds and are read unpaired.	Fixed in the generator, solid-ai-dev/solid-ai-benchmark#141, with a check: the record and the appendix name the unpaired contrasts and say their interval is for the difference of two independent means; no cell is called a paired difference. This report is regenerated here once #141 merges.

[braboj]

Codex review, round 3

Two findings: one repeated from round 2 and answered there, one fixed in the generator.

Severity	Finding	What became of it
P2	"Better at … data protection" while both data-protection rows are held (as in round 2).	By design, as answered in round 2: the bullets restate the radar's axes, the verdict reads the deciding rows, and the page says so.
P2	The method paragraph says every interval is a bootstrap, but the install rate's row is Newcombe's score interval; the ‡ note does not name it.	Fixed in the generator, solid-ai-dev/solid-ai-benchmark#142, with a check: the method paragraph names the install rate's score interval, from the Wilson interval of each arm's share, paired over the blocks' agreement or from the two shares on their own. This report is regenerated here once #142 merges.

[braboj]

Codex review, round 4

One finding, fixed in the generator; the PR's CI agrees.

Severity	Finding	What became of it
P2	Three voided trials' reasons in the record end in a bare URL, which fails the Markdown lint (MD034) and so the Lint Markdown job.	Fixed in the generator, solid-ai-dev/solid-ai-benchmark#143, with a check: a voided trial's reason is shown as inline code. This report is regenerated here once #143 merges.

Also: the PR now carries changelog:none, as the earlier report PRs did; the missing label failed its check.

[braboj]

Codex review, round 5

No findings; every CI check passes. The loop ends here: a round with nothing new.

Review loop summary

5 rounds, 6 findings, all P2. Four were faults of the report generator, fixed in the benchmark with a check each and the report regenerated here; one is by design; one was the missing changelog label.

Finding	What became of it
Two builds that would not start were described as part of the one low-scoring run.	Fixed, solid-ai-dev/solid-ai-benchmark#140.
"Better at data protection" while both data-protection rows are held (rounds 2 and 3).	By design: the bullets restate the radar's axes, the verdict reads the deciding rows.
The method paragraph said every interval was paired, while seven cross-round contrasts are unpaired.	Fixed, solid-ai-dev/solid-ai-benchmark#141.
The install rate's Newcombe interval was not named.	Fixed, solid-ai-dev/solid-ai-benchmark#142.
Bare URLs in the voided trials' reasons failed the Markdown lint.	Fixed, solid-ai-dev/solid-ai-benchmark#143.
No changelog label.	changelog:none added.