[LinuxDevil]

Why

M4 ticket S20 (Harness² rows 13 and 14). Harness² uses a blind LLM selector with a rubric (harness2/core/selection.py:71-153, harness2/core/prompts.py:1040-1104). The M3 plan dropped it as an authority, because hidden checks beat an LLM reading outputs. This adds it back only as an advisory tie-break among harnesses that already passed the gate. It checks output quality that a binary check can't see. The gate stays the authority, and the selector can't choose anything that failed. Per decision 5 (section 7 of the M4 plan) it is opt-in and off by default, so cost and behaviour are unchanged. Builds on S18 (#974) and S19 (#978).

Changes

• ImproveOptions.select?: boolean (default false).
  • With it on, the gate judges every candidate instead of stopping at the first pass: the composite and each eligible proposal in agg, or each checkpoint in seq.
  • The list of who is eligible doesn't change. Proposals that made no change, duplicates, and ones that failed training are still listed and not gated.
• New internal src/improve/select.ts. With two or more passers:
  • Blind labels. Labels become A, B, … in a seeded shuffle: the sorted labels, shuffled by a PRNG seeded from sha1 of the cycle (the base harness, and the training and held-out ids).
  • Session. One read-only session (ImproveRun.ask: fs read tools only, on an in-memory workspace) reads context/candidates/<id>.md.
  • What it reads. Each passer's final answers on the training runs the gate used:
    • ProbeRun.answer when the probe returns one; otherwise the last assistant message of the transcript, without tool calls;
    • wrapped as untrusted and unmarked;
    • unscored runs left out;
    • capped at 4k per answer and about 40k per file.
  • Prompt. selectPrompt follows Harness²'s rubric order: delivery, conformance, fidelity, consistency, fitness, finish. It replies with <why> and a choice fence.
• Fallback. If the selector fails (including a provider outage) or its choice isn't one of the ids, the cheapest passer is kept: the one with the fewest tokens per training run. A budget stop or an abort is never swallowed.
• Types:
  • ImproveResult.selection?: { chosen; why; fallback }. why passes through plain(), which audit.ts now exports internally, and chosen is a GateVerdict['label'].
  • New ProbeRun.answer?: string.
  • The reason names how many harnesses passed and what the selector chose.
• winningNotes follows selection.chosen, so changes carries the chosen editor's reasons.
• formatImproveResult adds a ## Selection section.
• Docs: an "An advisory selector" section in docs/improve.md, covering what the selector sees, the fallback, and the extra held-out runs and budget. The API report is updated.

Acceptance. With select off, the result is unchanged. A test compares select: undefined with select: false and pins the exact reason and gate list. Every existing S18 and S19 test, including the k = 1 snapshot, passes unchanged.

Testing

• TDD: new src/improve/select.test.ts (14 tests), written first and failing. It covers the ticket's four cases:

  • with one passer no selector runs
  • labels are shuffled deterministically
  • an invalid choice falls back to the cheapest
  • the selector never sees a rejected harness or a held-out case (no rejected answer, held-out answer, held-out id or provenance label reaches it)

  It also covers:

  • a valid choice of the dearer harness, with markdown and HTML stripped from why;
  • fallback on a thrown error and on a provider outage;
  • abort and budget stopping rather than falling back;
  • a lower-case choice, and a reply with no <why>;
  • the transcript fallback (final answer only, no tool calls, unscored runs left out);
  • selection among seq checkpoints;
  • changes following the chosen editor.

  A new format.test.ts case covers the card.

• npx vitest run src/improve src/cli/improve: 532 passed after rebasing onto feat(improve): applyProposal, the one writer #976, feat(improve): rollbackHarness and the revisions, rollback and audit commands #981, fix(forge): split LabelsDialog under the fallow complexity limit #988, fix(forge): templates screens under the fallow limits #991, feat(improve): regression corpus store and growth #987 and feat(improve): corpus guard #989 (plain() keeps feat(improve): rollbackHarness and the revisions, rollback and audit commands #981's control-character stripping). Coverage on src/improve is 98.4% of lines, with select.ts at 97%.

• The first CI run failed pack-smoke: 1041 tarball entries against a 1040 cap. Main has since raised the cap to 1150 in feat(improve): rollbackHarness and the revisions, rollback and audit commands #981, so this PR no longer changes scripts/pack-smoke.ts.

• A CI run failed size-check: 19,771,956 bytes unpacked against a 19,768,292 budget. feat(improve): corpus guard #989 (S14) has since raised the budget to 20,760,183, above that measurement, so this PR doesn't change scripts/size-budget.json either.

• Standard checks pass:

  • typecheck (src, tests, examples), lint and build;
  • api:update/api:check, docs:llms/docs:llms:check and docs:verify-snippets -- --skip-build;
  • docs:ratchet, at the 1724 baseline;
  • scoped coverage on src/improve, then npx fallow audit --base origin/main: no issues.

• An independent review found no High issues. It confirmed that the selector can't override the gate, sees no held-out data or provenance labels, and that select off is unchanged in both modes. Fixed:

  • Medium. The transcript fallback clipped away the final answer. It now shows the last assistant message.
  • Medium. Missing tests for outage, abort, a missing <why>, and an exact select-off result.
  • Low.
    • An empty why on a valid choice.
    • A strict, case-sensitive choice parser.
    • Tool calls in the transcript revealing how candidates differ.
    • Unbounded candidate files beyond read_file's cap.
    • Unscored runs shown as answers.
    • Doc gaps on what "cheapest" means, outage fallback, budget and the seed.
  • Nit. selection.chosen is now typed as a gate label.

• Declined: using the gate's costOnPassingCases for "cheapest". It isn't defined when no case passes under both harnesses. Tokens per training run is, and the docs now name it.

🤖 Generated with Claude Code