Repository navigation
Dual-review should distinguish infrastructure failure from reviewer verdict #157
Description
Goal
Classify dual-review infrastructure failures truthfully and retain usable non-certifying evidence when one review leg cannot produce a verdict.
Problem
Startup, authentication, subscription quota, timeout, invalid execution, and substantive reviewer findings can currently collapse into a generic review failure. A completed provider result may also be lost when its peer fails. That weakens the 1.0 honest-failure contract.
Required behavior
- Distinguish
invocation,auth,quota,timeout,invalid_execution, andreviewer_verdictoutcomes. - Preserve a completed provider result when the peer leg fails.
- Write a partial structured receipt for infrastructure failure, explicitly marked non-certifying.
- Recommend retry only for transient infrastructure failures; never imply that substantive findings should be retried away.
- Keep certification fail-closed unless every required reviewer completed successfully and the ordinary verdict rules pass.
Acceptance criteria
- One successful fake reviewer plus one startup failure produces a partial non-certifying receipt with an infrastructure-failure class.
- Auth, quota, timeout, malformed output, forbidden tool use, and substantive
NOT CERTIFIEDare distinguishable. - A completed peer result is retained without being promoted to joint certification.
- Retry guidance appears only for explicitly transient infrastructure classes.
- Focused integration tests and the canonical proof suite pass.
Boundaries and dependencies
- fix(review): stop tool-free cross-model reviews when Claude invokes advisor #158 owns immediate rejection and process cleanup for forbidden reviewer tool-use events.
- Long-running heartbeat/progress UX is tracked in Emit bounded provider progress during long dual reviews #164 and is not a 1.0 blocker.
- This classification contract must land before Complete the bounded incremental-review pilot and final candidate gate #109's final completion gate is trusted.
Activity
[-]Dual-review should expose provider progress and distinguish invocation failure from a review verdict[/-][+]Dual-review should distinguish infrastructure failure from reviewer verdict[/+]BaseInfinity commentedon Aug 22, 2026
1.0 release audit (2026-08-22): narrowed this issue to the release-blocking honest-failure contract. Long-run heartbeat/progress UX was split to #164 in 1.1.0.
#158 owns forbidden tool-use detection and process cleanup. This issue owns the general invocation/auth/quota/timeout/invalid-execution taxonomy, preservation of a completed peer result, and a partial explicitly non-certifying receipt.
BaseInfinity commentedon Aug 22, 2026
#158 is integrated via PR #165, so its rejection/process-cleanup dependency is satisfied. This issue is now the first open 1.0 release blocker and the active engineering slice.
Keep scope to the existing acceptance contract: truthful failure taxonomy, retained completed-peer evidence, explicitly non-certifying partial receipts, transient-only retry guidance, and fail-closed certification. ROADMAP cold-session synchronization belongs with this slice; #109 remains frozen until this lands.
BaseInfinity commentedon Aug 22, 2026
Implementation checkpoint (2026-08-22): audited origin/main at af03760 and created clean worktree /private/tmp/codex-sdlc-issue157 on fix/issue-157-review-outcomes. Root checkout remains untouched. Current seam: runReviewPair rejects on either leg and main writes only failed review-run state, so an already completed peer result is lost and no explicitly non-certifying partial dual-review receipt exists. Frozen allowlist: outcome taxonomy for invocation/auth/quota/timeout/invalid_execution/reviewer_verdict; preserve completed peer evidence; partial non-certifying receipt; transient-only retry guidance; all-required-reviewers fail-closed certification; ROADMAP cold-session/order sync. Excluded: #164 heartbeat/progress UX and #158 tool-use cleanup changes. Next evidence: add the narrow fake-reviewer integration case for completed Fable plus Sol startup failure, prove RED, then implement the smallest receipt path.
BaseInfinity commentedon Aug 22, 2026
TDD RED checkpoint: added the narrow fake-reviewer integration case inside the existing bounded dual-review test. Scenario: Fable completes CERTIFIED, then Sol exits during invocation. Focused command invoked only test_dual_review_is_independent_bounded_and_candidate_bound from tests/test-adapter.sh. RED is honest: exit 1 because .git/codex-sdlc/dual-review.json is absent, followed by Sol review INVALID_VERDICT. No production code or ROADMAP edits yet; canonical broad proof has not been run.
BaseInfinity commentedon Aug 23, 2026
Focused GREEN checkpoint: the executable acceptance matrix now distinguishes invocation, auth, quota, timeout, invalid_execution (including forbidden tool use and malformed output), and reviewer_verdict; retains completed peer evidence; writes schema-v4 status=partial, certifying=false, joint_verdict=NOT CERTIFIED receipts; and emits retry guidance only for quota/timeout or explicitly transient invocation infrastructure. Normal certification still requires both completed reviewer verdicts. ROADMAP mechanical RED then GREEN synchronized #157 before frozen #109, removed closed #158 from the queue, matched the eight open 1.0 blockers, and added current #162/#163/#164. Focused proof: tests/test-adapter.sh 115/115 and tests/test-roadmap.sh 14/14. Canonical broad proof and final reviews have not run yet.
BaseInfinity commentedon Aug 23, 2026
Frozen-candidate checkpoint: exact staged tree 34a6fe5ee4c3ffc46142b42018800b4de1cf6af5 on base af03760. Author self-review completed with no blocking finding. The one canonical proof passed 11/11 and wrote a fresh reviewed proof at 2026-08-23T00:26:52.924Z with workspace fingerprint sha256:0bb3768920c0a5b0441827c519fea8559574545725d56631ccce69508214e9c9. Staged paths are exactly dual-review.cjs, ROADMAP.md, test-adapter.sh, and test-roadmap.sh; no unstaged tracked changes. Next action is the normal required Sol High + Fable High dual-review gate against origin/main; no exception.
BaseInfinity commentedon Aug 23, 2026
Corrective-loop checkpoint (2026-08-22): the first required Sol High + Fable High gate failed closed and produced no certification. Fable invoked the forbidden
advisortool and was rejected asINVALID_VERDICT; the supervised pair then cancelled Sol. The schema-v4 partial receipt safely remainedcertifying: false/NOT CERTIFIEDand retained both reviewer outcomes, but its top-level outcome incorrectly selected Sol's cancellation (invocation) instead of the causal Fable failure (invalid_execution). This is a candidate-born #157 truthfulness defect. I am adding a narrow RED for causal-failure precedence, then will correct, re-prove a newly frozen tree, and rerun the required reviewers. No exception and no retry of the unchanged candidate.BaseInfinity commentedon Aug 23, 2026
Corrective-loop GREEN: added a deterministic race where Fable emits forbidden
advisoruse while Sol is still running. RED reproduced the defect (Sol review INVALID_VERDICT/ top-levelinvocation). The minimal fix now selects a non-cancelled peer failure as causal instead of allowing the supervisor-induced cancellation to mask it. The owning test passed 1/1, and the full focused adapter suite passed 115/115. The corrected partial receipt reports top-levelinvalid_execution,retry.recommended: false, preserves Sol asCANCELLED, and preserves Fable asINVALID_VERDICT. Next: corrected-tree self-review, freeze, fresh canonical 11/11 proof, then a new required Sol High + Fable High gate.BaseInfinity commentedon Aug 23, 2026
Corrected frozen-candidate checkpoint: staged tree
db2236d8d7a4f4de46e5aaadaf5a1cdce1c28678has no unstaged tracked changes and passed the fresh canonical proof 11/11. Proof created2026-08-23T00:50:38.997Z; workspace fingerprintsha256:caa244ff703278dc116bd47779e8fea6abdb26c1ecf70294565ba3d16b2d0c35. Check timings: diff 26ms, adapter 329.6s, packaging 16.8s, npm 80.7s, skill 1.9s, release 497ms, roadmap 2.4s, benchmark 783ms, e2e 19ms, setup 270.5s, update 640.7s. The proof stamp verifies against the frozen index. Next gate is the required normal Sol High + Fable High review; certification remains impossible unless both required reviewers complete and verdict rules pass.BaseInfinity commentedon Aug 23, 2026
Required-review checkpoint: corrected tree
db2236d8d7a4f4de46e5aaadaf5a1cdce1c28678again failed closed with a schema-v4 partial/non-certifying receipt. Sol returned an explicit usage-limit error twice (try again at Aug 25th, 2026 10:28 AM), causing Fable to be cancelled; no reviewer completed and no certification was issued. The receipt correctly marked the failure transient/retryable, but mislabeled the explicit quota as genericinvocationinstead ofquota. That is a second candidate-born #157 classification defect. I am adding a narrow Sol-quota RED and correcting it before reproving. The external quota may still block the final required Sol High + Fable High gate until reset; no review exception or fallback will be used.BaseInfinity commentedon Aug 23, 2026
Sol-quota corrective GREEN: deterministic structured Codex usage-limit events reproduced the real failure. RED showed top-level
invocation; the minimum fix reuses the existing quota recognizer when constructing reviewer failure outcomes. The owning dual-review test passed 1/1 and the full adapter suite passed 115/115. A Sol quota now produces an explicitly partial/non-certifying receipt withclassification: quota,transient: true, retry guidance, and the Fable peer retained asCANCELLED. Next: restage/self-review the changed candidate, mint a fresh exact-tree canonical proof, then hold the live required review until the receipt-authorized quota retry window.8 remaining items
BaseInfinity commentedon Aug 23, 2026
Corrective TDD checkpoint — protocol output vs forbidden tool use
RED 1: captured Claude Code 2.1.221’s real schema return (
assistant.message.content[] -> tool_use/StructuredOutput) and required it to be accepted only when explicitly enabled for the review lane and its input validates against the review contract. Existing code rejected it.RED 2: integrated candidate mutation during review had to produce an explicitly partial, non-infrastructure, non-retryable
invalid_executionoutcome. Existing code mislabeled it as infrastructureinvocation.GREEN: both focused contracts pass. The availability probe still rejects
StructuredOutput; malformedStructuredOutputstill rejects; a valid schema output followed byserver_tool_use/advisorstill rejects advisor; every other reviewer tool remains forbidden. Candidate dirty/mutation and empty-patch preconditions are taggedinvalid_execution. Incomplete initial evidence is retained only underinitial, withfinal: {}.New staged candidate tree:
299c868c000a496308426c04e591e9eab9677e3b. The prior proof/review evidence is stale by design. Broad proof, self-review, a fresh canonical 11/11 stamp, and the required joint gate remain pending. No exception is in use.BaseInfinity commentedon Aug 23, 2026
Corrected candidate proof checkpoint
Frozen staged tree
299c868c000a496308426c04e591e9eab9677e3bremains based on unchangedorigin/mainaf037604a6f8d8ac2b9a3279b176aebed61aefb5.Author self-review found no new candidate defect. Full adapter regression is 115/115. Fresh canonical proof after the correction is 11/11 PASS: created
2026-08-23T05:25:31.267Z, workspace fingerprintsha256:684f82601f4b11bddb75960cfddc4406ae5fa8843a5d8218f87a7b6fcc3b7e8f.The required Sol High + Fable High joint gate is next on this changed/proven tree. The prior partial receipt is stale and non-certifying; no exception or unchanged-input retry is in use.
BaseInfinity commentedon Aug 23, 2026
Required review verdict checkpoint — normal NOT CERTIFIED, second bounded correction
Candidate
299c868c000a496308426c04e591e9eab9677e3bcompleted both Sol High and Fable High normally. Advisor remained disabled; validatedStructuredOutputcompleted cleanly. Initial verdicts split, the one allowed verbatim cross-feed ran, and the joint receipt endednot_certified,certifying: false, outcomereviewer_verdict, retry not recommended.Blocking Sol P1: when a transient reviewer failure coexists with a completed peer’s P0/P1 findings, the partial receipt retains the blocker but still recommends retrying the unchanged candidate. That guidance must be suppressed.
Conceded P2 truthfulness gaps in the same #157 allowlist: auth preflight over-labels unrelated CLI/malformed-output failures as auth; an unconfirmed availability probe replaces the original failed-review runtime; and the outer catch discards reviews completed before a candidate-integrity failure.
I am taking the second and final bounded corrective round for those P1/P2 findings, with RED fixtures first. I will also make the directly adjacent fallback/cancelled/invalid-verdict metadata internally consistent where the same fixtures own it. If P0/P1 remains after the next required joint gate, this plan stops; no waiver or third correction will be used.
BaseInfinity commentedon Aug 23, 2026
Second/final corrective TDD checkpoint — GREEN
RED reproduced the Sol P1: a completed Fable
NOT CERTIFIEDresult with retained P1 evidence plus a transient Sol quota failure produced retry guidance for the unchanged candidate. Companion REDs covered auth transport vs malformed output, failed-review/probe runtime retention, completed reviews before candidate mutation, fallback-cause masking, and cancelled-peer infrastructure metadata.GREEN:
availability_probe;Both focused integrated groups pass. New staged candidate tree:
795f021afac0437768d49fe3659e810c1eaec68d. This is the second and final corrective round under the current plan. Broad proof, self-review, fresh canonical stamp, and one final required joint gate remain; any remaining P0/P1 stops the plan.BaseInfinity commentedon Aug 23, 2026
Final corrective candidate proof checkpoint
Frozen staged tree
795f021afac0437768d49fe3659e810c1eaec68dremains based on unchangedorigin/mainaf037604a6f8d8ac2b9a3279b176aebed61aefb5.Focused second-round RED/GREEN passed. Full adapter suite passed 115/115. Author self-review found no remaining blocker. Fresh canonical proof passed 11/11: created
2026-08-23T05:58:08.890Z, workspace fingerprintsha256:c2694e606faeb3f891c5759dbc4c911c5843888f03c02c02f3d1e1cbef6ae926.The final allowed Sol High + Fable High joint gate is next. Any remaining P0/P1 stops the current plan; no third correction or review exception is available.
BaseInfinity commentedon Aug 23, 2026
STOPPED — final gate NOT CERTIFIED; correction budget exhausted
Final frozen candidate:
/private/tmp/codex-sdlc-issue157fix/issue-157-review-outcomesaf037604a6f8d8ac2b9a3279b176aebed61aefb5795f021afac0437768d49fe3659e810c1eaec68d2026-08-23T05:58:08.890Z, fingerprintsha256:c2694e606faeb3f891c5759dbc4c911c5843888f03c02c02f3d1e1cbef6ae926The final allowed Sol High + Fable High gate completed normally. Advisor remained disabled; both reviewers completed initial and the single reconciliation round. Receipt: schema v4,
status: not_certified,certifying: false, outcomereviewer_verdict, retry false. Both final reviewers returnedNOT CERTIFIED.Remaining P1s:
confirmClaudeReviewerUnavailable()treats a nonzero availability probe carryingAUTH_OR_INPUT_BLOCKEDorPROVIDER_OR_TRANSPORT_FAILUREasinvalid_executionbecause those terminal states fall through the unconfirmed branch. The nested probe runtime and aggregate classification contradict each other.The current plan used its second/final corrective round. Per the SDLC contract, stop here: no third correction, waiver, exception, unchanged-input retry, commit, push, PR, or delivery occurred. #109 remains frozen.
A future continuation requires explicit human approval to replan a newly scoped correction. If approved, start with narrow REDs for (a) probe auth/transport terminal-state classification and (b) simultaneous transient + non-transient pair failures, then use the normal focused proof → self-review → fresh 11/11 → required joint gate. Do not reset or silently extend the exhausted correction budget.
BaseInfinity commentedon Aug 23, 2026
Human-approved narrow replan
After the prior STOP checkpoint exhausted its correction budget, the maintainer explicitly directed this release train to keep going. This is a new scoped plan; it does not erase, reset, waive, or silently extend the prior correction history.
Authorized scope is limited to the two remaining P1s from the final gate:
invalid_execution;Execution remains RED first, focused GREEN, full proof, self-review, one fresh canonical 11/11 stamp, then the normal required Sol High followed by Fable High. No review exception is available. #109 remains frozen until #157 integrates.
BaseInfinity commentedon Aug 23, 2026
Human-approved replan TDD checkpoint — GREEN
The two newly scoped P1s were reproduced before implementation:
Minimum GREEN on staged tree
e670ecdc1a66eee7f59114fbafea3a754d7cd526:The dirty root checkout remains untouched. Full proof and author self-review are next, followed by one fresh canonical 11/11 proof and the normal required Sol High → Fable High gate.
BaseInfinity commentedon Aug 23, 2026
Replanned candidate proof checkpoint
Frozen staged tree
e670ecdc1a66eee7f59114fbafea3a754d7cd526remains based on unchangedorigin/mainaf037604a6f8d8ac2b9a3279b176aebed61aefb5. The dirty root checkout remains untouched.Author self-review of the exact new delta found and corrected one cancelled-probe precedence edge through an additional RED → GREEN check. Focused supervision and isolated partial-receipt/fallback integration pass. Adapter is 115/115.
Fresh proof-stamped canonical suite passed 11/11: created
2026-08-23T07:26:40.444Z, workspace fingerprintsha256:e17dfb65a2d3304f67d8099ece910ac138a8a6634fc66e27578fdb46a75f63f2. An earlier direct 11/11 preflight also passed but did not write proof state and is explicitly non-certifying; only this guarded stamp is being supplied to reviewers.The normal required candidate-bound Sol High + Fable High joint gate is next. No exception, waiver, unchanged-input retry, or broadened scope is in use.
BaseInfinity commentedon Aug 23, 2026
Required joint gate — normal NOT CERTIFIED; first bounded correction in approved replan
Frozen tree
e670ecdc1a66eee7f59114fbafea3a754d7cd526completed both Sol High and Fable High normally, including the single allowed reconciliation round. Receipt: schema v4,status: not_certified,certifying: false, outcomereviewer_verdict, retry false. No auth/quota/timeout/tool-use/invalid-execution exception occurred.Both final reviewers converged on one P1: after confirmed Fable unavailability and a failed Opus fallback, the partial receipt substitutes the requested Fable identity and drops the preferred-review plus availability-probe evidence chain while retaining only fallback runtime. That is candidate-born and inside #157’s truthful requested-versus-actual reviewer contract.
Directly adjacent P2: a clean availability probe that contradicts the original suspected outage is labeled
invalid_execution; it should remain fail-closed but classify as transient provider inconsistency.I am taking the first bounded corrective round in this human-approved replan, with REDs first for those two behaviors. P3 timing/schema/late-metadata/vacuous-empty observations remain out of scope and non-blocking. After GREEN: exact-delta self-review, a fresh proof-stamped 11/11 run, then the normal required joint gate again. No waiver or exception.
BaseInfinity commentedon Aug 23, 2026
First replanned corrective round — focused GREEN
RED reproduced both reviewed defects:
invalid_executioninstead of transient invocation infrastructure.GREEN on new staged tree
0aa05dac28b7945ff68b1bac1de70dd8a6c0a2bb:preferred,configured, orfallback);preferred_reviewplusavailability_probe;fallback_reason: quota_exhausted;invocation;Focused supervision tests and the isolated fallback/partial-receipt integration group pass. Exact-delta author review and syntax checks are clean. One fresh guarded canonical 11/11 proof is next, then the required joint gate. P3 findings remain unchanged and out of scope.
BaseInfinity commentedon Aug 23, 2026
Corrected candidate proof checkpoint
Frozen staged tree
0aa05dac28b7945ff68b1bac1de70dd8a6c0a2bbremains based on unchangedorigin/mainaf037604a6f8d8ac2b9a3279b176aebed61aefb5; dirty root remains untouched.The first replanned correction passed focused RED/GREEN, exact-delta self-review, syntax checks, and the single guarded canonical proof run: 11/11 PASS, adapter 115/115, created
2026-08-23T07:53:02.692Z, workspace fingerprintsha256:a30d17726b2a1dc9997ad977ca5ad2b4118ae2cdbfda2dcd6e818fb352fcc3a0.The corrected receipt contract now retains attempted preferred/configured/fallback identity and the full failed-fallback runtime chain. The final required Sol High + Fable High joint gate is next. No exception, waiver, or unchanged-input retry is in use.
BaseInfinity commentedon Aug 23, 2026
Final required gate — CERTIFIED
Frozen tree
0aa05dac28b7945ff68b1bac1de70dd8a6c0a2bbcompleted the normal required Sol High + Fable High gate with clean initial agreement and no reconciliation round.Receipt: schema v4,
status: certified,certifying: true,joint_verdict: CERTIFIED; both reviewer outcomes completed withCLEANruntime andCERTIFIEDverdicts. Proof binding: created2026-08-23T07:53:02.692Z, fingerprintsha256:a30d17726b2a1dc9997ad977ca5ad2b4118ae2cdbfda2dcd6e818fb352fcc3a0. Reviewer identity: Sol High configured plus preferred Fable High; no fallback or exception.Fable recorded two P2 and two P3 non-blocking observations; neither reviewer reported P0/P1. They remain explicit residual follow-up risk and are not being silently repaired after certification.
Local time is Sunday after-hours, so the fixed-argv reviewed GitHub delivery boundary is next. It will commit only this certified tree, create/verify CI, and integrate only if the base remains unchanged and checks pass.
BaseInfinity commentedon Aug 23, 2026
Integrated terminal checkpoint
PR #166 merged and closed #157. Certified commit and remote
main:a07fc3d3fb77767de4c6e677e24bce628256668a; tree0aa05dac28b7945ff68b1bac1de70dd8a6c0a2bb.Both required Windows checks passed:
32627038910:windows-npmSUCCESS;32627153269:windows-npmSUCCESS in 2m13s.The 1.0.0 milestone now has 7 open blockers / 13 closed. #109 is the authoritative next issue and may now resume from its frozen v4 pilot without resetting its cumulative correction budget. Dirty root remained untouched.