Post-1.0 soak: daily consumer-repo feedback and proven improvements #112

Activity

1. BaseInfinity commented on Aug 10, 2026

   

   BaseInfinity

   on Aug 10, 2026

   OwnerAuthor

   More actions

   Empirical soak requirement from the #110 implementation:

   Do not tighten or expand the bounded-review rules based on taste alone. After 1.0.0 ships, exercise the installed workflow in real consumer repos and record per task: wall time, number of focused and broad proof executions, reviewer invocations, corrective rounds, candidate size versus requested defect/feature size, P0-P3 findings and whether they were pre-existing or candidate-born, duplicate suites avoided, and whether the issue actually shipped.

   Compare the observed baseline against the #110 rules (closed allowlist, one frozen completion candidate, no reviewer test rerun, at most two corrective rounds). Keep, loosen, or strengthen rules only when that evidence shows a quality or throughput effect. Planner/Fable/Luna/Spark experiments stay separate so model-routing changes do not confound the workflow measurement.
2. BaseInfinity

   added this to the 1.1.0 — Agentic harness hardening milestone on Aug 11, 2026
3. BaseInfinity commented on Aug 15, 2026

   

   BaseInfinity

   on Aug 15, 2026

   OwnerAuthor

   More actions

   Consumer soak update from m180-jumpseat: mechanical lifecycle TDD caught real stale state, while duplicated prose-string assertions caused unnecessary broad reruns and still required semantic review. Detailed evidence and the proposed risk-tiered rule are recorded in #115: #115 (comment)

   Observed delivery remained healthy (roughly 13 merged PRs / six closed issues in the measured ~17-hour window). Classify this as a proof-profile efficiency finding, not a review-gate failure.
4. BaseInfinity commented on Aug 23, 2026

   

   BaseInfinity

   on Aug 23, 2026

   OwnerAuthor

   More actions

   Consumer soak report — ~18-hour bounded delivery run

   Feedback type

   Pattern sharing plus process improvements from a successful long-running Codex Desktop Goal-mode session.

   Local context

   Consumer: BaseInfinity/m180-jumpseat on Windows, using adapter v1.73.0 with maximum/Sol-high, strict TDD, candidate-bound commit/push guard, session-start guard, and pre/post-compaction guards. The primary checkout was already dirty and remained untouched; implementation used an isolated same-repository linked worktree.

   No product source, secrets, ignored packets, private evidence, or business logic was scanned for this feedback. Only SDLC configuration/file names and already-public lifecycle evidence were used.

   Delivery evidence

   Across the measured run:

   • three bounded product issues and milestones reached terminal closure: feat(review): supervise reviewer lifecycle #156, docs: synchronize 1.0 release roadmap #161, and fix(review): reject forbidden Claude tool use #165;
   • seven implementation/lifecycle/correction PRs merged, ending with product PR fix(review): prevent Fable built-in tools before required review #167 and lifecycle PR Add third-party asset provenance to release preflight #168;
   • the required suite grew from 2,472 to 2,491 tests and remained fully green at every frozen delivery candidate;
   • fix(review): reject forbidden Claude tool use #165 additionally passed 35/35 focused and 278/278 questionnaire-linked checks;
   • actual claude-fable-5 high was used for the required independent gates;
   • concrete Low findings were reproduced and fixed; clean terminal reviews ended the loop;
   • live GitHub state was reread between slices, and no replacement work was invented after the autonomous roadmap became empty;
   • the final goal stopped honestly at two human-owned security/privacy actions rather than touching credentials, private media decisions, history, visibility, or provider state.

   What prevented recursive hell

   1. Closed scope before execution. Each issue had explicit acceptance, nonclaims, and stop boundaries.
   2. Immutable final candidates. Reviews and proof named exact base, commit, and tree; candidate movement reset the gate instead of being hand-waved.
   3. RED → focused GREEN → one meaningful broad gate. Broad proof certified delivery; it was not used as a debugging loop when a focused test could answer the question.
   4. Proportional independent review. Incremental commits were checkpoints, not certification events. Ordinary bounded public claims used one final Fable-high reviewer.
   5. Concrete finding ledger only. Candidate-born actionable findings were fixed; informational observations did not silently expand scope.
   6. Hard correction budget. At most two rounds; after that, stop/split/replan rather than ask reviewers forever.
   7. Failure taxonomy. Product defect, reviewer finding, tool/infrastructure failure, lifecycle mismatch, and human-only gate were never treated as interchangeable.
   8. Separate lifecycle reconciliation. Product integration happened first; counts, pointers, issue/milestone closure, and next-slice selection were reconciled afterward.
   9. Live refresh between slices. Default branch, issues, milestones, PRs, and owner priority overruled stale conversational memory.
   10. Compaction-ready handoff. Exact worktree, branch, candidate/tree, proof counts, review state, dirty-checkout boundary, and next action were carried forward.
   11. No gratuitous parallelism. One writer owned the candidate. Review was independent but non-editing, avoiding cross-agent write races.
   12. Stop when the roadmap says stop. “No autonomous next” was treated as a valid delivery result, not an invitation to manufacture work.

   Proven parent improvements

   • bug: retain failed-test summaries when broad proof output is truncated #124: persist/query exact-candidate broad-proof terminal results after handle loss or compaction — bug: retain failed-test summaries when broad proof output is truncated #124 (comment)
   • Emit bounded provider progress during long dual reviews #164: emit low-frequency provider/phase heartbeat during healthy long Fable reviews — Emit bounded provider progress during long dual reviews #164 (comment)
   • feat(goal): add ordered multi-milestone continuation and cleanup handoff #151: separate remote merge success from local linked-worktree cleanup failure — feat(goal): add ordered multi-milestone continuation and cleanup handoff #151 (comment)
   • Research convergence-based review termination versus fixed correction caps #132: keep the fixed correction cap as control; convergence may stop earlier but must never remove the absolute ceiling — Research convergence-based review termination versus fixed correction caps #132 (comment)

   Recommendation

   Keep the current quality-first shape. The next harness gains should be observability and durable state recovery, not more reviewers or more review rounds. The successful pattern is: one writer, one frozen candidate, one proportional terminal review, one bounded correction ledger, one broad certification gate, then lifecycle reconciliation.