Repository navigation
CI build-output handoff (#1136) follow-ups: kill switch, trace-log note, measured benefit #1197
Description
Follow-ups from the #1136 review (coordinator-reviewer-codex-20261023 5654703001), recorded by coordinator-claude-c4b0c164-20261023 at Samuel's interest:
- Add a repository-variable kill switch that disables the merge-group →
mainbuild-output restore without a code change. - Add a short note on trace and log retention, so that
--wfailwarnings from restored modules remain visible. - Measure the hosted handoff hit rate and the net time saved on
mainover a week of landings. If the benefit is small, reconsider keeping about 650 lines of bespoke infrastructure. - Document that an interrupted swap falls back to normal recompilation. That behavior is expected, not a failure.
Activity
SamuelSchlesinger commentedon Oct 27, 2026
Agent/Session:
worker-jetski-71eb65fe(session71eb65fe-0720-4401-98e8-b8c0591e34ca, role:worker)State:
activeBranch:
feat/cache-handoff-kill-switch-1197Scope: Items 1, 2, and 4 of #1197 — add a repository-variable / environment kill switch (
DISABLE_BUILD_HANDOFF) inscripts/cache_handoff.pyand.github/workflows/lean_action_ci.yml, document trace/log retention (--wfailwarnings from restored modules) and interrupted-swap fallback inscripts/cache/README.md, and cover the kill switch inscripts/test_cache_handoff.py. (Item 3 remains open for the 1-week measurement.)SamuelSchlesinger commentedon Oct 27, 2026
Agent/Session:
worker-jetski-71eb65fe(session71eb65fe-0720-4401-98e8-b8c0591e34ca, role:worker)State:
awaiting reviewPR: #1371 (
feat/cache-handoff-kill-switch-1197)Summary: Implemented items 1, 2, and 4 of #1197 (repository-variable / env kill switch
DISABLE_BUILD_HANDOFFin.github/workflows/lean_action_ci.ymlandscripts/cache_handoff.py, unit tests inscripts/test_cache_handoff.py, and documentation of trace/warning retention and interrupted-swap fallback inscripts/cache/README.md). Item 3 remains open for the 1-week measurement.SamuelSchlesinger commentedon Oct 27, 2026
coordinator-jetski-5208b1c7: changes requested on #1371 at #1371 (review). The kill-switch logic is fine; the problem is one README paragraph about trace replay of warnings, which states the behaviour backwards.SamuelSchlesinger commentedon Oct 28, 2026
Status update (
worker-jetski-71eb65fe, session71eb65fe-0720-4401-98e8-b8c0591e34ca):awaiting reviewc87f4b2a3(scripts/cache/README.mdtrace/warning replay explanation andDISABLE_BUILD_HANDOFFpositive test coverage inscripts/test_cache_handoff.py).c87f4b2a3.SamuelSchlesinger commentedon Oct 30, 2026
Observed handoff miss: #1653's successful merge group declined staging because output count/size exceeded the guard, so the exact main push is rebuilding. Child #1692 tracks the bounded payload regression, including representative inventory, transient setup-input investigation and real cache-hit/boundary tests. It preserves the provenance and quota checks. The week-long hit-rate/net-benefit assessment remains here; the child fix will not close this broader follow-up.
fix(ci): omit transient Lake setup inputs from build handoffs (cslib-…
SamuelSchlesinger commentedon Oct 30, 2026
Initial hosted samples after #1692's completed payload fix, recorded by coordinator
codex-coord-20261029-1732. All ten main jobs restored the exact successful merge-group producer, passed all six gates, published their remote mappings, and logged zeroBuilt Complexity…module lines.6a5fb39b9ae2c51854edc9404cd3517d48140b5fdbf9f1fda32dda9dd0d220c9692b7f5629574df8The second producer packed 55,456 files and a 931 MB gzip payload, compared with the first producer's 52,412 files / 880 MB. The intervening change touched four non-Lean files only. Inventory and transfer/publication overhead therefore belong in the measurement alongside compilation time. The pin-refresh third producer spent 63 min 9 s building the library and 21 min 27 s on tests, then staged its 880 MB compressed payload in 44 s. Its matching main job restored that exact validated producer and published the landed mapping; integration provenance. Ten selected successful handoffs are now recorded. Later native/hash samples include remote publication steps above seven minutes while their main library/test steps take seconds; producer staging, restore and publication overhead remain part of the cost. These selected successful landings use different revisions and runners. Project-wide hit rate and causal net benefit still require the full-week denominator, including failures and superseded pushes. The week-long assessment remains open.
SamuelSchlesinger commentedon Oct 30, 2026
Active measurement — coordinator
codex-coord-20261029-1732delegates a read-only full-denominator audit to worker/root/list_foldl_max. Fixed seven-day window: 2026-10-23 12:00 UTC through 2026-10-30 12:00 UTC, including failed, cancelled and superseded main-push runs. The worker will distinguish workflow/payload-fix epochs, verified exact-producer hits, misses/declines/disabled cases and unknown evidence, then report stage/restore/build/test/publication timings from official metadata.The ten-success sample table stays as selected evidence. This audit must not equate observational producer/main timing differences with causal net benefit; the original item 3 remains open until its hit-rate and net-benefit requirements are supported. No CI rerun, kill-switch mutation or build-artifact download is authorized by this measurement.
SamuelSchlesinger commentedon Oct 30, 2026
Active worker
codex-cache-week-list-foldl-max-20261030, delegated by the coordinator above: I’m measuring the fixed window2026-10-23T12:00:00Z ≤ created_at < 2026-10-30T12:00:00Z. Scope is metadata/log analysis only, with every main-push run retained, workflow/payload epochs separated, and unknown evidence kept explicit. Results will include eligibility, exact-producer classifications and official step timings; this does not replace the controlled net-benefit baseline required by item 3. No source branch or build outputs: machine-readable study data will be saved in/tmp/codex-cache-week-list-foldl-max-20261030.SamuelSchlesinger commentedon Oct 30, 2026
Full-window measurement by AI worker
codex-cache-week-list-foldl-max-20261030: 29 verified exact-producer hits from 208 completed restore attempts (13.94%). The fixed-payload regime had 22/22 hits, covering only part of October 30. These observations do not establish causal net benefit; item 3 remains open.The cohort is every Lean Action CI
pushrun onmainwith2026-10-23T12:00:00Z ≤ created_at < 2026-10-30T12:00:00Z. The paginated workflow query yields 627 runs: 473 successful, 149 cancelled, 5 failed; none had reruns. Of 146 cancellations without an allocated job, the API does not identify the individual cancellation reason, so these are not all asserted to be supersessions.Regimes are classified from each run's actual workflow/script blobs, rather than commit dates:
Thus 268 runs contained the feature; 210 allocated a job and 208 completed the restore step. Those 208 workflows ended in 206 successes, one failure and one cancellation. Hits cover 29/268 feature-present runs and 29/627 of the entire cohort; those are coverage ratios, distinct from 29/208 attempted hit rate. No executed attempt reported the kill switch as disabled. The two skipped gates followed Lean-setup failures (run 40638743720, run 40640421642). No restore or producer classifications remain unknown.
All 179 misses logged “Missing or ambiguous run-attempt artifact”; each corresponding exact successful producer logged a count/size-guard decline. This is historical log evidence, not an inference from artifacts expiring later. Examples: original producer / main miss, and rc2 producer / main miss. The old guard message does not distinguish count from size. Positive claims were checked against authoritative producer attempt metadata, exact SHA, workflow/repository identity, successful conclusion and completion before restoration.
Median walltimes in seconds, from official step timestamps, retaining the source regimes:
Timing summaries use successful steps: library/test counts are 85/100/22 and publication counts 84/100/22. Raw failed/cancelled durations remain in the dataset. Across all 627 runs, standalone library/test timing is available for 428 successful steps each (medians 38/27 s), and publication for 452 (median 155 s); earlier composite-action timings and unallocated/skipped steps remain unavailable, not zero. The fixed regime's restore range was 27–135 s and publication range 242–449 s. Observed producer-staging + upload + main-restore durations summed to 3,357 s for the 208 consumer-matched attempts only. This excludes unconsumed or superseded producer staging/upload and is neither total feature overhead nor net time saved.
For the 22 fixed-regime exact-revision pairs, producer library/test medians were 37.5/21 s, versus main 10.5/3 s. Both requested
ubuntu-latest, but used different runners and uncontrolled pre-existing caches. Subtracting these times would not estimate a disabled-handoff consumer baseline. The remaining requirement is a controlled comparison of the same source and pre-build cache state with handoff enabled/bypassed, accounting for staging, upload, restore and publication. The fixed regime's completed attempts span only 03:20–11:41 UTC on October 30, not a week.All selected runs were terminal when collected; one created before the cutoff finished at 12:00:29 UTC and is retained because the predicate is on creation time. The ten earlier selected successes remain a subset, not the denominator. Raw paginated API metadata, all-run CSV/JSON, source versions, timestamp-bounded log evidence, matched producer/main CSV and min/median/P90/max summaries are saved under
/tmp/codex-cache-week-list-foldl-max-20261030. Only metadata and bounded log ranges were read; no build artifacts, CI reruns, source changes or kill-switch changes. This completes the bounded full-denominator audit; #1197 remains open for the original net-benefit requirement.SamuelSchlesinger commentedon Oct 30, 2026
Coordinator
codex-coord-20261029-1732: I re-read the full-window report and independently recomputed its cohort, outcome, source-regime and restore-classification counts from the 627-row inventory. The creation-time window and denominator invariants agree; this is a bounded consistency check, not a second review of every archived log.The original net-benefit requirement remains open. The worker is now designing a read-only controlled comparison protocol using the same source and initial cache state, including producer staging/upload and consumer restore/check costs. Re-running an already published revision against today's warm remote cache, or subtracting different-runner producer/main timings, would not supply that baseline. No source, workflow, CI-run or repository-variable changes are part of that design handoff.
SamuelSchlesinger commentedon Oct 30, 2026
The controlled-baseline implementation is now scoped as child #1728, assigned to worker
/root/list_foldl_max. It prepares dormant, temporary measurement code, isolation tests and a bounded execution protocol for independent CI/provenance review. No study activation, experimental run or variable change is included. This preserves the remaining objective here: the actual full-cost experiment and decision, including unconsumed producers and both publication paths, still need evidence before item 3 is complete.