Repository navigation
feat(improve): lousho improve --watch with budgets, rate limits, lease and kill switch - #997
Merged
Merged
Conversation
…e and kill switch Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Retry after an outage or a blocked cycle, renew the lease during a tick, a per-directory lease id, meter the watcher's own spend into the day's usage, check stop and pause between steps, re-read settings each tick, and keep the Telegram owner when telegram.json is empty. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
S24 of the M4 plan (
m3-notes/m4/parity.md): E3, E4 and E5. Self-healing means the cycle runs without anyone starting it, so it needs hard limits: a kill switch, one watcher per directory, daily budgets, a cooldown after a rollback, dedupe, rate limits, and a clean stop.Changes
src/improve/watch.ts:watchTick(options, context), in order:.lousho/improve.stop,LOUSHO_IMPROVE=offorimprove.jsonenabled: false(re-read every tick) stops the tick, audited once aspause(remembered instate.json); turning it off auditsresume.acquireLease('improve-<sha of dir>', '<pid>:watch:<uuid>', 2 × interval)on a file store in.lousho/watch(a store without leases is refused); a second watcher getsleased. The lease is renewed every third of its TTL while the tick runs; losing it aborts the tick at the next step. healOnce'scycle.lockstill blocks concurrent cycles.checkApplied(S10); then the harvest below.guard.everyHours(rollback: true, each run of one revision gets its ownfirstRepeatrange from 200).cooldownUntilis in the future, and unless the failure-group signatures, the revision, the failed-run count and the newest failed trace equal those of the last cycle that did work (dedupe; an outage, busy, budget, cooldown, paused or interrupted cycle doesn't set the key, so the next tick retries).healOncegets the day's budget left after the watcher's own guard and corpus runs asmaxTokens; healOnce enforces cycles/day, applies/day, tokens/day and the audited-rollback cooldown under its lock.outagecycles in a row write the stop file and auditpause.stop()are re-checked between every step (monitor, harvest, guard, cycle); settings are re-read every tick (reloadSettings; the CLI keeps its--approval).watchAgent(options)ticks everyintervalMs(default 15 min); a tick that throws is reported viaonErrorand the watcher keeps going.stop()aborts between steps (an apply and its guard always finish), waits for the tick, and releases the lease.improveSchedule(options): adefineSchedule()schedule whoserunis one tick.withRateLimitatrateLimit.rpm(new setting, default 20).src/improve/watchSteps.ts: watcher state, the harvest (once per apply: when the monitor sayskept,voteOnOutcome('kept')and admit the training cases the edit fixed to the corpus asorigin: 'fixed'after 6/6 fresh runs; onrolled-back,voteOnOutcome('rolled-back')), and the scheduled guard. Their probes are metered against the day's budget left and their spend is recorded even when they stop early;usageTodaynow includes it, so healOnce (and a one-shotlousho improve) see it too. The harvest claim is a check-and-set under the state lock.src/improve/state.ts:state.jsonread/update under.lousho/state.lockwith an atomic write, now shared bymonitor.tsand the watcher.src/improve/pause.ts+lousho improve pause <dir> [--reason] [--by]/resume <dir> [--by]: write / remove the stop file, audited.lousho improve <dir> --check <m> --watch [--interval 15m] [--agent X]; Ctrl-C/SIGTERM stops after the step.--watchrefuses a dry run,--max-tokensand--out;--intervalneeds--watch.src/cli/init/telegramTemplate.ts) wrote.lousho/state.jsonwithout the lock and would clobber the loop's state; it now keeps its own.lousho/telegram.json(atomic rename) and still reads the owner from an oldstate.json. Bridges generated before this change still writestate.jsonwithout the lock; the S27 release note should tell Telegram users to regeneratebridge.ts.state.ts(.lousho/state.json),pause.ts(.lousho/improve.stop), the watcher's lease files under.lousho/watch/(viafileStore).Review (independent): one blocker (the dedupe key was saved after an outage or a blocked cycle, so the next tick never retried) and should-fix items (lease renewal, per-dir lease id and refusing lease-less stores, metering the watcher's own spend, checking stop/pause between steps, re-reading settings, Telegram
{}fallback, missing budget tests) are fixed in this PR. Not done: fake timers in tests (the tests drivewatchTickdirectly andwatchAgentwith real short intervals); a paused watcher also skips the monitor's auto-rollback, which S27 should document.Testing
src/improve/watch.test.ts(mock editor model, temp agent directory, real observe db): the stop file prevents any cycle and is audited once (and resume is audited);LOUSHO_IMPROVE=off/enabled: false; the second watcher on the same dir does nothing;maxAppliesPerDay 1blocks a second apply; the monitor's cooldown blocks cycles; the token budget passed to healOnce shrinks as spent; same signatures twice skip the second cycle; three outages in a row (no new failures between ticks) pause with one audit entry; maxCyclesPerDay, the daily token cap and an audited rollback stop the cycle; the watcher's own guard tokens count toward the day; the lease is renewed during a long tick so a second watcher stays out; the scheduled guard runs onceeveryHourspassed; a kept edit's fixed cases join the corpus once;improveSchedule;watchAgent.stop()during a cycle exits after it with nothing applied. Cycles/day and tokens/day caps are covered byheal.test.ts.src/cli/improve.test.ts:--watchruns until stopped, refuses--dry-run,--intervalparsing;pause/resumeaudited and the paused CLI cycle writes nothing.api:update+api:check,docs:llms+docs:llms:check,docs:verify-snippets -- --skip-build,docs:ratchet(1724), vitestsrc/improve src/cli src/observe,npx fallow audit --base origin/mainafter scoped coverage.🤖 Generated with Claude Code