[LinuxDevil]

Why

M4 ticket S10 (capability row E7, and Paper B's "retire on regression", B8 and B13). The gate and the canary each judge only a handful of cases. Real traffic is the final check, and this monitor is what makes auto-apply reversible without a person. The decisions taken in S0 apply here:

• 20 runs, a δ of 0.05, one-sided 95%;
• automatic rollback in both auto and manual mode;
• with too few runs, the monitor holds rather than passes.

Replaces #992. That PR closed when its branch was deleted after a merge attempt that failed on a conflict. This PR has the same commits, rebased on the corpus guard (#989), which now holds the size-budget change.

Changes

New src/improve/monitor.ts, exported from lousho/improve:

/** Compares runs under the current revision with the runs before it was applied, and rolls back on a regression. */
export async function checkApplied(db: SqlDatabase, dir: string, settings: Pick<ImproveSettings, 'monitor' | 'cooldownHours'>, options: { agent: string; now?: number }): Promise<MonitorVerdict>

MonitorVerdict.state is one of idle, pending, kept, kept-unverified, regressed or rolled-back. It carries reason, runs, baselineRuns, before, after and to, where before and after are WindowStats. It is an interface, not a union, so every field has a JSDoc and the docs ratchet stays at 1724.

What it watches. It watches only when the newest apply or rollback audit entry is an apply whose to equals currentRevision(dir). A directory a person edited by hand is idle and never touched. After a rollback there is nothing left to watch.

What it compares. It reads only the given agent's runs. The name is resolved to exactly one agent_id: an empty name or a name two agents share is refused. Revision ids are content hashes, so agents built from the same template share them. Runs are ordered by end time, so a late-finishing run can't move an earlier look's window:

• after is the first runs under the applied revision that started after the apply;
• before is the last 3 × window runs of the parent revision that ended before the apply.

With fewer than max(minRuns, window) runs on either side, the verdict is pending.

Serialisation and state.

• The whole check runs under .lousho/monitor.lock, so two concurrent checks audit a single decision.
• state.json is read, merged and written under .lousho/state.lock with an atomic write, and other tickets' keys are kept.
• A state.json that is not a JSON object is refused rather than overwritten.
• A stored verdict is keyed by the apply's audit seq, so a revision applied again is watched again.

Statistics. The ticket's rule as written does not meet its own acceptance bars:

• Its rule is "after more than maxDrop below before, and after's Wilson upper bound below before's pass rate". It treats before as exact, and in a Monte Carlo run it rolls back 17% of stable 0.8 agents at 20 runs each.
• Even a calibrated two-sample test at one-sided 5% has only about 75% power for 0.9 → 0.6 at 20 vs 20.

So the monitor uses a group-sequential design:

• Looks: up to three, at 20, 40 and 60 runs.
• Rollback: a look rolls back when the pass rate fell by more than maxDrop and Newcombe's interval (from Wilson bounds) for the difference excludes 0 at a per-look z of 2.1.
• Waiting: a drop that is not yet significant waits for the next look. It is kept at the third. This is also the "hold rather than pass" decision.
• Continuous scores: for values that are not pass/fail, a mean drop by more than maxDrop that the S9 trajectory gate (at the same z) reads as a regression also rolls back.

Measured over 20,000 seeded simulations with this code:

Agent	Rolled back	Bar
Stable 0.8	3.5%	at most 5%
Stable 0.9	2.4%	—
True drop 0.9 → 0.6	95%	at least 90%

Pocock's 2.004 gives 4.3%, which is too close to the bar for a 1,000-draw test, hence 2.1.

What each outcome writes. The verdict is stored in .lousho/state.json (monitor: { revision, at, verdict }), so a decided revision is not judged or audited again.

Outcome	Audit and state
Rolled back	cooldownUntil (from settings.cooldownHours) written first, then rollbackHarness(dir, parent, { actor: 'monitor', reason, expectedFrom: current }), which audits rollback, then an audit retire with evidence.score. lastGoodRevision now treats a retired revision as failed
Rollback fails for any reason (a refusal, a partial write, I/O)	audit guard-fail once (beside the writer's own refusal) and settle regressed, so it is not retried every tick
autoRollback: false	audit guard-fail and report regressed; files unchanged
Kept	audit guard-pass, so lastGoodRevision sees it
No verdict after giveUpDays	audit a new unverified action once, and report kept-unverified; never rolled back

Other changes:

• src/improve/audit.ts: a new unverified action.
• src/improve/apply.ts: lastGoodRevision counts a retire as a failure for the retired revision.
• src/observe/store.ts: finishedRuns gains endedBefore and byEnd on the agent id-or-name path.
• src/observe/continuousScore.ts: pointOf is shared by scoreSeries and the monitor.
• src/observe/continuousScore.ts: scoreSeries's agent is optional (a revision is enough), and WindowStat's fields gain JSDoc. WindowStat is exported from lousho/improve.

R12: products WG.2 (rollback for product layers) reuses this through rollbackHarness.

Testing

src/improve/monitor.test.ts uses a real observe db seeded per revision and a temporary agent directory applied with the real applyProposal:

• pending below minRuns;
• kept when the pass rate holds (a guard-pass entry);
• rolls back on a 30-point drop with 20 runs each (files byte-equal to the base, audit apply, rollback, retire, and the chain verifies);
• does not roll back a 5-point dip within noise;
• waits for another look when a drop is not yet significant, then keeps it at the third;
• does not touch a hand-edited directory;
• idle with nothing applied;
• autoRollback false only reports;
• rollback writes a cooldown;
• a mean score that falls without crossing 0.5 is caught by the trajectory gate;
• kept-unverified after giveUpDays, audited once and never rolled back;
• a decided verdict is not judged or audited again;
• nothing left to watch after a rollback;
• the test defaults equal loadImproveSettings defaults;
• Monte Carlo, 1,000 seeded draws each: false rollback is at most 5% on a stable 0.8, and detection is at least 90% for a true drop from 0.9 to 0.6. A continuous score at 0.8 ± 0.2 also rolls back at most 5% of the time, and a fall to 0.65 is caught at least 90% of the time.
• Added after review:
  • state.json keeps foreign keys, and lastGoodRevision skips the retired revision;
  • a corrupt state.json is refused and left as it was;
  • a revision applied again is watched again;
  • only the agent's own runs count, found by registered name;
  • a run that started before the apply and finished after it is left out;
  • a refused rollback is audited once and reported;
  • two concurrent checks audit one decision;
  • a continuous drop between 0.05 and 0.15 is caught.

Checks run: typecheck (src, tests, examples), lint, build, api:update + api:check (api/improve.api.md gains checkApplied, MonitorVerdict and WindowStat), docs:llms + docs:llms:check, docs:verify-snippets -- --skip-build, docs:ratchet (1724), vitest over src/improve, src/observe and the improve, improveAdmin and observe CLI tests (978 passed), and npx fallow audit --base origin/main after scoped coverage.

Review: an independent review found two blockers, both fixed:

• a re-applied revision was never watched again;
• runs were pooled across agents.

Nine should-fixes were also fixed:

• a refused rollback was retried every tick;
• a failure right after a rollback could lose the cooldown;
• concurrent checks were not serialised;
• state.json handling could lose other tickets' keys;
• lastGoodRevision still returned the retired revision;
• late-finishing runs moved the comparison windows;
• cooldownHours was ignored;
• an undocumented type appeared in the API;
• continuous drops below 0.15 were missed.

A re-check found four more should-fixes, all fixed:

• an empty agent pooled every agent;
• a shared name pooled agents;
• the exact-id query ignored the end-time filters;
• non-configuration rollback errors were retried.

The CHANGELOG nit is out of scope, because the release job writes it.

Follow-up for S24: the generated Telegram bridge template (src/cli/init/telegramTemplate.ts) also writes .lousho/state.json (relative to its cwd), without the lock or an atomic write.

Docs: S27.

Notes for later tickets:

• S24 calls checkApplied(db, dir, settings, { agent }) each tick, reads cooldownUntil from .lousho/state.json, and should write state.json under .lousho/state.lock too.
• S25 shows state.monitor.verdict, and warns while it is pending or kept-unverified.
• S12 may reuse judgeApplied for its forward trial.

🤖 Generated with Claude Code