[LinuxDevil]

Why

S12 (M4 parity plan, E6 and Paper B's B7). Some failures can't be replayed: there's no recorded input, or the tools aren't safe to simulate. Paper B handles these with a forward trial on later sessions. That is weaker evidence than replay, so per decision 1 (S0) this canary is built but off by default. It only runs when improve.json sets canary.traffic. It serves an unproven harness to real users, and the docs say so.

Changes

• src/improve/trafficCanary.ts (exported from lousho/improve):
  • startCanary(db, dir, { result, base, replay?, cycle?, agent?, maxAlwaysOnChars? }, settings) audits canary-start and then writes .lousho/canary.json. p0 is the base's flagged rate over its last 20 runs, or 1 with no history. It refuses unless canary.traffic is set. It also refuses when share > 0.25 or sessions > 50, when paused (stop file, LOUSHO_IMPROVE=off, enabled: false), when a canary is already running, during a rollback cooldown (a rollback audit entry within cooldownHours, or .lousho/state.json cooldownUntil), when the replay canary failed, when the result is not a proposal, when the base is stale, or when the candidate fails validateHarness.
  • canaryVerdict(db, dir, settings) is read-only. Candidate runs come from runsByRevision (S8) after startedAt and are grouped by session (gen_ai.conversation.id). A session is flagged if any of its runs is. The trial promotes once sessions sessions exist with p̂ = 0 or p̂ ≤ p0 − 0.05 (compared with a 1e-9 tolerance) and no new failure class compared with the base's last 20 runs. It retires early:
    • on any new failure class;
    • as soon as the flagged runs rule out a promotion;
    • after monitor.giveUpDays without enough runs;
    • when the directory changed, the loop was paused, or the opt-in was removed.
  • It reuses S9/S10 instead of new statistics. Runs come from finishedRuns (ordered by end time; the base history ends before the start). The trial also retires when significantDrop finds the candidate sessions pass significantly less often than the base, or when judgeApplied (S10) judges the candidate's scores a regression. These two checks can only retire, never promote.
  • settleCanary(db, dir, settings) acts on the verdict. In auto mode a promote goes through applyProposal(…, { mode: 'auto', by: 'canary' }), which re-runs the stale and validation checks. In manual mode it goes through savePending. A refused promote retires the canary. Every ending removes canary.json and audits canary-end.
  • stopCanary(dir, { actor, reason }) is for the kill switch or a person.
  • readCanary and routesToCandidate are exported too.
• src/improve/canaryArm.ts serves the canary. Per run it re-checks the file stamps, so a deleted canary.json, the stop file, LOUSHO_IMPROVE=off or an opt-out takes effect on the next run.
  • It serves only a canary.json that passes these checks: its hashes match its revisions, it is within the caps, the candidate passes validateHarness (with the result's failed checks), and the latest canary-* audit entry is the matching canary-start on an intact hash chain. Its base must also equal the harness the agent was loaded with. Anything else, any error included, serves the base.
  • Routing is stable per session: sha256(sessionId) mod 1000 < share × 1000. A run with no session id gets the base. The share is the smaller of canary.json and improve.json, capped at 0.25. Served sessions (hashed) are kept in .lousho/canary-sessions.txt, so the cap holds across restarts and processes on one host. The file is cleared at start and at end.
• createAgent: new harnessCanary?: (ctx) => HarnessArm | undefined option. It is honoured only with static string instructions. When the callback returns an arm, the run serves that arm's instructions and skills and the span carries its revision. A throw serves the base. With no option set, the static-spec path is unchanged.
• loadAgentDir attaches harnessCanary only in this case:
  • improve.json opts in;
  • the instructions come from instructions.md with no instructions/<family>.md;
  • the caller overrides none of instructions, prompt, skills or harnessCanary.
• docs/improve.md: a short "Live traffic canary (opt-in)" section with the warning. Full docs come in S27.

Products alignment (R12): none. This is a single-agent-directory feature.

Wiring for S23 (healOnce, #994) and S24 (watchAgent, #997)

S23 and S24 are now on main. This PR does not change them; the follow-up is:

• src/improve/heal.ts decide(), after if (!canary.passed) return rejected(...) and the recheck(state) pause check:
  • when state.settings.canary.traffic is set, call startCanary(db, state.dir, { result, base: state.harness, replay: canary, cycle: state.cycle.id, maxAlwaysOnChars: MAX_ALWAYS_ON }, settings);
  • return a new HealCycle outcome such as 'canary' instead of applyAndGuard or queue;
  • leave the path unchanged when canary.traffic is unset.
• src/improve/watch.ts steps(), after checkApplied:
  • call settleCanary(options.db, options.dir, options.settings);
  • on applied, count it toward maxAppliesPerDay and run guardCorpus, as applyAndGuard does; check the daily cap before settling, because settleCanary doesn't check it;
  • notify (S25) on queued and retired;
  • skip runCycle while readCanary(dir) returns a canary, because there is one canary per agent.
• pauseImprove may also call stopCanary(dir, { actor: 'scheduler', reason }). Serving already stops on the next run once the stop file exists, and settleCanary retires a paused canary.
• The cooldown already lines up: startCanary reads .lousho/state.json cooldownUntil, an ISO string written by S10/S24, and also refuses to start within cooldownHours of a rollback audit entry.
• The replay canary's canary-end audit entry from healOnce doesn't end a running traffic canary. Only a canary-end for the same base and candidate does. There is a test for this.

Known limits

• A session that already started on the candidate keeps the candidate's instructions in its history after the canary ends. This is existing session behaviour: the system prompt is fixed at the first turn. Its skills and revision stamp go back to the base.
• Concurrent processes appending to the sessions file at the same moment can overshoot the cap by the number of racing processes. Multi-host coordination is out of scope, as the ticket says.
• In manual mode the candidate reaches users before a person approves it; only the apply waits. This follows the ticket, and the docs say so.
• Nothing calls settleCanary until S24 wires it in. The docs say to call it on a timer.

Testing

• New tests, written first:

  • src/improve/trafficCanary.test.ts (41 tests): routing stability and a 10% share; off by default; p0; cooldown; one canary per agent; refusals; pending, promote and apply as canary; manual mode queues; retire on a new class; early abort; directory changed, paused and opted out; giveUpDays; a forged canary.json is never served or promoted; a stale promote retires; live abort in the arm; the caps.
  • src/createAgentCanary.test.ts (6 tests): byte-identical system prompt and revision with no canary.json, opted in or not; a routed session gets the candidate and its revision stamp, other sessions get the base; new sessions stop getting the candidate after the canary ends; a caller override turns the canary off; createAgent({ harnessCanary }) serves per run; a throwing callback serves the base.

• npx vitest run src/improve src/agentDir src/createAgent* src/observe src/handoffs.test.ts src/session src/execution

• typecheck (src, tests, examples), lint, build, api:update + api:check, docs:llms + docs:llms:check, docs:verify-snippets -- --skip-build, docs:ratchet (1724, at baseline), npx fallow audit --base origin/main after scoped coverage (no issues). pack-smoke: 15.4 MB unpacked, under the 20 MiB cap.

• Independent safety review: no blockers. These were fixed in the second commit:

  • the session cap now persists across processes;
  • the trial counts sessions, not runs;
  • the floating-point boundary at p̂ = p0 − 0.05 promotes as the ticket says;
  • canary-start is audited before canary.json appears, and the audit log is part of the arm's cache key;
  • a symlinked .lousho is refused for the canary lock;
  • the audit hash chain is verified;
  • failed checks are always re-checked;
  • the trial window was widened;
  • the docs now cover the settle cadence, opt-in at load and manual mode;
  • after rebasing on S23, a replay canary-end for another candidate no longer stops serving.

  Each fix has a test.

🤖 Generated with Claude Code