Skip to content

Say what ssh said when a host cannot be swept - #829

Merged
bborn merged 1 commit into
mainfrom
fix/remote-sweep-ssh-stderr
Oct 8, 2026
Merged

bborn merged 1 commit into
mainfrom
fix/remote-sweep-ssh-stderr

Conversation

@bborn

@bborn bborn commented Oct 8, 2026

Copy link
Copy Markdown
Owner

Why

After #828 shipped, every sweep of ivo failed with Could not end tasks' sessions on host; will retry host=ivo … error="exit status 255" and backed off to 15m. ssh exits 255 for every failure of its own: refused, unreachable, session lost, or the remote command killed by a signal. The bare status can't tell those apart.

What

  • endRemoteTaskSessions now wraps the error with the end of ssh's stderr (last 300 bytes, whitespace folded to one line, valid UTF-8): exit status 255: <what ssh said>.
  • That error goes into the daemon warning as before. On the first failure of a run of failures (the one backed off the least), each affected task's log also gets Could not reach <host> to end this task's agent session; will retry: <error>. Later retries in the same run don't add more log lines.

Investigating ivo's 255 (from Bruce)

I used the same ssh options and ControlPath, sh -lc wrapping, and hostSweepScript built from ivo's 28 real blocked runs.

  • Couldn't reproduce. A dry run (kills replaced by echo) under my shell and under the daemon's exact environment (env -i, the daemon's PATH, no SSH_AUTH_SOCK) exits 0 in 0.2–0.6s. It matches no windows and no processes. Auth works with and without an agent.
  • The daemon reached ivo every time. ivo's sshd logged an accepted login from Bruce at each failing sweep (15:29:38, 15:31:09, 15:33:10, 15:37:38, 15:45:38, 16:01:09 UTC). Each session lasted about 60s (the ControlPersist master), so the 255 came after authentication, not from an unreachable host. A process watcher on ivo during the 16:01:09 attempt saw the session but no command from the sweep under it.
  • Self-signal audit, as asked. I found no path by which the sweep can signal its own shell or its pipeline subshells:
    • processes are only listed for this uid;
    • the script itself ($$) and every ancestor are excluded;
    • anything with the script as an ancestor (the $(…) subshells and ps) is held;
    • the kill loops, readlink and resolve run after the snapshot, so they are never candidates;
    • agents and tmux, including a tmux: server title, are excluded;
    • on Linux only processes whose cwd is inside a recorded worktree can match, and the sweep runs from $HOME (sshd-session's cwd is unreadable, so it is skipped).
  • Controlled experiment on ivo. I started a sleep 900 with its cwd in one recorded worktree and ran the real script for that one task, with a session name that doesn't exist. It killed exactly that sleep (P 5653 <pid>) and exited 0.
  • Then it fixed itself. The daemon's 16:16:38 sweep of ivo succeeded with the deployed code and the same remote state (no tmux server, the same stale socket since Oct 6). 30 of ivo's 31 blocked tasks lost their stale session and run rows. Nothing was running to end, so no "Suspended" lines were logged.

So the cause was transient and is still unknown. If it comes back, this PR makes the warning and the task logs say why.

Tested

  • TestRemoteSweepFailureSaysWhatSSHSaid: a fake host whose ssh prints a reason on stderr and exits 255. The error carries both the status and the reason. Over two failed sweeps, the task log says so exactly once. Fails without the change.
  • TestWithStderrKeepsTheEnd: only the end of a long stderr is kept, on one line, and other errors pass through unchanged.
  • golangci-lint v2.13.2: 0 issues. go test -race -p 4 ./...: only the failures that also occur on main on this Mac (the /private/var executor tests, reaper TestLivePanePIDs_HonorsAgentSocket, ui TestRemoteAttachDropsALeftOverViewPairing).

🤖 Generated with Claude Code

ssh exits 255 for every failure of its own, so 'exit status 255' could not
tell a refused session from an unreachable host or a remote command killed
by a signal. The sweep's error now carries the end of ssh's stderr (last
300 bytes, one line), in the daemon warning and, once per run of failures,
in each affected task's log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bborn
bborn merged commit 0716411 into main Oct 8, 2026
4 checks passed
@bborn
bborn deleted the fix/remote-sweep-ssh-stderr branch October 8, 2026 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant