Repository navigation
Find what still makes the macOS test shards slow after #857 #894
Description
This was generated by AI during triage.
After #857, the macOS shards still spend 65-70 minutes of test time per CI run. #857 removed 22% of process spawns in its benchmark test on Linux. The macOS saving on main has been smaller and within the ±15% run-to-run noise #857 itself measured:
| Shards 1 / 2 / 3 / 4 test-step seconds | Sum | |
|---|---|---|
main before #857, 39337614995 (from #857) |
1107 / 1120 / 969 / 1191 | 4,387 |
main before #857, 39328254506 (from #857) |
1193 / 937 / 1174 / 972 | 4,276 |
main after #857, 39424072554 |
1189 / 1063 / 959 / 1037 | 4,252 |
main after #857, 39424576921 |
1076 / 935 / 986 / 1019 | 4,020 |
main after #857, 39430050727 |
769 / 1099 / 1066 / 982 | 3,920 |
In 39430050727, nextest flagged 93 tests over 30 s and 16 over 60 s. By module: spec_run 47, architect_run 9, security_review 7, security_run 5, pickup_run 5, merge_run 5, base_fix 3, bin 3, then run_notification, codex_sessions, ci (2 each), and stop, show_logs, issue_number (1 each). Linux runs the whole suite in about 8.5 minutes.
#857's analysis came from strace on x86_64 Linux. No one has yet attributed the macOS time itself: how much is process spawn, how much is the launch-side --git-common-dir rechecks #858 holds, and how much is something else, such as fixed sleeps, real-time polling or timeouts in the test harness.
Related, not duplicated
- shorten macos CI time #846 (closed) / Inspect Worktree ownership with fewer git processes to shorten macOS CI #857 (merged 2026-10-20 02:10 UTC, shipping in 0.12.3 via Release 0.12.3 #892): fewer git processes per ownership inspection.
- Decide whether Worktree ownership inspection may recheck the launch repository less often #858 (
needs-triage): whether ownership inspection may recheck the launch repository less often. That is a security design decision for the Day shift and stays there; this issue must not change it. - Run the full macOS suite on main and nightly, with a one-job macOS gate on PRs into main #893 changes which PRs pay this cost, not the cost itself.
Activity
JacobStephens2 commentedon Oct 20, 2026
Agent Brief
Category: enhancement
Summary: Attribute the remaining macOS test time and remove the causes that are not Worktree ownership rechecks
Current behavior:
After #857, the four macOS shards spend 3,900-4,250 seconds of test time per CI run, against about 8.5 minutes for the whole suite on Linux. 93 tests run over 30 seconds on macOS, 47 of them in
spec_run. The only profile so far was taken on Linux, and it pointed at Worktree ownership inspection's git spawns.Desired behavior:
spec_runones), measure where the wall time goes on macOS: spawned processes by program and caller, time spent in fixed sleeps, polling intervals or timeouts, and anything else that dominates. Use a macOS runner, aworkflow_dispatchjob, or a temporary diagnostic step in a PR. Record the method and numbers indocs/ci-performance.mdunder "Where macOS test time goes".Key interfaces:
gh,claudeandcodexbuilt from the test fakes, and the helpers that start the binary against real git.Directory::verify_repositoryand Worktree ownership inspection are read-only for this issue.Acceptance criteria:
docs/ci-performance.mdgives a macOS breakdown of the slowest tests' wall time by cause, with the runs or commands that produced it.mainrun after it merges, is compared in the PR body against the three post-Inspect Worktree ownership with fewer git processes to shorten macOS CI #857mainruns (3,920 / 4,020 / 4,252 s), with the ±15% run-to-run noise stated.cargo nextest run,cargo test,cargo clippy --all-targets -- -D warningsandcargo fmt --checkpass.Out of scope: