Skip to content

Run macOS tests with Xcode's git and Spotlight off, and record where macOS test time goes - #928

Merged
JacobStephens2 merged 2 commits into
mainfrom
issue-894
Oct 21, 2026
Merged

JacobStephens2 merged 2 commits into
mainfrom
issue-894

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Summary

The macOS shards' remaining time is git processes. Most come from Worktree ownership inspection and the launch-side recheck (#858). Homebrew's git also costs about 13 ms per process on the runner, where Xcode's costs 9 ms. This PR leaves the checks alone and makes each git process cheaper in macOS CI:

 test-macos-gate / test-macos-shard (macos-26)
   checkout, rustup, rust-cache
+  sudo mdutil -a -i off                    # Spotlight stops indexing the tests' repos
+  link "$(dirname "$(xcrun -f git)")"/git* into $RUNNER_TEMP/xcode-git
+  put $RUNNER_TEMP/xcode-git first on PATH # Xcode's git 2.50.1, not Homebrew's 2.56.0
   cargo nextest run …

Where the time goes (26 tests alone on macos-26, dtrace; caller split from Linux strace of the same tests):

wall time of a slow test
└── git processes alive: 77–99% (88% summed)
    ├── 41%  rev-parse --git-common-dir    launch recheck, #858, untouched
    ├── 43%  ownership inspection          untouched
    └── 17%  everything else
fakes 1–3% · shells <2% · fsync <0.1 s · sleeps overlap git work

Evidence

macOS test-step seconds (the cargo nextest run step, build included, as in #894's table). Run-to-run noise on main is ±15% (#857):

Run Shards 1 / 2 / 3 / 4 Sum
main after #857, 39424072554 959 / 1189 / 1063 / 1037 4,252
main after #857, 39424576921 1076 / 1019 / 986 / 935 4,020
main after #857, 39430050727 1099 / 769 / 1066 / 982 3,920
This PR, 39559241797 622 / 619 / 607 / 575 2,423
This PR, final run, 39559241308 (re-run) 406 / 435 / 522 / 379 1,742
First main run after merge not yet run; add when it lands

Both of this PR's runs, on the same commit, are well outside the noise: the first is 38–43% below the three post-#857 main runs and the final one 56–59% below. Build excluded, their nextest time was 2,198 s and 1,590 s. The gap between the two is runner variance, larger than #857 measured.

Opening the PR with test:macos already applied started an opened and a labeled run. The first was cancelled, leaving cancelled checks on the head, which docs/ci-performance.md warns about. Re-running it replaced them and gave the second measurement. In that re-run, Linux test failed once in tests/console_install.rs:269, unit_preserves_literal_paths_with_shell_and_systemd_metacharacters: Text file busy (ETXTBSY) executing a freshly copied binary. That file is untouched here, and test passed in the first run and on re-running the job. It is a pre-existing race, left for the Day shift.

Diagnostic passes (nextest time, build excluded, from docs/ci-performance.md): five main runs summed 3,327–3,595 s. With Spotlight off alone they summed 3,209 s, inside the noise. With Xcode's git alone, 2,132 s.

Before: in pass 4, with Xcode's git, security_run an_unavailable_github_cli_has_a_safe_pre_audit_dependency_cause and missing_node_fails_before_any_session_and_names_the_prerequisite failed: git-upload-pack: command not found.
After: both pass on Linux. On macOS they pass in this PR's full-suite run (39559241797: all four shards passed).

Local x86_64 Linux: cargo nextest run 2,168 passed; cargo test 2,171 passed; cargo clippy --all-targets -- -D warnings and cargo fmt --check clean.

Merge Danger

Door: two-way. Revert the two CI steps to get Homebrew's git and Spotlight back.

Blast Radius: CI-only.

macOS CI now tests against Xcode's git 2.50.1 instead of Homebrew's 2.56.0. A bug that shows only with Homebrew's newer git would not show on macOS. Linux CI keeps Ubuntu's git. No src/ change, and Worktree ownership checks are unchanged.

Unaddressed findings

Standards

  • Duplicated Code: the two new steps repeat in test-macos-gate and test-macos-shard. Declined. ci.yml already repeats each job's toolchain and cache steps per job, for example "Install the Rust version…" in test, test-macos-gate and test-macos-shard. The gate's comment points to the shards' explanation.
  • which git lookup duplicated with tests/pickup_run.rs::install_git_wrapper. Left as is. That duplication predates this PR, and the reviewer marked it optional.

Spec

  • Scope creep: the gate gets the same two steps. Declined. The gate runs the macOS-specific tests as the full suite runs them. Giving it a different git from the main and nightly shards would test those tests on a git the full suite no longer uses. macos-plan is untouched, so no event's macOS work changes (Run the full macOS suite on main and nightly, with a one-job macOS gate on PRs into main #893 stays out of scope).
  • Homebrew's git no longer runs anywhere in CI. No change. This is the trade-off under Merge Danger, recorded at the end of the doc section.

Review coverage: both reviewers read all three changed files. No generated output was excluded.

The diagnostic workflow lives on the issue-894-profile branch, which the doc's run links point to. Delete it whenever its logs are no longer needed.

Closes #894

🤖 Generated with Claude Code

Built with claude · claude-opus-5-5 · medium

Security review outcome

No unaddressed introduced findings.

JacobStephens2 and others added 2 commits October 21, 2026 01:35
Profiling the macOS shards (#894) found the slowest tests are git
processes, each costing about 13 ms under Homebrew's git, and Spotlight
indexing the tests' repositories. Xcode's git alone took 38% off the
shards' summed test time. docs/ci-performance.md records the method, the
runs and the breakdown. Two security_run tests that run thirdshift with
only git on PATH now also link the transport programs Xcode's git needs
beside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fifth profile pass traced the 19 slow tests the first passes left out,
including three of the 16 over 60 seconds, so the breakdown table now
covers all 26 with their Linux git counts beside the macOS ones. The CI
comment now says why the git-* programs are linked, and link_git links git
before the optional transport programs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JacobStephens2 JacobStephens2 added the test:macos On a PR into main: run the full 4-shard macOS suite instead of the one-job macOS gate (#893) label Oct 21, 2026
@JacobStephens2
JacobStephens2 merged commit 671f291 into main Oct 21, 2026
40 of 48 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-894 branch October 21, 2026 07:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:macos On a PR into main: run the full 4-shard macOS suite instead of the one-job macOS gate (#893)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Find what still makes the macOS test shards slow after #857

1 participant