Skip to content

fix(vm): give macOS VM jobs the cache proxies - #211

Merged
luthermonson merged 2 commits into
mainfrom
fix/macos-vm-proxy-env
Oct 5, 2026
Merged

luthermonson merged 2 commits into
mainfrom
fix/macos-vm-proxy-env

Conversation

@luthermonson

Copy link
Copy Markdown
Contributor

Problem

The fleet's Mac pulls ~130 GB/day of WAN, and its caches (gomod, cargo, ghrel, composer) are still at 0 bytes after #208 fixed the address the proxies advertise.

The reason: cache-proxy env (GOPROXY, RUSTUP_DIST_SERVER, GHREL_PROXY, COMPOSER_REPO_PACKAGIST, ...) is only ever applied to containers (pkg/runtime/runtime.go:1116). A macOS VM job has no container. Its runner is started over SSH inside a fresh macOS VM (macOSRunnerSetupScript), and nothing passed it any proxy env. All 38 jobs on the Mac since 10-02 were macOS VM jobs, so every one of them fetched straight from the internet.

Fix

The env reaches the job through the runner's .env file, which the GitHub runner loads into every job's environment.

  • vm.MacOSVMConfig.JobEnv carries the cache-proxy env, already rewritten to the host's Vz NAT address (192.168.64.1) by resolveJobProxyEnv from fix(proxies): advertise an address jobs can actually reach #208.
  • Verified from inside the guest. Before the runner starts, one bounded SSH command (runSSHCommand, 90s cap) probes every proxy address from inside the VM and stages the env in /tmp/ephemerd-runner.env only if all of them answer. The result is logged per VM, as either cache proxies reachable from the macOS VM; advertised to the job or a warning that names the unreachable address.
  • All-or-nothing. Go falls back on |direct, but cargo and rustup don't fall back on an unreachable proxy. A partial env could fail a build that would succeed uncached.
  • The runner setup script appends the staged file to the runner's .env right before starting the runner.
  • Fail-safe. Any failure (SSH error, timeout, unreachable proxy, an address that can't be verified) leaves nothing staged, and the job runs exactly as before, just uncached.

The script builder lives in an untagged file (pkg/vm/macos_proxyenv.go). The darwin VM file can't compile off macOS (vz needs cgo), and keeping the logic out of it is what makes it testable in CI.

Tests

pkg/vm/macos_proxyenv_test.go executes the generated script with real sh and curl against an httptest server:

  • all addresses reachable: env staged byte-for-byte, and the status line reports the count;
  • one dead address: nothing staged (not even GOSUMDB=off), no temp file left behind, and the dead address is named;
  • values containing $HOME, quotes, backticks and $(...) are written unexpanded (quoted heredoc);
  • malformed, newline-smuggling and heredoc-terminator entries produce no script;
  • an unparseable or unsafe URL drops the whole env rather than being staged unverified. A first draft got this wrong (EnvHostPorts silently skips unparseable URLs), and this test caught it.

Passes locally on Windows/Git Bash. The darwin wiring is compile-checked by CI's macOS build.

Verify on metal after rollout

  1. The Mac's ephemerd log shows cache proxies reachable from the macOS VM; advertised to the job for each macOS job.
  2. The cache directories under /var/lib/ephemerd grow and log hits.
  3. The Mac's en1 WAN delta per day drops. Whatever doesn't drop (Homebrew, Xcode, Actions downloads) is traffic with no proxy yet.

Container jobs get GOPROXY, RUSTUP_DIST_SERVER, GHREL_PROXY and the rest
from the container runtime (pkg/runtime). macOS VM jobs have no
container: the runner is started over SSH in a fresh macOS VM, and
nothing passed it any proxy env. So every crate, module, toolchain and
release asset a macOS job fetched went straight to the internet. The
fleet's Mac runs almost nothing but macOS VM jobs, which is why its
caches stayed at 0 bytes after #208 fixed the advertised address, while
the node kept pulling ~130 GB/day.

The env now reaches the job through the runner's .env file, which the
runner loads into every job's environment:

- MacOSVMConfig.JobEnv carries the cache-proxy env, already rewritten to
  the host's Vz NAT address by resolveJobProxyEnv.
- Before the runner starts, a bounded SSH command probes every proxy
  address from INSIDE the guest and stages the env only if all of them
  answer (all-or-nothing: cargo and rustup do not fall back on an
  unreachable proxy). The outcome is logged per VM.
- The runner setup script appends the staged env to the runner's .env.

Any failure leaves nothing staged and the job runs exactly as before,
uncached. The script builder lives in an untagged file so its tests,
which execute the generated script with real sh and curl, run on every
platform.
@ephpm

ephpm Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

ePHPm Preview — removed

Preview deployment has been torn down.

@ephpm
ephpm Bot temporarily deployed to preview-pr-211 October 5, 2026 19:50 Inactive
@ephpm
ephpm Bot temporarily deployed to preview-pr-211 October 5, 2026 19:50 Inactive
@ephpm
ephpm Bot temporarily deployed to preview-pr-211 October 5, 2026 19:50 Inactive
golangci-lint on Linux flagged macOSProxyEnvStage as unused: the untagged
file declared it, but only the darwin VM code references it.
@ephpm
ephpm Bot temporarily deployed to preview-pr-211 October 5, 2026 20:15 Inactive
@ephpm
ephpm Bot temporarily deployed to preview-pr-211 October 5, 2026 20:15 Inactive
@ephpm
ephpm Bot temporarily deployed to preview-pr-211 October 5, 2026 20:15 Inactive
@luthermonson
luthermonson merged commit 460febc into main Oct 5, 2026
4 checks passed

This branch was previously deployed

1 inactive deployment
preview-pr-211 — 72c17b84 Deployed Oct 5, 2026 by ephpm[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant