Repository navigation
fix(vm): give macOS VM jobs the cache proxies - #211
Merged
Merged
Conversation
Container jobs get GOPROXY, RUSTUP_DIST_SERVER, GHREL_PROXY and the rest from the container runtime (pkg/runtime). macOS VM jobs have no container: the runner is started over SSH in a fresh macOS VM, and nothing passed it any proxy env. So every crate, module, toolchain and release asset a macOS job fetched went straight to the internet. The fleet's Mac runs almost nothing but macOS VM jobs, which is why its caches stayed at 0 bytes after #208 fixed the advertised address, while the node kept pulling ~130 GB/day. The env now reaches the job through the runner's .env file, which the runner loads into every job's environment: - MacOSVMConfig.JobEnv carries the cache-proxy env, already rewritten to the host's Vz NAT address by resolveJobProxyEnv. - Before the runner starts, a bounded SSH command probes every proxy address from INSIDE the guest and stages the env only if all of them answer (all-or-nothing: cargo and rustup do not fall back on an unreachable proxy). The outcome is logged per VM. - The runner setup script appends the staged env to the runner's .env. Any failure leaves nothing staged and the job runs exactly as before, uncached. The script builder lives in an untagged file so its tests, which execute the generated script with real sh and curl, run on every platform.
|
ePHPm Preview — removed Preview deployment has been torn down. |
golangci-lint on Linux flagged macOSProxyEnvStage as unused: the untagged file declared it, but only the darwin VM code references it.
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The fleet's Mac pulls ~130 GB/day of WAN, and its caches (gomod, cargo, ghrel, composer) are still at 0 bytes after #208 fixed the address the proxies advertise.
The reason: cache-proxy env (
GOPROXY,RUSTUP_DIST_SERVER,GHREL_PROXY,COMPOSER_REPO_PACKAGIST, ...) is only ever applied to containers (pkg/runtime/runtime.go:1116). A macOS VM job has no container. Its runner is started over SSH inside a fresh macOS VM (macOSRunnerSetupScript), and nothing passed it any proxy env. All 38 jobs on the Mac since 10-02 were macOS VM jobs, so every one of them fetched straight from the internet.Fix
The env reaches the job through the runner's
.envfile, which the GitHub runner loads into every job's environment.vm.MacOSVMConfig.JobEnvcarries the cache-proxy env, already rewritten to the host's Vz NAT address (192.168.64.1) byresolveJobProxyEnvfrom fix(proxies): advertise an address jobs can actually reach #208.runSSHCommand, 90s cap) probes every proxy address from inside the VM and stages the env in/tmp/ephemerd-runner.envonly if all of them answer. The result is logged per VM, as eithercache proxies reachable from the macOS VM; advertised to the jobor a warning that names the unreachable address.|direct, but cargo and rustup don't fall back on an unreachable proxy. A partial env could fail a build that would succeed uncached..envright before starting the runner.The script builder lives in an untagged file (
pkg/vm/macos_proxyenv.go). The darwin VM file can't compile off macOS (vz needs cgo), and keeping the logic out of it is what makes it testable in CI.Tests
pkg/vm/macos_proxyenv_test.goexecutes the generated script with realshandcurlagainst anhttptestserver:GOSUMDB=off), no temp file left behind, and the dead address is named;$HOME, quotes, backticks and$(...)are written unexpanded (quoted heredoc);EnvHostPortssilently skips unparseable URLs), and this test caught it.Passes locally on Windows/Git Bash. The darwin wiring is compile-checked by CI's macOS build.
Verify on metal after rollout
cache proxies reachable from the macOS VM; advertised to the jobfor each macOS job./var/lib/ephemerdgrow and log hits.en1WAN delta per day drops. Whatever doesn't drop (Homebrew, Xcode, Actions downloads) is traffic with no proxy yet.