Skip to content

ci(runner): bump the embedded actions runner automatically - #212

Open
luthermonson wants to merge 1 commit into
mainfrom
ci/runner-bump-workflow
Open

luthermonson wants to merge 1 commit into
mainfrom
ci/runner-bump-workflow

Conversation

@luthermonson

Copy link
Copy Markdown
Contributor

Why

ephemerd embeds a pinned actions/runner. GitHub deprecates old runner versions and then refuses their connections, so a stale pin silently stalls every job on that pool. The macOS pool once ran zero jobs for 7 days this way. The last bump (#183) came from an outside contributor, and nothing on our side was watching for new releases. actions/runner v2.338.0 is already out, so the first run after merge should open a bump PR.

What

.github/workflows/runner-bump.yml runs daily at 07:23 UTC and on workflow_dispatch (optional version input). It runs on GitHub-hosted ubuntu-latest, so it never takes a self-hosted fleet slot.

  • It reads the newest non-draft, non-prerelease actions/runner release (/releases/latest, flags re-checked) and compares it to RunnerVersion as numeric semver.
  • If the release is newer, go run ./mage/runnerbump edits the pins. The job then runs gofmt -l mage/ and go build ./mage/..., commits, and opens or updates a PR on the fixed branch bot/actions-runner-bump.
  • Idempotency:
    • No newer release: nothing happens.
    • Same release as the bot branch: nothing is pushed. The bumped files are compared with the remote branch, so a rerun doesn't retrigger CI on the fleet.
    • A newer release: the branch is force-pushed from fresh main, and the existing PR is retitled and re-described. No second PR is opened.
    • Main reaches the latest release some other way (a manual bump): the stale bot PR is closed and its branch deleted. This only happens on unpinned runs.
  • concurrency: runner-bump (no cancel). The workflow gets contents: write and pull-requests: write, and nothing else.

mage/runnerbump is a small Go program that uses only the standard library. It does the edits; no sed is involved.

  • What it edits:
    • RunnerVersion.
    • The // GitHub Actions runner (actions/runner vX) comment.
    • The five actions-runner-{linux-x64,linux-arm64,win-x64,osx-x64,osx-arm64} entries in pinnedSHA256.
    • ARG RUNNER_VERSION in images/runner-ci-{linux,macos,windows}/Dockerfile.
    • After editing, it runs gofmt on the result to realign the map.
  • Safety checks: each site must match exactly once, and no trace of the old version may survive. A refactor of download.go therefore fails the bump loudly instead of producing a half-edited file.
  • Checksums:
    • They come from the release API's per-asset digest (sha256:...).
    • If a digest is missing, the asset is downloaded and hashed.
    • They are cross-checked against the release notes' BEGIN SHA markers when those exist. A disagreement fails the bump; the notes are never the only source.
    • Any of the five archives missing from the release fails the bump.
  • Flags: -version, -check, -root, -release-json (for offline testing) and -body-out. It can be run by hand: go run ./mage/runnerbump.
  • main_test.go covers the rewrite, unknown-shape refusal, a missing asset, a notes/digest mismatch, the Dockerfile rewrite and numeric semver.

mage/download/download.go: only the checksum-source comment changed, to point at the digest source and the bot.

Token: needs a GitHub App to be fully hands-off

The ephpm org has "Allow GitHub Actions to create and approve pull requests" switched off, for this repo and for ephpm/ephpm (can_approve_pull_request_reviews: false). The GITHUB_TOKEN house pattern (sdk-bump.yml / turso-pin-watch.yml) therefore cannot open PRs: ephpm/ephpm's SDK Bump failed with GitHub Actions is not permitted to create or approve pull requests. Default-token PRs would not trigger CI anyway. So the job uses a GitHub App installation token when one is configured:

Kind Name Value
Actions variable (repo or org) BOT_APP_CLIENT_ID the App's Client ID
Actions secret (repo or org) BOT_APP_PRIVATE_KEY a private key (.pem) for that App

The App needs Contents: read & write and Pull requests: read & write, and must be installed on ephpm/ephemerd. Don't reuse the ephemerd runtime App's key. Org-level names let ephpm/ephpm's bump workflows use the same App later.

Without the App, the job falls back to GITHUB_TOKEN. It still pushes bot/actions-runner-bump, then fails with an ::error:: that gives a compare link and both fixes. The alternative fix is to enable the Actions setting above, but CI would then not run on the bot PR until someone closes and reopens it.

Other version references (grep for the version and 2.33x)

  • Moved by the bot:
  • Deliberately not edited:
    • pkg/runner/runner_test.go: synthetic version-parse and compare fixtures.
    • pkg/runtime/runtime_test.go: a /data/runners/2.333.1-linux path fixture.
    • pkg/runner/runner.go: example comments.
  • Follow-up, not version strings: pkg/vm/macosvm_darwin.go (two sites) and pkg/vm/macos_install_darwin.go pick the first runners/* dir that has a run.sh. Those dirs are named <ephemerd-version>-<os> and old ones are never pruned, so a macOS host can keep injecting an older ephemerd's runner after an upgrade. pkg/runner.findTarball already selects the highest version; these globs should too.
  • Follow-up: the same pattern could cover the CNI, containerd and runc pins. Their checksum sources are per-project sidecar files, so that is left for a separate change.

Validation

  • Diff shape:
  • No-op cases:
    • Same version: no change, bump=false.
    • An older target (2.335.1): no change.
    • Bump to latest (2.337.0 -> 2.338.0), then a rerun: no change.
    • -check: reports the bump without editing.
  • Failure and fallback paths (-release-json):
    • One digest stripped: that asset is downloaded and hashed, and the result is identical to the API-digest bump.
    • win-x64 removed: fails and names the asset; tree untouched.
    • prerelease: true: refused.
    • Forged digest: refused by the release-notes cross-check.
  • Push idempotency: simulated against a local bare remote. The sequence 2.338.0, 2.338.0, 2.339.0, 2.339.0 gave push, skip, push, skip.
  • Checks: go test ./mage/runnerbump passes; go build ./mage/... and CGO_ENABLED=0 GOOS=linux go vet ./mage/... are clean.
  • actionlint 1.7.12: clean, apart from two false positives on create-github-app-token@v3's client-id input. actionlint's bundled metadata predates v3, whose action.yml defines client-id and deprecates app-id.

After merge

The workflow can only be dispatched once it is on main:

gh workflow run runner-bump.yml -R ephpm/ephemerd            # latest stable
gh workflow run runner-bump.yml -R ephpm/ephemerd -f version=2.338.0
gh run list -R ephpm/ephemerd --workflow runner-bump.yml --limit 1

ephemerd embeds a pinned actions/runner. GitHub deprecates old runner
versions and then refuses their connections, so a stale pin silently stalls
every job on a pool (the macOS pool once ran zero jobs for a week). The last
bump came from an outside contributor; nothing on our side watched for it.

Add .github/workflows/runner-bump.yml, a daily (and dispatchable) job on
GitHub-hosted ubuntu-latest that compares the newest non-draft,
non-prerelease actions/runner release with RunnerVersion and, when it is
newer, opens or updates a PR on the fixed branch bot/actions-runner-bump.

The edit is done by mage/runnerbump, a stdlib-only Go program, not sed. It
rewrites RunnerVersion, the actions/runner pin comment, the five runner
pinnedSHA256 entries and ARG RUNNER_VERSION in the three CI runner-image
Dockerfiles, then gofmts. Every site must match exactly once and no trace of
the old version may survive, so a refactor of download.go fails the bump
instead of half-editing it. Checksums come from the release API's per-asset
digest (download-and-hash when one is missing), cross-checked against the
release notes' BEGIN SHA markers; a missing archive or a disagreement fails.

Run against the pre-#183 tree with -version 2.337.0 it reproduces #183's
download.go and Linux Dockerfile byte for byte. It also syncs the macOS and
Windows image ARGs, which were left at 2.333.1 by #97 and #183.

The job uses a GitHub App token when BOT_APP_CLIENT_ID / BOT_APP_PRIVATE_KEY
are configured (so CI runs on the PR) and falls back to GITHUB_TOKEN.
@ephpm

ephpm Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

ePHPm Preview — deployed (health check pending)

URL https://ephpm-ephemerd-pr-212.preview.ephpm.dev
Framework PHP
PHP 8.5
Deployed in 61.8s

Preview updates automatically on each push to this PR.

@ephpm
ephpm Bot temporarily deployed to preview-pr-212 October 11, 2026 02:48 Inactive
@ephpm
ephpm Bot temporarily deployed to preview-pr-212 October 11, 2026 02:48 Inactive

This branch was successfully deployed

1 active deployment
preview-pr-212 — fa27d0bc Deployed Oct 11, 2026 by ephpm[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant