Skip to content

ci: run Linux CI on both Ubuntu LTS images; fork-free reclaim decoys with liveness checks - #718

Open
rominf wants to merge 2 commits into
mainfrom
fix/preflight-decoy-multicall-coreutils
Open

rominf wants to merge 2 commits into
mainfrom
fix/preflight-decoy-multicall-coreutils

Conversation

@rominf

@rominf rominf commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

GitHub repointed ubuntu-latest from ubuntu-24.04 to ubuntu-26.04 (image ubuntu26/20261012.163). The last merge group to pass before the switch, #710, ran on 24.04; every merge-group run after it went red, with no change in this repository. Ubuntu 26.04 ships uutils coreutils, whose /bin/sleep is a multicall binary. The reclaim decoys were copies of it named llama-server, so they printed coreutils: unknown program 'llama-server' and exited at once. That failed preflight_runs_the_real_reclaim_assert_clear in Test (affected crates) and Coverage, and reclaim-gpu.sh --self-test in build-and-test. #712 fixed the decoys by copying bash and running a sleep 1 loop.

This PR builds on #712 in two ways.

1. Fork-free decoys that must be seen alive

  • No copied binary and no child process. The xtask test runs /bin/sh with the decoy path as argv[0] (CommandExt::arg0). The shell waits in the read builtin on a stdin pipe the test holds. spawn_decoy in reclaim-gpu.sh runs its own bash the same way, via exec -a, and reads a FIFO it holds open for both reading and writing. Killing the one pid leaves nothing behind; fix(ci): keep the GPU reclaim decoys alive on multicall coreutils #712's loop left a sleep 1 child running for up to a second. The xtask decoy also exits by itself if the test dies before killing it, because the pipe closes. Running the real sleep under another argv[0] is not an option: uutils dispatches on argv[0] on some targets and on the executed path on others.
  • Liveness checks. Before reading any decoy, both tests check it is alive. A dead decoy fails the test with a message naming it, instead of surfacing later as a gate or selection failure. The xtask test also checks that the decoy outlived the scan before it trusts a clear verdict. Run against the original cp /bin/sleep decoy on ubuntu-26.04, the self-test now opens with:
    FAIL: pre-warm decoy (pid=13) is not running; it exited at spawn, so the checks that read it test nothing
    
    and the xtask test with the decoy engine process (pid=…) exited at spawn (exit status: 1) ….

2. Linux CI on both LTS images instead of ubuntu-latest

Job Images Check names
build-and-test 24.04 + 26.04 build-and-test, build-and-test (ubuntu-26.04)
test 24.04 + 26.04 Test (affected crates), Test (affected crates) (ubuntu-26.04)
e2e 24.04 + 26.04 E2E tests, E2E tests (ubuntu-26.04)
remote-ssh 24.04 + 26.04 remote control channel (containerised), … (ubuntu-26.04)
every other hosted Linux job, in all workflows 24.04 unchanged

Required checks. Branch protection matches required checks by job name. The 24.04 leg of each split job therefore keeps its exact current name, using name: ${{ matrix.os == 'ubuntu-24.04' && '<name>' || format('<name> ({0})', matrix.os) }}. I checked that every required context is still produced by exactly one job. changes is the exception: it appears twice, once in ci.yml and once in e2e-selfhosted.yml, and was already duplicated on main. For an admin: to make 26.04 blocking, add these contexts as required:

  • build-and-test (ubuntu-26.04)
  • Test (affected crates) (ubuntu-26.04)
  • E2E tests (ubuntu-26.04)

remote control channel (containerised) (ubuntu-26.04) is not required today, and neither is its 24.04 leg.

Single-LTS jobs. Each of these has a trailing comment on its runs-on saying why one image is enough:

  • lint, metadata and docs jobs: changes, prek, license headers, lychee, Sphinx, commit signatures, skill checks, third-party notices, xfail hint, CodeQL;
  • report merging and publishing jobs;
  • clippy, which runs the same pinned toolchain on either image;
  • coverage, which is one measurement against the committed floors (Test (affected crates) runs the tests on both images);
  • the PR, nightly and release binary builds, which run in the manylinux container. Moving them off ubuntu-latest also keeps the image they run on from changing silently.

Caches keyed by image. rust-cache keys only on Linux-x64, and sccache keys on the compiler binary, which is the same rustup download on every image. Without the image in the key, a target directory or proc macro linked against 26.04's glibc would be restored on 24.04, and caches written by 26.04 runs under ubuntu-latest would be restored onto the jobs now pinned to 24.04. Every hosted Linux setup-rust-toolchain now passes cache-key with the image. The sccache prefixes become rocm-cli/build-and-test/<image>; clippy shares the 24.04 one. The coverage and cargo-about actions/cache keys now include ubuntu-24.04. As a result, the first run after merge starts with cold caches.

E2E gate. E2E tests used to run its scenarios only when build-and-test succeeded. That result now combines both images, so a failure on 26.04 alone would have let the required 24.04 leg report green without running anything. The gate now runs the scenarios whenever build-and-test reached a verdict.

E2E artifacts. The 24.04 leg still uploads e2e-report, which feeds E2E consolidated report. The 26.04 leg uploads ubuntu-26.04-e2e-results, which is for inspection only. The consolidated report has a single mock platform, and two inputs labelled as it would collide. Showing both images in that grid needs parse_descriptor support and is left as a follow-up.

Contract tests in xtask/src/workflow_contract.rs:

  • no_workflow_runs_on_the_moving_ubuntu_latest_label scans every workflow file. moving_ubuntu_label_detector_flags_values_but_not_comments covers the detector itself.
  • lts_split_jobs_keep_their_required_name_on_24_04 pins the exact name expression, runs-on: ${{ matrix.os }}, both images and fail-fast: false for the four split jobs.
  • hosted_linux_compile_caches_are_keyed_by_runner_image checks that rust-cache cache-key, SCCACHE_AZURE_KEY_PREFIX and cargo actions/cache keys name the job's image.

I mutation-checked each test. Restoring ubuntu-latest on one job, using a plain matrix name, dropping 26.04 from the matrix, dropping fail-fast: false, and removing the image from a rust-cache key, the sccache prefix or the coverage key each made the matching test fail.

Testing

  • Decoys: reclaim-gpu.sh --self-test (plain, and with a redirected TMPDIR) and preflight_runs_the_real_reclaim_assert_clear pass in ubuntu:24.04 (GNU coreutils), in ubuntu:26.04 (Ubuntu's uutils), and in ubuntu:24.04 with the uutils 0.12.0 release binary bind-mounted over /usr/bin/sleep. No decoy process is left running afterwards.
  • cargo test -p xtask (321 passed), cargo fmt --check, both clippy invocations from CI with -D warnings, and cargo check -p xtask --tests --target x86_64-pc-windows-gnu.
  • prek on the changed files passes, except for the hawkeye hook, whose local binary cannot parse licenserc.toml (a local version mismatch that also fails on main). The License header check (hawkeye) CI job covers it.
  • actionlint 1.7.12 adds no new findings apart from label "ubuntu-26.04" is unknown. That release predates the label, which is listed in actions/runner-images. It already reported the self-hosted labels before this change.

This change affects CI and test fixtures only. CLI behaviour does not change, so no Gherkin scenario applies.

@rominf
rominf requested a review from a team as a code owner October 21, 2026 12:32
@rominf
rominf requested a review from volen-silo October 21, 2026 12:32

@jussielo-amd jussielo-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice fix — the FIFO-parked-shell decoy is a cleaner way to dodge the multicall-coreutils cmdline matching problem than the old fs::copy approach, and I confirmed the match logic (cmdline_matches_rule) only ever looks at /proc/<pid>/cmdline, never the file content, so dropping that copy step doesn't lose any coverage. Cleanup, the selftest_require_alive wiring across the other decoy-based fixtures, and the zombie fixture correctly staying unaffected all check out too.

One small thing I noticed, non-blocking: in the self-test liveness check (workflow_contract.rs around line 2868), there's a narrow TOCTOU window — if the decoy process exits between try_wait() and the /proc/<pid>/cmdline read, the code falls into the assert!(named, ...) branch and panics claiming the process "is alive but its command line does not start with {argv0}", when actually it just exited. It's a sub-millisecond race on an already-rare path, so it'd only ever mislead someone debugging a future flake rather than cause a false pass/fail — not something that needs to hold up this PR, but might be worth a follow-up if you want the panic message to stay accurate under that edge case.

Two things before this is mergeable, separate from the code review itself:

  • GitHub currently shows this PR as having merge conflicts with the base branch — will need a rebase.
  • The only check reporting here is the Read the Docs build; I don't see the usual cargo test/clippy gates on this PR, so I can't confirm they've actually run.

Approving on the logic/design itself — thanks for tracking down the root cause on ubuntu-26.04. Just flagging the conflict and CI status so they don't get missed before merge.

The reclaim decoys now copy no binary and fork no child. The xtask
preflight test runs /bin/sh under the decoy path as argv[0], parked in
the `read` builtin on a pipe the test holds; reclaim-gpu.sh's
spawn_decoy runs its own bash the same way via `exec -a`, reading a
FIFO it holds open read-write. Killing the one pid leaves nothing
behind (the bash-loop decoys left a `sleep 1` child for up to a
second), and the xtask decoy exits by itself if the test dies first.

A decoy that dies at spawn - as a copied multicall `sleep` does on
ubuntu-26.04 - used to surface as a gate or selection failure. Both
tests now check each decoy is alive before reading it and fail naming
the decoy, and the xtask test checks the decoy outlived the scan
before trusting a clear verdict.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
ubuntu-latest moved from 24.04 to 26.04 without notice and turned every
merge-group run red. Name the images instead.

build-and-test, Test (affected crates), E2E tests and the remote-ssh
lane run on ubuntu-24.04 and ubuntu-26.04. Branch protection matches
required checks by job name, so the 24.04 leg keeps each job's bare
name and only the 26.04 leg gets an " (ubuntu-26.04)" suffix. Every
other hosted Linux job is pinned to ubuntu-24.04 with a comment saying
why one LTS is enough: lint, metadata, reporting and publishing jobs,
clippy, the coverage measurement, and the release builds, which run in
the manylinux container.

Compile caches are now keyed by image. rust-cache keys on "Linux-x64"
only and sccache on the compiler binary, the same rustup download on
every image, so a proc macro or build script linked against 26.04's
glibc would otherwise be restored on 24.04.

E2E tests now runs its scenarios whenever build-and-test reached a
verdict, not only on success: that result aggregates both images, and a
26.04 failure must not let the required 24.04 leg report green without
running. The 26.04 leg's report is uploaded under a name the
consolidated report does not read, since that report has one mock
platform.

New workflow contract tests: no workflow names ubuntu-latest, the split
jobs keep their required names on 24.04, and hosted Linux compile
caches name the runner image.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the fix/preflight-decoy-multicall-coreutils branch from a81658b to db1e550 Compare October 21, 2026 13:46
@rominf rominf changed the title fix(xtask): keep the preflight decoy alive on multicall coreutils (ubuntu-26.04) ci: run Linux CI on both Ubuntu LTS images; fork-free reclaim decoys with liveness checks Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants