Skip to content

fix(ci): keep the GPU reclaim decoys alive on multicall coreutils - #712

Open
rominf wants to merge 2 commits into
mainfrom
fix/selftest-decoys-on-multicall-coreutils
Open

rominf wants to merge 2 commits into
mainfrom
fix/selftest-decoys-on-multicall-coreutils

Conversation

@rominf

@rominf rominf commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

Why

ubuntu-latest now resolves to Ubuntu 26.04, which ships uutils coreutils. There /bin/sleep is a single multicall binary that chooses the tool to run from argv[0]. The GPU reclaim self-test builds its stand-in engine processes by copying /bin/sleep to a path like .../e2e-prewarm/.../llama-server, so on 26.04 every one of them prints coreutils: unknown program 'llama-server' and exits immediately.

With no live decoys:

  • bash scripts/reclaim-gpu.sh --self-test fails three checks (could not read a decoy's command line, pre-warm engine process was not selected, bystander decoy is not running) and refuses to run the real reclaim. That fails build-and-test and skips its remaining steps.
  • The xtask test preflight_runs_the_real_reclaim_assert_clear, which copies /bin/sleep the same way, sees the real --assert-clear scan pass with "no leaked processes". That fails Test (affected crates) and Coverage.

This hits every PR and merge-queue run that lands on a 26.04 runner, whatever the change.

What changes

The decoys are copies of bash instead of sleep. bash ignores the name it runs under, so argv[0] is still the real-shaped path the reclaim rule is meant to match and the decoy keeps the shape a live engine process has. The decoy body is a sleep 1 loop rather than sleep 300: bash execs a lone simple command in place, which would put sleep back in argv[0]. Those sleep 1 children name no E2E root, so they are never selected, and SIGTERM still ends the decoy. The xtask test's decoy is detached from the test's stdio, as the script's already is, so a sleep 1 child that outlives the decoy's SIGKILL does not hold the test's output open.

Verification

  • reclaim-gpu.sh --self-test, both plainly and with TMPDIR redirected as CI runs it: fails on main and passes with this change in an ubuntu:26.04 container; passes in ubuntu:24.04 and on a 24.04-style host.
  • preflight_runs_the_real_reclaim_assert_clear: the test binary built from main fails inside ubuntu:26.04 and the one from this branch passes; it also passes natively.
  • cargo test -p xtask (313 tests), cargo clippy -p xtask --all-targets -- -D warnings, cargo fmt --check and shellcheck are clean.

Not verified locally: the build-and-test steps that run after the self-test (release-readiness self-test, pinned-key check, install-lifecycle e2e). They have been skipped on every 26.04 run so far, so this PR's CI is their first run on the new image.

Scope

This changes test fixtures only, so there's no user-observable CLI behaviour and no e2e scenario applies (AGENTS.md §3). It doesn't pin runners back to 24.04; the decoys now work on both images. Every other hosted job already passed on 26.04.


  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows.
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md (the clap declaration and dispatch wiring staying in main.rs/lib.rs is expected, not a violation). (N/A: no new subcommand.)
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will not happen. (N/A: no user-facing message changed.)

ubuntu-latest now resolves to Ubuntu 26.04, which ships uutils
coreutils. Its /bin/sleep is one multicall binary that picks the tool
to run from argv[0], so the decoys made by copying it to
.../llama-server print "coreutils: unknown program 'llama-server'" and
exit at once. The reclaim self-test then fails three checks and the
xtask contract test that runs the real --assert-clear passes a scan it
should fail, which turns build-and-test, Coverage and Test (affected
crates) red for every run on the new image.

Copy bash instead: it ignores the name it runs under, so argv[0] stays
the real-shaped path the rule is meant to match. The body is a loop
because bash execs a lone `sleep 300` in place, which would put
`sleep` back in argv[0]. Its `sleep 1` children name no E2E root, so
they are never selected, and SIGTERM still ends the decoy.

Verified with the self-test (plain and with TMPDIR redirected) and the
xtask test in ubuntu:26.04 and ubuntu:24.04 containers; both failed on
26.04 before this change.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The delimiter decoy's comment said bash would swallow an extra argument
as $0; it does not, and the argument reaches /proc/<pid>/cmdline. It is
a separate helper only because spawn_decoy passes no arguments through.

Detach the xtask test's decoy from the test's stdio, as the script's
decoy already is: its sleep 1 child outlives the decoy's SIGKILL by up
to a second and would hold the inherited output open meanwhile.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf requested a review from a team as a code owner October 20, 2026 08:55
@rominf
rominf requested a review from jussielo-amd October 20, 2026 08:55

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the decoy-spawn fix for the uutils multicall /bin/sleep breakage.

Traced the actual reclaim-matching logic (cmdline_matches_rule, same_selected_process in reclaim-gpu.sh) against the new bash -c 'for _ in {1..300}; do sleep 1; done' decoy body: the trailing args don't collide with any E2E_ROOTS/ENGINE_MARKERS substring, and the static loop body (no embedded pid/timestamp) keeps pid-recycling detection correct. SIGTERM still kills the bash wrapper immediately since no trap is installed (SIG_DFL disposition), and the orphaned sleep 1 grandchild is already accounted for by the added stdio-detach on the xtask side. A repo-wide grep confirms no other cp /bin/sleep-style decoy was left unfixed.

Diff matches the PR description's claims exactly, scoped to the two call sites, no CI workflow changes, no scope creep.

One non-blocking note: the bash-vs-sleep rationale is restated in prose at three separate comment sites (reclaim-gpu.sh twice, workflow_contract.rs once). Reasonable given the shell/Rust split, not asking for a change.

The two failing checks (Strix Halo Windows E2E download timeout and its downstream consolidated-report failure) look like the known repo.amd.com download-infra flake, not caused by this change.

LGTM, approving.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants