Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions tests/e2e-cucumber/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,17 @@ path = "src/bin/rocm-demo-env.rs"
name = "fake-tailscale"
path = "src/bin/fake-tailscale.rs"

# Stand-ins for `uv` and the runtime's Python interpreter, planted by the
# `rocm comfyui install` scenarios so they run deterministically and offline
# on every platform cargo builds for (see `src/fixture_binary.rs`).
[[bin]]
name = "fake-uv"
path = "src/bin/fake-uv.rs"

[[bin]]
name = "fake-python"
path = "src/bin/fake-python.rs"

[dependencies]
axum.workspace = true
cucumber = { version = "0.23", features = ["output-json", "output-junit"] }
Expand Down
9 changes: 9 additions & 0 deletions tests/e2e-cucumber/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,15 @@ The `.feature` file is both the spec and the test input — cucumber reads it at
3. Implement step functions: Given = setup, When = action, Then = assertion.
4. Register the module in `tests/e2e.rs` with `pub mod <name>_steps;`.
5. Run to verify.
6. If a step needs to fake a subprocess the CLI shells out to (`uv`, a runtime's
`python`, `tailscale`), add a `[[bin]]` under [`src/bin/`](src/bin/) rather
than writing a shell/batch script: it builds on every platform cargo already
targets. [`fixture_binary`](src/fixture_binary.rs)'s `fixture_binary()`
locates it and `fixture_target_name()` names it; `install_fixture_binary` in
[`tests/e2e.rs`](tests/e2e.rs) plants it (it needs the `rocm-core`
dev-dependency the library target can't reach). See
[`fake-uv`](src/bin/fake-uv.rs), [`fake-python`](src/bin/fake-python.rs),
and [`fake-tailscale`](src/bin/fake-tailscale.rs) for the pattern.

## Design principles

Expand Down
30 changes: 21 additions & 9 deletions tests/e2e-cucumber/features/comfyui.feature
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,8 @@ Feature: ComfyUI install reports progress and makes failures actionable
#
# The runtime, its rocm_sdk probe, the Python interpreter and `uv` itself are
# all planted so the scenario is hermetic: no GPU, no network, no real
# package resolution. Linux-only because the planted shims are POSIX shell
# scripts.
@id:comfyui-uv-install-failure-names-log @requires-os:linux
# package resolution.
@id:comfyui-uv-install-failure-names-log
Scenario: comfyui-01 - A failed dependency install names the log it wrote
Given a ready ROCm runtime with a ComfyUI checkout pending dependencies
And the ComfyUI dependency install with uv fails
Expand All @@ -32,7 +31,7 @@ Feature: ComfyUI install reports progress and makes failures actionable
# both `install` and `status` output -- a user-observable line that AGENTS.md
# requires a Gherkin scenario for, not just the unit tests that assert each
# command's `folder:` line in isolation.
@id:comfyui-uv-install-progress-streamed @requires-os:linux
@id:comfyui-uv-install-progress-streamed
Scenario: comfyui-02 - A dependency install streams progress output
Given a ready ROCm runtime with a ComfyUI checkout pending dependencies
And the ComfyUI dependency install with uv prints progress and succeeds
Expand Down Expand Up @@ -60,9 +59,8 @@ Feature: ComfyUI install reports progress and makes failures actionable
#
# No GPU is needed: runtime readiness is filesystem + manifest state, so the
# scenario plants two ready wheel runtimes and asserts the refusal names every
# remediation and lists both keys. Linux-only because the planted rocm_sdk stub
# uses `.so` library names; the selection logic it exercises is platform-agnostic.
@id:comfyui-ambiguous-runtime-actionable @requires-os:linux
# remediation and lists both keys.
@id:comfyui-ambiguous-runtime-actionable
Scenario: comfyui-03 - ComfyUI install refuses an ambiguous runtime choice and names every remediation
Given two ready ROCm runtimes and no active default
When the user installs ComfyUI without choosing a runtime
Expand All @@ -89,6 +87,11 @@ Feature: ComfyUI install reports progress and makes failures actionable
# label-truncation fix in `cli_progress::assemble_status_line` — the
# tarball scenario in `download_progress_pty.feature` is the regression
# test for that.
# Linux-only: this scenario drives `rocm` under a real PTY, the same
# suite-wide limitation `dash.feature`'s header comment documents
# (portable-pty compiles on Windows via ConPTY, but that path isn't yet
# promoted to a blocking contract) — unrelated to this file's other
# scenarios, whose own Linux gate is explained below.
@id:comfyui-source-download-shows-live-progress @requires-os:linux @serial
Scenario: comfyui-04 - The source-archive download spinner renders progress and clears on completion
Given a paced ComfyUI source archive fixture
Expand All @@ -109,8 +112,17 @@ Feature: ComfyUI install reports progress and makes failures actionable
# started during the download), an interrupted reinstall, a user's entry set
# aside, a failure after the new code is in place, and a download that is
# not a ComfyUI release. Each asserts what the CLI prints together with what
# is actually on disk afterwards. Linux-only because the planted runtime
# uses `.so` names and a POSIX-shell Python stand-in.
# is actually on disk afterwards. Only three of these still need the POSIX
# shell: comfyui-11's GPU-check failing transition plants a `#!/bin/sh`
# script (its passing-again half uses the Windows-portable `fake-python`
# stand-in, like every other scenario here), and comfyui-08/comfyui-12 each
# start their stand-in running ComfyUI process through `sh -c` (see
# `spawn_stand_in`). comfyui-05/-06/-07/-09/-10/-13 plant the same
# Windows-portable `fake-python` stand-in `comfyui-01`/`-02`/`-03` use and
# have no shell dependency of their own, but stay `@requires-os:linux` too:
# their reinstall behavior has not been run on Windows yet. Ungating them
# is tracked separately (#697), to keep this fix scoped to #629's
# comfyui-01/02/03 ask.
#
# rocm-cli's saved record of a ComfyUI it started is unreadable here. `stop`
# cannot read it either, so it must not block the reinstall; the reinstall
Expand Down
48 changes: 48 additions & 0 deletions tests/e2e-cucumber/src/bin/fake-python.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
// Copyright © Advanced Micro Devices, Inc., or its affiliates.
//
// SPDX-License-Identifier: MIT

//! A stand-in for the runtime's Python interpreter, for the ComfyUI install
//! scenarios' two probes.
//!
//! `apps/rocm/src/comfyui.rs` shells out to the runtime's Python twice:
//! `-c <script>` for the pre-install torch-stack version probe (answered here
//! with empty JSON, so the probe reads "nothing pinned") and
//! `<script-path> <result-path>` for the post-install GPU check (answered by
//! writing a fixed, GPU-having result to the given path). One binary answers
//! both forms unconditionally, so a scenario that only reaches the first
//! probe and one that reaches both work against the same stand-in.
//!
//! A Rust binary rather than a shell script so the scenarios run on Windows
//! too — a `.bat` stand-in can't take the `-c` form at all: its argument is a
//! genuine multi-line script, and Rust's CVE-2024-24576 mitigation rejects
//! any newline-containing argument once the target resolves to a batch file.
//!
//! Only the two forms `probe_comfyui` and the torch-stack probe actually use
//! are accepted; anything else fails loudly rather than guessing, the same
//! rule `fake-tailscale` follows for an unexpected command.

const GPU_PROBE_JSON: &str = r#"{"torch_version": "2.4.0", "torch_cuda_available": true, "device_count": 1, "devices": ["Fake GPU"]}"#;

// The probe script path is always `probe_comfyui`'s own internally-generated
// `probe.py` (never user input), so a case-sensitive extension check is fine.
#[allow(clippy::case_sensitive_file_extension_comparisons)]
fn main() {
let args: Vec<String> = std::env::args().collect();
match args.get(1).map(String::as_str) {
// `python -c <script>`: the pre-install torch-stack version probe.
Some("-c") if args.len() == 3 => {
print!("{{}}");
}
// `python <probe.py> <result.json>`: the post-install GPU check.
Some(script) if script.ends_with(".py") && args.len() == 3 => {
let result_path = &args[2];
std::fs::write(result_path, GPU_PROBE_JSON)
.unwrap_or_else(|e| panic!("failed to write GPU probe result: {e}"));
}
_ => {
eprintln!("fake-python: unexpected invocation: {args:?}");
std::process::exit(1);
}
}
}
28 changes: 28 additions & 0 deletions tests/e2e-cucumber/src/bin/fake-uv.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
// Copyright © Advanced Micro Devices, Inc., or its affiliates.
//
// SPDX-License-Identifier: MIT

//! A stand-in for `uv`, for the ComfyUI dependency-install scenarios.
//!
//! `rocm comfyui install` shells out to `uv` to resolve ComfyUI's Python
//! dependencies. A scenario that has to be deterministic and offline plants
//! this instead and controls its outcome through two env vars:
//!
//! `FAKE_UV_STDOUT` — a line to print to stdout before exiting (unset: none).
//! `FAKE_UV_EXIT_CODE` — the exit code to return (unset: `0`).
//!
//! A Rust binary rather than a shell script so the scenarios run on Windows
//! too.

fn main() {
if let Ok(line) = std::env::var("FAKE_UV_STDOUT") {
println!("{line}");
}
let code: i32 = match std::env::var("FAKE_UV_EXIT_CODE") {
Ok(value) => value.parse().unwrap_or_else(|e| {
panic!("FAKE_UV_EXIT_CODE={value:?} is not a valid exit code: {e}")
}),
Err(_) => 0,
};
std::process::exit(code);
}
47 changes: 47 additions & 0 deletions tests/e2e-cucumber/src/fixture_binary.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
// Copyright © Advanced Micro Devices, Inc., or its affiliates.
//
// SPDX-License-Identifier: MIT

//! Locate a fixture stand-in built as a `[[bin]]` of this crate.
//!
//! A scenario that needs a fake subprocess — `fake-tailscale`, `fake-uv`,
//! `fake-python` — declares it a normal cargo `[[bin]]` instead of
//! hand-writing a shell or batch script, so it runs on every platform cargo
//! already builds for and gets normal incremental, cached builds. Cargo puts
//! it next to this test binary itself, so finding it is a matter of walking
//! up from `std::env::current_exe()`.
//!
//! Only the lookup lives here: planting the located binary at a scenario's
//! chosen path needs `rocm_core::fake_executable::write_fake_executable` to
//! avoid reopening the `ETXTBSY` race, and that helper is a dev-dependency —
//! unavailable to this crate's own library target. See `install_fixture_binary`
//! in `tests/e2e.rs`, the test binary that can reach it.

use std::path::PathBuf;

/// The path cargo built the `[[bin]]` named `name` at, alongside this test
/// binary.
pub fn fixture_binary(name: &str) -> PathBuf {
let mut dir = std::env::current_exe().expect("test binary path");
dir.pop();
if dir.ends_with("deps") {
dir.pop();
}
let candidate = dir.join(fixture_target_name(name));
assert!(
candidate.exists(),
"the {name} stand-in was not built at {}; it is a [[bin]] of this crate",
candidate.display()
);
candidate
}

/// The filename a planted fixture executable needs on this platform: `stem`
/// unchanged on unix, `stem.exe` on Windows.
pub fn fixture_target_name(stem: &str) -> String {
if cfg!(windows) {
format!("{stem}.exe")
} else {
stem.to_owned()
}
}
1 change: 1 addition & 0 deletions tests/e2e-cucumber/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

pub mod capability;
pub mod expectation;
pub mod fixture_binary;
pub mod harness_diagnostics;
pub mod http_server;
pub mod installer_fixture;
Expand Down
28 changes: 28 additions & 0 deletions tests/e2e-cucumber/tests/e2e.rs
Original file line number Diff line number Diff line change
Expand Up @@ -754,6 +754,34 @@ pub fn rocmd_binary() -> PathBuf {
})
}

/// Plants the `[[bin]]` fixture `name` (see `e2e_cucumber::fixture_binary`)
/// at the exact path `dest`.
///
/// Parent dirs are created as needed. Shared so the three stand-ins
/// (`fake-tailscale`, `fake-uv`, `fake-python`) don't each repeat the same
/// create-then-write body.
///
/// Lives here rather than in `e2e_cucumber::fixture_binary` because it needs
/// `rocm_core::fake_executable::write_fake_executable` to avoid reopening the
/// `ETXTBSY` race a plain `std::fs::copy` would — holding a write fd on `dest`
/// for the duration of the copy is exactly the window that race exploits —
/// and `rocm-core` is a dev-dependency this crate's library target cannot
/// reach, only this test binary.
pub fn install_fixture_binary(name: &str, dest: &Path) {
if let Some(parent) = dest.parent() {
std::fs::create_dir_all(parent)
.unwrap_or_else(|e| panic!("failed to create {}: {e}", parent.display()));
}
let contents = std::fs::read(e2e_cucumber::fixture_binary::fixture_binary(name))
.unwrap_or_else(|e| panic!("failed to read the {name} stand-in: {e}"));
rocm_core::fake_executable::write_fake_executable(dest, contents).unwrap_or_else(|e| {
panic!(
"failed to install the {name} stand-in at {}: {e}",
dest.display()
)
});
}

/// Spawn the real `rocm` binary with the scenario's isolated env.
///
/// Returns `(stdout, stderr, rc)`. Every scenario goes through here, so this is
Expand Down
Loading
Loading