Skip to content

test(e2e): stop quit_and_wait deadlocking on a slow quit prompt - #713

Merged
jussielo-amd merged 2 commits into
mainfrom
fix/e2e-quit-confirm-resend-race
Oct 21, 2026
Merged

jussielo-amd merged 2 commits into
mainfrom
fix/e2e-quit-confirm-resend-race

Conversation

@rominf

@rominf rominf commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

Why

dash-quit-confirm-when-serving (dash-26) sometimes times out on the mock e2e lane with timed out after 30s waiting for the TUI to exit after repeating "q", and the last screen shows the "quit anyway?" prompt open. The dashboard itself is fine; the cause is the driver's quit loop.

TuiSession::quit_and_wait sends q, waits 500 ms for the process to exit, and sends y instead once the prompt is on screen. The open prompt reads q as Cancel. When the dashboard takes between one and two resend intervals to act on a key, which happens on a loaded runner, each decision is made with one earlier q still in flight. That q flips the prompt after the screen was read, so the chosen key lands on the other state, and the loop repeats q, q, y until the deadline. The sibling scenario dash-esc-menu-quit-row-confirms-when-serving has failed the same way on other branches.

What changes

The wait between sends doubles after each attempt where the process is still running. Once the wait outlasts the TUI's latency, nothing is in flight when the screen is read, so the key sent matches the state it lands on. The first resend is still sent after 500 ms, so a quit key swallowed by a transient state is retried as quickly as before. The last wait is still cut to the step deadline.

The loop moves into the e2e-cucumber library as quit_and_wait, next to send_until, with the PTY I/O passed in as closures. The driver keeps its error messages for an unsuccessful exit, a reader-thread panic and a timeout unchanged.

Verification

  • A unit test drives the loop against a simulated dashboard that acts on each key 80 ms after it is sent, with a 50 ms resend interval. With the old fixed cadence it times out repeating q, q, y; with doubling it quits. Further tests cover a responsive prompt (q then y), an unsuccessful exit, a failed send, a child that never exits, and a long interval cut to the deadline. Removing the deadline cut or the send-error propagation each fails its test.
  • The dash and chat e2e scenarios (cargo xtask e2e -- -n '(dash|chat)-'): 43 of 44 pass. The exception, dash-reports-wsl-gpu-telemetry, waits for GPU telemetry rather than quitting, and fails identically on main on the WSL host used.
  • cargo test --workspace --all-targets, cargo clippy --workspace --all-targets -- -D warnings, python scripts/smoke_local.py and a Windows-target cargo check -D warnings are clean.

Not verified: the original timeout locally. Running dash-26 three times pinned to one CPU core alongside eight busy loops did not push the dashboard's key latency past 500 ms, so the reproduction is the simulated dashboard above.

Scope

This is a test-harness change only, with no user-observable CLI behaviour, so no new e2e scenario is needed (AGENTS.md §3); dash-26 and its siblings are the scenarios it stabilises. It deliberately doesn't wait for the prompt marker before deciding: that would still need a timed fallback for a swallowed key, and the growing wait covers both cases.


  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows.
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md (the clap declaration and dispatch wiring staying in main.rs/lib.rs is expected, not a violation). (N/A: no new subcommand.)
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will not happen. (N/A: no user-facing message changed.)

@rominf
rominf marked this pull request as ready for review October 21, 2026 06:06
@rominf
rominf requested a review from a team as a code owner October 21, 2026 06:06
@rominf
rominf requested a review from juhovainio October 21, 2026 06:06
quit_and_wait resends the quit key every 500 ms and answers the
quit-confirm prompt with "y" once it sees it. When the dashboard takes
between one and two resend intervals to act on a key, every decision is
made with one earlier "q" still in flight. That "q" flips the prompt
after the screen was read, so the chosen key lands on the other state
and the loop repeats "q, q, y" until the 30 s deadline. On a loaded
mock lane this timed out dash-26 with the prompt open on screen.

Double the wait after each attempt instead. Once it outlasts the TUI's
latency nothing is in flight when the screen is read, so the key sent
matches the state it lands on. The first resend still happens after
500 ms, so a quit key swallowed by a transient state is retried as
quickly as before.

The loop moves into the e2e-cucumber library, next to send_until, so a
unit test can drive it against a simulated dashboard with a fixed key
latency: it fails with the old fixed cadence and passes with this one.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Both branches were untested: dropping the error from a failed send, or
the clamp that cuts the last wait to the deadline, left every test
green. The clamp matters more now the wait doubles, since an unclamped
final wait can run as long as the whole step timeout past it.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the fix/e2e-quit-confirm-resend-race branch from 7d3b866 to 4f872dd Compare October 21, 2026 07:14
@rominf
rominf enabled auto-merge October 21, 2026 07:33
@rominf
rominf disabled auto-merge October 21, 2026 07:33

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the quit_and_wait extraction and the new regression tests (failed-send path, deadline clamp). No blocking issues.

A couple of non-blocking notes for your consideration, neither should hold up merge:

  1. The resend cadence moves from a fixed 500ms to exponential backoff (500ms -> 1s -> 2s -> ... ), which cuts total resend attempts within the default 30s timeout from ~60 to ~6. That's a good fix for the phase-lock race this PR targets, but it does mean the original "key dropped by transient state" case now gets fewer, more widely-spaced retries. Probably fine since that case tends to resolve on attempt 1-2, but worth confirming it doesn't regress recovery time there.

  2. The new loop skeleton in quit_and_wait.rs (deadline/remaining/clamp arithmetic) is structurally similar to send_until.rs's; RetryTiming itself is shared but the clamp logic is duplicated rather than factored into one shared primitive both call into. Not a blocker for this PR, just a possible future unification.

Everything else checks out: backoff/clamp arithmetic is correct (including the first-iteration and overflow cases), error messages are unchanged byte-for-byte from the pre-refactor code, both call sites are unaffected, and the new tests exercise the deadline clamp and failed-send paths as described.

@jussielo-amd
jussielo-amd added this pull request to the merge queue Oct 21, 2026
Merged via the queue into main with commit 6186f15 Oct 21, 2026
32 checks passed
@jussielo-amd
jussielo-amd deleted the fix/e2e-quit-confirm-resend-race branch October 21, 2026 13:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants