Repository navigation
test(e2e): stop quit_and_wait deadlocking on a slow quit prompt - #713
Conversation
quit_and_wait resends the quit key every 500 ms and answers the quit-confirm prompt with "y" once it sees it. When the dashboard takes between one and two resend intervals to act on a key, every decision is made with one earlier "q" still in flight. That "q" flips the prompt after the screen was read, so the chosen key lands on the other state and the loop repeats "q, q, y" until the 30 s deadline. On a loaded mock lane this timed out dash-26 with the prompt open on screen. Double the wait after each attempt instead. Once it outlasts the TUI's latency nothing is in flight when the screen is read, so the key sent matches the state it lands on. The first resend still happens after 500 ms, so a quit key swallowed by a transient state is retried as quickly as before. The loop moves into the e2e-cucumber library, next to send_until, so a unit test can drive it against a simulated dashboard with a fixed key latency: it fails with the old fixed cadence and passes with this one. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Both branches were untested: dropping the error from a failed send, or the clamp that cuts the last wait to the deadline, left every test green. The clamp matters more now the wait doubles, since an unclamped final wait can run as long as the whole step timeout past it. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
7d3b866 to
4f872dd
Compare
juhovainio
left a comment
There was a problem hiding this comment.
Reviewed the quit_and_wait extraction and the new regression tests (failed-send path, deadline clamp). No blocking issues.
A couple of non-blocking notes for your consideration, neither should hold up merge:
-
The resend cadence moves from a fixed 500ms to exponential backoff (500ms -> 1s -> 2s -> ... ), which cuts total resend attempts within the default 30s timeout from ~60 to ~6. That's a good fix for the phase-lock race this PR targets, but it does mean the original "key dropped by transient state" case now gets fewer, more widely-spaced retries. Probably fine since that case tends to resolve on attempt 1-2, but worth confirming it doesn't regress recovery time there.
-
The new loop skeleton in quit_and_wait.rs (deadline/remaining/clamp arithmetic) is structurally similar to send_until.rs's; RetryTiming itself is shared but the clamp logic is duplicated rather than factored into one shared primitive both call into. Not a blocker for this PR, just a possible future unification.
Everything else checks out: backoff/clamp arithmetic is correct (including the first-iteration and overflow cases), error messages are unchanged byte-for-byte from the pre-refactor code, both call sites are unaffected, and the new tests exercise the deadline clamp and failed-send paths as described.
Why
dash-quit-confirm-when-serving(dash-26) sometimes times out on the mock e2e lane withtimed out after 30s waiting for the TUI to exit after repeating "q", and the last screen shows the "quit anyway?" prompt open. The dashboard itself is fine; the cause is the driver's quit loop.TuiSession::quit_and_waitsendsq, waits 500 ms for the process to exit, and sendsyinstead once the prompt is on screen. The open prompt readsqas Cancel. When the dashboard takes between one and two resend intervals to act on a key, which happens on a loaded runner, each decision is made with one earlierqstill in flight. Thatqflips the prompt after the screen was read, so the chosen key lands on the other state, and the loop repeatsq, q, yuntil the deadline. The sibling scenariodash-esc-menu-quit-row-confirms-when-servinghas failed the same way on other branches.What changes
The wait between sends doubles after each attempt where the process is still running. Once the wait outlasts the TUI's latency, nothing is in flight when the screen is read, so the key sent matches the state it lands on. The first resend is still sent after 500 ms, so a quit key swallowed by a transient state is retried as quickly as before. The last wait is still cut to the step deadline.
The loop moves into the
e2e-cucumberlibrary asquit_and_wait, next tosend_until, with the PTY I/O passed in as closures. The driver keeps its error messages for an unsuccessful exit, a reader-thread panic and a timeout unchanged.Verification
q, q, y; with doubling it quits. Further tests cover a responsive prompt (qtheny), an unsuccessful exit, a failed send, a child that never exits, and a long interval cut to the deadline. Removing the deadline cut or the send-error propagation each fails its test.cargo xtask e2e -- -n '(dash|chat)-'): 43 of 44 pass. The exception,dash-reports-wsl-gpu-telemetry, waits for GPU telemetry rather than quitting, and fails identically onmainon the WSL host used.cargo test --workspace --all-targets,cargo clippy --workspace --all-targets -- -D warnings,python scripts/smoke_local.pyand a Windows-targetcargo check -D warningsare clean.Not verified: the original timeout locally. Running dash-26 three times pinned to one CPU core alongside eight busy loops did not push the dashboard's key latency past 500 ms, so the reproduction is the simulated dashboard above.
Scope
This is a test-harness change only, with no user-observable CLI behaviour, so no new e2e scenario is needed (AGENTS.md §3); dash-26 and its siblings are the scenarios it stabilises. It deliberately doesn't wait for the prompt marker before deciding: that would still need a timed fallback for a swallowed key, and the growing wait covers both cases.
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows.docs/architecture.md(the clap declaration and dispatch wiring staying inmain.rs/lib.rsis expected, not a violation). (N/A: no new subcommand.)