Skip to content

drm/asahi: g17: stop cancelling behind a failed command once it has retired - #556

Merged
RyanTheTide merged 1 commit into
aurora-silicon:integration/aurorafrom
jgoyvaerts:m4pro/g17-failed-record-retirement
Oct 31, 2026
Merged

RyanTheTide merged 1 commit into
aurora-silicon:integration/aurorafrom
jgoyvaerts:m4pro/g17-failed-record-retirement

Conversation

@jgoyvaerts

Copy link
Copy Markdown
Contributor

Problem

Each g17 queue recorded the earliest failed ordinal per engine (failures[engine]) and kept it for the queue's lifetime. Any later command whose prefix covered that ordinal was cancelled. GL submits with a full prefix (every command orders after every earlier one), so after one failed command, every later command of that queue was cancelled too, forever.

Since cancellation is no longer reported as a device loss, the client never notices. Seen on a J616s (M4 Pro):

  • CS2 hangs the GPU.
  • One Hyprland render that sampled the game's failed buffer is cancelled.
  • Every later Hyprland render is cancelled as well: about 47 a second, all fences ECANCELED.
  • The screen keeps the game's last frame until Hyprland or the machine restarts. New buffers do not help, because the queue itself is poisoned.

Fix

The record exists so that nothing runs before a failed producer that the firmware may still own or replay. That only holds while the command keeps its VM pin:

  • complete() and fail_unpublished() release the pin.
  • fail_while_owned() keeps it until release_after_retirement().

FailureRecord now tracks the failed commands that still hold their pin, and drops each one once it has released the pin. A failure that cannot be tracked because memory runs out still cancels for good, as before.

Testing

  • KUnit (asahi_g17_dependency, new): 4 tests. Three of them fail on the old keep-forever behaviour. 188/188 pass on integration/aurora with the change.
  • Live, J616s, 7.3 + this change:
    • CS2 hung the GPU and lost its device.
    • Hyprland kept drawing: its fences signalled successfully, 0 cancellations, about 380 renders in 4 s after the recovery.
    • Without the change, the same repro froze the desktop every time (4 times today).

Relation to #433

This is independent of #433 and applies cleanly to integration/aurora without it. #433 keeps innocent clients alive across another client's recovery. This change stops a client's own queue from staying poisoned after one cancelled command. The two are complementary: with #433 alone, the desktop still froze after a CS2 hang.

Assisted-by: LLM

…etired

A queue remembered the earliest failed ordinal of each engine for good,
and every later command whose prefix covered it was cancelled. GL orders
each submission after every earlier one, so once one command of a queue
failed, every later command of that queue was cancelled too. With
cancellation no longer reported as a lost device, the client never
noticed: after a CS2 GPU hang, one Hyprland render that sampled the
game's failed buffer was cancelled, then every later Hyprland render
(about 47 a second, all fences ECANCELED), and the screen kept the
game's last frame until Hyprland or the machine restarted. New
buffers did not help; the queue itself was poisoned.

The record exists so that nothing runs before a failed producer the
firmware may still own or replay. That holds only while the command
keeps its VM pin: complete() and fail_unpublished() release it,
fail_while_owned() keeps it until release_after_retirement(). Track the
failed commands that still hold their pin and stop counting each one
once it has released it. A failure that cannot be tracked for lack of
memory keeps cancelling for good, as before.

Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com>
Assisted-by: LLM
@RyanTheTide
RyanTheTide merged commit 41c4055 into aurora-silicon:integration/aurora Oct 31, 2026
@RyanTheTide

Copy link
Copy Markdown
Contributor

Merged into integration/aurora. Hardware: on the J616s a second client submitted a hanging dispatch while glmark2 looped (4 runs around 7000, 6928 after): the other client kept completing, no stray cancels, no GPU faults, journalctl -k -p 3 empty, no WARN or DART faults. The M2, M1 Pro and Neo passed with it in the integration candidate, including 3 concurrent glmark2 clients for 60 s with 0 GPU errors. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants