Repository navigation
drm/asahi: retire faulted G17 compute commands on the firmware's fault record - #496
Merged
DjDeveloperr merged 1 commit intoOct 27, 2026
Merged
Conversation
…t record When a compute command fails on the GPU the firmware writes a zero-duration completion record for it, which quarantines its queue, and reports the failure with a firmware event of type 7 carrying the queue and the stamp the command would have stored. It never stores that stamp. The driver ignored the event, so the quarantine waited for a retirement witness that could not arrive, and the queue kept its pool, its traversal scratch and its share of the lower GPU VA window for as long as the module stayed loaded. Under ray query stress that fails this way every few seconds (for example ray_query.stress.geometry_shader beside other clients) the retained queues filled the lower window within minutes. Every later allocation there failed: publications returned EBUSY, and gnome-shell crashed on ENOMEM creating a 16K texture after about 7 minutes. Decode the event as ComputeFault and accept its stamp as the faulted command's completion stamp for the quarantine's retirement witness, while it is newer than the stored one. The witness still requires the queue's ring to be fully consumed, so commands published after the failed one keep waiting for their own stamp.
This was referenced Oct 26, 2026
RyanTheTide
added a commit
that referenced
this pull request
Oct 27, 2026
Pin the public #496/#504/#505 ports and distinguish shared retention fixes from firmware-qualified fault witnesses and descriptor changes. Document raw observability and the first-fault/idle-floor evidence needed before claiming achieved-clock, CTS or remaining performance parity. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
RyanTheTide
added a commit
that referenced
this pull request
Oct 27, 2026
…rfaces Keep host resource/ownership tests compiling the production methods after #496/#505. Include the actual fault-stamp comparison, model the new queue field, and supply inert tracing/capacity notifications. The owner-ordering fixture models no executed unstamped pair; that predicate and its lifetime are covered in kernel KUnit. Preserve each original negative control. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a compute command fails on the GPU, the T8140 firmware writes a zero-duration completion record for it and reports the failure with a firmware event of type 7. That event carries the queue (word 3) and the completion stamp the command would have stored (word 4). The firmware never stores that stamp. The driver treated type 7 as an unknown event and ignored it, so the queue's quarantine waited for a retirement witness that could not arrive. The queue then kept its pool, its ray traversal scratch and its share of the lower GPU VA window for the rest of the boot.
This change decodes type 7 as a compute fault and accepts its stamp as the failed command's completion stamp for the quarantine's retirement witness, while it is newer than the stored stamp. The witness still requires the queue's ring to be fully consumed, so commands published after the failed one keep waiting for their own stamp. A type 7 record with a layout other than this (queue number of 128 or more, or a misaligned stamp) stays an unknown event and is ignored as before.
This is a partial fix.
Measured effect
Neo (A18 Pro), Mesa agx3 at #62's head, 10-minute runs of the five-lane stress harness plus two loops of
dEQP-VK.ray_query.stress.geometry_shader.{triangles,aabbs}. The baseline is integration/aurora at #423's merge.On integration/aurora this load fills the lower VA window within about 5 minutes. After that every allocation there fails: publications return EBUSY, and in one run gnome-shell crashed on ENOMEM creating a 16 KiB texture after about 7 minutes, which ended the desktop session. With this change the window still runs out under the same stress, only later and with fewer refusals. Whether the desktop survives still depends on which client hits ENOMEM first, so this change does not prevent it.
Remaining case (follow-up)
The queues that still leak hold a command published behind the failed one, and their context holds a quarantined render pair with work behind the same failure. The firmware never runs or reports that later work, so only a kill of the context returns them. The closed-context kill path does not reach them today:
A follow-up will hold a failed context's own quarantined pairs, settle their work when the firmware acknowledges the kill, and release the hold then.
Separate pre-existing issue
When a ray query geometry-shader command fails beside other clients, the firmware also fails the compute of other clients that were running at the same moment. It sends a type 7 event with their own queue numbers, including a pure ALU dispatch with no traversal state, and the driver reports those clients as lost devices. This happens on integration/aurora as well: there, all 11 lost dispatches of a long-running ALU client coincided with a type 7 event for its own queue. It is not caused by this change, which only uses the queue and stamp from the firmware's record to retire that queue. It needs its own investigation.
Testing