Skip to content

drm/asahi: retire faulted G17 compute commands on the firmware's fault record - #496

Merged
DjDeveloperr merged 1 commit into
integration/aurorafrom
fix/g17-quarantine-leak-20261026
Oct 27, 2026
Merged

DjDeveloperr merged 1 commit into
integration/aurorafrom
fix/g17-quarantine-leak-20261026

Conversation

@DjDeveloperr

Copy link
Copy Markdown
Contributor

When a compute command fails on the GPU, the T8140 firmware writes a zero-duration completion record for it and reports the failure with a firmware event of type 7. That event carries the queue (word 3) and the completion stamp the command would have stored (word 4). The firmware never stores that stamp. The driver treated type 7 as an unknown event and ignored it, so the queue's quarantine waited for a retirement witness that could not arrive. The queue then kept its pool, its ray traversal scratch and its share of the lower GPU VA window for the rest of the boot.

This change decodes type 7 as a compute fault and accepts its stamp as the failed command's completion stamp for the quarantine's retirement witness, while it is newer than the stored stamp. The witness still requires the queue's ring to be fully consumed, so commands published after the failed one keep waiting for their own stamp. A type 7 record with a layout other than this (queue number of 128 or more, or a misaligned stamp) stays an unknown event and is ignored as before.

This is a partial fix.

Measured effect

Neo (A18 Pro), Mesa agx3 at #62's head, 10-minute runs of the five-lane stress harness plus two loops of dEQP-VK.ray_query.stress.geometry_shader.{triangles,aabbs}. The baseline is integration/aurora at #423's merge.

integration/aurora with this change
quarantined compute queues awaiting a witness (peak) 23–27 10–13
lower-window capacity refusals 9486 (15 min) 166–2356
publications failed with EBUSY 426 (15 min) 4–101
GPU recoveries / firmware crashes 0 / 0 0 / 0

On integration/aurora this load fills the lower VA window within about 5 minutes. After that every allocation there fails: publications return EBUSY, and in one run gnome-shell crashed on ENOMEM creating a 16 KiB texture after about 7 minutes, which ended the desktop session. With this change the window still runs out under the same stress, only later and with fewer refusals. Whether the desktop survives still depends on which client hits ENOMEM first, so this change does not prevent it.

Remaining case (follow-up)

The queues that still leak hold a command published behind the failed one, and their context holds a quarantined render pair with work behind the same failure. The firmware never runs or reports that later work, so only a kill of the context returns them. The closed-context kill path does not reach them today:

  • it requires the context's render pools to be released, which happens only after the kill;
  • admission holds only idle render slots for the kill;
  • kill holds are released only when the firmware stops.

A follow-up will hold a failed context's own quarantined pairs, settle their work when the firmware acknowledges the kill, and release the hold then.

Separate pre-existing issue

When a ray query geometry-shader command fails beside other clients, the firmware also fails the compute of other clients that were running at the same moment. It sends a type 7 event with their own queue numbers, including a pure ALU dispatch with no traversal state, and the driver reports those clients as lost devices. This happens on integration/aurora as well: there, all 11 lost dispatches of a long-running ALU client coincided with a type 7 event for its own queue. It is not caused by this change, which only uses the queue and stamp from the firmware's record to retire that queue. It needs its own investigation.

Testing

  • Regression gate passes.
  • Targeted CTS (5368 cases): 4687 pass, 681 not supported, 0 fail.
  • Ray query and ray tracing pipeline CTS groups (57190 cases): 47100 pass, 0 fail. Two host-built 1M-primitive acceleration-structure cases were killed for memory exhaustion, as on the base.
  • KUnit: event 7 decoding and stamp ordering.

…t record

When a compute command fails on the GPU the firmware writes a
zero-duration completion record for it, which quarantines its queue,
and reports the failure with a firmware event of type 7 carrying the
queue and the stamp the command would have stored. It never stores
that stamp. The driver ignored the event, so the quarantine waited for
a retirement witness that could not arrive, and the queue kept its
pool, its traversal scratch and its share of the lower GPU VA window
for as long as the module stayed loaded.

Under ray query stress that fails this way every few seconds (for
example ray_query.stress.geometry_shader beside other clients) the
retained queues filled the lower window within minutes. Every later
allocation there failed: publications returned EBUSY, and gnome-shell
crashed on ENOMEM creating a 16K texture after about 7 minutes.

Decode the event as ComputeFault and accept its stamp as the faulted
command's completion stamp for the quarantine's retirement witness,
while it is newer than the stored one. The witness still requires the
queue's ring to be fully consumed, so commands published after the
failed one keep waiting for their own stamp.
@DjDeveloperr
DjDeveloperr merged commit 7a27c9c into integration/aurora Oct 27, 2026
RyanTheTide added a commit that referenced this pull request Oct 27, 2026
Pin the public #496/#504/#505 ports and distinguish shared retention fixes
from firmware-qualified fault witnesses and descriptor changes. Document
raw observability and the first-fault/idle-floor evidence needed before
claiming achieved-clock, CTS or remaining performance parity.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
RyanTheTide added a commit that referenced this pull request Oct 27, 2026
…rfaces

Keep host resource/ownership tests compiling the production methods after
#496/#505. Include the actual fault-stamp comparison, model the new queue
field, and supply inert tracing/capacity notifications. The owner-ordering
fixture models no executed unstamped pair; that predicate and its lifetime
are covered in kernel KUnit. Preserve each original negative control.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant