[DjDeveloperr]

Follow-up to #496 (stacked on it). This handles the case #496 left open: a closed context whose compute command failed, with render work queued behind the failure.

What the firmware does

When a compute command fails, the firmware holds the commands its context published after it and never runs them. A kill of the context is the only thing that returns them. The context's render work behind the same failure behaves differently:

• It runs about a second later and consumes its ring entries.
• It returns its free-list blocks.
• Its completion stamp is never stored, and the job stays linked in the context's work list until the firmware processes a kill.

Before this change none of that work could retire. The context, its pools, its render pairs and its memory stayed allocated until the GPU firmware restarted, which on Neo means until reboot. On #496 a ten-minute fault storm left 12 such teardowns pending.

What this change does

1. The kill can proceed. For a closed context whose compute failed:

   • Each quarantined render pair whose commands all ran counts as idle for the kill: both pass timestamps written, rings consumed, free list balanced.
   • Admission holds such pairs for the kill together with their unretired commands.
   • The kill does not wait for the context's render pools to be released, because that only happens after the kill.
   • A faulted compute queue with no commands behind the failure is a kill candidate too. Without this, single-command faulted queues were stranded.

2. Memory is freed only after the firmware acknowledges the kill. The unstamped render work of such a pair is retired only once the context's scheduler page reports the kill. Until then its graph memory stays allocated and out of the reuse cache. This is enforced on the retire path itself, and again at the point where a detached context's graphs are handed to reclaim: a context whose work list still links a node, with no acknowledged kill, never has its graphs freed or recycled.

3. The context's work storage is retained. The firmware leaves a killed context's last job linked in its work list even after the kill. The 32 KiB work object is therefore kept mapped and out of reuse until the firmware stops, instead of being handed to the next context. Kept objects are counted in debugfs (firmware_linked_objects), and a warning is printed past 256.

4. Killed pairs are reused last. A killed context's render slots are released once its scheduler release is consumed, instead of at the next firmware stop. A new owner then takes, in this order:

   • a clean released pair, least recently used first;
   • an unowned slot, but only while the physical queue table can still hold both of the new pair's identities. The runtime updates admission whenever the table changes;
   • otherwise the killed pair released longest ago.

   Siblings and spares never take killed pairs, and a sibling doesn't claim an unowned slot once the table is full.

5. Closed owners only. The executed-witness retirement applies only to pairs whose owner has closed. A live owner's quarantined pair stays quarantined as before.

6. The kill waits for a moment without other clients' compute, within a memory bound.

   • A failed context's kill is published only when no other context's compute command is in flight; render work isn't waited for. A GPU that retired nothing for the liveness interval counts as stopped, and the kill then goes ahead.
   • When GPU memory (user committed plus kernel resident) comes within an eighth of the device limit, one waiting kill skips the wait, at most one per 250 ms. Each kill's memory is then freed before the next is considered.
   • Nothing is freed before the acknowledgement, so the waiting kill doesn't use recycled memory.
   • debugfs reports the waits in buckets, plus the memory-pressure kills and a live gauge (failed_kills waiting= longest_wait_ms=). A wait over 5 s is warned about once per kill.

Why the order matters

Earlier versions retired and freed the pair's memory first and published the kill a few milliseconds later, or minutes later with a gate. Both made the firmware use memory that had already been handed out again:

• Freed first, killed a few ms later: GPU recoveries 4–7 ms after the kill, and occasional firmware crashes, in a five-lane stress harness.
• Freed first, killed minutes later: a storm of compute watchdog timeouts while the kill waited.

Killing first and freeing after the acknowledgement removes both. Each of the other rules is there because a simpler version failed on the same harnesses:

• Killed pairs first. A diagnostic run that handed killed pairs to new owners first stranded the teardowns of 24 contexts, and a render stall behind a failed context took the desktop down.
• Killed pairs last, ignoring the queue table. The killed pairs kept their physical queue IDs installed, and the table ran out after about 50 kills: 34 publications failed with EBUSY. The queue-aware order reuses a killed pair once the table is full. In the pairs run below the table reached 127 of 128 with no refusals.
• No wait. With kill-before-free alone, kills published beside other clients' running compute made the firmware recover the GPU twice in ten minutes of the five-lane load (67 faults). drm/asahi: retire faulted G17 compute commands on the firmware's fault record #496 never kills, and had 0 recoveries in 414 faults on the same harness.
• Unbounded wait. Under sustained compute the dead clients' memory reached the device GPU memory limit (67% of RAM, 5082 MiB on Neo) in both legs. gnome-shell's next allocation failed with ENOMEM and it crashed. The memory exception in (6) prevents that.

Measured effect

Neo (A18 Pro), Mesa agx3, final version of this change.

Five-lane stress, 10 minutes (render, render2, long ALU, churn and global-priority CTS, plus two loops of dEQP-VK.ray_query.stress.geometry_shader.{triangles,aabbs}):

	#496	with this change
faults (event 7)	44	52
failed contexts killed (acknowledged, pairs retired)	0	15 (2 under memory pressure)
GPU recoveries / firmware crashes	0 / 0	0 / 0
queue publications refused with EBUSY	101	0
scheduler teardowns pending after the run	12	0
render pairs still quarantined after the run	12	0
GPU memory peak (device limit 5.0 GiB)	not sampled	4.3 GiB, no allocation refused
gnome-shell	alive	alive

Pairs harness, 10 minutes (two ray query stress loops beside heavy geometry):

• 51 faults and 51 kills (3 under memory pressure), all acknowledged, with 0 recoveries, 0 EBUSY and 0 ENOMEM.
• The physical queue table was full at 127/128 with 0 refusals.
• 4 compute watchdog timeouts between the two traversal clients. That is the long-job contention case drm/asahi: keep G17 long jobs alive beside other long jobs #499 addresses, and this branch doesn't include drm/asahi: keep G17 long jobs alive beside other long jobs #499.
• 4 render-only teardowns were left pending.
• gnome-shell stayed up.

Without the memory-pressure exception, this harness ended in a desktop death from ENOMEM while kills waited.

Known limits

• The work storage of a killed context (32 KiB) stays allocated until the GPU firmware restarts.
• While other clients keep compute running, a faulted client's kill waits, and its memory stays held until the GPU first has no foreign compute in flight, or until GPU memory nears the device limit. Under ordinary desktop load that moment comes almost at once. Under the stress harnesses kills waited up to 305 s, and GPU memory sat just under the pressure threshold.
• A kill published through the memory exception runs beside foreign compute, which risks a GPU recovery. There were 5 such kills over both runs, with no recovery after any of them.
• Render-only remainders: a closed context whose compute side is clean but whose render pair holds executed, unstamped work isn't killed yet. These are the 4 teardowns left pending in the pairs run. A kill transaction that doesn't need a compute exit candidate would close this. It is follow-up work.
• A firmware-safe kill beside running compute would remove the wait. It is not understood yet.

Testing

• Regression gate on the final build: PERFORMANCE-REGRESSION-PASS.
• Targeted CTS on the final build (5368 cases): 4687 pass, 681 not supported, 0 fail.
• Ray query and ray tracing pipeline CTS (57190 cases), on an earlier revision with the same kill path, before killed-last ordering and the memory exception: 47100 pass, 10088 not supported, 0 fail. Two host-built 1M-primitive acceleration-structure cases were killed for memory exhaustion, as on drm/asahi: retire faulted G17 compute commands on the firmware's fault record #496. Both pass when rerun serially on the final build (build.cpu.level_geometries.aabbs_1_1048576_1, build.cpuht_max.level_geometries.triangles_1_1048576_1).
• KUnit (asahi_g17*, 117 cases), new tests:
  • executed-and-consumed snapshot;
  • kill hold ends at the consumed release, and the killed pair is reused last;
  • killed pairs are taken last, oldest first;
  • a full queue table reuses killed pairs before unowned slots, without a sibling;
  • linked unkilled work is retained.
• An asahi_g17_teardown tracepoint (static key) records fault, retire, kill (marking memory-pressure kills), acknowledgement, detach and slot-owner events for diagnosing teardown on hardware. It is only called on teardown paths.