Skip to content

drm/asahi: g17: stop withholding compute doorbells - #573

Merged
DjDeveloperr merged 1 commit into
aurora-silicon:integration/aurorafrom
jgoyvaerts:m4pro/g17-no-doorbell-withholding
Oct 29, 2026
Merged

DjDeveloperr merged 1 commit into
aurora-silicon:integration/aurorafrom
jgoyvaerts:m4pro/g17-no-doorbell-withholding

Conversation

@jgoyvaerts

Copy link
Copy Markdown
Contributor

Since #468, a G17 compute publication with two or more unretired commands of its queue ahead of it withholds its doorbell, and only the compute scan of a later event pass rings it (once at most one command ahead lacks a completion record). Nothing guarantees that such a pass comes. When the last event pass for the commands ahead runs before that condition holds, the firmware sits idle on a command it was never told about, and everything waiting on that command stalls until some unrelated firmware message runs the scan.

What it looks like: Hearthstone (DXVK on Honeykrisp, which runs buffer copies as compute) in a muvm guest freezes for 330-400 ms, about ten times a minute while hovering over cards. In the guest, every freeze is DXVK's queue thread blocked in vkWaitSemaphores on one GPU fence; everything else waits behind it. On the host, kprobes on apple_mbox_send/apple_mbox_recv_irq show the GPU firmware sending no message at all for the whole freeze (7 of 7 freezes in one capture). Each gap opens right after a compute publication from the VMM's GPU thread and ends when scan_compute rings a withheld compute doorbell, after which the queued renders publish at once. Hyprland's own frames stall with it, so this is GPU-wide, not per-client.

The fix: remove the withholding and everything built on it (#468's six commits), so every compute publication rings its doorbell again, as before. The follow-ups in #468 closed some ways into the stuck state (timeouts, recovery, a record waiting for its stamp) but not the lost wakeup itself. This gives back the few microseconds per hop the withholding saved in the cross-queue semaphore ping-pong benchmark; a 400 ms stall costs more than that saves.

Tested: J616s (T6040).

  • This change on top of our daily kernel: the Hearthstone freezes no longer reproduce, and the system runs normally.
  • An earlier form of the same fix (withholding disabled by its depth constant, same behaviour) on our daily kernel: the Hearthstone freezes no longer reproduce.
  • KUnit (asahi kunitconfig, arm64 QEMU): 187 passed, 1 skipped (needs CONFIG_UCLAMP_TASK). The driver builds with no new warnings.

Not covered by a test: I could not reproduce the stall synthetically. About 180,000 bursts of small compute copies, cross-queue ping-pongs and copies queued behind long renders (native and in muvm) never stalled on the withholding kernel either, so there is no test that fails without this change. The evidence is the Hearthstone trace above. Not benchmarked on Neo.

🤖 Generated with Claude Code

A compute publication with two or more unretired commands of its queue
ahead of it withholds its doorbell. Only the compute scan of a later
event pass rings it, once at most one command ahead lacks a completion
record. Nothing guarantees that such a pass comes: if the last event
pass for the commands ahead runs before that condition holds, the
firmware sits idle on a command it was never told about, and every
waiter on that command stalls until an unrelated firmware message runs
the scan.

Under a Vulkan game in a muvm guest (DXVK on Honeykrisp, which runs
buffer copies as compute) this shows up as GPU-wide stalls of 330 to
400 ms, about ten a minute under steady input. Probes on the GPU mailbox
show the firmware sending nothing for the whole stall. Each one opens
right after a compute publication and ends when the compute scan rings
a withheld compute doorbell, after which the queued work runs at once.
The compositor's own frames stall with it.

The follow-up commits closed some of the ways into this state (job
timeouts, recovery, a record waiting for its stamp) but not the lost
wakeup itself. Remove the withholding and everything built on it, so
that every compute publication rings its doorbell again. This gives
back the few microseconds per hop the withholding saved in a
cross-queue semaphore ping-pong.

This reverts:
  3ff8c49 ("drm/asahi: withhold G17 compute doorbells behind two unretired commands")
  b1f5dc6 ("drm/asahi: drop G17 compute doorbells that another doorbell announced")
  d38cce2 ("drm/asahi: announce withheld G17 compute doorbells whenever retirement stalls")
  7012a59 ("drm/asahi: announce withheld G17 compute doorbells after a recovery")
  5de99fc ("drm/asahi: keep withholding G17 compute doorbells while a stamp is awaited")
  83ff3a9 ("drm/asahi: restart a withheld G17 compute doorbell another doorbell covered")

Fixes: 3ff8c49 ("drm/asahi: withhold G17 compute doorbells behind two unretired commands")
Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com>
Assisted-by: LLM
@DjDeveloperr

Copy link
Copy Markdown
Contributor

Tested on J700 (Neo). The module was built from this PR merged onto integration/aurora aeae3d9 (#570–#572) and loaded on that kernel.

  • KUnit asahi* (QEMU arm64): 172 passed, 1 skipped (the same skip as the base).

  • Regression gate: PASS. Targeted CTS: 4687 pass / 681 not supported / 0 fail.

  • CTS synchronization: 28647 pass / 0 fail. On the same kernel without this change, one run had a signal_order timeline case killed by the 30 s watchdog. That hasn't recurred here, though one clean run of an intermittent bug proves little. CTS synchronization2: 36516 pass / 0 fail.

  • gpubench-stress --all: no stalls, timeouts or faults. The only failures are xq.exit and life.churn, on the ENOENT teardown log lines that every current kernel prints (a separate fix is coming). queue.depth: pass.

  • Lost-wakeup hunt: reproduced on the Neo, and this change removes it. 90 s per pattern, desktop quiet. Each cell is the 99.99th percentile / worst wait:

    pattern integration/aurora (withholding) this PR
    one fill per submit 0.16 / 1.1 ms 0.15 / 5.1 ms
    three command buffers per submit 15.8 / 17.0 ms 0.18 / 0.7 ms
    fill + clear-only render pass + fill per submit 14.0 / 16.9 ms 0.17 / 1.7 ms
    two queues, one waiting on the other 0.18 / 2.2 ms 0.18 / 2.7 ms

    Only multi-command submits stall. On the Neo something periodic, at about display rate, runs the compute scan within ~17 ms, so a lost wakeup there is a 15–17 ms hitch rather than your 330–400 ms. Median waits move by +0.5–2.4% (63→64, 92→93, 88→89, 95→97 µs).

The hunt tool times every wait on GPU compute work. It submits bursts of 3–6 small compute fills on one queue with random gaps between them, waits for each burst's timeline value, and counts waits over 20/50/200 ms. It has four patterns:

  • one fill per submit;
  • three fills per submit;
  • fill + clear-only render pass + fill per submit, like DXVK's mixed submits;
  • two queues, the second waiting on the first.

A likely way into the stuck state, for whoever brings withholding back later. Publication withholds on active.len, which counts unretired commands. The release only runs in a compute scan and counts commands without a completion record. A predecessor that already has its record and only waits for its stamp still makes a new publication withhold. It raises no further event, though, so no scan comes to release the doorbell. With #572's stamps at node drain, that is every command but the last in a multi-command submit. Withholding only behind recordless predecessors, the same predicate the release uses, should close that. We'd measure that separately. Merging the revert now, as discussed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants