Repository navigation
drm/asahi: harden work-queue lifetime across teardown (does not yet fix the M1 Pro firmware crash) - #431
Closed
RyanTheTide wants to merge 1 commit into
Closed
drm/asahi: harden work-queue lifetime across teardown (does not yet fix the M1 Pro firmware crash)#431RyanTheTide wants to merge 1 commit into
RyanTheTide wants to merge 1 commit into
Conversation
…nvalidated When a queue is destroyed while it still has jobs on the GPU, drm_sched_fini() cancels the scheduler jobs, but the firmware keeps running the commands it already holds. Each work queue then stays alive only through its event slot and is dropped from the completion path as soon as its last stamp is seen. Dropping it frees its queue info (ring state, ring and notifier list), and the last command frees the queue's notifier and threshold. Those come from the shared and private kernel allocators, so the memory can be reused right away. The GPU context that the firmware still associates those structures with is only invalidated at the next alloc(), so until then the firmware can still refer to a work queue whose memory has already been handed out. The context data already keeps the tile vertex buffer alive until it is invalidated. Do the same for the queue structures. A GpuContext now keeps a list of retained objects. A work queue moves its queue info there when it is dropped, and the queue adds its notifier when it is created. The list travels with the context data to the garbage list and is freed once DestroyContext has completed. The queue info points to the context weakly now, since the context owns it from then on. This closes the lifetime window for those two objects. On its own it does not stop the sustained-teardown firmware crash on the M1 Pro (G13X): that crash still reproduces with this change, so at least one other kernel object on the teardown path is read by the firmware after it is freed and reused. Investigation continues. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
This was referenced Oct 26, 2026
Contributor
Author
|
Merged into aurora-wip through #534 (2026-10-28 consolidation). aurora-wip is the home for WIP from here on. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Under sustained Vulkan work that creates and destroys queues while the GPU is busy, the M1 Pro (G13X) firmware crashes with a translation fault, after which every submit returns ENODEV until a reboot. The crash always follows repeated "Job canceled on DRM scheduler teardown" bursts, which is the kernel cancelling scheduler jobs for a queue that is being destroyed with work still in flight.
Root cause
A user queue owns a per-pipe work queue for vertex, fragment and compute. Each work queue has a firmware
QueueInfostructure (its ring state, the ring itself and the notifier list), and the queue has a notifier. The firmware learns about a work queue the first time it is run and keeps referring to it, by the GPU address of thatQueueInfo, until the queue's GPU context is invalidated with a DestroyContext command.When a queue is destroyed with jobs still on the GPU, drm_sched_fini() cancels the scheduler jobs, but the firmware keeps running the commands it already has. The work queue stays alive only through its event slot, and the completion path drops it as soon as the last stamp is seen. Dropping it frees the
QueueInfoand the notifier. Those come from the shared and private kernel allocators, which hand the memory straight back out, so another queue or object can take it over.The GPU context is a separate object. It is only invalidated at the next
GpuManager::alloc(), after the queue structures may already be gone and reused. Between the work queue being freed and the context being invalidated, the firmware still holds the oldQueueInfoaddress and can read it. If the memory has been reused, it reads a structure that now holds something else and can fault on a pointer that is no longer valid.The context data already keeps the tile vertex buffer alive until it is invalidated, through
GpuContextData. The queue structures did not get the same treatment.The fix
Give the GPU context a list of objects to keep until it is invalidated. A work queue moves its
QueueInfointo that list when it is dropped, and the queue adds its notifier to the list when it is created. The list travels with the context data to the garbage list and is freed right after DestroyContext completes, in the samealloc()pass that already invalidates idle contexts. The firmware's view of a queue therefore stays valid for as long as the firmware can still refer to it, and nothing reuses that memory early. The rawQueueInfonow points at its context weakly, since the context owns it from then on.There is no extra timer and no idle-time cost beyond a few tens of KB per retired context, which is freed at the next GPU activity. G17 does not use this path and is unaffected.
Hardware result
Built as 7.3.0-aurora-73w-t3018 (73w recipe, config identical to 73w) and run on the
M1 Pro with the fastest known trigger (the full affected-VK v2 CTS list under the
submit-only Mesa overlay, from a fresh boot). It still crashed the firmware, at chunk 65
of 125 (firmware crash, 8 submit ENODEVs, 77 teardown lines, no kernel warnings). Stock
crashed at chunk 58 and the earlier candidate at 54 and 61 on other boots, so chunk 65
is within the boot-to-boot band. The change does not resolve the crash.
What is established: the fault is a firmware access to a kernel-private GPU object the
driver no longer owns, always after queues are destroyed with jobs in flight. The kernel
heap defers frees to a garbage list and reuses the VA at a later cache-flush pass, so
under sustained teardown load a private object the firmware still references is reused
and the firmware reads stale memory. Retaining the QueueInfo and notifier until context
invalidation was not enough, so another private object on this path (per-command
microsequences and barriers, the InitBuffer bind command, or event-stamp slot reuse) is
the next suspect. A diagnostic kernel with a reuse quarantine and a free-log is being run
to name it.
🤖 Generated with Claude Code