Skip to content

drm/asahi: harden work-queue lifetime across teardown (does not yet fix the M1 Pro firmware crash) - #431

Closed
RyanTheTide wants to merge 1 commit into
integration/aurorafrom
fix/asahi-teardown-pending-jobs-20261025
Closed

RyanTheTide wants to merge 1 commit into
integration/aurorafrom
fix/asahi-teardown-pending-jobs-20261025

Conversation

@RyanTheTide

Copy link
Copy Markdown
Contributor

Status: DRAFT, not a fix. This hardens an object-lifetime gap found while chasing the
M1 Pro (G13X) firmware crash under sustained Vulkan CTS, but on hardware it does NOT
stop the crash. See "Hardware result" below. Posted for review of the lifetime change
and the direction; the crash is still being diagnosed under t-3018.

What this changes

Under sustained Vulkan work that creates and destroys queues while the GPU is busy, the M1 Pro (G13X) firmware crashes with a translation fault, after which every submit returns ENODEV until a reboot. The crash always follows repeated "Job canceled on DRM scheduler teardown" bursts, which is the kernel cancelling scheduler jobs for a queue that is being destroyed with work still in flight.

Root cause

A user queue owns a per-pipe work queue for vertex, fragment and compute. Each work queue has a firmware QueueInfo structure (its ring state, the ring itself and the notifier list), and the queue has a notifier. The firmware learns about a work queue the first time it is run and keeps referring to it, by the GPU address of that QueueInfo, until the queue's GPU context is invalidated with a DestroyContext command.

When a queue is destroyed with jobs still on the GPU, drm_sched_fini() cancels the scheduler jobs, but the firmware keeps running the commands it already has. The work queue stays alive only through its event slot, and the completion path drops it as soon as the last stamp is seen. Dropping it frees the QueueInfo and the notifier. Those come from the shared and private kernel allocators, which hand the memory straight back out, so another queue or object can take it over.

The GPU context is a separate object. It is only invalidated at the next GpuManager::alloc(), after the queue structures may already be gone and reused. Between the work queue being freed and the context being invalidated, the firmware still holds the old QueueInfo address and can read it. If the memory has been reused, it reads a structure that now holds something else and can fault on a pointer that is no longer valid.

The context data already keeps the tile vertex buffer alive until it is invalidated, through GpuContextData. The queue structures did not get the same treatment.

The fix

Give the GPU context a list of objects to keep until it is invalidated. A work queue moves its QueueInfo into that list when it is dropped, and the queue adds its notifier to the list when it is created. The list travels with the context data to the garbage list and is freed right after DestroyContext completes, in the same alloc() pass that already invalidates idle contexts. The firmware's view of a queue therefore stays valid for as long as the firmware can still refer to it, and nothing reuses that memory early. The raw QueueInfo now points at its context weakly, since the context owns it from then on.

There is no extra timer and no idle-time cost beyond a few tens of KB per retired context, which is freed at the next GPU activity. G17 does not use this path and is unaffected.

Hardware result

Built as 7.3.0-aurora-73w-t3018 (73w recipe, config identical to 73w) and run on the
M1 Pro with the fastest known trigger (the full affected-VK v2 CTS list under the
submit-only Mesa overlay, from a fresh boot). It still crashed the firmware, at chunk 65
of 125 (firmware crash, 8 submit ENODEVs, 77 teardown lines, no kernel warnings). Stock
crashed at chunk 58 and the earlier candidate at 54 and 61 on other boots, so chunk 65
is within the boot-to-boot band. The change does not resolve the crash.

What is established: the fault is a firmware access to a kernel-private GPU object the
driver no longer owns, always after queues are destroyed with jobs in flight. The kernel
heap defers frees to a garbage list and reuses the VA at a later cache-flush pass, so
under sustained teardown load a private object the firmware still references is reused
and the firmware reads stale memory. Retaining the QueueInfo and notifier until context
invalidation was not enough, so another private object on this path (per-command
microsequences and barriers, the InitBuffer bind command, or event-stamp slot reuse) is
the next suspect. A diagnostic kernel with a reuse quarantine and a free-log is being run
to name it.

🤖 Generated with Claude Code

…nvalidated

When a queue is destroyed while it still has jobs on the GPU,
drm_sched_fini() cancels the scheduler jobs, but the firmware keeps
running the commands it already holds. Each work queue then stays alive
only through its event slot and is dropped from the completion path as
soon as its last stamp is seen. Dropping it frees its queue info (ring
state, ring and notifier list), and the last command frees the queue's
notifier and threshold. Those come from the shared and private kernel
allocators, so the memory can be reused right away. The GPU context that
the firmware still associates those structures with is only invalidated
at the next alloc(), so until then the firmware can still refer to a
work queue whose memory has already been handed out.

The context data already keeps the tile vertex buffer alive until it is
invalidated. Do the same for the queue structures. A GpuContext now
keeps a list of retained objects. A work queue moves its queue info
there when it is dropped, and the queue adds its notifier when it is
created. The list travels with the context data to the garbage list and
is freed once DestroyContext has completed. The queue info points to the
context weakly now, since the context owns it from then on.

This closes the lifetime window for those two objects. On its own it
does not stop the sustained-teardown firmware crash on the M1 Pro
(G13X): that crash still reproduces with this change, so at least one
other kernel object on the teardown path is read by the firmware after
it is freed and reused. Investigation continues.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Merged into aurora-wip through #534 (2026-10-28 consolidation). aurora-wip is the home for WIP from here on.

@RyanTheTide
RyanTheTide deleted the fix/asahi-teardown-pending-jobs-20261025 branch October 28, 2026 05:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant