Skip to content

drm/asahi: add G17 accounting and fix GPU resource lifetimes - #151

Closed
DjDeveloperr wants to merge 22 commits into
integration/aurorafrom
fix/g17-telemetry-20261019
Closed

DjDeveloperr wants to merge 22 commits into
integration/aurorafrom
fix/g17-telemetry-20261019

Conversation

@DjDeveloperr

@DjDeveloperr DjDeveloperr commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Neo’s G17 GPU needs usable load and memory accounting, and concurrent clients expose several driver resource-admission failures. This series adds accounting, fixes those failures while preserving queue capacity and batching, and reduces unnecessary render-target storage.

Changes:

  • Expose G17 firmware utilization with power-state freshness, standard DRM per-client memory fdinfo, and categorized kernel allocation extents. Full raw firmware capture remains an explicit diagnostic read with a two-second expiry. GEM reporting is an Asahi opt-in for the Rust DRM core.
  • Let unused speculative render pairs yield under admission pressure. Recover only tracked, never-published initialization QIDs for compute; preserve published identities, the render minimum and the shared 128-ID physical capacity.
  • Find render-pool backing addresses free in the canonical and both alias VMs. Keep ordinary best-fit placement as the fast path, use ordered common holes on collision, and roll partial aliases back safely.
  • Keep pending compute-pool release transactions polled and treat their admission as bounded temporary backpressure. Preserve the control-consumer witness before reuse; waiting releases the device mutex and preparation lease.
  • Remove unused render-state storage and use one 32 KiB TVB page as the minimum target-record stride, retaining the computed layout size and all sixteen generations. With forty held clients and the same 124 target objects, target allocation extents fall from 259.625 MiB to 42.625 MiB: a 217 MiB reduction under that workload.
  • Replace the exact-node devres list search with tracked device-list membership. Clear membership under the device lock on every detach path, including moves to private group/release-all lists, so a concurrent Rust Devres drop cannot steal a release callback’s node. Explicit single-copy pointer access also covers a lookup using a different device. This removes quadratic GPU process-exit cleanup without limiting object creation or concurrency. Device-resource nodes grow by one pointer.

Validation on Neo G17P B1 uses the unified 7.2.10 integration tree with Asahi as a module. The complete kernel Image, modules and device trees build, including strict modpost checking. The installed kernel, embedded module and loaded module build ID are verified before hardware tests.

The exact final reviewed build passes forty held independent GLES clients with redraw and fresh-client checks; repeated contexts within one retained VM; three simultaneous 300-second render/compute/mixed workloads with concurrent context churn and layered rendering; Vulkan four-sample color/depth/stencil checks; additional large layered-target growth during concurrent stress (144 pixel checks and 200 layered readbacks); four compute clients with changing integer readbacks; twelve four-client launches around retained-pool expiry; and 2,000 contexts across 100 retained-VM cycles. No GPU client failure, submit failure, GPU fault, BUG or panic appears. Four compute clients performing thirty seconds of checked work finish and exit in 31.46 seconds total, compared with 86.97 seconds using the same compact layout before the devres correction.

Source-derived controls cover all 388,800 target layouts, retaining capacity and consumer bounds, and all 262,144 occupancy combinations of a three-VM common-address oracle. Devres host controls exercise initialization, replacement, ordinary/action/group removal, private release-list ownership, and 1,000 concurrent detach/remove/wrong-device races with ASan/UBSan. A stale-private-list-membership negative control fails as expected. Existing telemetry decoding, snapshot concurrency/expiry, allocation lifetime, DRM opt-in and shmem callback controls pass. Added kernel KUnit assertions have not been run.

Performance measurements remain bounded. Three six-second fresh-boot runs per mode give final-series medians of 680.04/205.36/160.63 iterations per second for render/compute/mixed, versus 682.98/206.89/156.71 for the original target-floor control. These unpaired samples decline with warming and do not establish broad performance neutrality. Explicit full raw firmware capture costs about 3% in a same-boot comparison and remains opt-in. Low/medium firmware priority mappings are preserved; high/realtime remain reserved, and strict tail-latency guarantees have not been established.

Broader game validation remains incomplete. Ryujinx reaches Brilliant Diamond’s 3D introduction around 30 FPS and accepts controller input, but the scene shows excessively bright highlights and visible aliasing, and character naming stalls in the software-keyboard applet. This is not a visual-correctness or gameplay pass. Focused controls pass distinct 2×/4× HDR sample uploads and average resolves in RGBA16F and R11G11B10F, sRGB roundtrips, and independently read-back raster coverage, sample masks and floating-point additive blending. Those controls do not establish correctness of the game’s full rendering pipeline. The reported visual defects remain under investigation; the captured naming trace also exposes a foreground applet returning Success before it completes. A separate emulator fix handles a buffer upload into a four-sample depth image, which violates VUID-vkCmdCopyBufferToImage-dstImage-07973; that change is outside this kernel PR. A GLES four-sample default-pbuffer readback failure also reproduces on the unmodified packaged 7.1.12 kernel with identical Mesa, while single-sample and layered checks pass. Gameplay and other GPU families still need broader validation, so this PR remains draft.

KDE discovery requires a separate userspace Asahi backend. Allocation extents and shared/imported handle totals are distinct from unique physical RAM usage. Firmware power residency is kept separate from GPU execution; per-client engine time, temperature and power are not inferred.

@DjDeveloperr DjDeveloperr changed the title drm/asahi: add G17 GPU telemetry and reduce render-state memory drm/asahi: add G17 telemetry and reclaim retained render capacity Oct 19, 2026
@DjDeveloperr
DjDeveloperr marked this pull request as draft October 19, 2026 23:27
@DjDeveloperr DjDeveloperr changed the title drm/asahi: add G17 telemetry and reclaim retained render capacity drm/asahi: add G17 telemetry and fix GPU resource admission Oct 20, 2026
Wire Rust DRM file operations and drivers into the standard fdinfo memory
helper. Observe shmem residency without populating pages or taking the
reservation lock, including cached pages after device unpinning.

Add residency controls and a client-memory reporter which deduplicates
shared client FDs and keeps shared/imported memory distinct from private
handle accounting.
Track GEM allocations across GPU generations and classify kernel backing
without counting client aliases again. Expose current and peak extents
through a diagnostic memory view; keep imported wrapper accounting explicit.

Remove two unused pages from each render generation and the unused register
shadow and alias from each render pair. Preserve generation depth, work
capacity and embedded descriptor register arrays. The cold Neo comparison
saves 528 KiB per pair, or 7.22 MiB with the same fourteen installed pairs.
Decode the four utilization percentages independently and use the aggregate
reported by firmware. Order power witnesses in the firmware clock domain,
clear load after power-off, and require fresh utilization after wake. Mark
device loss and statistics-stream failure unavailable rather than retaining
a plausible busy value.

Expose a read-only per-device snapshot and document its validity and units.
Keep completed power residency separate from execution time. Full raw
payload capture is enabled only by an explicit diagnostic read and expires
after two seconds, preserving the normal report-consumption path.

Add decoder, snapshot-concurrency and capture-expiry controls, plus a load
reader which tolerates a transient snapshot collision. On Neo the default
path sustains the earlier compute throughput; explicit raw capture costs
about three percent in the matched comparison.
Signed-off-by: Dj <43033058+DjDeveloperr@users.noreply.github.com>
A canonical low address may still be occupied by a retained alias in
another VM. RenderBacking::new previously propagated that fixed-address
collision as ENOSPC, failing accepted render work during shared-VM queue
replacement even though backing memory and other addresses were available.

Place fresh backing in the canonical and both alias VMs before initializing
the pool tables. Keep ordinary best-fit placement on the fast path; after a
collision, search lowest-address holes in the refusing VM and retry with
the same backing. Roll partial aliases back before releasing the canonical
reservation. Keep guards, physical capacities, generation depth and
firmware retirement requirements intact.

Add source-derived controls for rollback, non-capacity errors, guards and
best-fit holes, plus an independent occupancy oracle covering all 262144
combinations of three six-page address spaces.

On Neo G17P B1, the first-run 40-client/shared-VM reproduction, changing
pixel checks, three simultaneous 300-second workloads, concurrent layered
rendering, Vulkan multisample readbacks and four compute owners pass with
no kernel GPU errors. A further 2000-context shared-VM test passes and
returns tracked kernel memory below its starting level after VM close.

Signed-off-by: Dj <43033058+DjDeveloperr@users.noreply.github.com>
@DjDeveloperr
DjDeveloperr force-pushed the fix/g17-telemetry-20261019 branch from 4000ee0 to 8f429c0 Compare October 20, 2026 02:55
A released compute queue cannot be rebound while its USC pool release is
still awaiting control-consumer observation. At a full physical QID
namespace, all retained queues can occupy this state together: admission
then returns ENOSPC even though no owner or command holds those queues.

Keep the event worker polling until every published pool release has been
observed. Classify namespace exhaustion with a pending release as temporary
backpressure and let admission retry outside the device mutex and the
preparation lease. Bound that wait and return EBUSY if it does not settle.

Preserve the firmware release witness, physical identity lifetimes, warm
retention policy and normal admission with spare capacity. Source-derived
controls cover the captured four-queue transition, wake removal after
settlement, genuine exhaustion and error propagation; both original
failure modes fail the same checks as negative controls.

On Neo G17P, the full cold-start pressure/geometry/multisample gate and
three simultaneous 300-second workloads pass. Twelve further four-client
compute launches, including the ten-second pool-expiry boundary, verify
420,110,336 changing integer words with no kernel GPU errors.

Signed-off-by: Dj <43033058+DjDeveloperr@users.noreply.github.com>
Rust Devres drops remove an exact node, but devres_node_remove scans the
entire device list to find it. Repeated removal of older GPU SG-table
resources therefore does quadratic work while holding devres_lock. After
large concurrent GPU workloads this keeps exiting clients CPU-bound for
roughly a minute.

Track the device-list owner in each node and remove a matching node
directly under the device lock. Clear membership on every detach path,
including replacement, ordinary and action removal, group removal, and
moves to the private release-all/group callback list. A concurrent drop
must leave nodes already owned by that release path alone. Preserve
callback ordering and lifetime handling.

Use single-copy membership accesses so a simultaneous comparison using a
different device does not race through an ordinary pointer access. The
membership field adds one pointer per resource node.

Source-derived ASan/UBSan controls cover the removal APIs, replacement,
open/closed/nested groups, private-list ownership and 1000 concurrent
removal/detachment/different-device lookup races. Retaining stale private
list ownership makes the negative control fail.

The combined series passes Neo G17P concurrent GPU stress, expiry cycles
and retained-VM context churn. Four clients performing 30 seconds of
checked compute work finish and exit in 31.46 seconds total, versus 86.97
seconds with the same compact target layout before this correction.

Signed-off-by: Dj <43033058+DjDeveloperr@users.noreply.github.com>
The target-record stride already accounts for its computed layout and UAT
alignment, but its 256 KiB minimum wastes backing on smaller targets. Use
one 32 KiB TVB page as the minimum instead, keeping the computed used size,
consumer bounds, growth handling and all sixteen generations unchanged.
Queue capacities, batching and synchronization policy are unchanged.

Source-derived controls validate 388800 layouts against the original
capacity and consumer bounds; 114426 need less backing. At forty held GPU
clients with 124 target objects on Neo, allocation extents fall from
272236544 to 44695552 bytes, saving 217 MiB for that workload.

The compact layout passes changing-pixel and layered readbacks, target
growth, concurrent render/compute/mixed stress, Vulkan multisample checks,
compute pool expiry and 2000 retained-VM contexts. Brilliant Diamond also
reaches its language menu around 30 FPS in a bounded emulator run; broader
gameplay and application performance validation remain separate.

Signed-off-by: Dj <43033058+DjDeveloperr@users.noreply.github.com>
@DjDeveloperr DjDeveloperr changed the title drm/asahi: add G17 telemetry and fix GPU resource admission drm/asahi: add G17 accounting and fix GPU resource lifetimes Oct 20, 2026
@DjDeveloperr
DjDeveloperr marked this pull request as ready for review October 21, 2026 11:41
G17 queue IDs never change engine once published (the driver has no
firmware operation that uninstalls one), and the driver rebinds a
physical render pair or compute queue only after its logical owner
closes. A live logical queue therefore keeps its compute queue and up to
two render pairs while idle. Several Vulkan devices with eight queues
each (the synchronization multi_queue cases, six processes), or 50 held
single-queue clients, exceed both pools with idle owners alone:

 - Render: 63 admission slots, two per owner. A new owner is refused in
   acquire_render_with_ids() (admission ENOSPC, or no QIDs for a fresh
   pair) and its accepted job fails at publication ("reported at
   g17/runtime/backend.rs:39"), even when the holders are only warming
   spares or finishing a job.
 - Compute: reserve_compute() fails and build_compute() returns ENOSPC
   from the SUBMIT ioctl, which is fatal to the client.
 - Compute could take any QID beyond a 16-identity render minimum. Those
   QIDs stay compute for the device lifetime, so open queues could be
   left without a render pair.

Treat held identities as contention:

 - Compute takes any free QID beyond a render floor of one pair for each
   open logical queue (existing pairs included). Below that floor it only
   replaces the unpublished initial reservations, which no render pair
   can use, and cancels unused render reservations only beyond the floor.
   Contention-free clients keep one compute queue each, as before.
 - When neither a fresh QID nor a released queue exists, publication may
   move an idle installed compute queue of another live owner (no active,
   replayed, quarantined, exiting, growing or superseded state; populated
   pool large enough) through the existing bind_owner() path, preferring
   the same VM and the least recently used queue. It publishes under the
   same device mutex hold. Render surplus held by installed pairs never
   becomes compute capacity, so it does not prevent the takeover. The
   previous owner keeps its context and binds again at its next
   publication.
 - Submission no longer fails when no physical compute queue is free. It
   leaves binding to publication, which retries for at most the job
   timeout and then fails the packet like any other unpublished error.
   One binder per owner prevents a submission and a publication from
   giving the same owner two queues.
 - A refused render owner retries reclamation after each release for a
   bounded time instead of failing at once: unused warming first, then the
   least recently used idle installed pair of a live owner that keeps
   another installed pair. The donor claims no other pair while it is
   open, not even the one it gave away: its work-state channel ordinals
   stay bound to the pairs it already used, and the rebind displaces its
   graph with the context still open, so that graph stays retained for the
   donor's scheduler until its own release, armed only by its close. The
   recipient returns its empty reservations and ends with exactly that
   pair.
 - A VM keeps one parameter-buffer view for each of its graphs on a pair
   instead of one per pair. The displaced graph of an open queue keeps its
   view until that queue's release, so with one view per pair any other
   queue of the same VM (one DRM file with several queues) offered the
   freed pair could not install a graph there before the donor closed.
 - The render admission wait for a pick is bounded instead of waiting
   indefinitely where the scheduler watchdog cannot fail the job.
 - A compute queue rebound from a detached owner that closed without
   publishing now returns to service instead of keeping that owner, its
   QID and VM aliases until processor stop.

Published identities keep their lifetime rules: no QID is cancelled or
changes engine, and every rebind still requires the queue or pair to be
idle with no pending firmware witness, as for a closed owner's queue.

Without contention, publication is unchanged apart from a registry lookup
under the already held device mutex, and no queue changes owner.

The resource host tests cover the floor for open queues, initial-identity
replacement, render forfeit guards and recency, compute takeover states,
the unpublished close, 40 held plus 12 fresh clients sharing 20 compute
queues with all QIDs leased and render surplus beyond the floor, the
survivors of 47 render clients after closes restore that surplus, a
compositor that gives its idle sibling to a held client and keeps drawing
on its other pair while the recipient holds it and after it exits, and
the other queues of a donor's VM installing and drawing on the freed
pair; the close, admission, donor and shared-VM tests fail without their
fixes.

Signed-off-by: DjDeveloperr <hey@djdev.me>
When a logical queue closes, its scheduler release is published once its
work is idle. The firmware consumes the release against the queues that
scheduler last ran. After the host has seen it consumed, the teardown parks
the scheduler's render pairs, and a later owner reinstalls a parked pair
with fresh queue records.

A successor on the same VM could find such a pair still installed after
the release was published but before its consumption was seen. It then
rebound the pair in place, rewriting the job-list head, pid and scheduler
fields of queue records the firmware had run for the released scheduler,
and published its first command, which carries the owner change.

A GL client that recreates its context every few milliseconds next to
three Vulkan clients slowed the release enough to open this window. Its
successor published the first commands on both of its pairs inside it,
and the GPU firmware crashed about 3 ms later without writing a crash log.
At the crash the release had been consumed. Both pairs were rebound in
place and their first commands for the new owner were still in flight.

Order owner changes and scheduler releases:

 - A render pair whose context has an armed or unconsumed scheduler
   release is not rebound. The new owner retries until the teardown has
   parked the pair, then installs it with fresh records. The check is
   made when the rebind is chosen and again when it is committed, because
   a live donor may close while one of its pairs is prepared for another
   owner.
 - A scheduler release waits while a render pair or compute queue that
   changed owner away from that context still has the new owner's first
   command in flight. A pair records at publication which scheduler its
   queues last ran for. A borrowed pair keeps a snapshot of that state.
 - The first command after an owner change waits while the scheduler the
   pair last ran for is being released. This happens only when a live
   donor closes between giving up a pair and the recipient's first
   command.
 - The same holds for compute queues: a queue whose bound scheduler has an
   armed or unconsumed release is not rebound, re-armed for another owner
   or taken over, and the first command after a compute owner change
   waits while the scheduler the queue was bound to is being released.
 - A pair parked without a consumed release, an unused warm pair moved to
   another owner, keeps the scheduler it last ran for, so its next owner
   change still waits for that release. A park after the consumed release
   of the pair's scheduler clears it.

The normal path is unchanged. A pair whose owner does not change compares
one word at publication. A release is delayed only while an owner change
is in flight. After a context churn, a same-VM successor's first render
waits until the previous context's release has been consumed and its
pairs parked, which successors on another VM already did.

Measured on Neo with diagnostic switches that apply the first two rules:
 - four-client churn: 3 of 3 runs crashed within 10 to 120 s without
   them, 2 runs of 90 s passed with them
 - GLES: 15160 vs 15175 iterations, 5054 vs 5059 contexts per 90 s
 - Vulkan 4x MSAA render: 212643 vs 214050 iterations

tools/asahi/g17-owner-release-order-tests.py compiles the release table,
pair switch tracking and release readiness predicates against host stubs
and checks the guard placement in the rebind and publication paths.

Signed-off-by: DjDeveloperr <hey@djdev.me>
A render command whose admission and queue IDs succeeded can still fail
during graph construction or preparation with ENOSPC: parameter-buffer
or counter identities, render free-list IDs, retired-binding room or the
shared lower GPU VA window may all be held by other open owners during a
burst of new clients. The packet then fails at publication within a
fraction of a second although these resources free as warming graphs are
reclaimed, published work retires and displaced scheduler releases
complete. Construction of an optional warming sibling fails the command
the same way, even though its own pair is ready.

Retry such a refusal after a release for the same bounded time as
admission, with the device mutex and preparation lease dropped; a failed
build or discarded preparation leaves its reservation unchanged. Treat
exhausted identities, VA or memory as a soft refusal for warming. On the
publication path, a compute takeover that cannot map its aliases falls
back to the off-lock binder instead of failing the command.

A pair that is borrowed, still settling, or waiting for its previous
scheduler's release to be consumed and the pair parked (a same-VM
successor after a context churn) returned ETIMEDOUT after 5000 polls,
about a second. Bound that wait by the same time instead, and kick the
event worker, which observes the release and parks the pair, on every
poll.

Name each exhausted resource with its site once per module load, and log
one census of render capacity (admission slots, render entries, retired
bindings, scheduler releases by phase, queue IDs, compute queues and
parameter-buffer, counter and free-list identities) when a render
capacity wait first starts and when one first expires, so the remaining
refusals can be attributed on hardware.

Signed-off-by: DjDeveloperr <hey@djdev.me>
A render packet that waits in publication for graph capacity already holds
its admission slot, and the entry of a pair under construction has no
installed pair. When such a wait outlived the two-second scheduler
watchdog, timeout_render() looked the pair up for an unpublished packet,
failed with EIO, and the timeout handler stopped the runtime for every
client. A burst of new clients whose render free lists did not fit the
shared lower GPU VA window did exactly that two seconds after the first
wait began.

Each retry of such a build also populated the full free-list backing,
tens of MiB with ray-query scratch reservations, before the window was
searched, and freed it again after the refusal. Sixteen concurrent
builders reached several GiB of transient kernel allocations.

 - A render or compute watchdog expiry for a packet that is not
   published returns without action: its publisher owns the bounded wait
   and the result. If the collector cannot be allocated, the expiry is
   retried at the next watchdog period instead of stopping the runtime.
 - A refused free-list placement is not attempted again for the same or a
   larger size until free-list backing is freed, apart from one probe per
   100 ms across all callers, so waiting builders allocate nothing.
 - While the window refuses free-list backing, idle retained compute pools
   are given back regardless of the warm cap.
 - The capacity census reports the installed render and compute free-list
   backing in MiB, and the largest render free list.

The host resource tests check the placement gate and the watchdog guard
placement.

Signed-off-by: DjDeveloperr <hey@djdev.me>
Every logical queue leases one of the 58 application contexts of the
TTBAT (5..62) when it is created. A closed queue keeps that lease until
its scheduler release has been consumed and the reclaim worker returns
it; warm retained compute queues and render pairs keep only storage, not
the context. After heavy device churn, contexts of queues closed moments
earlier can therefore still be leased while fewer than 58 queues are open,
and a new queue failed QUEUE_CREATE with ENOSPC without any kernel
message. Userspace creates its kernel queue at the first submission and
reports that as a lost device.

When the context allocation is refused and some leased contexts belong to
closed queues, kick the event worker and retry for a bounded time with
neither the device mutex nor the preparation lease held. Only open queues
then count against the limit. Refusals that remain are reported once as
exhausted logical queues or execution contexts.

Add debugfs asahi-g17/capacity with one line of queue resource counts:
open queues, leased contexts and those held by closing queues, admission
slots, render entries, retired bindings, scheduler releases by phase,
queue IDs, compute queues and pools, and parameter-buffer, counter and
render free-list identities. The render capacity census prints the same
line.

Signed-off-by: DjDeveloperr <hey@djdev.me>
A pair installs the client alias of each kick ring into its VM when the
pair is built, and the VM keeps that alias until it closes: a client that
gets the pair back later reuses it. Under identity pressure, an installed
pair whose identities were never published is reclaimed and its rings are
dropped. That frees the rings' shared lower addresses while the building
client still aliases the old pages there. A later ring placed at the same
address and handed to that client cannot be aliased: its preparation finds
the client's mapping naming other pages and fails with EFAULT, and the
client's render is reported as a lost device. Held GLES clients hit this
every few seconds while fresh clients came and went.

Keep the storage of a dropped ring and reuse it for later rings. An
address then keeps naming the same pages, so every alias a client keeps
stays valid; a reused ring is zeroed and keeps an existing alias. Storage
of a ring that ever wrote an entry is retained and never reused. A new
ring is counted only once its storage exists, reserving room to return it
then, so a dropped ring never allocates and a failed creation leaves no
reservation behind.

Signed-off-by: DjDeveloperr <hey@djdev.me>
…out of the submit ioctl

A publication that meets contention waits in several phases, each with its
own budget: the admission pick (8 s), two render capacity waits (8 s each),
borrowed or settling pairs (8 s), up to 1000 work-state polls, and for
compute 2 s each for binding and ring backpressure, 200 ms for a spared
queue's witness and 2 s for pool growth. One packet could wait about 30 s
before failing. Dependency-free renders are published from the submit ioctl,
which holds the logical queue's mutex, so a client killed while its ioctl
waited stalled for that long. Submission-time compute binding also slept for
a spared queue's witness and failed the submit with EBUSY after 200 ms.

Give each publication from the scheduler's worker a single deadline, four
job timeouts after it starts, shared by every contention wait including the
admission pick, which keeps its own shorter budgets within it. Expiry still
fails the packet with the error of the phase that waited.

The submit ioctl no longer sleeps for contention. An ioctl publication that
would wait (capacity, a borrowed or settling pair, work states, a closed
preparation epoch during recovery, an admission pick) returns its admission
lease and its publication claim before anything became firmware-visible,
and leaves the command to the scheduler like a command with dependencies;
later commands of the queue then wait behind it in the backlog. Optional
sibling warming treats this like any other skipped build. Submission-time
compute binding is a single attempt and leaves binding to publication.

Compute publication now also waits for a free work state, as render does,
instead of failing the packet with EBUSY.

A queue creation waiting for contexts of closing queues returns ERESTARTSYS
when a signal is pending; nothing of the new queue is visible at that point.

A publication that gives up at its deadline names what it waited for, the
source line and the owner's pid in one warning, at most once a second with
a count of the suppressed ones, so a lost device reported for contention can
be attributed from the kernel log alone. For a busy render pair it also
names the pair and the state that kept it busy.

Signed-off-by: DjDeveloperr <hey@djdev.me>
…ir is borrowed

Event passes observe only render pairs that are installed in their entry.
A pair that preparation, growth or an observation has borrowed is skipped,
and nothing requested another pass when it came back: the completion of
work published on that pair waited for an unrelated later firmware event,
or for the one-millisecond rescan when no further event arrived.

Record a completion event that arrives for a borrowed pair and have the
borrower request an event pass when it returns the pair. Growth needs no
request of its own; its work item always queues one when it finishes.

Signed-off-by: DjDeveloperr <hey@djdev.me>
Publication retries with a fixed 200 us sleep when the pair it needs is
busy. The usual cause is a borrow: an event pass observing the pair, or a
preparation or growth of it. Each returns the pair under the device mutex
and then notifies `changed`, typically within microseconds, so a submission
that collided with an observation slept for ~200 us for no reason.

When the pair is borrowed, wait on `changed` (bounded by one tick) under
the device mutex, so the return wakes the submission at once. Other
busy causes keep the 200 us poll. Both stay within the publication's
deadline. A publication from the submit ioctl does not wait at all: it
hands the command to the scheduler, as for any other contention.

Signed-off-by: DjDeveloperr <hey@djdev.me>
Admission prefers an idle render pair of the owner over stacking onto a
busy one, so a stream of renders that each depend on the previous one
alternates between the owner's two pairs. Each render's fragment kick then
waits for the previous fragment kick on the other pair through an explicit
cross-queue dependency, which the firmware resolves only after completing
that kick, and the next fragment work always runs on another queue.

When the command's queue prefix already covers the owner's newest live
render, the sibling pair cannot run it any earlier: its barrier waits for
that render either way. Queue it on that render's pair instead, where the
queue's own order provides the dependency, as long as the pair is healthy
and has room. Independent renders keep preferring an idle pair, so they can
still overlap on two pairs; attachment hazards keep their planned pair.

Signed-off-by: DjDeveloperr <hey@djdev.me>
Every compute command got a fresh page for its firmware start/end words:
a GEM object, its write-combined kernel mapping and a GPU mapping were set
up in the submit ioctl, and torn down again when the command's completion
was dropped. On Neo a lone dispatch spends ~95 us from its fence signal to
the next command's ioctl, against ~17 us for a render, which has no such
per-command allocation.

Keep up to eight retired pages per context, mapped in its VM, and hand them
out again with both words cleared. A page returns only when its completion
is dropped, which follows exact retirement or a kill witness, so the
firmware no longer writes it. Commands beyond the retained count still
allocate and free their page.

Signed-off-by: DjDeveloperr <hey@djdev.me>
…it ioctl

Renders without dependencies are published from the submitting ioctl;
compute commands always went through the scheduler's worker, which costs a
worker wakeup per command: ~30 us from ioctl to doorbell for a lone
dispatch on Neo, against ~12 us for a render.

Publish a compute command from the ioctl under the same conditions as a
render (no dependencies, a shallow queue) when its queue is already bound,
fits it and has room, and no earlier compute command of the owner still
waits in the scheduler. Count scheduled compute commands separately from
renders, so the existing render condition is unchanged.

The ioctl makes one attempt under one device mutex hold and never sleeps.
Binding, takeover, pool growth, replays of recovered commands, a missing
work state, ring backpressure and recovery all return the publication
claim before anything became visible and leave the command to the
scheduler, whose path is unchanged. The ioctl's call chain therefore does
not include queue binding.

Signed-off-by: DjDeveloperr <hey@djdev.me>
…nds deep

A dependency-free command is published from its ioctl only while fewer
than two commands of its engine are still in flight; deeper ones go
through the scheduler's worker, a hop of ~100 us per command on Neo. With
four dispatches in flight a dependency-free compute stream spends that hop
on every command: on Neo it takes 71.7 us per dispatch with the limit of
two and 56.9 us when those publish inline.

Allow inline publication up to four commands deep for compute. Renders
keep the limit of two, so render publication is unchanged. Ordering is
unchanged: inline publication still requires that no earlier command of
the engine waits in the scheduler, and it still makes one attempt that
never binds or sleeps.

Signed-off-by: DjDeveloperr <hey@djdev.me>
@DjDeveloperr

Copy link
Copy Markdown
Collaborator Author

Pushed 13 commits on top of aa5996f: stability fixes found by stress-testing this PR on Neo (A18 Pro, G17P), plus a per-pass latency series.

Stability fixes

  • Order G17 queue owner changes after scheduler releases. A same-VM GL successor rebound its predecessor's render pairs while that context's release was still unconsumed, which crashed the GPU firmware (and Hyprland with it) under GL context churn next to MSAA or Vulkan clients. Before: 3/3 crashes. After: 0 across every run since. Compute queues follow the same rule.
  • Keep queue-identity contention from failing clients. Exhausting the 128 firmware QIDs used to wedge clients or lose devices. Now takeover happens under the publication lock. A forfeit donor never takes a pair back while its displaced graph still holds that slot, which previously stalled the compositor. Each VM keeps one view record per graph per pair, so another queue of the donor's VM can still install on the freed pair.
  • Wait for exhausted render resources and name them; keep those waits from stopping the runtime. Capacity failures are bounded, named ENOSPC (contexts, render capacity, shared lower VA) instead of runtime stops.
  • Wait for contexts of closing queues at queue creation. Hitting the 58-context limit was a silent DEVICE_LOST.
  • Keep render kick-ring storage for the life of the device. Reclaiming an unpublished pair freed a ring address that a client VM still aliased, which lost that client's device with EFAULT (object.rs:142).
  • Bound publication waits by one deadline and keep them out of the submit ioctl. The ioctl never sleeps for contention, and a rate-limited warning names the origin of any EBUSY or ETIMEDOUT.

Per-pass series

  • Rerun the event pass for completions missed while a pair is borrowed, and wait for the borrowed pair to return instead of a fixed 200 µs sleep.
  • Dependent renders queue behind their predecessor's pair.
  • Reuse compute timestamp pages.
  • Publish dependency-free compute from the ioctl, up to 4 deep.

Fresh-boot results (×3): compute dispatch round trip −18%, compute depth 4 −21%, dependent render passes −12%. No row regressed.

Qualification on Neo (final tip 639fef4)

  • Full stress battery: firmware-crash repro; 300 s mixed GL/VK churn; 40 held + 12 fresh GL clients; Vulkan device churn; 20 compute clients; render50; qid48; 100-process context cycles; pool expiry; exit timing; accounting.
  • 0 firmware crashes, 0 runtime stops, 0 EFAULT/EBUSY/ETIMEDOUT, and 0 compositor losses across three held passes plus the full battery.
  • render50 and qid48 are limited only by the named ENOSPC ceilings: 58 execution contexts and render capacity.
  • The real-application mix (Chromium WebGL on GL and Vulkan alongside MSAA render, GL context churn and compute churn) passes with the desktop alive.
  • Every commit builds and passes the G17 host resource tests. Stack: probe path 10,000 B (pre-series 9,936), submit ioctl 19.1 KB.

@greptile-apps

greptile-apps Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[High impact] No newly introduced issue found in this review blocks merging.

Summary

Adds G17 load and memory reporting, improves queue admission under pressure, reduces render storage, and replaces device-resource list scans with tracked membership.

  • G17 render and compute commands reuse eligible idle queues when shared capacity runs short.
  • G17 load reports show firmware activity separately from GPU power state.
  • GPU memory reports separate client handles from driver-owned allocation extents.
  • G17 render targets start smaller, and each render owner drops unused backing storage.
  • Device-resource removal checks tracked ownership instead of searching the device list.

Known limits stated by DjDeveloperr: broader gameplay and GPU-family validation remain incomplete; the reported game rendering defects and naming stall remain under investigation; performance neutrality and strict tail-latency guarantees are not established; added KUnit assertions have not been run. These acknowledged limits were not reported again.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Submit command] --> B{Resources ready?}
    B -->|Yes| C[Publish command]
    B -->|No| D[Leave publication to scheduler]
    D --> E[Wait within publication budget]
    E --> F[Reclaim unused spare or bind retained queue]
    F --> C
    C --> G[Observe retirement or release witness]
    G --> H[Reuse eligible resources]
Loading

Reviews (1) · Last reviewed commit: "drm/asahi: publish G17 compute from the ..." · Reviewed by Greptile

@DjDeveloperr

Copy link
Copy Markdown
Collaborator Author

Closing. Most of this landed through #229 and #244 on integration/aurora. What didn't, the G17 telemetry/utilization export and a few submit-path pieces, can come back as a fresh PR against the current tree if it's still wanted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant