Repository navigation
drm/asahi: improve G17 submission, queue admission and completion - #423
Merged
DjDeveloperr merged 65 commits intoOct 26, 2026
Merged
Conversation
Contributor
|
Heads up while you iterate. This doesn't build with the universal recipe (Fedora asahi base + asahi.config + olddefconfig, LLVM=1, so -Werror is on) on top of 73w (integration/aurora 420a1c8 + #394).
They're only called from Rust, and trace.h is just the TRACE_EVENT header, so nothing in C declares them. An external module build doesn't turn -Werror on, so it passes there. A small trace_api.h with the 8 prototypes, included by trace.c (and usable for bindgen too), would fix it. I'm carrying on with the qualification (one-shot boot on the Neo, full applicable Vulkan CTS with #62, stress, benchmarks) on a local commit that only adds those prototypes to trace.c. It isn't pushed to your branch. Full results will follow in one comment when they're done. |
A compute command on a bound queue went through every admission step on each submission. ensure_compute took the device mutex for a replay and witness pass, shared_compute took a preparation lease and the mutex again, and build_compute did both once more to find its queue already bound. compute_packet then took a third lease and the mutex a fourth time, replaying compute again before publishing. Compute also never published from its submit ioctl, so even a command with nothing to wait for went through the scheduler worker first. Its completions never fed the CPU-capacity detector, so the host loop feeding a compute-bound GPU ran at the lowest CPU performance level. Check what ensure_compute establishes under the one device-mutex section that publishes, and run the admission passes only when that check fails. Publish a dependency-free compute command from its submit ioctl when its queue is bound and nothing has to wait, as renders do. Keep a separate backlog per engine so that a command waiting in one engine's scheduler does not stop the other engine's inline publication. Report compute dispatch envelopes to the detector as render passes are.
A render whose pair was out on loan slept 200 us before trying again. The event worker lends each pair with completions out for an off-lock observation after every completion event. At queue depths of two and more, completions and submissions interleave, so a submission often found its pair on loan for a few microseconds and then slept the full 200 us. Count the device's change notifications. Every pair return already notifies under the device mutex. Wait for the notification under that mutex, unless one arrived since the attempt started, with the same deadline as before.
Object::initialize already clears the complete descriptor before calling its encoder. Fill that zeroed representation directly instead of clearing the same 16 KiB again. Name the encoder write_zeroed to make its precondition explicit. The firmware bytes and validation remain unchanged. The matched module passed the targeted compute/synchronization gate and checked GPU readbacks.
A publisher waiting for a physical compute channel reclaimed idle USC backing even though freeing backing cannot create another QID. The normal warm-pool limit could also evict a retired channel while another runnable publisher was waiting to reuse it. Subsequent handoffs repeatedly allocated and cleared fresh pool backing. Distinguish channel pressure from backing allocation pressure. Track unbound compute admission through publication, cancellation and error, and retain backing for the existing compute channels while that demand remains. The last publisher schedules normal idle reclamation. Allocation-pressure reclamation, the idle warm-pool limit and expiry remain in force. Expose the admission count in the capacity diagnostics. On Neo, the matched 16-client probe drops pool allocations from 4888 to 53 and median submission time from 257 to 109 us. The Asahi KUnit suites pass 67/67; the targeted compute/synchronization gate passes 116 with 18 skips. Full mustpass qualification remains pending.
Fence arrays require every member to signal. During syncobj timeline walks, a pending predecessor already proves that the array is incomplete, yet the lazy signaled callback polls every successor too. Return false immediately at that predecessor, keeping callback accounting and the enabled-signaling path unchanged. Add checks for out-of-order completion, avoiding successor polls, and callback installation after a partial lazy poll. Remove the obsolete signal-on-any description from the constructor documentation.
Compute-only contexts that yield their physical channel still take the device mutex in the inline submit path before falling back to admission. Track an advisory binding hint to skip that attempt. Actual publication retains the locked VM, epoch and binding checks, and scheduler admission ordering is unchanged. Mark contexts with validated rendering before any batch member is queued, so mixed-engine queues retain the original inline path. Validated with 69 Asahi KUnit cases, targeted compute/sync CTS (116 pass, 18 skip), mixed device churn and 16-client concurrent GPU readbacks. Matched 16-process quick submission time improves from 117.27 to 89.55 us; paired render follow-ups remain at control performance.
Only bypass an unbound compute-only inline attempt when multiple publishers are waiting for admission. A lone admission retains the original probe and its latency behavior. Actual publication still uses the existing locked state checks. Matched quick controls retain queue signaling latency (174.1 to 174.5 us) while 16-process submission time improves from 117.3 to 90.7 us. All 20 performance rows complete successfully. Checked mixed/concurrent GPU readbacks, 69 Asahi KUnit cases and targeted compute/sync CTS also pass.
Recognize eligible singleton compute producers through their exported aggregate fences. Order queued consumers after actual producer publication using the existing DRM scheduled fence, then validate the current physical ticket and emit firmware waits. Preserve CPU waits for unsupported dependencies and host outputs, and revalidate recovery epochs before acquiring consumer capacity. Retain source binding loans through consumer retirement, propagate source errors before dependent completion, and release failed submission pins when the last runtime owner retires even if exported fence references survive. Keep bounded member counters in byte atomics so submission state retains its original allocation class. Validated with 78 KUnit tests, the compute/synchronization CTS gate, chained readback, process exits and mixed compute/render pressure. Fixed-core ordinary compute matches the control while the queue-hop path improves. Existing intermittent unrelated-client loss during forced recovery remains unresolved.
… queued Compute retirement removes the producer from its queue's active list under the device mutex, but signals the producer's fence only after the lock is dropped. A consumer published in between found neither an active producer nor a completed one, fell back to the scheduler, and every later command in the chain then waited for its producer's publication through the scheduler. Mark the singleton firmware source retired as its successful retirement is queued (errors are already recorded there), after rechecking its own inputs. sync.signal queue-to-queue (timeline, 2 queues): scheduler fallbacks for chain consumers 57% -> 0.1%. (cherry picked from commit a3be7f63500c76d117000ee1474713e27889aabe)
Early publication was limited to queues with fewer than two outstanding commands, so every deeper queue paid a scheduler worker hop per command and cross-queue chains advanced one scheduler round trip per link. The backend already falls back to the scheduler behind any backlog and on every capacity miss, which preserves submission order, so the depth limit only added latency. sync.signal queue-to-queue (timeline, 2 queues) 94.1 -> 56.2 us/hop together with the preceding change; compute and render depth rows unchanged. (cherry picked from commit 8508510641edcedb5ae1417f560cdff4da646bda)
Render descriptors already come from the retained object pool with a firmware-cached mapping; compute descriptors were mapped uncached, so the firmware read every compute descriptor from memory on each command. Allocate the compute descriptor ring from the same pool (fully cached prefix, retained until processor stop) and give the hardware an uncached canonical low alias for the descriptor address it fetches. compute.light d64 27.4 -> 24.3 us/submit, queue-to-queue hop 58.1 -> 54.8 us; compute/sync CTS gate, cross-queue order/data 10k, dependent reads under pressure and mixed churn pass. (cherry picked from commit c85c050ae2ef77cbeebddb70c6602db477675090)
The compute channel budget was static: compute could only take a QID while every one of the 58 possible execution contexts could still install a render pair, which always left exactly ten compute channels. A client without a channel then made every idle channel of every live client shareable before binding one, so sixteen compute processes evicted each other on nearly every submission (map/unmap and TLB invalidation churn, ComputeReserve refused 52k times in one benchmark pass). Installed QIDs never change engine before processor stop, so a compute channel still permanently narrows render. Size the render reserve by the logical queues that are actually open plus 24 queues of headroom, and refuse a new logical queue only when its dedicated render pair would no longer fit. With every context open this is the previous rule. Rebinding now prefers a channel of a closed owner, then a vacant one to re-arm, then a fresh QID, and only then shares the least recently used idle channel of a single live client. gpubench on Neo (us/submit, matched baseline): compute.light procs d16 92.0 -> 33.4; compute.light queue rows unchanged (d64 27.4 -> 26.5). Stress gate passes. (cherry picked from commit bf7ae5bf04b7d3f14b5ec94ef1f8c1434b4fe494)
A render whose attachment spans overlapped a live render of another logical queue in the same VM waited in admission, on the host, until that render retired. gpubench render.light queues renders every queue into one image, so all queues ran one render at a time: run_job blocked for (N-1) renders and the 2/4/8-queue rows stayed at one serialized step (174/201/217 us/submit). Order such overlaps at publication instead. The device mutex already orders publications; the later render's fragment kick now waits in firmware for the newest published overlapping fragment kick of every other pair, so its tiling kick stays concurrent and no host wait or scheduler stall is involved. Published QIDs keep their kick rings, so these stamps only advance. An admission bound on other pairs of the VM keeps the barrier count within its fixed room. Cross-VM overlaps through shared objects keep the host wait, so one VM's fault can never hold another VM's kick in firmware. render.light queues d2/d4/d8: 173.8/201.3/216.9 -> 81.4/66.9/62.1 us/submit. Other rows unchanged; the mild stress gate (CTS compute/sync, cross-queue order/data, slow-read pressure, mixed churn) passes. (cherry picked from commit 809d95cfdd685b6f94570cd313b1723c7f01f466)
Each compute command writes a queue configuration record, an optional predecessor and completion-wait records, and an announcement; the firmware read all of them through an uncached mapping. Render keeps its equivalent configuration records cached and only its announcements uncached. Move the configuration, predecessor and completion-wait records into a retained pool object with a firmware-cached mapping at the same per-slot offsets; announcements stay uncached. compute.light d1 161.0 -> 152.2 us, d64 24.3 -> 23.1 us/submit, queue-to-queue hop 54.8 -> 51.2 us; compute/sync CTS gate, cross-queue order/data 10k, dependent reads under pressure and mixed churn pass. (cherry picked from commit b3d5b9d6294bb46d63778264205da9d3b750651e)
…found Every event pass with an unobserved compute command walked all 2048 records of the compute completion ring under the device mutex to find the few it needed; that walk was half of the event worker's CPU time. Resume each scan after the last record the previous scan needed and stop as soon as every requested identity has an observation. A scan that finds nothing still wraps the whole ring once, as before. scan_compute leaves the top of the 16-process profile; benchmark rows are unchanged within noise (they are not bound by the scan). Compute/sync CTS gate, cross-queue order/data 10k, dependent reads under pressure and mixed churn pass. (cherry picked from commit 67a49c9c40b31b96a7834c8c185dc20e0bac1fa2)
…mutex Every compute publication took the device mutex, which the event worker also holds for each service pass, and repeated device-wide work on every submit (replay scan, render-prefix scan, binding checks). Give each physical compute queue its own lock (a cell the registry owns; device-mutex holders lock cells one at a time) and move the device-wide state a publication needs into a per-session port that is reached under a preparation lease: compute work-ring slot reservations, QoS accounting under its own spinlock, the recovery generation, the full-effort flag and the doorbell, whose RTKit borrows are serialized by a mailbox lock. The steady-state path (bound, installed, no input sources, no render prefix) publishes from the submit ioctl or run_job with only the queue's lock and the port's small locks; everything else falls back to the device-mutex path, which now reserves work-ring slots through the same port. A command published while the event worker is scanning is not part of that scan's completion snapshot and is left unobserved until the next one. gpubench on Neo (us/submit, vs matched baseline): compute.light d1 166.3 -> 156.2, d64 27.4 -> 25.8, procs d16 33.4 -> 33.0 (firmware per command time now bounds these rows). Stress gate passes. (cherry picked from commit 8fc91b2117de925e362342a842b0d0befbba5312)
Render emits its queue configuration record only for an install, a context or QoS update, or when the configuration differs from the last one the firmware consumed in the current recovery epoch. Compute appended the record to every command, so the firmware reapplied unchanged queue state each time. Use the same rule for compute, caching the last consumed configuration with its epoch and forgetting it when the item ring is reset. compute.light d2 61.9 -> 51.5, d8 25.9 -> 24.3, d64 23.9 -> 22.8 us/submit; queue-to-queue hop 51.1 -> 49.6 us. Compute/sync CTS gate, cross-queue order/data 10k, dependent reads under pressure and mixed churn pass. (cherry picked from commit 0e59b1f92effb58c87f705907959de45abe77623)
Waits for shared capacity sampled a condition and slept 1 ms before checking again. Add availability domains: a waiter samples the domain's generation before its failed check and registers a fence against that sample (or retries when a release intervened); the release that provides the resource bumps the generation and signals only that domain's waiters. The fences can serve as scheduler prepare_job dependencies or be waited on directly. The preparation gate can carry a domain that its reopening, failure and removal signal. Logical-queue creation is the first user: a creator refused for want of a queue slot or an execution context held by a closing queue now waits for the returned slot or the reclaimed context instead of sleeping in 1 ms steps, under the same overall deadline. Tested on Neo as part of compute-wait2 (stress gate passes). (cherry picked from commit 79cf4be7732d549b0f5d1f903bed7effdb80a4fb)
Compute admission and publication retried with sleeps: 1 ms for an unpublished input source, a 20 us..1 ms backoff for a work state, up to 200 x 1 ms for recovery, 1 ms per ring backpressure retry, 200 x 1 ms for a spared drain witness, and 1 ms per growth and channel capacity retry. Use two availability domains for compute admission; the release that provides the resource signals only the waiters of its own domain: - channels: a queue becomes reusable, shareable or re-armable (checked on each transition in the retirement scan and pool service), a scheduler release is consumed, an owner closes, or an unpublished slot and QID are returned; - progress: any compute retirement, finished replays, pool release observation, and the preparation gate reopening after recovery. prepare_job now returns the channel, work-state or recovery fence a compute packet still lacks, so run_job finds its capacity ready, and a prepared packet claims its bound channel so the channel is not shared away before run_job. Retries that remain inside publication block on the same fences; the admission slice only bounds cancellation checks. Each event pass also rechecks the level while waiters exist, which covers capacity that appears through render (QIDs, work states, closed queues). gpubench on Neo (us/submit, matched baseline): unchanged steady state (procs d16 33.1, d64 25.9, d1 162); procs d2 93.2 -> 69.2. With the channel budget forced back to ten channels (diagnostic build), sixteen processes now share them at 51.9 us/submit instead of the old 92.0, with no stall. Stress gate passes, also under the forced shortage. (cherry picked from commit 25fb5800d97db7457f093ff41206047fbd9bddcc)
…record A consumer of a firmware-visible compute source carried a completion-wait record before its descriptor and also named the source in its kick entry. The second dependency flagged the command as entry-signalled, so it also appended a predecessor record, and the firmware resolved the same wait twice. Keep only the completion-wait record, which orders the consumer behind its source by itself. Queue-to-queue hop 49.6 -> 46.0 us. Cross-queue order and data checks (10k iterations each), dependent reads under pressure, the compute/sync CTS gate and mixed churn pass. (cherry picked from commit 58ef65a530b707fb77878e9fdc68d577a27759fc)
Add tracepoints for each command's publication (doorbell rung, with its out fence and the CPU counter), its retirement (with the firmware's GPU start and end timestamps), physical queue binds and event worker pass boundaries. With the counter and the GPU's system counter timestamps on the same events, a trace places firmware dispatch, GPU execution, firmware completion and every host stage of a command on one timeline. The events are disabled by default and cost a static-key test each. (cherry picked from commit 565dae906223948df97a252cc3274e8db6183363)
Compute rang its pipe and priority doorbell (pipe 2 in bits 1:0, class in bits 3:2) only for the first publication of a priority class, then the generic work doorbell for every command. Render rings its own pipe doorbell for every publication, and so does the M5 compute path. Ring the compute pipe doorbell for every publication instead of the generic one. gpubench on Neo (us/submit, on the integration branch): compute.light d2 60.3 -> 53.2, d64 22.8 -> 21.5, d1 163 -> 154..160, procs d16 unchanged (27.2), hop unchanged (48.7). Stress gate passes. (cherry picked from commit 398141cf95c44ac4d0073f3620a99d1235f31309)
Render keeps its queue records in a firmware-cached prefix; the compute queue record, job-list head and item ring were mapped uncached. Allocate the compute graph from the retained object pool with its 32 KiB queue record, job-list head and item ring in a firmware-cached prefix, and move the producer/consumer words to their own uncached page after it, as native does. compute.light d64 22.2 -> 19.7 us/submit, 16-process compute 27.2 -> 25.7, queue-to-queue hop 47.5 -> 42.4 us. Compute/sync CTS gate, cross-queue order/data 10k, dependent reads under pressure and mixed churn pass. (cherry picked from commit 488708add18c39f709d6a9b53736d339428cf5d0)
Render admission and publication still slept or polled: a 20 us..1 ms backoff when every work state of the context was taken, up to 200 x 1 ms while recovery was observed, an unconditional 1 ms sleep for render capacity (through the compute capacity wait), and a 1 ms condvar timeout as the wakeup for a pair on loan, firmware room or a pending teardown. Block on the event that ends each wait instead: - a render availability domain, signalled by every admission-pool return (lease drop, owner close, quarantine change, exit-kill hold release, processor stop), every pair return from a loan, and on each event pass while a render publisher waits, since firmware ring room, consumed releases and teardown steps have no event of their own; - a per-context work-state domain, signalled whenever a work state returns; - the preparation gate: a publisher that observed recovery waits for its closure (close() now wakes the gate's waiters) and then blocks in enter() until recovery reopens it. Render capacity first returns idle backing, then waits on the domain. Waiters sample a domain only after a failed check and retry once before blocking, so steady-state publication takes no extra lock. Deadlines only bound the waits; 10 ms slices only bound cancellation checks. Steady state is unchanged (render d1 135.7, d32 41.4, queues d8 62.6 us/submit); the mild stress gate passes. (cherry picked from commit a3fc054bc52c2bc82350e6073366045a1c3c9489)
A logical render queue owns exactly one physical pair, whose active ring holds RENDER_DEPTH (16) commands, but its scheduler had 32 credits. Jobs 17 to 32 entered run_job and parked there in the admission pool (or in the 1 ms busy wait) until a retirement freed a ring entry, holding the scheduler worker. With the credit limit equal to the pair depth the scheduler itself waits for the credit that the retirement returns, and run_job never blocks on a full pair. render.light d32 41.15 -> 41.26 us/submit (unchanged; the row is firmware bound), all other rows unchanged, mild stress gate passes. (cherry picked from commit 15de925f91cdb2a875e09fc1c6b64af79fa37ae2)
…is busy A render pool whose last render retired sent its final FreeListRelease right away: a control-ring record and a control doorbell for every render of a one-deep queue. The firmware processes each such message serially before it returns to the work rings, so with several queues active the releases stretched every completion burst (one release per render in render.light queues: about 44.6k per run). When another pair still has published work, leave a registration whose references drop to zero lingering instead; the owner's next render reuses it like an overlapping render, with no release and no new generation. A lingering registration ends, and its release is published, when its owner closes or moves to another pool (each event pass checks) or after the 16 ms firmware idle hold, for which the event worker schedules a timed pass. On an otherwise idle device the release is still sent at once: reusing a registration that went idle costs the firmware more than the release does (render d1 147 -> 184 us when every idle registration lingered). render.light queues d2/d4/d8: 80.8/67.3/62.6 -> 76.9/62.4/58.2 us/submit; render d1 148, d32 41.3 (unchanged). Releases per queues-d8 run: 44574 -> 18 (measured with every idle registration lingering). Mild stress gate passes.
A render publication into an owner's installed pair took the device mutex six times from the scheduler (seven from the ioctl): to fetch the admission pool (and, from the ioctl, again to try it), to check the pair's identities, to find the pair already built (constructing a throwaway set of build resources), to look for a sibling pair to warm, to lend the pair out and to publish it. The admission pool lives as long as the firmware session, so the logical queue keeps a reference to it. The identity check also reports whether the owner's pair is installed, idle and backed, in which case the first pass skips graph construction; every retry still goes through it. An owner has exactly one pair (kick.rs asserts it), so there is no sibling to warm. Steady state now takes the device mutex three times from either path. gpubench unchanged within noise (render d1 151.6, d32 42.8, queues d8 58.8 us/submit); the mild stress gate passes.
Every render publication held the device mutex, which the event worker also needs for each pass, across the whole publication: dependency and hazard collection, kick and descriptor commits, QoS accounting, both work-ring records and producers, and the doorbell. In the steady state (the owner's graph is installed and registered at its priority, last ran this owner's scheduler, needs no growth, has nothing to replay or settle, and the command has no compute prefix) the device mutex now only lends the pair out, together with its free-list preparation and a parameter-buffer reference, and takes it back. Preparation and publication in between take the port: a render publication lock that every render publication holds, with or without the device mutex (the tiling and fragment work rings keep one host producer at a time, and attachment overlaps keep their publication order), QoS accounting, the completion stamps, the doorbell and the full-effort witness. The port gains views of the render work rings and of the stamp page. The free-list lease leaves preparation, or is rolled back with the parameter-buffer reference, when the pair returns; while it is lent the event worker skips the pair, so no retirement can touch its commands or references. A recovery that starts during the publication fails it with EIO, as on the compute path. Anything else, including a first publication, growth and owner changes, still publishes under the device mutex. During gpubench render rows 306k of 320k publications took the new path. render.light d2/d8/d32 68.9/45.2/42.8 -> 65.9/43.5/41.1 us/submit, d1 150.2 (151.6), queues d8 58.7 (58.8); the rows stay firmware bound. Mild stress gate passes.
Kernels built with CONFIG_WERROR fail on the G17 changes. trace.c defines the eight tracepoint entry points that only Rust calls, so nothing in C declares them (-Wmissing-prototypes). The Rust side also leaves code without users: the compute host's generic doorbell, a recovery poll constant, the old capacity-wait helper and an unused import, and, without G16, the CPU timestamp mapping that only the G16 timestamp path uses. Declare the entry points in trace_api.h, included by trace.c, remove the unused items, and build the CPU timestamp mapping only with G16. Reported-by: RyanTheTide
The scheduler fails a published job two seconds after it started unless the job is a ray-tracing or acceleration-structure build. Under load that timer measures how the GPU is shared rather than whether the job makes progress. Beside four other clients, a 1.5 s compute dispatch took up to 5 s, and lower-priority work waited behind higher-priority work for seconds, while the GPU kept retiring other commands at least every 1.2 s. Those jobs failed with a lost device although nothing had hung. Renew a published job's deadline while any command on the GPU retired in the last five seconds, at most fourteen times (thirty seconds in all). A GPU that retires nothing for five seconds has stopped, and its jobs fail as before.
A client whose work failed closes its queues, and 250 ms later the driver asks the firmware to kill the closed context's remaining commands. While other contexts' commands are running, that kill makes the firmware restart the GPU and fail everything in flight, and a few repeats crash it. On an otherwise idle GPU the same kill is handled cleanly. Publish the kill only while no other context has a render pair or compute queue in flight, or once the GPU has retired nothing for five seconds, the watchdog's liveness window. Until then the closed context's queues stay quarantined and retained, and its commands retire like any others once the GPU reaches them, as commands with ray traversal scratch already do. Deferred kills are rechecked on every service pass.
…once After a firmware restart, the firmware retires every render group it had started on a pair without running it: the groups the recovery table lists, and those whose tiler had run while their fragment stage was still pending. Groups it had not started run as usual. The host republishes dropped groups only when every retained group was dropped, so that none runs after a live successor. Once a later group has run, the dropped ones never qualify, and in-order retirement cannot pass the unexecuted front either. The pair then waited for the render watchdog: about 2 s later before deadline renewal, up to 30 s with it while other work keeps retiring. Its owner then saw ETIMEDOUT and a poisoned VM, even when the client was innocent and none of its work had started. In a stress run with forced recoveries, 10 of 11 recoveries left such a pair. Replay was also wrong for a dropped group that a compute command waited on: the wait ends when firmware retires the group, so the consumer may read output the group never wrote while the replay runs it afterwards. A client in the same stress run read back the previous iteration's image this way without any error. When firmware has retired every group of the pair but a dropped group cannot run again in order (a later group already ran, the dropped group had started, or a compute command waited on it), fail the whole set at once, after the same double observation as a replay. The groups that ran fail too: they consumed output the dropped ones never wrote. The work is innocent, so it fails with ENODATA, which its owner sees, without poisoning the VM. Guilty and blamed sets are left to the timeout as before, and a group that ran ahead of every dropped one is still retired in order.
A compute doorbell is withheld while two or more unretired commands of its queue precede it, and any departure from ordinary retirement is meant to ring it at once. A recovery that touches no command of the queue left it withheld until those predecessors retired. Once the firmware has resumed from a restart, ring every withheld doorbell, one per priority class, before any replay is published.
Feature bit 17 is also used for render resources by the M5 userspace line, whose features take bits 17 to 28. Advertise cumulative signals as bit 29, which no line uses, so one Mesa can serve both kernels. A kernel and Mesa that disagree on the bit fall back to ordinary per-point signals, so either mix still works.
Contributor
Author
|
Thanks RyanTheTide. Status on each point:
|
…constructor The test that grows compute QIDs with free identities still called Registry::new() with the clusters and descriptor flags only, from before the constructor took the deflake setting and the firmware variant, so a KUnit build of the driver failed. Construct the registry as the other tests do, for one cluster on the 26A428 firmware.
…e budget The test that a firmware QID floor admits only the render pairs it leaves reserved every compute channel before opening any logical queue. Compute QIDs are now sized by the open logical queues, so on 25G83's floor that first reservation was refused and the test panicked. Open the 42 logical queues the floor allows without the up-front compute reservations, and check as before that the next one is refused when it is created.
…ot the clock The test that gate churn cannot renew a compute admission budget looped until the budget ran out and then required the whole loop to have taken less than 500 ms. The loop does not sleep, so that bound only measured how long the test thread was descheduled, and a KUnit run under KVM on a shared CI host exceeded it. Check the property instead: every admission granted came from an attempt that started before the deadline, and the first attempt after it is refused with EBUSY.
Contributor
Author
|
Broad applicable CTS on the final pair, as requested: this head's module (18b408b) with Mesa aurora-silicon/mesa#62 (309c3feea0e). Run on Neo, 7.3, 6 workers.
The 4 QualityWarnings are all in The two crashes were integration/aurora moved since this was tested, and the PR now conflicts in
The merged module (de1be1b) was checked against the tested one, both with Mesa #62:
|
DjDeveloperr
marked this pull request as ready for review
October 26, 2026 04:24
This was referenced Oct 26, 2026
Contributor
Author
|
A robustness issue in this driver under sustained ray-tracing load, reproduced on Neo with the merged module (integration 9cbdd07) and Mesa #62. This is relevant to the post-merge qualification on the distro bump PRs. It took a 15-minute run of the multi-client stress harness plus looped
A related wedge on an older build: a failed queue whose client was still open was never evicted, so later compute deferred behind it and timed out. Fix in progress: first decode event 7, which is probably the firmware reporting an aborted command. If it identifies the command, the driver can retire it with an error and drain its queue. If it doesn't, the fallback is a bounded eviction of quarantined queues the firmware hasn't drained, open clients included. The pass bar is the same harness with no capacity leak and no EBUSY wave. I'll link the PR here. |
Contributor
Author
|
Update on the robustness issue above: part of the fix is up as #496.
For the Neo sustained-stress part of the post-merge qualification: a long multi-client run that mixes RT stress with other clients can hit this, on the merged driver and with #496. |
RyanTheTide
added a commit
that referenced
this pull request
Oct 26, 2026
List every #423 commit separately and enumerate the related integration and aurora-wip history, including original/rebased aliases and Dj's contributions. Record current reachability, firmware/data gates, superseded policies, source evidence and a J616s measurement for each. Separate public measured Neo results from stack results and estimates. Document the applicable CLPC GPU-demand and PMP-voter ports and the protocol-specific paths that cannot be enabled by copying Neo data. Keep unmerged measured follow-ups outside the pinned-head claims. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
erikwb
pushed a commit
to erikwb/linux
that referenced
this pull request
Oct 28, 2026
Trace calibration, firmware admission, queueing, power, recovery and memory paths against the shared Neo backend. Separate changes already present through aurora-silicon#463 and aurora-silicon#423 from firmware ABI restrictions and missing hardware measurements. Record the public-source boundary for M5 Pro. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
G17 submission spends substantial time waiting for host scheduling, holding the device mutex, rebuilding firmware records and walking growing fence histories. This series moves steady-state work onto the queue that owns it and retains the existing recovery and fallback paths.
The series covers the finalized GPU work through
30f85560fd28, rebased ontointegration/auroraatd3cf19be28d5(includes the fabric DVFS fix, #440). It remains a draft while the remaining benchmark gaps are addressed and the final kernel/Mesa pair completes sustained stress and full applicable Vulkan CTS qualification.Changes
Recovery under overload (added 2026-10-25)
Qualification at six concurrent CTS workers found that overload could take down the whole GPU. Six concurrent
dEQP-VK.ray_query.stressclients reliably reproduce it. Three commits fix it:Result on the same load: the GPU stays online, the desktop survives and no client gets wrong data. The overloaded clients see a lost device, as they would on macOS, whose firmware timeouts are the same.
On this head plus the fabric DVFS fix (#440) and Mesa #62:
Userspace and compatibility
Companion: Mesa #62, including the cumulative queue-signal change originally qualified as
e3039f098426. Existing userspace keeps its current output path, and the companion retains its fallback on kernels without the feature. Cumulative completion does not add execution dependencies between otherwise independent GPU submissions.Integration notes
The target branch has gained additional G17 variants and recovery fixes since the original qualification:
f602f7151dd8is omitted because the target already supplies that behavior through645b4e55cf16and subsequent recovery fixes. Guilty-work guards, spared quarantine and the render scan before timeout attribution are retained.This PR excludes the mailbox hard-IRQ experiment, timestamp experiment and unfinished Mesa launch work.
Native-wait record bounds
The dynamic QID budget also raises the native-wait bound from 10 to 128. Compute wait records begin at offset
0x900in a0x1000-byte slot, with0x60bytes per wait. A 19th wait crosses into the next slot. The original 10-wait bound stayed within that allocation; the larger bound in this series exposes the problem.The record-bounds commit sizes the cached slots to
0x4000and validates each write against its own slot before forming the allocation offset. Announcement slots remain0x1000. This adds 384 KiB per physical compute queue and does not reduce queue concurrency or the input bound.The production offset helper and its test bodies pass on the host for all 128 waits in all 32 slots, including invalid-window and overflow cases; an old-size negative control fails. First-command native waits retain the existing
EAGAINfallback before record selection, so they do not use the inline record region. Extracted guard and window checks cover that fallback and the inline boundary as well. Real Vulkan tests now pass with 19 and 20 simultaneous native inputs, three rounds each: the native-input counter advances by exactly 19/20, every producer result and the consumer sum match CPU checks, adjacent result allocations remain intact, and no GPU fault is reported. These tests use a test-only 21-queue Mesa override; production queue limits are unchanged. Host coverage spans all 128 supported waits; hardware boundary coverage is 19/20, not an arbitrary-128-input claim.Follow-up CPU efficiency changes
The follow-up completion lookup indexes requested QID/kick identities before reading record payloads. It keeps the complete scan and retirement fallbacks, including missing-stamp and recovery handling. Five source test cases and a 10,000-record differential check against the original collector preserve exact matching, collision handling and error observations. In matched workload profiles, the compute scanner's self samples fell from 4.87% to 2.24% at depth 64 and from 3.16% to 1.93% in the queue-hop workload.
The validation cache retains page ranges that the VM already checked, including their permissions and mapping epoch. It keeps 128 persistent intervals while the transient parser remains bounded at 48 requests. This avoids the measured depth-64 working-set thrash without changing mapping lifetime or GPU concurrency. Repeated normal runs reduced compute depth-64 submit API elapsed time from 10.06 to 7.46 us and render depth-32 submit API elapsed time from 12.29 to 8.16 us. These benchmark metrics use a monotonic wall clock around the API call, so they include scheduling and lock waits; they are not per-thread CPU-clock measurements. Page/permission/epoch/overflow tests and a rotating 64-buffer test pass.
The scanner profile establishes a CPU-cost reduction; the submission benchmark establishes lower API elapsed time. The combined candidate measured 19.44 us at compute depth 64, 41.48 us at render depth 32, and 42.12 us per queue hop. They do not establish parity with the recorded macOS throughput targets.
The next frontier change checks the signalled flag of driver-private completion fences before taking a status lock. These fences are explicitly signalled and have no polling callback. Pending observations can only retain a frontier member conservatively; terminal statuses still use the normal locked API, and external inputs keep their generic fence handling. Seven depth-64 repetitions reduced median submit API elapsed time from 7.52 to 5.79 us. Matched profiles reduced frontier-scan samples from 1.83% to 0.25% of the machine. Throughput stayed near 19.5 us per command, so this is a CPU-efficiency improvement rather than a throughput-parity claim.
Validation
Original finalized stack on Neo:
Rebased candidate:
7.3.0-1-perf-ARCHkernel (coreaa79324715fb): passed; latest module build IDd1b39857cb8e51b1f5249d3f1c11af3fa3205f43, source5b25f951ab88. This is not a full boot of the target integration kernel.git diff --check: passed.3049029c1ebdwith Mesa051a: all 19 benchmark rows and the existing stress suite passed. A separate 35-second, 18-child pressure run explicitly used the selected ICD. Published software-fence ordering, deadline forwarding, 256-completion backlog and error propagation passed; 20 simultaneous native inputs passed three rounds of result and neighboring-allocation checks. No GPU faults were reported. These bounded checks do not replace the sustained stress and full applicable Vulkan CTS still required, alongside the remaining performance targets.5b25f951ab88with Mesa051a: all 19 benchmark rows, corrected candidate-bound short stress, software-fence ordering/deadline/backlog/injected-error tests and 20-source native fan-in passed. The small compute/synchronization CTS gate recorded 116 Pass and 18 Skip; this is not the full qualification suite. No new GPU faults were reported.Complete benchmark scope and required order
The identified full standard and RT run used source
5b25f951ab88, moduled1b39857cb8e51b1f5249d3f1c11af3fa3205f43, retained coreaa79324715fb,and Mesa
562393785267. It produced all 209 recorded configurations, with a livedesktop and zero new GPU faults. Against the exact recorded macOS run, 55 of
207 comparable primary medians strictly win, 143 are slower and 9 tie. The two
macOS pipeline skips remain explicitly noncomparable. The 19-row development
matrix and nine representative rows do not establish this full gate. These
are measured medians, not statistical significance or sustained qualification.
All CTS work, including preparation and small slices, is stopped until Linux
strictly exceeds macOS in every comparable benchmark. Then the final pair must
pass proper sustained stress, full applicable Vulkan CTS, and only then merge.
Both companion PRs remain draft and unmerged. Earlier CTS results below are
historical; they do not satisfy the final gate. Experimental scheduler and
query changes are not included in these published heads.
The companion Mesa PR now includes clean commit
d6d814afdb8, fixing missing relocation helpers and inactive/empty TLAS handling. Its exact rebuilt artifact passes all 17 TLAS phases / 136 ray-transform comparisons, all 68 BLAS/AABB phases / 544 ray checks, and 16 concurrent RT clients (680 phases / 5,440 comparisons), plus query/data/order and bounded compute/render pressure gates on this qualified kernel module. These fix correctness and do not establish an RT throughput gain. The complete 209-row record above belongs to the preceding Mesa562artifact; no full performance record is claimed for the new companion head.The companion now also includes
9bbce5b1798and clean head7c68adfc257, enabling existing uniform DMA for ordinary internal compute pipelines and using existing linear PBE writes for eligible G17P buffer fills of at least 64 MiB. Exact rebuilt library SHA2560a2c688bbc98650e6c7771602599abdb1aea8bff91ddfa35fa55a3bc574409ee, build ID53a9fb9740865d0f8426fc5f40d38a2b67f237d0, passed 24 query/error cases, 49 core rows, 28 fill/copy rows, 15 GPU boundary cases, eight concurrent fill clients (120 cases / 2,013,511,680 checked words), order/data checks and bounded mixed pressure on this unchanged kernel. Large-fill times improve about 2.9–4.0%; representative core results remain essentially flat. Linux still trails the recorded macOS fill results by about 28–29%. These bounded checks and partial gains do not satisfy the all-benchmark, sustained-stress or CTS gates. All generated precompiled helper C is unchanged from the preceding companion head; the earlier AS fixtures remain prior-head evidence.Completion scan reads only new records (2c0abe2)
While any command was in flight, the compute completion scan read all 2048 ring slots on every pass (about 3 µs before any fence was signalled), and its cursor never advanced while a queue stayed busy. It now folds only the records appended since the last pass, with a one-time full read as a fallback for a command whose stamp already covers it.
Same boot on Neo, patched cpufreq (#443), app on cpu5, against the previous scan (µs):
Gate: order/data 10k, 35 s pressure and mixed churn passed, with no new faults. Targeted CTS (synchronization2 basic + timeline_semaphore, api.command_buffers, compute.pipeline.basic) under a 6 GB cap gave 3,023 pass, 84 not supported and 0 failures.
Render submit path (39bdd5b..66ef051, 8 commits)
Render ioctl entry → doorbell goes from 13.67 to 11.23 µs (same-boot phase timers); compute goes from 8.90 to 8.36. The changes:
Same boot on Neo, three rounds alternating per row (µs per submit; app pinned to cpu5 unless noted):
Render d32, render queues and hop are unchanged; they are bounded by firmware completion work.
Gate passed. Targeted CTS (synchronization2 basic + timeline_semaphore, api.command_buffers, renderpass1/2 suballocation, dynamic_rendering primary basic/random): 5368 cases, 4687 pass, 681 not supported, 0 fail.
Drop redundant render queue configuration records (1aec5be)
A render published onto a drained pair forced a queue configuration record onto both item rings, even when the queue already held exactly the value being seeded. The firmware processes each record before starting the kick, at ~1.5 µs each. The record is now omitted when it would only reload the held completion. Recovery, replay and graph rebuilds still seed.
Same boot, alternated per row, 3 rounds (µs per submit):
Gate passed. Targeted CTS: 3027 pass, 84 not supported, 0 fail.
One CPU mapping per timestamp buffer (d0df244)
Every timestamped command mapped its destination buffer into the kernel per timestamp (taking the reservation lock), then queued a cleanup work item at completion. A timestamp buffer is now mapped once, when it is bound; its last-reference release is deferred once per buffer.
Same-boot phase timers, render d1 on cpu5:
End-to-end rows are unchanged within noise. Gate passed. Targeted CTS: 23,638 cases including all 18,270
query_pool.*, 22,847 pass, 791 not supported, 0 fail.Withhold compute doorbells behind unretired commands (3ff8c49, b1f5dc6, d38cce2)
A compute doorbell makes the firmware read its priority class's whole work ring. A doorbell it handles while finishing a command delays that command's completion stamp from about 12 µs to 24 µs, and with it every waiter. A publication with two or more unretired commands of its queue ahead cannot start soon. Its doorbell is now withheld until at most one command precedes it, and is dropped if a doorbell of the same class rung meanwhile already announced its slot. Any departure from ordinary retirement rings it at once: timeout, kill, quarantine, replay, recovery or exit.
Cross-queue hop (µs per hop):
Gate passed. 10k order/data iterations passed. Targeted CTS: 3555 pass, 0 fail.
Multi-client stress (added 2026-10-25)
An external five-lane stress run on this branch reported more GPU timeouts than on the base. A five-lane harness reproduces it on Neo: global-priority preemption CTS, process churn, render fill and geometry, and a client looping 1.5 s compute dispatches.
Cause:
Enabling firmware time-slicing (context-switch timer on compute, or on all engines) was tested and made things worse; it stays off.
Fix:
CONFIG_WERRORfailed: the tracepoint entry points had no C prototypes, and the series left unused Rust items. Reported by RyanTheTide.Same harness, 20.5 minutes active:
Ten gpubench rows of 444 missed their 300 ms measurement windows while sharing the GPU with the 1.5 s dispatches. No client lost its device.
Gate passed on the fixed build: cross-queue data, 29,476 mixed stress iterations with 0 bad, pressure cycle; no driver messages in the kernel log. Targeted CTS (synchronization2 basic + timeline_semaphore, api.command_buffers, renderpass1/2 suballocation and subpass dependencies, dynamic_rendering primary basic/random): 5368 cases, 4687 pass, 681 not supported, 0 fail, 0 crash. Each new commit builds with
-Werror.Full gpubench record on afbfbc0 with Mesa #62, built from this exact head: no driver errors, and gate-eligible provenance. A four-boot A/B on the same kernel core compared d0df244, the doorbell series alone, afbfbc0, and d0df244 again:
The stress fix costs nothing measurable. The remaining rows are unchanged.
Recovery: dropped render groups (added 2026-10-25)
Forced GPU recoveries under the same stress, with an oracle lane checking data, found two pre-existing render bugs:
31279ab: when the firmware has retired the whole set but a dropped group can't run again in order, the set fails at once with ENODATA, visible to its owner without poisoning the VM. That covers a later group that already ran, a group that had started, or a group a compute command waited on. Pure-render sets still replay, so innocent render clients survive.
7012a59: after recovery, any still-withheld compute doorbell is rung before replays publish.
With forced recoveries (14):
Production stress (600 s, no fault injection) was all-zero errors, with 45 CTS iterations clean and the oracle 38/38. Gate PASS; targeted CTS (query_pool included) 23,638 cases, 0 fail. A four-boot A/B against afbfbc0 is flat on every compute, render, hop and submit row.
Cumulative signals move to feature bit 29 (8e257af)
Bit 17 is also used by the M5 userspace line, whose features take bits 17–28. Cumulative signals move to bit 29 in lockstep with Mesa #62. On this kernel a kprobe counted 0 cumulative outputs with the old Mesa (it falls back to per-point signals) and 74,260 with the new one. Gate PASS, targeted CTS 4687 pass / 681 not supported / 0 fail, and an rb5-vs-rb6 A/B flat on every row.