[DjDeveloperr]

Stacked on #244. Mirrors aurora-silicon#555 (fa91d94), a G17 render fix found on the J700 Omarchy install. I put it in its own PR rather than on #244, which is already approved. It can be folded in at merge time if you prefer.

Problem

On the J700, about every other boot under a 6-worker Vulkan CTS load, one client's render fails with ETIMEDOUT about 30 s after it was submitted, and the client sees a lost device:

asahi: GPU work of a client failed (ETIMEDOUT, reported at g17/runtime.rs:...); the client sees a lost device

It reproduces with older qualified G17 modules too, so it isn't a regression. CTS sessions that start right after boot hit it more often, and so does a gallium desktop.

This also explains the GPU failure on the first boot of a fresh J700 Omarchy install: a render ETIMEDOUT 3.4 s after login. On the released 937b727 the timeout stopped the whole G17 runtime, because that tree lacks #229's timeout fixes. With #229 and #244 it fails only the client, after the watchdog; with this change the group retires.

Cause

A pair-state dump at the failure showed the firmware did finish the group:

• both rings were consumed (done == read == write);
• the pass record holds its tiler start and fragment end;
• on one pair, 13 later groups had executed as well.

Both queues' completion stamps, though, stop exactly one kick short. For example, slot 35's front ordinal 406 needs stamps covering kick 407, and both read 406 << 8. No stamp store, completion notification or later event ever moves them.

next_render_observation only observes a pair once the stamps cover its front, so the pair is never observed again. The client then waits through the watchdog and its 14 renewals before failing. Compute already tolerates the same firmware behavior for its records (STAMP_FALLBACK_NS, 50 ms).

Change

Render gets the same bound:

• The observation scan, and the scan before a render watchdog attribution, latch the time when a front group is executed and consumed while the stamps lag it.
• A latched pair stays in the 1 ms rescan without a notification. After 50 ms it is observed and the group retires on its pass record (retirement::poll_stampless).
• When the watchdog of such a front group itself expires, the group retires on its record instead of failing. Failing would also give up on the stamps.
• The first and every 64th such retirement are logged with a lifetime count.

The cherry-pick had one context conflict in queue/render.rs. aurora has a Pair::report_recovery() diagnostic next to the new methods, and this tree doesn't, so they follow qos_owners() here. The commit notes it.

Testing

On aurora (J700 on the Omarchy install, integration/aurora kernel with this change as a module, Mesa aurora-wip), each boot starts 6-worker targeted Vulkan CTS (synchronization2 basic/timeline, command buffers, render passes) right after login:

asahi module	boots	boots with a client ETIMEDOUT
integration/aurora 4460ba6, and diagnostic builds of it	15	9
the earlier qualified G17 module (b3a06ac), on two kernel cores	2	2
this change	6	0

With the change, every boot logged exactly one retirement on the pass record, 33-47 s after boot during the first heavy load:

asahi: G17: lifetime total 1 render groups retired on their pass record after 50 ms without completion stamps (latest: slot 11 ordinal 1544)

So the stall happens on most boots. Without the change, it only becomes a failure when a client waits on the stalled group. On the same stack: regression gate PASS, targeted CTS 4687 pass / 681 not supported / 0 fail, desktop alive, no new kernel errors.

On this branch:

• omacom KUnit: 1721 tests, 1718 pass, 3 skip. That includes three new asahi_g17_render_retirement cases that model the stalled pair as read on hardware.
• Kernel CI (manual dispatch): see the comment below.

[DjDeveloperr]

Kernel CI (manual dispatch) on 9b4e9dd passed: https://github.com/omacom/linux-aurora/actions/runs/40705851208 (plan, self-test, KUnit, package, boot, host tests). As for #244, compile, commits and device trees are skipped on a dispatch.

[DjDeveloperr]

Retargeted to integration/aurora: its base, #244, has merged. The diff now shows only this PR's commit.

[chriskearney]

Source-only question at 580d40d634410b22256e1aa8028434c9f0150b9b: what establishes cache-maintenance completion on the stampless retirement path?

poll_stampless() returns Complete from consumed queue windows and nonzero pass timestamps after the 50 ms latch. The normal path additionally requires visible(stamps) and explicitly says the pre-stamp snapshot cannot authorize export (lines 268–283); the new documentation likewise places the stamps after cache maintenance.

Does firmware finish that maintenance before advancing both queues’ done cursors to the window target? If so, documenting that ordering would explain why the fallback evidence is sufficient. Otherwise the 50 ms delay appears to be the remaining visibility assumption. Output::publish() exports all four timestamp destinations and signals the fence, so this matters for fence-then-CPU readback as well as render-carried occlusion availability.

The existing compute fallback and the reported successful runs are useful supporting evidence. I have not reproduced a failure or run this head on hardware; elapsed time alone is not evidence of a regression. The new KUnit cases model stalled completion stamps, but do not establish the cache-visibility ordering above.

[DjDeveloperr]

Good question, and you've read it right. Nothing on the stampless path proves the end-of-render maintenance has finished. poll_stampless() retires on three conditions: the pass record shows the group executed, both queues' done cursors reached the window target, and 50 ms have passed since that was first seen. The normal path waits for the stamps, which the firmware stores after the maintenance. We have no documentation or source showing whether the firmware finishes the maintenance before it advances the done cursors. So the 50 ms bound is the remaining visibility assumption, as you say. It covers both the fence and the timestamps that Output::publish() exports.

What we do have:

• In every stall we dumped, the pass record held the tiler start and fragment end, both rings were consumed, and later groups on the same pair had already executed. The stall is the stamp store going missing, not a group still running.
• Compute has used the same 50 ms fallback for its records.
• On the J700 qualification runs (6 boots, one stampless retirement each, during the first heavy CTS load), targeted CTS had 0 failures. That includes the query-pool and copy cases that read results back on the CPU after the fence.

That's evidence of no problem, not proof of the ordering. I'll update the comments on STAMP_FALLBACK_NS and poll_stampless() (comment-only, aurora-silicon/linux first, then here) to say exactly this: the fallback relies on the bound, not on a firmware ordering guarantee. If we can establish the ordering on hardware, I'll replace that wording with the evidence.

On the Mesa side, render-carried occlusion availability reads the same slots. A stampless retirement there is covered by the same bound, which also means it isn't independently proven.

[DjDeveloperr]

Follow-up: a new stress test (128 compute and 64 render submissions in flight on one queue) produced one stampless render retirement ("retired on their pass record after 50 ms without completion stamps") about 54 ms before a GPU firmware crash. That's a single sample and not established as the cause, but it's exactly the window your question is about, so please treat this PR as on hold. We're tracing it with the full G17 tracepoints, will report what we find, and will change the mechanism if it's implicated.

[DjDeveloperr]

@chriskearney follow-up on the earlier sample (a stampless render retire about 54 ms before a firmware crash under gpubench-stress queue.depth, with 128 compute and 64 render submissions in flight on one queue):

• Cause: a GPU-side hang, triggered by our driver admitting a context's ioctls to work states out of submission order across engines. Native Metal on the same firmware schedules in commit order (36 per queue) and survives the same mix.
• Fix: drm/asahi: g17: admit a context's ioctls to work states in submission order aurora-silicon/linux#571, mirrored as drm/asahi: g17: admit a context's ioctls to work states in submission order (stacked on #275) #276 (stacked on drm/asahi: g17: keep a work-state node for the other engine #275). It admits in submission order, with a bounded run-ahead of 8. With it the reproducer passes 4 × 30 s and a 60 s run (1.99M checked results) with no kernel log lines, so the stampless fallback no longer fires there. The performance A/B is neutral or better. Evidence: queue.depth: G17P stalls under deep mixed render+compute on one queue (driver trigger not yet found) aurora-silicon/gpubench#5.
• Separately: the 50 ms fallback still fires during a CTS synchronization pass (about 9.7–9.9k compute retirements per pass, on stock and fixed builds alike), with 0 CTS failures. The visibility assumption documented in this PR stands as written. Why sync CTS hits the fallback that often is a separate latency item we're looking into.

[DjDeveloperr]

@chriskearney a direct answer to your original question. It supersedes the last point of my previous comment. Your question found a real early-release bug, thank you.

• What we measured: in one CTS synchronization pass, all 9792 of 9792 pass-record fallback retirements retired a member of an ioctl whose work-state node was still running on the firmware. Its real stamp landed 4–300 ms later, with a median of 28 ms. So the fallback did release the member fence, the vm_job pin and the render lease before the post-maintenance stamp.
• Fix: drm/asahi: g17: count the stamp fallback from the drain of the command's node aurora-silicon/linux#572 (merged), mirrored as drm/asahi: g17: count the stamp fallback from the drain of the command's node (stacked on #264) #277, stacked on this PR's branch. The 50 ms bound now counts from the drain of the command's node. A drained node whose stamps still lag for 50 ms keeps the old rescue (the original render-stamp stall), and the watchdog last resort is unchanged. arm64: dts: apple: t8112-j493: Enable SIO for DP audio AsahiLinux/linux#572's description lists what retirement releases.
• Qualification on a fresh boot: 0 stampless retirements, compute or render, across the gate, the targeted CTS and the full synchronization and synchronization2 groups. The stock build had about 18.8k there. Performance is within noise, and KUnit passes 130/130 on drm/asahi: g17: count the stamp fallback from the drain of the command's node (stacked on #264) #277's tree.

[scottjones]

We read this at 72b737b against integration/aurora, and ran it only with #278, #275, #276 and #277 on top.

For the stall it targets, the logic does what it says. A front group that has executed with lagging stamps gets latched (g17/queue/render.rs:1044) and retires on its record after 50 ms.

The problem is what that 50 ms measures, and #277's numbers answer it. The firmware stores an ioctl's stamps only once its node drains, so a lagging stamp usually isn't late. In the synchronization2 pass, all 6 render fallbacks and all 9792 compute fallbacks had a node still running, and compute stamps landed up to 299 ms afterwards. At this head a retirement like that:

• completes the free-list and QoS accounting (g17/queue/render.rs:1342);
• releases the vm_job pin and the render lease, before the fence (g17/completion.rs:641, 643);
• runs memory.retire() (g17/queue/render.rs:1358), which can free retired target backing (g17/buffer/target.rs:337).

The node and out-fence are still held by any member that hasn't retired, but the backing and accounting aren't. We haven't seen a failure from this, and we don't want this head in integration/aurora without the gate.

So we'd ask for:

• #277 folded in, or merged directly after with no release between them.
• A clear statement on whether the hold from your earlier comment is lifted. The title and body still say "stacked on #244", which has merged.
• Landing after #278 and #276. A withheld doorbell can keep a node from draining, which is when this fallback fires.

Smaller, not blocking: expire_stamp_wait() skips the bound. note_stamp_wait() can latch at that moment, and g17/queue/render.rs:1085 sets the latch to 0, so a group can retire with no wait at all. The comment at g17/queue/render/retirement.rs:171 says this path relies on the bound.

The mirror is faithful: 9b4e9dd matches fa91d94 apart from where the new methods sit, and 72b737b is byte-identical to 4176f53. Scope is g17 only, but all the evidence is J700.

Our runs don't test this head alone. With #277 on top, on a Neo with our clang 22 build: CTS synchronization and synchronization2 0 fail, then queue.depth 5/5 and sparse.churn 3/3 with omacom/mesa#20, and no stampless retirement. #278, #275 and #276 without this passed those same stress runs with no stampless retirement, so queue.depth doesn't need this PR and this needn't hold them. That's one job per kernel on one Neo. #277 folded in or merged directly after is what would turn this into an approval, along with what we ask for on #277.