Skip to content

drm/asahi: g17: keep innocent clients alive across another client's recovery - #803

Merged
RyanTheTide merged 2 commits into
aurora-silicon:aurora-wipfrom
jgoyvaerts:m4pro/g17-innocent-clients-recovery-wip
Nov 2, 2026
Merged

RyanTheTide merged 2 commits into
aurora-silicon:aurora-wipfrom
jgoyvaerts:m4pro/g17-innocent-clients-recovery-wip

Conversation

@jgoyvaerts

Copy link
Copy Markdown
Contributor

Replaces #433. That PR no longer applied: the G17 rework (#423) replaced the submission path it changed, and its hold on #438 went away when #438 closed. This is the same fix, ported onto aurora-wip and kept on top of #556. It includes the ordering and cancellation fixup from #433 (Ryan's 986ec63).

Problem. One client's GPU hang took down other clients. A CS2 hang cancelled 42 commands, Hyprland got ECANCELED from fail_unpublished(), Honeykrisp treated that as a lost device, and the desktop froze until a reboot.

1. Keep innocent clients alive across another client's recovery

  • ECANCELED is no longer reported as the client's own failure from fail_unpublished(), ordered output publication, or fail_while_owned() for spared work. Work the firmware listed as interrupted still reports its failure.
  • A publication behind a recovery, or behind this owner's draining spared pair, waits up to RECOVERY_WAIT_MS (26 s) instead of 2 s. A guilty pair's publication still fails after a second.
  • Commands with failed inputs or predecessors stay cancelled, because a failed fence does not prove its producer retired.

2. Cancel consumers of a failed producer whatever it failed with

inherited_error() only turned EIO, ENODEV and ETIMEDOUT into ECANCELED. A render group that a recovery dropped fails with ENODATA, and that passed through to its consumers as their own lost device. In one case Xwayland died this way, and a VM started 85 minutes later failed on Xwayland's buffers without any recovery.

G17 consumers now always get ECANCELED. This is G17 only: G15 keeps inherited_error() unchanged, and no file under g15/ changes. An interrupted fence wait is not a producer failure, so it keeps its own error.

Host tests changed. Four tests pinned the old 2 s budget, so they move to the new one:

  • g17-compute-deadline-tests.py: a pending recovery now ends at 26 s, and a new no-recovery-extension mutant fails if the extension is removed.
  • g17-admission-deadline-tests.py: a spared quarantine now waits for the caller's deadline.
  • g17-compute-constructor-tests.py and g17-compute-growth-deadline-tests.py: fixture and mutant updates only.

Tests

  • KUnit on this tree (arm64, DRM_ASAHI + DRM_ASAHI_G16): 212 pass, 1 skipped of 213. The skip is apple_g17_rx_util_min_test, which needs CONFIG_UCLAMP_TASK. Three new asahi_g17_wait_clock cases cover the render wait deadline.
  • tools/asahi/run-host-tests.py: 159 of 161. apple-dcp and g15-retention-tests --mutations fail here, and fail the same on clean aurora-wip 52f0a6b on this machine.
  • The driver also builds with DRM_ASAHI_G15=y and DRM_APPLE=m.
  • checkpatch --strict: clean.

Tested on hardware (J616s, M4 Pro). The G17 logic here has run on this machine's daily kernels since 2026-10-28, on an older aurora-wip base:

  • CS2 reason-0 recovery: only CS2 failed. Hyprland survived, and CS2 ran 7 more times.
  • The kernel running now went through 3 reason-2 recoveries caused by test programs, and Hyprland kept running.

Known limits

  • Innocent work the firmware listed as interrupted still reports a lost device, by design. Those recoveries did this to t3code and Hearthstone.exe when their commands were in flight. So a recovery still costs a client that was running on the GPU at that moment; this PR protects the ones that were only queued or waiting.
  • The owner of groups a recovery dropped still sees ENODATA.

Not tested

  • This exact tree has not been booted. It differs from the daily kernel in two small refactors: PRODUCER_FAILED replaces the shared helper for G17, and the wait deadline moves into WaitClock::deadline so it can be unit tested.
  • No new recovery campaign, and no run on M1 Pro, M2 or the Neo. The M1 Pro and M2 checks in drm/asahi: g17: keep innocent clients alive across another client's recovery #433 were for the old code and do not carry over.

Diagnosed and written by Claude Opus 5.5.

🤖 Generated with Claude Code

…ecovery

A recovery for one client's hang still killed the others: a CS2 reason-0
recovery cancelled 42 commands and Hyprland got ECANCELED from
fail_unpublished(), which Honeykrisp treats as a lost device, so the
desktop froze until a reboot.

- ECANCELED is not reported as this client's failure from
  fail_unpublished(), the ordered output publication, or
  fail_while_owned() for spared work. Work the firmware listed as
  interrupted keeps reporting its failure.
- Publication waits for a pending recovery, or for this owner's spared
  pair to drain, for up to RECOVERY_WAIT_MS (26 s) instead of the usual
  2 s: the compute admission deadline extends (Admission::
  extend_for_recovery) in ensure_compute, in the recovery waits of
  compute_packet including the input-source wait, and in
  Gate::enter_until, whose gate only a recovery closes; the render waits
  keep a WaitClock whose usual bound starts afresh once the recovery
  ends, and render admission extends its deadline while a recovery is
  pending. A spared quarantined pair waits until the caller's deadline;
  a guilty pair's publication still fails after a second.
- Commands with failed inputs or predecessors stay cancelled: a failed
  fence does not prove its producer retired.

The compute deadline host tests move to the longer recovery budget, and
a mutant now checks that removing the extension fails them.

Link: aurora-silicon#433
Co-developed-by: Ryan Murray <ryan@aurorasilicon.org>
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com>
Assisted-by: LLM
…led with

A command whose input fence, ordered predecessor or compute source failed
inherits the failure through inherited_error(). Only EIO, ENODEV and
ETIMEDOUT became ECANCELED; any other error passed through unchanged and
was reported as the consumer's own failure, which userspace treats as a
lost device.

A GPU recovery fails the render groups it dropped and cannot replay with
ENODATA. One such recovery (render slot 10, 11 groups of a muvm GPU
worker) took down Xwayland, whose next command consumed one of the
worker's buffers. Xwayland's own commands then failed, so its buffers
kept that failure in their fences, and 85 minutes later a newly started
muvm GPU worker that imported them failed with ENODATA too, without any
recovery.

Cancel the consumer for every producer failure. ECANCELED is not
reported as a device loss and does not poison the consumer's VM; its
fence still carries the error, as the UAPI documents. An interrupted
fence wait is not a producer failure and keeps its own error.

G15 reports every input failure as a client failure anyway, so it keeps
inherited_error() unchanged.

Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com>
Assisted-by: LLM
@RyanTheTide

Copy link
Copy Markdown
Contributor

Merged into aurora-wip. This exact tree was booted on the Neo (A18 Pro, G17P) as a one-shot of 7.3.0-aurora-prr1103: aurora-wip c243830 plus this PR and the other open aurora-wip PRs.

  • A guilty Vulkan client ran a compute shader that never ends. It got VK_ERROR_DEVICE_LOST, with one kernel line: "GPU work of a client failed (ETIMEDOUT, reported at g17/runtime.rs:1300); the client sees a lost device".
  • An innocent glmark2-wayland --off-screen running at the same time in the GNOME session finished with rc 0 (score 11811). gnome-shell kept the same PID, and a second glmark2 after the recovery scored 11436.
  • The boot had k -p3 0 and 0 AVCs, and its warning set is no worse than fleet14o.
  • Build: the full kernel is clean. The full host suite gives 174 of 175 on both the test merge and plain c243830 (apple-dcp, since fixed by PR 861).

The J616s run of this tree is queued with PR 837 in the next J616s slot. Thanks Jonathan.

mchalunderscore pushed a commit to mchalunderscore/linux that referenced this pull request Nov 4, 2026
…nto integration/aurora

Bring integration/aurora up to the aurora-wip tip 9ff2b9a. After this
merge the two branches have the same tree.

The range contains the fleet14o source c243830 (aurora-silicon#857, aurora-silicon#858, aurora-silicon#795,
aurora-silicon#859), which was promoted on all four machines on 2026-11-02 and 03, and
the merges after it: aurora-silicon#861 (apple-dcp selftest stubs for aurora-silicon#858), aurora-silicon#860
(SEP keybag in the shared xART window, the aurora-wip copy of aurora-silicon#856),
aurora-silicon#724 (J616s boot NOR, both parts), aurora-silicon#779 (ALS colour temperature ABI),
aurora-silicon#862 (J700 GD25UF16E boot NOR, the Neo bless fix), aurora-silicon#863 (J616s ANE H16
start stall), aurora-silicon#797 (T6000 DCS whole-word writes) and aurora-silicon#803 (G17 keeps
innocent clients alive across another client's recovery).

aurora-silicon#858 and aurora-silicon#860 make the same changes integration/aurora already had as
aurora-silicon#855 and aurora-silicon#856, so they merge without conflicts.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants