Repository navigation
Merge train: G17 hang confinement (#928) plus its host-test fixture - #989
Merged
Merged
Conversation
"GPU work of a client failed" did not say which client: on a desktop with a browser, a game under Wine and a compositor, 31 such lines in one boot could not be attributed. Record the creating process (tgid and comm) with the VM status and print it. The message keeps its prefix, which tooling matches. Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com> Assisted-by: LLM
…s timeout On the M4 Pro (25G83) every recovery slot key reads 0, so recovery attribution cannot check the firmware's blame and always keeps it. A fragment shader spinning on a lock then takes down unrelated clients: a restart drops only the guilty pair's started group, its next queued group spins again, and each following recovery 83 ms later blames whichever queue the spinner starved. The driver fails that queue's oldest render with EIO, which poisons its VM. On 11-03 that killed Hyprland, quickshell and T3 Code in two cascades of deqp-vk hangs. In five of the six innocent blames the blamed render had never started (tiler start and fragment end unwritten). When slot identity cannot judge a timeout (not a page fault), read the host's own pass records at the same halted instant: if the blamed pair's oldest published render has not started, and exactly one VM has a published render that started and did not finish, and no other VM holds compute work, blame that VM's queues instead. Groups an earlier recovery listed or blamed are skipped; the restart dropped them, so their records stay half-written. In every other case the firmware's blame stands, as before. A blamed compute queue keeps the firmware's blame: only render pairs have the pass records this reads. This does not stop the guilty pair's queued groups from running after the restart, so a client with many of them can still reach the restart storm limit. Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com> Assisted-by: LLM
…atus
A command a GPU restart interrupted fails with ECANCELED. Unless a
recovery found its VM guilty, it does not poison the VM, but it was still
passed to VmStatus::report_failure, which stores it in the VM status
that GET_PARAMS VM_STATUS and the status mirror return. Honeykrisp's
check_status treats any nonzero status as a lost device, so every Vulkan
client with work in flight lost its device in another client's
recovery, while GL clients, which never read the status that way,
survived. The UAPI documents the opposite: a job cancelled because
another VM's recovery discarded it is not a failure of this VM, and its
error stays 0.
Report a non-poisoning ECANCELED to nobody, as was already done for work
cancelled by its owner. The fence still carries ECANCELED, the restart
is still logged ("interrupted by a GPU recovery after it started"), and
a guilty VM is still poisoned with it. Under Vulkan such a submission's
output is undefined, as the UAPI says, instead of the device being lost.
A restart that finds nobody guilty (no blamed queue and no guilty table
entries, or owners it cannot resolve) now fails all interrupted work this
way, the culprit's too, so a culprit that keeps hanging is stopped only
by the restart storm limit.
Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com>
Assisted-by: LLM
…t storm A firmware restart drops the groups that had started, but a hung client's other published groups run as usual afterwards. A client that queued the same spinning draw many times (deqp-vk on 11-03 had 17 renders retained on one pair) therefore hangs the GPU once per group, 83 ms apart. Once the blame lands on that client, its VM is poisoned and it can publish nothing new, so the cascade ends with its queued work. Yet the eighth restart within 30 s reaches the storm limit and stops the GPU for every client. When every VM a recovery finds guilty had already failed before it, and the recovery table matched one of their commands by its exact stamp (so the restart dropped it), take that restart back from the storm count. At most RENDER_DEPTH restarts are forgiven per window, so an unknown firmware loop still reaches the limit. A first failure always counts. The host cannot stop the queued groups themselves: 25G83 has no recovery target word, and a ContextKill is only sent for closed contexts. Releasing a killed live context's commands would depend on the kill acknowledgement waiting for a running spinner, which is unverified. Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com> Assisted-by: LLM
When a GPU restart drops an innocent pair's groups and they cannot run
again in order, fail_dropped fails them with ENODATA without poisoning
the VM. The failure was still passed to VmStatus::report_failure, which
stores it in the status that GET_PARAMS VM_STATUS and the status mirror
return, and Honeykrisp's check_status treats any nonzero status as a
lost device. In the 16-group spin repro on m4.119, vkmark lost its
device this way twice ("failing 2 groups ... ENODATA ... the client
sees a lost device") although no recovery blamed it.
Treat a non-poisoning ENODATA like a non-poisoning ECANCELED: the fence
carries the error, the VM status stays 0. Every other ENODATA source
(compute replay and spared quarantine) is likewise work a recovery
dropped, and a poisoning ENODATA is still recorded. The VM_STATUS
UAPI comment now names ENODATA beside ECANCELED.
Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com>
Assisted-by: LLM
A pair whose groups a restart drops untouched has them republished, up
to MAX_REPLAYS times. In a cascade of restarts another client causes,
the same innocent groups are dropped on every restart, and the fourth
drop failed them with EIO recorded on their VM: T3 Code lost its device
in the 16-group spin repro on m4.119 ("12 groups retired without
execution; replay limit reached, failing them").
replay::decide() only replays groups that are innocent, not blamed and
never ran, so none of them can have caused the hang. Fail them like
fail_dropped does: spared, with ENODATA, which stays out of the VM
status. The replay limit still bounds how often they are republished.
Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com>
Assisted-by: LLM
… work Each recovery classifies every VM it does not find guilty as spared. A hung client's VM that an earlier recovery found guilty therefore turned spared as soon as one later recovery blamed someone else. The next recovery that blamed it again then found its oldest work spared, took the blame as collateral to that recovery, found no guilty VM, and so did not treat the restart as leftover work of a failed VM: it counted toward the restart storm limit. In the 16-group spin repro on m4.119, gens 9 and 10 blamed glmark2 and quickshell, and the four correct blames on the spinning client after them all counted; the next restart reached the limit and stopped the GPU for everyone. Remember on the VM status that a recovery found it guilty, and keep such a VM's work guilty in every later recovery. A VM that failed only for its own client is still spared. Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com> Assisted-by: LLM
render_culprit moves a keyless timeout's blame off a render that never started only when exactly one VM has a render that started and did not finish, and no other VM holds compute work. Under load that rarely holds: bystanders' tilers start and their fragments then wait behind the spinning one. In the 16-group spin repro on m4.119, gens 9 and 10 blamed glmark2 and quickshell, whose blamed renders had neither pass written, and both lost their device. If a VM an earlier recovery found guilty still has such a render, blame it first: its groups queued before that recovery keep hanging the GPU one per restart, and it has already failed. Only otherwise does the one-running-VM rule apply. Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com> Assisted-by: LLM
render_culprit moved a keyless timeout's blame only when the blamed pair's oldest render had not started. A bystander's render whose tiler started and whose fragment pass then waits behind the spinning one is just as starved. On m4.120, in the 16-group spin repro, 15 of the 16 recoveries landed on the spinning client, 3 of them through render_culprit. In the 16th, the firmware blamed T3 Code, whose render had its tiler start written and no fragment end, while the spinner's last group ran. T3 Code lost its device. When a VM an earlier recovery found guilty still has a render that started and did not finish, blame it whether or not the blamed render had started: that VM has already failed, and its queued groups end with the restarts they cause. The one-running-VM rule, which can blame a VM that has not failed, still needs a blamed render that never started. Known limit: if a second VM hangs while a guilty VM still has started renders, each restart blames the guilty VM until those renders are gone, and those restarts do not count toward the restart storm limit. The evidence here cannot tell that second hang from a starved bystander, which is the case this change is for. Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com> Assisted-by: LLM
"drm/asahi: name the client in failure reports" gives VmStatus the creating process's tgid and comm, and the g17 recovery commits add recovery_guilty. The host test declares its own VmStatus and extracts the production status bodies into it, so it stopped compiling (no field `comm`, no `BStr`) and run-host-tests lost all three g15-input-error-tests runs. Add the three fields with a fixed host client, and a BStr stand-in that prints the bytes, so the extracted failure report formats as in the kernel. Assisted-by: LLM Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Contributor
Author
|
Merge train (MTL, 06/11). Gates passed on test tree 0412501d (integration/aurora 2661d71 + this batch):
|
This was referenced Nov 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Carries jgoyvaerts' #928 (drm/asahi: g17: confine a GPU hang to the client that caused it) unchanged, plus one commit of ours on top. Merging this also merges #928, with his commits and authorship intact.
#928 gives
VmStatusthe creating client'stgidandcomm, plusrecovery_guilty.tools/asahi/g15-input-error-tests.pydeclares its ownVmStatusand extracts the production status bodies into it, so on its own #928 breaks all three g15-input-error-tests runs in run-host-tests (no fieldcomm, noBStr). The extra commit adds the fields and aBStrstand-in to the fixture. All three runs (plain,--mutations,--ref) pass again.Review notes on #928 (not blocking):
FORGIVE_LIMIT = RENDER_DEPTH(32) againstLIMIT = 8means up to about 39 restarts per 30 s window before the GPU stops. Still bounded, but a cap nearerLIMITwould match the commit message better.🤖 Generated with Claude Code