Documentation: amdgpu lowered lockup_timeout in kernel 7.x affects InvokeAI

[darksylinc]

Is there an existing issue for this problem?

• I have searched the existing issues

Install method

Invoke's Launcher

Operating system

Linux

GPU vendor

AMD (ROCm)

GPU model

Radeon R9700 AI PRO

GPU VRAM

32

Version number

v6.13.6

Browser

No response

System Information

Irrelevant

What happened

I started getting some repeatable GPU timeouts on AMD:

ring gfx_0.0.0 timeout
Process Xorg ... Xorg:cs0
Starting gfx_0.0.0 ring reset
Ring gfx_0.0.0 reset succeeded
[drm] device wedged, but recovered through reset

The screen would freeze and not recover. InvokeAI was still running in the background until it finished.
I could SSH into the machine and sudo service restart lightdm to recover.

But I noticed InvokeAI was fine on Kernel 6.17, though the screen would lock up for roughly 2 seconds while finishing VAEs (which was at the time the system would freeze in 7.0).

The culprit?

lockup_timeout was lowered from 10 seconds to 2 seconds:

6.17

modinfo amdgpu | grep -E 'lockup_timeout|timeout_period|timeout_fatal_disable|gpu_recovery'
parm:           lockup_timeout:GPU lockup timeout in ms (default: 10000 for all jobs. 0: keep default value. negative: infinity timeout), format: for bare metal [Non-Compute] or [GFX,Compute,SDMA,Video]; for passthrough or sriov [all jobs] or [GFX,Compute,SDMA,Video]. (string)
parm:           gpu_recovery:Enable GPU recovery mechanism, (1 = enable, 0 = disable, -1 = auto) (int)
parm:           timeout_fatal_disable:disable watchdog timeout fatal error (false = default) (bool)
parm:           timeout_period:watchdog timeout period (0 = timeout disabled, 1 ~ 0x23 = timeout maxcycles = (1 << period) (uint)

7.0

parm:           lockup_timeout:GPU lockup timeout in ms (default: 2000. 0: keep default value. negative: infinity timeout), format: [single value for all] or [GFX,Compute,SDMA,Video]. (string)
parm:           gpu_recovery:Enable GPU recovery mechanism, (1 = enable, 0 = disable, -1 = auto) (int)
parm:           timeout_fatal_disable:disable watchdog timeout fatal error (false = default) (bool)
parm:           timeout_period:watchdog timeout period (0 = timeout disabled, 1 ~ 0x23 = timeout maxcycles = (1 << period) (uint)

It may be relevant to document that AMDGPU users on Linux should set amdgpu.lockup_timeout kernel parameter back to something higher like 10 seconds or else InvokeAI may "crash" the Xorg server.

Technically this is not InvokeAI's fault, but it is customary on Windows for commercial software to explain that TDR setting needs to be increased given that it's relevant to the application.

What you expected to happen

The system to continue working normally.

How to reproduce the problem

Just run any workload with SDXL at around 1320x1320 on Kernel 7.0 and the display will freeze.

Additional context

No response

Discord username

No response