Skip to content

media: apple: avd: t8103: size the VP instruction FIFO limit correctly - #585

Closed
kcirtapfromspace wants to merge 62 commits into
AsahiLinux:asahi-wipfrom
kcirtapfromspace:avd-t8103-fifo-mask
Closed

media: apple: avd: t8103: size the VP instruction FIFO limit correctly#585
kcirtapfromspace wants to merge 62 commits into
AsahiLinux:asahi-wipfrom
kcirtapfromspace:avd-t8103-fifo-mask

Conversation

@kcirtapfromspace

Copy link
Copy Markdown

t8103_configure_stream() programs AVD_V3_VP_INSN_FIFO_MASK to 0x100000
while fifo_size() allocates 0x100000 * 12, so the limit describes a twelfth
of the buffer it points at. A frame whose instruction stream exceeds 1 MiB wraps
inside the larger allocation and the hardware never signals completion:
avd_watchdog_func() fires two seconds later with Frame processing timed out!
and the client gets zero decoded frames.

Every real 1080p H.264 keyframe is large enough to hit this, so H.264 failed on
the first frame of essentially any real-world stream. HEVC and VP9 were
unaffected even at larger coded sizes — a 1080p VP9 clip with a 127 KiB keyframe
decodes cleanly while H.264 fails 10/10 at 140 KiB. That asymmetry is what
pointed here: it rules out shared buffers and anything codec-agnostic, and
t8103 is the only variant that programs a nonzero value in this register.

Deriving it from fifo_size() rather than hardcoding a second constant keeps the
two from drifting apart again.

Measured

MacBook Pro 13" M1 (j293/t8103), on asahi-wip at ccce11a, 10 decode runs per
clip via ffmpeg + VA-API:

clip keyframe before after
synthetic 1080p 89 KiB 10/10 10/10
synthetic 1080p 140 KiB 0/10 10/10
real-world 1080p 384 KiB 0/10 10/10

Correctness against libavcodec, per-frame PSNR (inf = bit-exact):

stream before after
H.264 real-world 1080p no output y:inf u:inf v:inf
HEVC real-world 1080p y:inf u:inf v:inf y:inf u:inf v:inf
VP9 real-world 1080p y:inf u:inf v:inf y:inf u:inf v:inf

Reproducer, if useful:

ffmpeg -y -f lavfi -i "testsrc2=size=1920x1080:rate=30:duration=1,noise=alls=8:allf=t" \
       -c:v libx264 -profile:v main -pix_fmt yuv420p -qp 31 big.mp4 </dev/null
ffmpeg -y -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -i big.mp4 \
       -frames:v 5 -pix_fmt nv12 -f rawvideo /dev/null </dev/null

Notes

Testing used a VA-API driver package
built around sofus13's libva-v4l2_request fork, plus a package for the avd-fw
firmware, since neither is currently packaged for Arch Linux ARM.

Five other hypotheses were built and measured before this one and all were
falsified — the 8x8 transform path, pps_tile[0] sizing, the multi-slice
readl_poll_timeout, a lost mailbox interrupt under IRQF_ONESHOT, and the
H.264 slice-offset scan. Recording that in case any of them looks tempting later.

Two smaller things noticed while working in this file, not included here and
happy to send separately if wanted:

  • stream_scaling() dereferences run->scaling_matrix without a NULL check,
    and that control is optional (run->scaling_matrix = ctrl ? ctrl->p_cur.p : NULL). A client that sets pic_scaling_matrix_present_flag without supplying
    the control would oops.
  • H.264 has no data_byte_offset (HEVC does), so stream_slice() re-derives it
    by scanning for emulation-prevention bytes with an unbounded data[off] read.
    It measured correct on every stream tested, but it is an out-of-bounds read on
    malformed input.

I also independently hit the weighted-prediction bug that #581 fixed, and can
confirm that fix resolves it — weighted_bipred_idc being applied to P slices
put luma about 54 dB off. Arrived at the same place from the other direction.

jannau and others added 30 commits June 20, 2026 17:01
Apple Silicon based laptop use SPI as transport for HID. Add support for
SPI-based HID devices and and Apple keyboard and trackpad devices.
Intel based laptops using the keyboard input driver applespi use the
same HID over SPI protocol and can be supported later.

This requires SPI keyboard/mouse HID types since Apple's intenal
keyboards/trackpads use the same product id.

Signed-off-by: Janne Grunau <j@jannau.net>
Apple M2* chips have an embedded MTP processor that handles all HID
functions, and does not go over a traditional bus like SPI. The devices
still have real IDs, so add them here.

Signed-off-by: Hector Martin <marcan@marcan.st>
Signed-off-by: Hector Martin <marcan@marcan.st>
This maximum is arbitrary. Recent Apple devices have some vendor-defined
reports with 16384 here which fail to parse without this, so let's bump
it to that.

This value is used as follows:

report->size += parser->global.report_size * parser->global.report_count;

[...]

/* Total size check: Allow for possible report index byte */
if (report->size > (max_buffer_size - 1) << 3) {
	hid_err(parser->device, "report is too long\n");
	return -1;
}

All of these fields are unsigned integers, and report_count is bounded
by HID_MAX_USAGES (12288). Therefore, as long as the respective maximums
do not overflow an unsigned integer (let's say a signed integer just in
case), we're safe. This holds for 16384.

Signed-off-by: Hector Martin <marcan@marcan.st>
Apple MacBook keyboards started using HID over SPI in 2015. With the
addition of the SPI HID transport they can be supported by this driver.
Support all product ids over with the Apple SPI vendor id for now.

The Macbook Pro (M1, 13-inch, 2020) uses the same function key mapping
as other Macbook Pros with touchbar and dedicated ESC key.
Apple silicon Macbooks use the same function key mapping as the 2021 and
later Magic Keyboards.

Signed-off-by: Janne Grunau <j@jannau.net>
We use BUS_HOST for MTP HID subdevices

Signed-off-by: Hector Martin <marcan@marcan.st>
This mode is added to ease adding new xkeyboard configs for Apple
silicon Macbook keyboards. The existing ones have strange quirks [1] and
as the keyboard sends a key code for the 'fn' there is desire to use it
as additional modifier [2].

[1]: https://pagure.io/fedora-asahi/remix-bugs/issue/17
[2]: https://asahilinux.org/docs/project/help-wanted/ (Keyboard layout cleanup)

Signed-off-by: Janne Grunau <j@jannau.net>
Signed-off-by: Janne Grunau <j@jannau.net>
Signed-off-by: Janne Grunau <j@jannau.net>
Will be used for supporting MacBook trackpads connected via SPI.

Signed-off-by: Janne Grunau <j@jannau.net>
The trackpads in Macbooks beginning in 2015 are HID devices connected
over SPI. On Intel Macbooks they are currently supported by applespi.c.
This chang adds support for the trackpads on Apple Silicon Macbooks
starting in late 2020. They use a new HID over SPI transport driver.
The touch report format differs from USB/BT Magic Trackpads. It is the
same format as the type 4 format supported by bcm5974.c.

Signed-off-by: Janne Grunau <j@jannau.net>
Apple M2 devices expose the multi-touch device over the HID over
DockChannel transport, which we represent as the HOST bus type. The
report format is the same, except the legacy mouse header is gone and
there is no enable request needed.

Signed-off-by: Hector Martin <marcan@marcan.st>
The trackpad has to request multi touch reports during resume.

Signed-off-by: Janne Grunau <j@jannau.net>
On at least some SPI devices (e.g. recent Apple Silicon machines), the
Broadcom touch controller is prone to crashing. When this happens, the
STM eventually notices and resets it. It then notifies the driver via
HID report 0x60, and the driver needs to re-enable MT mode to make
things work again.

This poses an additional issue: the hidinput core will close the
low-level transport while the device is closed, which can cause us to
miss a reset notification. To fix this, override the input open/close
callbacks and send the MT enable every time the HID device is opened,
instead of only once on probe. This should increase general robustness,
even if the reset mechanism doesn't work for some reason, so it's worth
doing it for USB devices too. MTP devices are exempt since they do not
require the MT enable at all.

Signed-off-by: Hector Martin <marcan@marcan.st>
For SPI/MTP trackpads, query the dimensions via HID report instead of
hardcoding values.

TODO: Does this work for the USB/BT devices? Maybe we can get rid of the
hardcoded sizes everywhere?

Signed-off-by: Hector Martin <marcan@marcan.st>
Keyboard and trackpad of Apple Sillicon SoCs (M1, M1 Pro/Max) laptops
are are HID devices connected via SPI.

This is the same protocol as implemented by applespi.c. It was not
noticed that protocol is a transport for HID. Adding support for ACPI
based Intel MacBooks will be done in a separate commit.

How HID is mapped in this protocol is not yet fully understood.

Microsoft has a specification for HID over SPI [1] incompatible with the
transport protocol used by Apple.

[1] https://docs.microsoft.com/en-us/windows-hardware/drivers/hid/hid-over-spi

Contains "HID: transport: spi: apple: Increase receive buffer size"

The SPI receive buffer is passed directly to hid_input_report() if it
contains a complete report. It is then passed to hid_report_raw_event()
which computes the expected report size and memsets the "missing
trailing data up to HID_MAX_BUFFER_SIZE (16K) or
hid_ll_driver.max_buffer_size (if set) to zero.

Co-developed-by: Hector Martin <marcan@marcan.st>
Signed-off-by: Hector Martin <marcan@marcan.st>
Signed-off-by: Janne Grunau <j@jannau.net>
DockChannel is a simple FIFO interface used to communicate between SoC
blocks. Add a driver that represents the shared interrupt controller for
the DockChannel block, and then exposes probe and data transfer
functions that child device drivers can use to instantiate individual
FIFOs.

Signed-off-by: Hector Martin <marcan@marcan.st>
Apple M2 devices have an MTP coprocessor embedded in the SoC that
handles HID for the integrated touchpad/keyboard, and communicates
over the DockChannel interface. This driver implements this new
interface.

Signed-off-by: Hector Martin <marcan@marcan.st>
This driver can be used for coprocessors that do some background task or
communicate out-of-band, and do not do any mailbox I/O beyond the
standard RTKit initialization.

Signed-off-by: Hector Martin <marcan@marcan.st>
jannau and others added 24 commits August 26, 2026 10:46
Certain Broadcom bluetooth chips (bcm4377/bcm4378/bcm438) need ACL
streams carrying audio to be set as "high priority" using a vendor
specific command to prevent 10-ish second-long dropouts whenever
something does a device scan. This patch sends the command when the
socket priority is set to TC_PRIO_INTERACTIVE, as BlueZ does for audio.

Signed-off-by: Sasha Finkelstein <fnkl.kernel@gmail.com>
The current approach of silently disabling all rust drivers if the
toolchain is missing results in users that try to compile their own
kernels getting a "successful" build and then being confused about where
did their drivers go. In comparison, missing openssl results in a build
failure, not a disappearance of everything that depends on it.

This also means that allyesconfig will depend on rust, but since the
rust experiment concluded with "rust is here to stay", i believe that
allyesconfig should be building rust drivers too.

Signed-off-by: Sasha Finkelstein <k@chaosmail.tech>
Apple M3 Pro and Max devices are using 'gp00' keys for GPIO in addition
to 'gP00' keys. Add a second compatible to handle this keys with an
additional macsmc-gpio instance.

Signed-off-by: Janne Grunau <j@jannau.net>
Add support for SMC GPIO keys with a lower letter 'p' via the
"apple,smc-low-gpio" compatible. This adds support for a second
macsmc-gpio controller using 'gp00' keys.
These keys are used on Apple M3 Pro and Max MacBooks in the controller
for keyboard and trackpad and for the built-in DisplayPort to HDMI
converter.

Signed-off-by: Janne Grunau <j@jannau.net>
Apple M3 Pro and Max devices are using 'gp00' keys for GPIO in addition
to 'gP00' keys. These keys are handled by an additional macsmc-gpio
instance using the "apple,smc-low-gpio" compatible.

Signed-off-by: Janne Grunau <j@jannau.net>
Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
I feel fairly confident they dont do anything, would be fun to find out
why apple uses them

Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
turns out AVD works with interchange/compressed buffers internally and
only decompressed if we ask it to

Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
Names are still wrong tho, mostly seg and the other

Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
t8103_configure_stream() programmed AVD_V3_VP_INSN_FIFO_MASK to 0x100000
while fifo_size() allocates 0x100000 * 12, so the limit described a
twelfth of the buffer it points at. A frame whose instruction stream
exceeded 1 MiB wrapped inside the larger allocation and the hardware
never signalled completion, leaving avd_watchdog_func() to fire two
seconds later with "Frame processing timed out!" and the client to see
zero decoded frames.

Every real 1080p H.264 keyframe is large enough to hit this, so H.264
failed on the first frame of essentially any real-world stream while
HEVC and VP9 were unaffected even at larger coded sizes. t8112 and t8122
do not program a limit in the equivalent register; only t8103 did.

Derive it from fifo_size() so the two cannot drift apart again.

Measured on a MacBook Pro 13" M1 (j293/t8103), 10 decode runs per clip:

                            before      after
  synthetic 1080p  89 KiB   10/10       10/10
  synthetic 1080p 140 KiB    0/10       10/10
  real-world 1080p 384 KiB   0/10       10/10

Decoded output is bit-exact against libavcodec (per-frame PSNR reports
inf) for real-world H.264, and HEVC and VP9 are unchanged.

Signed-off-by: Patrick Deutsch <105461352+kcirtapfromspace@users.noreply.github.com>
{
w32(AVD_V3_VP_INSN_FIFO_IOVA + (fifo_idx * 4), addr >> 8);
w32(AVD_V3_VP_INSN_FIFO_MASK + (fifo_idx * 4), 0x100000);
/*

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is it here, put it in the commit message instead.

@WhatAmISupposedToPutHere

Copy link
Copy Markdown
Member

you do not need to generate a wall of slop to describe a one line change, rewrite the pr description in your own words, it should not be difficult.

@sofus13

sofus13 commented Aug 30, 2026

Copy link
Copy Markdown

thats kinda weird, hevc's instructions should be bigger than h264.
Would you mind testing with 0 instead?
I would prefer getting dart errors until i size fifo_size correctly

@jannau

jannau commented Aug 30, 2026

Copy link
Copy Markdown
Member

Writing 0 works as well, fixed in asahi-7.1.12-1 as fixup commit. Thanks for finding this issue

@jannau jannau closed this Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants