media: apple: avd: t8103: size the VP instruction FIFO limit correctly - #585
Closed
kcirtapfromspace wants to merge 62 commits into
Closed
media: apple: avd: t8103: size the VP instruction FIFO limit correctly#585kcirtapfromspace wants to merge 62 commits into
kcirtapfromspace wants to merge 62 commits into
Conversation
Apple Silicon based laptop use SPI as transport for HID. Add support for SPI-based HID devices and and Apple keyboard and trackpad devices. Intel based laptops using the keyboard input driver applespi use the same HID over SPI protocol and can be supported later. This requires SPI keyboard/mouse HID types since Apple's intenal keyboards/trackpads use the same product id. Signed-off-by: Janne Grunau <j@jannau.net>
Apple M2* chips have an embedded MTP processor that handles all HID functions, and does not go over a traditional bus like SPI. The devices still have real IDs, so add them here. Signed-off-by: Hector Martin <marcan@marcan.st>
Signed-off-by: Hector Martin <marcan@marcan.st>
This maximum is arbitrary. Recent Apple devices have some vendor-defined
reports with 16384 here which fail to parse without this, so let's bump
it to that.
This value is used as follows:
report->size += parser->global.report_size * parser->global.report_count;
[...]
/* Total size check: Allow for possible report index byte */
if (report->size > (max_buffer_size - 1) << 3) {
hid_err(parser->device, "report is too long\n");
return -1;
}
All of these fields are unsigned integers, and report_count is bounded
by HID_MAX_USAGES (12288). Therefore, as long as the respective maximums
do not overflow an unsigned integer (let's say a signed integer just in
case), we're safe. This holds for 16384.
Signed-off-by: Hector Martin <marcan@marcan.st>
Apple MacBook keyboards started using HID over SPI in 2015. With the addition of the SPI HID transport they can be supported by this driver. Support all product ids over with the Apple SPI vendor id for now. The Macbook Pro (M1, 13-inch, 2020) uses the same function key mapping as other Macbook Pros with touchbar and dedicated ESC key. Apple silicon Macbooks use the same function key mapping as the 2021 and later Magic Keyboards. Signed-off-by: Janne Grunau <j@jannau.net>
We use BUS_HOST for MTP HID subdevices Signed-off-by: Hector Martin <marcan@marcan.st>
This mode is added to ease adding new xkeyboard configs for Apple silicon Macbook keyboards. The existing ones have strange quirks [1] and as the keyboard sends a key code for the 'fn' there is desire to use it as additional modifier [2]. [1]: https://pagure.io/fedora-asahi/remix-bugs/issue/17 [2]: https://asahilinux.org/docs/project/help-wanted/ (Keyboard layout cleanup) Signed-off-by: Janne Grunau <j@jannau.net>
Signed-off-by: Janne Grunau <j@jannau.net>
Signed-off-by: Janne Grunau <j@jannau.net>
Will be used for supporting MacBook trackpads connected via SPI. Signed-off-by: Janne Grunau <j@jannau.net>
The trackpads in Macbooks beginning in 2015 are HID devices connected over SPI. On Intel Macbooks they are currently supported by applespi.c. This chang adds support for the trackpads on Apple Silicon Macbooks starting in late 2020. They use a new HID over SPI transport driver. The touch report format differs from USB/BT Magic Trackpads. It is the same format as the type 4 format supported by bcm5974.c. Signed-off-by: Janne Grunau <j@jannau.net>
Apple M2 devices expose the multi-touch device over the HID over DockChannel transport, which we represent as the HOST bus type. The report format is the same, except the legacy mouse header is gone and there is no enable request needed. Signed-off-by: Hector Martin <marcan@marcan.st>
The trackpad has to request multi touch reports during resume. Signed-off-by: Janne Grunau <j@jannau.net>
On at least some SPI devices (e.g. recent Apple Silicon machines), the Broadcom touch controller is prone to crashing. When this happens, the STM eventually notices and resets it. It then notifies the driver via HID report 0x60, and the driver needs to re-enable MT mode to make things work again. This poses an additional issue: the hidinput core will close the low-level transport while the device is closed, which can cause us to miss a reset notification. To fix this, override the input open/close callbacks and send the MT enable every time the HID device is opened, instead of only once on probe. This should increase general robustness, even if the reset mechanism doesn't work for some reason, so it's worth doing it for USB devices too. MTP devices are exempt since they do not require the MT enable at all. Signed-off-by: Hector Martin <marcan@marcan.st>
For SPI/MTP trackpads, query the dimensions via HID report instead of hardcoding values. TODO: Does this work for the USB/BT devices? Maybe we can get rid of the hardcoded sizes everywhere? Signed-off-by: Hector Martin <marcan@marcan.st>
Keyboard and trackpad of Apple Sillicon SoCs (M1, M1 Pro/Max) laptops are are HID devices connected via SPI. This is the same protocol as implemented by applespi.c. It was not noticed that protocol is a transport for HID. Adding support for ACPI based Intel MacBooks will be done in a separate commit. How HID is mapped in this protocol is not yet fully understood. Microsoft has a specification for HID over SPI [1] incompatible with the transport protocol used by Apple. [1] https://docs.microsoft.com/en-us/windows-hardware/drivers/hid/hid-over-spi Contains "HID: transport: spi: apple: Increase receive buffer size" The SPI receive buffer is passed directly to hid_input_report() if it contains a complete report. It is then passed to hid_report_raw_event() which computes the expected report size and memsets the "missing trailing data up to HID_MAX_BUFFER_SIZE (16K) or hid_ll_driver.max_buffer_size (if set) to zero. Co-developed-by: Hector Martin <marcan@marcan.st> Signed-off-by: Hector Martin <marcan@marcan.st> Signed-off-by: Janne Grunau <j@jannau.net>
DockChannel is a simple FIFO interface used to communicate between SoC blocks. Add a driver that represents the shared interrupt controller for the DockChannel block, and then exposes probe and data transfer functions that child device drivers can use to instantiate individual FIFOs. Signed-off-by: Hector Martin <marcan@marcan.st>
Apple M2 devices have an MTP coprocessor embedded in the SoC that handles HID for the integrated touchpad/keyboard, and communicates over the DockChannel interface. This driver implements this new interface. Signed-off-by: Hector Martin <marcan@marcan.st>
This driver can be used for coprocessors that do some background task or communicate out-of-band, and do not do any mailbox I/O beyond the standard RTKit initialization. Signed-off-by: Hector Martin <marcan@marcan.st>
Certain Broadcom bluetooth chips (bcm4377/bcm4378/bcm438) need ACL streams carrying audio to be set as "high priority" using a vendor specific command to prevent 10-ish second-long dropouts whenever something does a device scan. This patch sends the command when the socket priority is set to TC_PRIO_INTERACTIVE, as BlueZ does for audio. Signed-off-by: Sasha Finkelstein <fnkl.kernel@gmail.com>
The current approach of silently disabling all rust drivers if the toolchain is missing results in users that try to compile their own kernels getting a "successful" build and then being confused about where did their drivers go. In comparison, missing openssl results in a build failure, not a disappearance of everything that depends on it. This also means that allyesconfig will depend on rust, but since the rust experiment concluded with "rust is here to stay", i believe that allyesconfig should be building rust drivers too. Signed-off-by: Sasha Finkelstein <k@chaosmail.tech>
Signed-off-by: Janne Grunau <j@jannau.net>
Apple M3 Pro and Max devices are using 'gp00' keys for GPIO in addition to 'gP00' keys. Add a second compatible to handle this keys with an additional macsmc-gpio instance. Signed-off-by: Janne Grunau <j@jannau.net>
Add support for SMC GPIO keys with a lower letter 'p' via the "apple,smc-low-gpio" compatible. This adds support for a second macsmc-gpio controller using 'gp00' keys. These keys are used on Apple M3 Pro and Max MacBooks in the controller for keyboard and trackpad and for the built-in DisplayPort to HDMI converter. Signed-off-by: Janne Grunau <j@jannau.net>
Apple M3 Pro and Max devices are using 'gp00' keys for GPIO in addition to 'gP00' keys. These keys are handled by an additional macsmc-gpio instance using the "apple,smc-low-gpio" compatible. Signed-off-by: Janne Grunau <j@jannau.net>
Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
I feel fairly confident they dont do anything, would be fun to find out why apple uses them Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
turns out AVD works with interchange/compressed buffers internally and only decompressed if we ask it to Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
Names are still wrong tho, mostly seg and the other Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
Signed-off-by: sofus <sofus.c@icloud.com>
t8103_configure_stream() programmed AVD_V3_VP_INSN_FIFO_MASK to 0x100000
while fifo_size() allocates 0x100000 * 12, so the limit described a
twelfth of the buffer it points at. A frame whose instruction stream
exceeded 1 MiB wrapped inside the larger allocation and the hardware
never signalled completion, leaving avd_watchdog_func() to fire two
seconds later with "Frame processing timed out!" and the client to see
zero decoded frames.
Every real 1080p H.264 keyframe is large enough to hit this, so H.264
failed on the first frame of essentially any real-world stream while
HEVC and VP9 were unaffected even at larger coded sizes. t8112 and t8122
do not program a limit in the equivalent register; only t8103 did.
Derive it from fifo_size() so the two cannot drift apart again.
Measured on a MacBook Pro 13" M1 (j293/t8103), 10 decode runs per clip:
before after
synthetic 1080p 89 KiB 10/10 10/10
synthetic 1080p 140 KiB 0/10 10/10
real-world 1080p 384 KiB 0/10 10/10
Decoded output is bit-exact against libavcodec (per-frame PSNR reports
inf) for real-world H.264, and HEVC and VP9 are unchanged.
Signed-off-by: Patrick Deutsch <105461352+kcirtapfromspace@users.noreply.github.com>
WhatAmISupposedToPutHere
requested changes
Aug 30, 2026
| { | ||
| w32(AVD_V3_VP_INSN_FIFO_IOVA + (fifo_idx * 4), addr >> 8); | ||
| w32(AVD_V3_VP_INSN_FIFO_MASK + (fifo_idx * 4), 0x100000); | ||
| /* |
There was a problem hiding this comment.
why is it here, put it in the commit message instead.
Member
|
you do not need to generate a wall of slop to describe a one line change, rewrite the pr description in your own words, it should not be difficult. |
|
thats kinda weird, hevc's instructions should be bigger than h264. |
Member
|
Writing 0 works as well, fixed in asahi-7.1.12-1 as fixup commit. Thanks for finding this issue |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
t8103_configure_stream()programsAVD_V3_VP_INSN_FIFO_MASKto0x100000while
fifo_size()allocates0x100000 * 12, so the limit describes a twelfthof the buffer it points at. A frame whose instruction stream exceeds 1 MiB wraps
inside the larger allocation and the hardware never signals completion:
avd_watchdog_func()fires two seconds later withFrame processing timed out!and the client gets zero decoded frames.
Every real 1080p H.264 keyframe is large enough to hit this, so H.264 failed on
the first frame of essentially any real-world stream. HEVC and VP9 were
unaffected even at larger coded sizes — a 1080p VP9 clip with a 127 KiB keyframe
decodes cleanly while H.264 fails 10/10 at 140 KiB. That asymmetry is what
pointed here: it rules out shared buffers and anything codec-agnostic, and
t8103 is the only variant that programs a nonzero value in this register.
Deriving it from
fifo_size()rather than hardcoding a second constant keeps thetwo from drifting apart again.
Measured
MacBook Pro 13" M1 (j293/t8103), on
asahi-wipat ccce11a, 10 decode runs perclip via ffmpeg + VA-API:
Correctness against libavcodec, per-frame PSNR (
inf= bit-exact):y:inf u:inf v:infy:inf u:inf v:infy:inf u:inf v:infy:inf u:inf v:infy:inf u:inf v:infReproducer, if useful:
Notes
Testing used a VA-API driver package
built around sofus13's
libva-v4l2_requestfork, plus a package for the avd-fwfirmware, since neither is currently packaged for Arch Linux ARM.
Five other hypotheses were built and measured before this one and all were
falsified — the 8x8 transform path,
pps_tile[0]sizing, the multi-slicereadl_poll_timeout, a lost mailbox interrupt underIRQF_ONESHOT, and theH.264 slice-offset scan. Recording that in case any of them looks tempting later.
Two smaller things noticed while working in this file, not included here and
happy to send separately if wanted:
stream_scaling()dereferencesrun->scaling_matrixwithout a NULL check,and that control is optional (
run->scaling_matrix = ctrl ? ctrl->p_cur.p : NULL). A client that setspic_scaling_matrix_present_flagwithout supplyingthe control would oops.
data_byte_offset(HEVC does), sostream_slice()re-derives itby scanning for emulation-prevention bytes with an unbounded
data[off]read.It measured correct on every stream tested, but it is an out-of-bounds read on
malformed input.
I also independently hit the weighted-prediction bug that #581 fixed, and can
confirm that fix resolves it —
weighted_bipred_idcbeing applied to P slicesput luma about 54 dB off. Arrived at the same place from the other direction.