Repository navigation
media: apple: isp: add IMX958 P010 previews and firmware config discovery - #462
Merged
Merged
Conversation
Port the processed config-5 preview command sequence from our m1n1 J616s work, with contiguous V4L2 P010 strides and pool geometry. Query all advertised H17 sensor configs and retain the board defaults while appending consistent output modes. Keep NV12 the default and scope the new path to T6040/IMX958. Retire output and metadata leases after the report ACK, restart an empty P010 queue on requeue, bound ring servicing and latch malformed or timed-out command replies. Add command-byte, geometry, ownership and ring-wrap coverage. These source corrections still require sustained capture validation on 26.6.2. Builds on Jonathan Goyvaerts’ T6040 camera support in PR #384. Protocol reference: our m1n1 PR #20 at 9be380c8a5 and Linux PR #231 at a4e7ee8. No firmware payload is included. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Exercise every offered format and discrete size with long MMAP capture, repeated stream restarts and format switches. Record delivered rate, timing gaps and buffer errors, and fail on substitutions or stalls. Allow a mode filter for different nominal rates. A driver-generated sequence alone is not evidence of zero firmware drops. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Check the command deadline before advancing the ring cursor, including when an ACK arrives at the wait timeout boundary. Keep the failed channel latched until coprocessor stop. Cover the boundary in KUnit. Emit the capture stress result after cleanup so a failed STREAMOFF cannot leave a passing result. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Apply the empty-queue restart to both processed IMX958 formats. Exercise 10,000 empty-to-submitted transitions for each of P010 and NV12. Other sensors retain their queue policy. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Size the command table columns for the full H17 record sizes and fix the document title underline. The standalone RST parser now accepts the document with warnings treated as errors. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The buffer wire carries 64-bit addresses, while the proxy observed zero-extended low words on its firmware. Accept either complete canonical pair for a submitted lease and reject mixed or unrelated upper bits. Keep pool, plane and fresh-tag checks. Also assert that NV12 stays first in format enumeration. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Remove the inherited multiplanar module parameter. T6040/IMX958 supports normal V4L2 negotiation for contiguous NV12 and P010 through either API, with one memory plane containing the two ISP components. Keep NV12 first and initial; other sensors retain their shipped single-planar API. Do not advertise the old non-contiguous NV12M alias. Cache complete sensor configuration replies with the video state and print them on the standard log-status ioctl, so collecting the firmware list needs no driver or boot parameter. Free the cache on probe failure and final video release, and test failed discovery without publishing partial state. No module/kernel parameters or platform Kconfig options are added. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Acquire the validity word, copy the complete message object after the DMA read barrier and restore that captured word. Copying seven payload words through the arg1 field violates FORTIFY field bounds. Cover all eight copied words and keep the ownership ordering unchanged. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Remove the parameter-gated multi-planar hooks, which changed the queue type without reinitialising its cached plane semantics. Both contiguous NV12 and P010 use the existing single-planar queue. Keep format negotiation explicit through S_FMT. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Credit Ryan\x27s m1n1 PR #20, aurora-silicon/m1n1#20, at 9be380c8a5a94609fdbe62d693c7b2121478ad8f for the processed preview protocol. Record what each requested file establishes, the diagnostic RAW snapshot limitation and the missing per-config timing records. Distinguish the separate AOP audio sequence from ISP camera configuration. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Correct the reStructuredText underline and verify the complete document with warnings treated as errors. Source protocol credit: Ryan Murray, m1n1 PR #20, aurora-silicon/m1n1#20, at 9be380c8a5a94609fdbe62d693c7b2121478ad8f. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The combined test kernel is published from
j616s/camera-r5-test, source734a4018f3a6b5e17bc2afc322099c477b67a66d, with display r5905dff77456191245d1aff3d093c2f41947983ffas its first parent and this camera PR at2114ed0c6245126cd90a2e5e96e2cddd4e0697e0as its other parent. The display driver and Apple DTS tree match r5, and the ISP tree matches this PR. All 12,014 resolved configuration settings match r5's published config. Release:7.3.0-aurora-j616s-r5-cam1. This provides one combined boot for display and camera qualification.The J616s camera driver exposes NV12 through two device-tree presets. This adds the processed P010 preview from our m1n1 J616s work at 1280x720 and 1920x1080, and queries the firmware's configuration list instead of treating those presets as the complete sensor capability list. NV12 stays first and initial because browsers/WebRTC and many applications take the first format or do not support P010. The new format and protocol checks apply to T6040/IMX958; the Neo keeps its shipped default formats, presets and command path.
This is draft. The code and host tests are ready for a J616s run, but RAW Bayer capture, nominal FPS for every firmware configuration, and sustained zero-drop capture are not established. Those are remaining parts of Ryan's expanded request, not claims made by this PR.
Built on Jonathan Goyvaerts' T6040 camera support in #384. The primary protocol source is Ryan's m1n1 #20,
j616sat9be380c8a5a94609fdbe62d693c7b2121478ad8f, especiallyisp_capture.py,isp_sensor.py,isp_profile.py,isp_command.pyanddocs/j616s-bringup.md. I also checked the pinned profile and IPC layouts in Linux #231,a4e7ee86bc5eed8ff9dab61ef9656adfe000b207.What #20 actually captured
Both processed sizes use firmware config 5, scaler 1, output format 0x12, two-plane 10-bit YCbCr 4:2:0. The words contain MSB-aligned ten-bit samples in little-endian 16-bit storage. The RGB PNG is a conversion of P010, not a Bayer capture.
The branch records completed 720p P010 frames, including an 80-frame run that wrapped both rings, with intermittent startup/streaming stalls still unresolved. The later 1080p trial also uses config 5. It completed one uniquely matched frame, two full matching 6,242,304-byte reads, CH_STOP and zero DART faults. That is evidence for a 1080p frame, not continuous 1080p or measured FPS.
T6040PreviewConfiguration.configure(..., width=1920, height=1080)is the reusable setup for it.All 13 sensor configurations were queried on 25G76, but the public branch does not contain their complete reply table or a completed processed-colour result from another config. The RAW trial recorded stable 4,644,864-byte surfaces after stop without a completed RAW buffer report. Its private orchestration, Bayer order and packing are not in the public branch. Combined RAW/YUV exceeded the retained allocation budget and was refused. I have not substituted an assumed SRGGB10/SRGGB12 format for those missing facts.
Public-source recheck
Re-read all ten requested files at the exact #20 head, plus the PR discussion, J700-to-J616s diff and RAW-documentation history. The pinned bring-up notes state that the RAW trial received no completed-buffer report and that capture orchestration remained private. Commit 743e4e75 adds a diagnostic RAW snapshot report, without a RAW receiver implementation. This does not dispute that the session produced a RAW snapshot; it distinguishes that evidence from a completed-capture ABI.
isp_capture.pyisp_pixels.pyisp_sensor.pyisp_profile.py,isp_t6040_25g76.jsonisp_vid.py,experiments/isp.pyisp_cmd.pyisp_command.pysrc/isp.cThe 13 runtime queries are documented, but their complete reply records are not saved in these files. The qualified helper requests 30/15 FPS AE limits. Neither that nor the sensor-clock field supplies a per-mode nominal-rate table. The kernel is therefore not given a guessed Bayer FOURCC, a fabricated mode list or invented nominal FPS.
Separate AOP map
The modern AOP map is 0x20 SPUApp, 0x21 wakehint, 0x22 audio, 0x23 voice trigger, 0x24 accelerometer, 0x25 gyroscope, 0x26 lid angle, 0x27 ALS, and 0x28-0x2b with unqualified roles. This differs from the legacy ENDPOINTS13 map. The checked audio path uses
aop-audioon endpoint 0x22 and EPIC v2. It establishes LP readiness with an owned adpx buffer and lpai/pdm0/adpx/lai attachment and power requests, then attaches hpai, requests pw1, prepares stream-10 NS RX0 DMA, starts DMA and requests hpai pwrd. Matching reports and stable full copies precede RX stop and return through pw1 to idle. The camera helper issues no AOP command; these audio operations are not transplanted into apple-isp. The ISP camera channel/command maps remain below.Linux formats and modes
P010 is
V4L2_PIX_FMT_P010. Both NV12 and P010 use the standard single-planar V4L2 API, with one contiguous memory plane containing Y and UV. UV follows Y immediately. The proxy page-aligned each component allocation independently; Linux passes its actual contiguous extents to the firmware pool, so it does not introduce an undocumented inter-plane gap into V4L2 buffers.The inherited
multiplanarmodule parameter is removed. There are no driver-defined module/kernel parameters, new platform flags or per-platform Kconfig options. Applications select P010 explicitly with S_FMT through the shipped single-planar API. The inherited multi-planar hooks changed the queue type without updating cached vb2 plane semantics, so those hooks and the non-contiguous NV12M alias are removed. No default-format parameter was added.Probe queries every advertised config, bounded at 64, using
CH_CAMERA_CONFIG_GETwith the 0x120-byte H17 record. A debug-gated dump preserves every reply, andVIDIOC_LOG_STATUSprints every complete cached reply, input/output sizes and the sensor-clock word without sending firmware commands. Collect that dump before a stress run; no probe-time parameter is needed. The sensor-clock word is not reported as FPS.The two DT presets retain their order and NV12 default. Config-5 P010 is enabled when the runtime input geometry matches the source profile; an unspecified P010 size chooses 720p first. Additional distinct sizes are generated from the firmware's output dimensions only when both copies of the geometry agree, the dimensions are nonzero/even and within input, and the output is at most 16384 in either dimension. The reported full output becomes that config's crop. The 1920x2160 input height alone does not create a processed 2160-row mode.
Configurations with duplicate dimensions are retained in the diagnostic list but do not become independently selectable modes. Their nominal rates and any sensor-specific RAW/processed restrictions still need the runtime replies. This does not yet fulfil exhaustive capture of every firmware configuration. No 4:2:2 or full-range format is advertised without evidence for its firmware encoding.
ISP endpoints
These are ISP shared-memory channels, separate from the AOP endpoint map. Counts below come from the 25G76 source fixture. Linux negotiates the current descriptor addresses and counts on each boot.
Descriptors are 0x100 bytes, with the u64 ring address at 0x50. Slots are 0x40 bytes, eight u64 words. Payload precedes arg0 ownership publication. The existing parser checks ring bounds and overlap against the channel table, boot/command storage and other rings. IO/DEBUG share one source bit; the three reply channels share another.
Command stream
An IO payload starts with a u32 zero/status and u32 opcode. Channel is zero. The source's discovery is CONFIG_GET (0x0003, 0x1c bytes), DSID setup (0x3206, 0x50), global START (0x0000, 0x0c), CH_INFO_GET (0x010d, 0x190), then every CH_CAMERA_CONFIG_GET (0x0106, 0x120).
DSID has eight slots, range 0x2fc, bases 0x2201d4000 / 0x2221d4000 / 0x2241d4000 / 0x2261d4000 and four U64_MAX sentinels. The proxy's global START argument is 0x11; the current working Linux H17 boot sends 0. This PR retains Linux's once-per-boot startup, including its existing print/PMP setup. It does not send a second global START when switching to P010.
The P010 channel setup then follows the proxy preview sequence:
Local RAW here feeds the ISP's processed preview internally. It does not expose a Bayer V4L2 stream.
BUF_H2T submits metadata and output before CH_START (0x0100, 0x0c). Linux keeps its variable-length combined batch, using a 16-byte type/count header and 64-byte descriptors. The proxy's builders used separately padded small batches. Each descriptor carries four u64 addresses, four u32 flags, plane count, pool and a u64 tag. Used-plane flags are 0x40000000. The firmware's completed P010 descriptor returns both plane addresses as zero-extended low 32 bits. Linux accepts either the exact full submitted address pair or the documented zero-extended pair, and rejects mixed/unrelated upper bits. Pool 9, two planes and the fresh tag must also match the submitted lease. BUF_T2H render status 0x10000000 means complete; submit/return-ACK statuses are 0x30000000/0x80000000. CH_STOP (0x0101) and CH_BUFFER_RETURN (0x0104) finish the stream.
The proxy requested AE limits 7680/3840, or 30/15. Linux retains its frame-rate and exposure-priority controls. Default is fixed 30/30; enabling
exposure_auto_prioritypermits 30/15. These are control limits, not nominal-rate measurements. P010 uses the minimal preview sequence without adding the NV12 path's setfile, CNR, MBNR, SIF-format or temporal-filter setup. Previously programmed NV12 tuning, including a loaded setfile, can remain resident across a format switch. The target run must validate those transitions; a matching command fixture does not prove the resulting firmware state or throughput.25G76 versus the current 26.6.2 driver
The old profile's firmware identity differs. It is not used to admit the current firmware.
The fixed TEXT/DATA reservations in the J616s DT should eventually be supplied through a bootloader handoff for the actual loaded image and its mapping lifetimes. This PR does not implement that handoff or change the DT reservations.
Stalls and validation
The source fixes an empty-queue requeue that could leave the last IMX958 buffer pending, releases P010 output and metadata leases only after the completion ACK, acquires validity before reading a reply payload and bounds P010 ring drains so control/log traffic cannot starve buffer returns. P010 command replies must have the expected length, zero status and unchanged header. Timeout and malformed replies latch failure; an ACK at or after the deadline cannot advance the ring. The existing forced coprocessor stop runs before uncertain leases are released. Resident firmware then requires reboot.
These are source-visible corrections, not proof that the proxy's intermittent stalls have been reproduced or cured on 26.6.2.
-Wall -Wextra -Werror. Six synthetic runner cases pass, including detection of a timing gap despite contiguous driver sequence numbers, stall/empty streams, substituted formats and failed STREAMOFF.W=eandDRM_WERROR=y. The full Fedora-derived build passes with r5's published config and globalCONFIG_WERROR=y. Both build Image, modules and DTBs on combined source734a4018f3a6b5e17bc2afc322099c477b67a66d. Actual command receipts confirm C-Werror, Rust-Dwarningsand fatal warnings on all seven ISP objects. FORTIFY remains enabled; the complete-message IPC copy is covered by KUnit.dtbs_checkanddt-validatereturn zero but report existing PCI/Wi-Fi compatible and unevaluated-property findings. This branch changes no DTS or binding. Both supplied DTBs match r5 byte for byte; full logs are included. These results are not described as a clean global schema validation.7.3.0-aurora-j616s-r5-cam1vermagic and reports no module parameters. The source-policy audit and warning-fatal RST parse pass.Published test kernel
Release:
7.3.0-aurora-j616s-r5-cam1, source734a4018f3a6b5e17bc2afc322099c477b67a66d, tree433a4e9b6cb09bba55d0993b505303c5ab0503ed.The local hardware-handoff artifact is published at:
It contains Image/Image.gz, matching signed/stripped modules, J616s and unchanged Neo DTBs, the DTB archive, config/System.map, source pins and bundle, build recipes/logs, ARM64 KUnit results, C stress-runner source and the public-source audit. No m1n1 Python implementation, Apple binary or firmware payload is copied into this artifact set.
SHA256SUMScovers 116 files; its SHA256 is807e78513f75aa384fda1856232bf95efb2c6214ee2ca6a5532d07429fe03ea4.READYwas written last after verification. The README includes explicit P010 long-run/cycle/switch commands. This replaces the earlier camera-only artifact for the combined display/camera run.Hardware routing
The preliminary shared
hw_requestfor devicej616s,uname -r, returnedValueError: unknown device 'j616s', before creating a request ID. The installed controller documents Neo and M1 Pro/M2 reference targets, without a J616s alias. No target transport, boot or capture was attempted outside the controller. J616s routing must be available before the full qualification request can be queued. This artifact is ready for that handoff; it is not completed RAW or zero-drop qualification.The manual stress runner enumerates offered formats/sizes, runs each for ten minutes, then repeated stop/restart and format-switch cycles. It records delivered FPS, maximum gap, sequence gaps and buffer errors and rejects substitutions. A driver-generated sequence cannot establish zero firmware-internal drops; that also needs a validated firmware frame counter or an independent rate oracle.
No target access was made by this worker. The J616s run must collect the 26.6.2 config replies first, verify both P010 images and measure each mode's sustained rate and gaps while audio is idle. The current RAW packing/order, independently selectable duplicate configs, exhaustive nominal FPS and zero-drop sensor performance remain open. No Apple binary, setfile contents or copied firmware payload is included.