Skip to content

media: apple: isp: add IMX958 P010 previews and firmware config discovery - #462

Merged
RyanTheTide merged 11 commits into
integration/aurorafrom
j616s/camera-sensor-modes
Oct 28, 2026
Merged

RyanTheTide merged 11 commits into
integration/aurorafrom
j616s/camera-sensor-modes

Conversation

@RyanTheTide

@RyanTheTide RyanTheTide commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

The combined test kernel is published from j616s/camera-r5-test, source 734a4018f3a6b5e17bc2afc322099c477b67a66d, with display r5 905dff77456191245d1aff3d093c2f41947983ff as its first parent and this camera PR at 2114ed0c6245126cd90a2e5e96e2cddd4e0697e0 as its other parent. The display driver and Apple DTS tree match r5, and the ISP tree matches this PR. All 12,014 resolved configuration settings match r5's published config. Release: 7.3.0-aurora-j616s-r5-cam1. This provides one combined boot for display and camera qualification.

The J616s camera driver exposes NV12 through two device-tree presets. This adds the processed P010 preview from our m1n1 J616s work at 1280x720 and 1920x1080, and queries the firmware's configuration list instead of treating those presets as the complete sensor capability list. NV12 stays first and initial because browsers/WebRTC and many applications take the first format or do not support P010. The new format and protocol checks apply to T6040/IMX958; the Neo keeps its shipped default formats, presets and command path.

This is draft. The code and host tests are ready for a J616s run, but RAW Bayer capture, nominal FPS for every firmware configuration, and sustained zero-drop capture are not established. Those are remaining parts of Ryan's expanded request, not claims made by this PR.

Built on Jonathan Goyvaerts' T6040 camera support in #384. The primary protocol source is Ryan's m1n1 #20, j616s at 9be380c8a5a94609fdbe62d693c7b2121478ad8f, especially isp_capture.py, isp_sensor.py, isp_profile.py, isp_command.py and docs/j616s-bringup.md. I also checked the pinned profile and IPC layouts in Linux #231, a4e7ee86bc5eed8ff9dab61ef9656adfe000b207.

What #20 actually captured

Both processed sizes use firmware config 5, scaler 1, output format 0x12, two-plane 10-bit YCbCr 4:2:0. The words contain MSB-aligned ten-bit samples in little-endian 16-bit storage. The RGB PNG is a conversion of P010, not a Bayer capture.

The branch records completed 720p P010 frames, including an 80-frame run that wrapped both rings, with intermittent startup/streaming stalls still unresolved. The later 1080p trial also uses config 5. It completed one uniquely matched frame, two full matching 6,242,304-byte reads, CH_STOP and zero DART faults. That is evidence for a 1080p frame, not continuous 1080p or measured FPS. T6040PreviewConfiguration.configure(..., width=1920, height=1080) is the reusable setup for it.

All 13 sensor configurations were queried on 25G76, but the public branch does not contain their complete reply table or a completed processed-colour result from another config. The RAW trial recorded stable 4,644,864-byte surfaces after stop without a completed RAW buffer report. Its private orchestration, Bayer order and packing are not in the public branch. Combined RAW/YUV exceeded the retained allocation budget and was refused. I have not substituted an assumed SRGGB10/SRGGB12 format for those missing facts.

Public-source recheck

Re-read all ten requested files at the exact #20 head, plus the PR discussion, J700-to-J616s diff and RAW-documentation history. The pinned bring-up notes state that the RAW trial received no completed-buffer report and that capture orchestration remained private. Commit 743e4e75 adds a diagnostic RAW snapshot report, without a RAW receiver implementation. This does not dispute that the session produced a RAW snapshot; it distinguishes that evidence from a completed-capture ABI.

Requested source What it establishes
isp_capture.py Config-5 P010 720p/1080p sequence; a generic pool-3 descriptor builder, without RAW packing, CFA or receiver commands
isp_pixels.py P010 decoder only; no Bayer decoder
isp_sensor.py Runtime IMX958 ID/count and indexed config query; no saved reply table or timing decoder
isp_profile.py, isp_t6040_25g76.json Boot segments, allocation geometry and seven channels; the fixture contains no camera-info or config replies
isp_vid.py, experiments/isp.py Older fixed 720p NV12 receiver
isp_cmd.py Legacy sensor-0x248 examples; temporary 3.75 FPS AE requests and a comment about older 30 FPS requests, not IMX958 per-config nominal rates
isp_command.py Serialized command ownership, replies and deadlines
src/isp.c Boot power/heap handoff, without a RAW client or mode table

The 13 runtime queries are documented, but their complete reply records are not saved in these files. The qualified helper requests 30/15 FPS AE limits. Neither that nor the sensor-clock field supplies a per-mode nominal-rate table. The kernel is therefore not given a guessed Bayer FOURCC, a fabricated mode list or invented nominal FPS.

Separate AOP map

The modern AOP map is 0x20 SPUApp, 0x21 wakehint, 0x22 audio, 0x23 voice trigger, 0x24 accelerometer, 0x25 gyroscope, 0x26 lid angle, 0x27 ALS, and 0x28-0x2b with unqualified roles. This differs from the legacy ENDPOINTS13 map. The checked audio path uses aop-audio on endpoint 0x22 and EPIC v2. It establishes LP readiness with an owned adpx buffer and lpai/pdm0/adpx/lai attachment and power requests, then attaches hpai, requests pw1, prepares stream-10 NS RX0 DMA, starts DMA and requests hpai pwrd. Matching reports and stable full copies precede RX stop and return through pw1 to idle. The camera helper issues no AOP command; these audio operations are not transplanted into apple-isp. The ISP camera channel/command maps remain below.

Linux formats and modes

Output Config Y/UV byte stride Contiguous sizeimage Rate evidence
1280x720 P010 5 2560 / 2560 2,764,800 Proxy requested AE 30/15; measured FPS pending
1920x1080 P010 5 3840 / 3840 6,220,800 One proxy frame; measured FPS pending
1280x720 NV12 5 1280 / 1280 1,382,400 Existing Linux mode; this candidate's measurement pending
1920x1080 NV12 5 1920 / 1920 3,110,400 #384 reports 30 fps; this candidate's measurement pending

P010 is V4L2_PIX_FMT_P010. Both NV12 and P010 use the standard single-planar V4L2 API, with one contiguous memory plane containing Y and UV. UV follows Y immediately. The proxy page-aligned each component allocation independently; Linux passes its actual contiguous extents to the firmware pool, so it does not introduce an undocumented inter-plane gap into V4L2 buffers.

The inherited multiplanar module parameter is removed. There are no driver-defined module/kernel parameters, new platform flags or per-platform Kconfig options. Applications select P010 explicitly with S_FMT through the shipped single-planar API. The inherited multi-planar hooks changed the queue type without updating cached vb2 plane semantics, so those hooks and the non-contiguous NV12M alias are removed. No default-format parameter was added.

Probe queries every advertised config, bounded at 64, using CH_CAMERA_CONFIG_GET with the 0x120-byte H17 record. A debug-gated dump preserves every reply, and VIDIOC_LOG_STATUS prints every complete cached reply, input/output sizes and the sensor-clock word without sending firmware commands. Collect that dump before a stress run; no probe-time parameter is needed. The sensor-clock word is not reported as FPS.

The two DT presets retain their order and NV12 default. Config-5 P010 is enabled when the runtime input geometry matches the source profile; an unspecified P010 size chooses 720p first. Additional distinct sizes are generated from the firmware's output dimensions only when both copies of the geometry agree, the dimensions are nonzero/even and within input, and the output is at most 16384 in either dimension. The reported full output becomes that config's crop. The 1920x2160 input height alone does not create a processed 2160-row mode.

Configurations with duplicate dimensions are retained in the diagnostic list but do not become independently selectable modes. Their nominal rates and any sensor-specific RAW/processed restrictions still need the runtime replies. This does not yet fulfil exhaustive capture of every firmware configuration. No 4:2:2 or full-range format is advertised without evidence for its firmware encoding.

ISP endpoints

These are ISP shared-memory channels, separate from the AOP endpoint map. Counts below come from the 25G76 source fixture. Linux negotiates the current descriptor addresses and counts on each boot.

Channel Type Source / doorbell bit Captured slots Traffic
TERMINAL 2 0 768 Firmware logs and terminal ACKs
IO 0 1 8 Host command payloads and in-place replies
DEBUG 0 1 8 Discovered, unused for capture
BUF_H2T 0 2 64 Metadata/output descriptor submissions
BUF_T2H 1 3 64 Buffer completion/return reports
SHAREDMALLOC 1 3 8 Firmware allocation and free requests
IO_T2H 1 3 8 Discovered, no capture handler installed

Descriptors are 0x100 bytes, with the u64 ring address at 0x50. Slots are 0x40 bytes, eight u64 words. Payload precedes arg0 ownership publication. The existing parser checks ring bounds and overlap against the channel table, boot/command storage and other rings. IO/DEBUG share one source bit; the three reply channels share another.

Command stream

An IO payload starts with a u32 zero/status and u32 opcode. Channel is zero. The source's discovery is CONFIG_GET (0x0003, 0x1c bytes), DSID setup (0x3206, 0x50), global START (0x0000, 0x0c), CH_INFO_GET (0x010d, 0x190), then every CH_CAMERA_CONFIG_GET (0x0106, 0x120).

DSID has eight slots, range 0x2fc, bases 0x2201d4000 / 0x2221d4000 / 0x2241d4000 / 0x2261d4000 and four U64_MAX sentinels. The proxy's global START argument is 0x11; the current working Linux H17 boot sends 0. This PR retains Linux's once-per-boot startup, including its existing print/PMP setup. It does not send a second global START when switching to P010.

The P010 channel setup then follows the proxy preview sequence:

Opcode Command Bytes Parameters after header
0x0024 FLICKER_SENSOR_SET 0x0c 0
0x013b CH_SBS_ENABLE 0x10 ch, 1
0x0107 CH_CAMERA_CONFIG_SELECT 0x10 ch, 5 for the two previews
0x010e CH_BUFFER_RECYCLE_MODE_SET 0x10 ch, 1, empty only
0x010f CH_BUFFER_RECYCLE_START 0x0c ch
0x080c CH_CROP_SCL1_SET 0x1c ch, 0, 0, width, height
0x0b09 CH_OUTPUT_CONFIG_SCL1_SET 0x38 ch, width, height, 1, 0x12, Y stride, UV stride, 0, 0, height, 0, width
0x0208 CH_AE_FRAME_RATE_MAX_SET 0x10 ch, FPS * 256
0x020a CH_AE_FRAME_RATE_MIN_SET 0x10 ch, minimum FPS * 256
0x0117 CH_BUFFER_POOL_CONFIG_SET, meta 0x9c pool 0, count 8, size/stride 0x8000, one block
0x0117 CH_BUFFER_POOL_CONFIG_SET, output 0x9c pool 9, count 2, actual Y/UV extents and strides, two blocks
0x0125 CH_LOCAL_RAW_BUFFER_ENABLE 0x10 ch, 1
0x0b0d CH_PREVIEW_STREAM_SET 0x10 ch, 1
0x0138 CH_MASTER_SLAVE_SYNC_MODE_SET 0x10 ch, 0

Local RAW here feeds the ISP's processed preview internally. It does not expose a Bayer V4L2 stream.

BUF_H2T submits metadata and output before CH_START (0x0100, 0x0c). Linux keeps its variable-length combined batch, using a 16-byte type/count header and 64-byte descriptors. The proxy's builders used separately padded small batches. Each descriptor carries four u64 addresses, four u32 flags, plane count, pool and a u64 tag. Used-plane flags are 0x40000000. The firmware's completed P010 descriptor returns both plane addresses as zero-extended low 32 bits. Linux accepts either the exact full submitted address pair or the documented zero-extended pair, and rejects mixed/unrelated upper bits. Pool 9, two planes and the fresh tag must also match the submitted lease. BUF_T2H render status 0x10000000 means complete; submit/return-ACK statuses are 0x30000000/0x80000000. CH_STOP (0x0101) and CH_BUFFER_RETURN (0x0104) finish the stream.

The proxy requested AE limits 7680/3840, or 30/15. Linux retains its frame-rate and exposure-priority controls. Default is fixed 30/30; enabling exposure_auto_priority permits 30/15. These are control limits, not nominal-rate measurements. P010 uses the minimal preview sequence without adding the NV12 path's setfile, CNR, MBNR, SIF-format or temporal-filter setup. Previously programmed NV12 tuning, including a loaded setfile, can remain resident across a format switch. The target run must validate those transitions; a matching command fixture does not prove the resulting firmware state or throughput.

25G76 versus the current 26.6.2 driver

The old profile's firmware identity differs. It is not used to admit the current firmware.

  • Both use a 0x290-byte boot block with the same populated field offsets, including camera scheme at 0xd0 and the no-optical-card flag at 0x120. Current H17 startup checks descriptor flag bit 1.
  • 25G76 pins TEXT/DATA sizes 0xca8000/0x1344000, remap 1<<40 and a 256 MiB shared aperture. Linux keeps the current reserved-memory layout, main-memory tail backing and 0xde600000 shared-size descriptor. No old segment size, heap request, IOVA or hash is transplanted.
  • The descriptor/slot widths, seven channel names, DSID layout, 0x190 channel info, scaler-1 crop/output, 0x9c pools and preview/local-RAW/sync/channel start/stop builders agree at source level. Global START's argument differs as described above. Config discovery adds the 0x120 H17 exchange; the unused legacy current-config helper still has its 0xdc layout. Appended H17 config bytes remain uninterpreted. The channel-info helper keeps the common 0x118-byte prefix of its 0x190-byte reply and does not replay the old preview validator's byte-0x138 check; scaler selection remains in the current T6040 descriptor.
  • Channel info reports metadata size 0x4d00. P010 uses the source's 0x8000 backing/pool size, eight metadata slots and two outputs. NV12 retains its existing pool parameters. T6040 has no capture-metadata pool or capture watchdog.
  • Linux keeps its three-DART IOMMU domain, bootloader address-filter setup, current reset-vector admission and resident-firmware lifecycle. It does not replay proxy DART tables, private power orchestration or the pinned profile's recovery scheme.

The fixed TEXT/DATA reservations in the J616s DT should eventually be supplied through a bootloader handoff for the actual loaded image and its mapping lifetimes. This PR does not implement that handoff or change the DT reservations.

Stalls and validation

The source fixes an empty-queue requeue that could leave the last IMX958 buffer pending, releases P010 output and metadata leases only after the completion ACK, acquires validity before reading a reply payload and bounds P010 ring drains so control/log traffic cannot starve buffer returns. P010 command replies must have the expected length, zero status and unchanged header. Timeout and malformed replies latch failure; an ACK at or after the deadline cannot advance the ring. The existing forced coprocessor stop runs before uncertain leases are released. Resident firmware then requires reboot.

These are source-visible corrections, not proof that the proxy's intermittent stalls have been reproduced or cured on 26.6.2.

  • ARM64 QEMU KUnit, 28/28 pass, including the 14-command preview bytes, failures at each command, geometry/config-list checks and failed-discovery cleanup, P010 storage/format checks, completion identity, 100,000 command-ring transactions, late-ACK cursor retention and 10,000 last-buffer requeues for each of P010/NV12.
  • The capture delivery runner builds with -Wall -Wextra -Werror. Six synthetic runner cases pass, including detection of a timing gap despite contiguous driver sequence numbers, stall/empty streams, substituted formats and failed STREAMOFF.
  • The full universal defconfig+asahi.config build passes with W=e and DRM_WERROR=y. The full Fedora-derived build passes with r5's published config and global CONFIG_WERROR=y. Both build Image, modules and DTBs on combined source 734a4018f3a6b5e17bc2afc322099c477b67a66d. Actual command receipts confirm C -Werror, Rust -Dwarnings and fatal warnings on all seven ISP objects. FORTIFY remains enabled; the complete-message IPC copy is covered by KUnit.
  • Both resolved profiles preserve all 127 requested Asahi settings. The published candidate matches every one of r5's 12,014 config settings, including the enabled power-allocator governor and global WERROR policy. The complete config comparison and input hashes are included.
  • The ISP binding check passes. J616s dtbs_check and dt-validate return zero but report existing PCI/Wi-Fi compatible and unevaluated-property findings. This branch changes no DTS or binding. Both supplied DTBs match r5 byte for byte; full logs are included. These results are not described as a clean global schema validation.
  • The installed ISP module matches its copy inside the archive, carries 7.3.0-aurora-j616s-r5-cam1 vermagic and reports no module parameters. The source-policy audit and warning-fatal RST parse pass.

Published test kernel

Release: 7.3.0-aurora-j616s-r5-cam1, source 734a4018f3a6b5e17bc2afc322099c477b67a66d, tree 433a4e9b6cb09bba55d0993b505303c5ab0503ed.

The local hardware-handoff artifact is published at:

/Volumes/AuroraS500/aurora-ctl/artifacts/j616s-camera-20261026/734a4018f3a6/

It contains Image/Image.gz, matching signed/stripped modules, J616s and unchanged Neo DTBs, the DTB archive, config/System.map, source pins and bundle, build recipes/logs, ARM64 KUnit results, C stress-runner source and the public-source audit. No m1n1 Python implementation, Apple binary or firmware payload is copied into this artifact set. SHA256SUMS covers 116 files; its SHA256 is 807e78513f75aa384fda1856232bf95efb2c6214ee2ca6a5532d07429fe03ea4. READY was written last after verification. The README includes explicit P010 long-run/cycle/switch commands. This replaces the earlier camera-only artifact for the combined display/camera run.

Hardware routing

The preliminary shared hw_request for device j616s, uname -r, returned ValueError: unknown device 'j616s', before creating a request ID. The installed controller documents Neo and M1 Pro/M2 reference targets, without a J616s alias. No target transport, boot or capture was attempted outside the controller. J616s routing must be available before the full qualification request can be queued. This artifact is ready for that handoff; it is not completed RAW or zero-drop qualification.

The manual stress runner enumerates offered formats/sizes, runs each for ten minutes, then repeated stop/restart and format-switch cycles. It records delivered FPS, maximum gap, sequence gaps and buffer errors and rejects substitutions. A driver-generated sequence cannot establish zero firmware-internal drops; that also needs a validated firmware frame counter or an independent rate oracle.

No target access was made by this worker. The J616s run must collect the 26.6.2 config replies first, verify both P010 images and measure each mode's sustained rate and gaps while audio is idle. The current RAW packing/order, independently selectable duplicate configs, exhaustive nominal FPS and zero-drop sensor performance remain open. No Apple binary, setfile contents or copied firmware payload is included.

Port the processed config-5 preview command sequence from our m1n1 J616s work, with contiguous V4L2 P010 strides and pool geometry. Query all advertised H17 sensor configs and retain the board defaults while appending consistent output modes. Keep NV12 the default and scope the new path to T6040/IMX958.

Retire output and metadata leases after the report ACK, restart an empty P010 queue on requeue, bound ring servicing and latch malformed or timed-out command replies. Add command-byte, geometry, ownership and ring-wrap coverage. These source corrections still require sustained capture validation on 26.6.2.

Builds on Jonathan Goyvaerts’ T6040 camera support in PR #384. Protocol reference: our m1n1 PR #20 at 9be380c8a5 and Linux PR #231 at a4e7ee8. No firmware payload is included.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Exercise every offered format and discrete size with long MMAP capture, repeated stream restarts and format switches. Record delivered rate, timing gaps and buffer errors, and fail on substitutions or stalls. Allow a mode filter for different nominal rates. A driver-generated sequence alone is not evidence of zero firmware drops.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Check the command deadline before advancing the ring cursor, including when an ACK arrives at the wait timeout boundary. Keep the failed channel latched until coprocessor stop. Cover the boundary in KUnit. Emit the capture stress result after cleanup so a failed STREAMOFF cannot leave a passing result.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Apply the empty-queue restart to both processed IMX958 formats. Exercise 10,000 empty-to-submitted transitions for each of P010 and NV12. Other sensors retain their queue policy.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Size the command table columns for the full H17 record sizes and fix the document title underline. The standalone RST parser now accepts the document with warnings treated as errors.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The buffer wire carries 64-bit addresses, while the proxy observed zero-extended low words on its firmware. Accept either complete canonical pair for a submitted lease and reject mixed or unrelated upper bits. Keep pool, plane and fresh-tag checks. Also assert that NV12 stays first in format enumeration.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Remove the inherited multiplanar module parameter. T6040/IMX958 supports normal V4L2 negotiation for contiguous NV12 and P010 through either API, with one memory plane containing the two ISP components. Keep NV12 first and initial; other sensors retain their shipped single-planar API. Do not advertise the old non-contiguous NV12M alias.

Cache complete sensor configuration replies with the video state and print them on the standard log-status ioctl, so collecting the firmware list needs no driver or boot parameter. Free the cache on probe failure and final video release, and test failed discovery without publishing partial state. No module/kernel parameters or platform Kconfig options are added.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Acquire the validity word, copy the complete message object after the DMA read barrier and restore that captured word. Copying seven payload words through the arg1 field violates FORTIFY field bounds. Cover all eight copied words and keep the ownership ordering unchanged.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Remove the parameter-gated multi-planar hooks, which changed the queue type without reinitialising its cached plane semantics. Both contiguous NV12 and P010 use the existing single-planar queue. Keep format negotiation explicit through S_FMT.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Credit Ryan\x27s m1n1 PR #20, aurora-silicon/m1n1#20, at 9be380c8a5a94609fdbe62d693c7b2121478ad8f for the processed preview protocol. Record what each requested file establishes, the diagnostic RAW snapshot limitation and the missing per-config timing records. Distinguish the separate AOP audio sequence from ISP camera configuration.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Correct the reStructuredText underline and verify the complete document with warnings treated as errors. Source protocol credit: Ryan Murray, m1n1 PR #20, aurora-silicon/m1n1#20, at 9be380c8a5a94609fdbe62d693c7b2121478ad8f.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
@RyanTheTide
RyanTheTide merged commit 99a7967 into integration/aurora Oct 28, 2026
@RyanTheTide
RyanTheTide deleted the j616s/camera-sensor-modes branch October 28, 2026 05:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant