rtsp: send interleaved frames in one write, stamp per frame, add G.711

[kasperiio]

On a small SoC (ARM1176 at 600 MHz) the RTSP sender spent more CPU on send() syscalls and on spinning over partial writes than the encoder spent encoding. In Frigate this showed as a stream that lagged and then fast-forwarded.

• One write per frame. Interleaved (TCP) packets are staged in a per-connection buffer and flushed once per frame or per 64 KB. A send() costs roughly 100 µs here regardless of size, and a frame is many packets.
• 512 KB socket send buffer, enough to absorb one keyframe. The kernel default (64 KB on these boards) cannot hold one, so the sender spun on EAGAIN for as long as the link took to drain — hundreds of ms of CPU per keyframe. SO_SNDBUFFORCE with a fallback to SO_SNDBUF.
• poll() instead of usleep(1000) on a partial write.
• One timestamp per frame, taken from the encoder's capture time. Previously only the marker packet was stamped, so every earlier packet of a frame carried the previous frame's time: receivers saw timestamps go backwards within a frame, and players stalled and then raced to catch up. Falls back to send time for HALs that leave the pack timestamp at zero.
• G.711 A-law (PT 8) as an alternative to MP3 for RTSP audio, selected by rtsp.audio_codec. The SDP now advertises 8000 Hz for PT 0/8 rather than 90000. This also allows the software MP3 encoder — the single most expensive thing on this SoC, ~24 % of a core at 48 kHz — to be skipped entirely when nothing consumes MP3.
• MP4 muxer: does not build moof/mdat when no HTTP client is connected (parameter sets are still cached, so a header is ready on connect), and resets its cached header when the stream configuration changes — without that a codec switch kept sending the old one.
• /api/rtsp and /api/onvif expose those settings, with the matching web UI section.
• bitbuf: memcpy instead of a byte-at-a-time loop.

Testing

Built for fh8856v100_lite (ARMv6). Runs as part of the combined branch this was split from, on three Fullhan FH8856 cameras — RTSP with both MP3 and G.711 audio, ONVIF, MP4 and MJPEG over HTTP. This branch on its own is build-tested, not separately run on hardware.

Sits on top of the four bug fixes in the companion PR; independent of the HAL PR.

[qodo-free-for-open-source-projects]

PR Summary by Qodo

Optimize RTSP frame delivery and add configurable G.711 audio

🐞 Bug fix ✨ Enhancement ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Batches interleaved RTP writes and uses capture timestamps to prevent lag on constrained devices.
• Adds configurable G.711 audio while bypassing unnecessary MP3 and MP4 processing.
• Exposes RTSP and ONVIF settings through HTTP APIs and the web interface.

Diagram

graph TD
  UI["Settings UI"] --> API["Settings APIs"] --> CFG["App Config"] --> MEDIA["Media Pipeline"]
  MEDIA -->|capture time| TS["Frame Timestamp"] --> RTP["RTP Sender"] --> BATCH["TCP Batch"] --> CLIENT["RTSP Client"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Frame-scoped writev batching

• ➕ Avoids copying RTP headers and payloads into a staging buffer.
• ➕ Can retain one syscall per frame when the iovec count stays within platform limits.
• ➖ Requires careful partial-write bookkeeping across many iovec entries.
• ➖ Packet lifetimes and IOV_MAX limits complicate large keyframe handling on embedded platforms.

2. Asynchronous socket writer

• ➕ Separates media production from network backpressure.
• ➕ Can queue, prioritize, or drop frames without blocking encoder threads.
• ➖ Adds queue ownership, memory limits, shutdown coordination, and latency policy.
• ➖ Represents a substantially larger architectural change than the observed bottleneck requires.

Recommendation: Keep the PR's bounded per-connection staging buffer, enlarged socket buffer, and poll-based backpressure handling. It directly addresses syscall and EAGAIN costs with limited architectural disruption; consider writev only if profiling shows the added memcpy has become significant.

Files changed (13) +337 / -11

Enhancement (8) +325 / -8

index.htmlAdd RTSP and ONVIF network settings controls +33/-0

Add RTSP and ONVIF network settings controls

• Loads RTSP and ONVIF settings during page initialization. Adds controls for service state, authentication, RTSP port, and selectable MP3 or G.711 A-law audio.

res/index.html

media.cAdd G.711 A-law output and conditional MP3 encoding +45/-1

Add G.711 A-law output and conditional MP3 encoding

• Downsamples PCM to 8 kHz, encodes 20 ms G.711 A-law packets, and routes them to RTSP when selected. Skips software MP3 encoding when no recording, streaming, HTTP, RTMP, or MP3-based RTSP consumer needs it.

src/media.c

rtp.cBatch RTP writes and stamp complete frames consistently +126/-5

Batch RTP writes and stamp complete frames consistently

• Stages interleaved RTP packets per connection, flushes once per frame or 64 KB, and waits with 'poll()' on socket backpressure. Uses capture-derived timestamps across every video-frame packet and adds payload type 8 G.711 packetization with an 8 kHz clock.

src/rtsp/rtp.c

rtsp.cConfigure RTSP clocks and larger socket send buffers +20/-2

Configure RTSP clocks and larger socket send buffers

• Advertises 8 kHz SDP clocks for static G.711 payload types while retaining 90 kHz for MPEG audio. Allocates connection staging buffers and requests 512 KB send buffers for RTP and accepted RTSP sockets.

src/rtsp/rtsp.c

rtsp.hAdd per-connection interleaved transmit storage +5/-0

Add per-connection interleaved transmit storage

• Extends RTSP connection state with a staged transmit buffer and length. Defines a 64 KB batch threshold for interleaved TCP output.

src/rtsp/rtsp.h

rtsp_server.hExpose the G.711 RTP sender +1/-0

Expose the G.711 RTP sender

• Declares the public RTSP server entry point for sending PCMA audio packets.

src/rtsp/rtsp_server.h

server.cExpose network APIs and avoid unused media processing +94/-0

Expose network APIs and avoid unused media processing

• Adds '/api/rtsp' and '/api/onvif' handlers and detects HTTP consumers before running MP3 or fragmented MP4 work. Continues caching parameter sets without MP4 clients, and applies an HTTP socket send timeout to prevent stalled clients from blocking the server.

src/server.c

server.hExpose HTTP audio consumer detection +1/-0

Expose HTTP audio consumer detection

• Declares the helper used by the media thread to determine whether HTTP MP3 or MP4 clients require encoded audio.

src/server.h

Bug fix (2) +6 / -1

moov.cCorrect MP3 object type for MPEG-1 sample rates +2/-1

Correct MP3 object type for MPEG-1 sample rates

• Emits the MPEG-1 audio object type indication for MP3 streams sampled at 32 kHz or above, improving MP4 decoder configuration accuracy.

src/fmt/moov.c

mp4.cInvalidate cached MP4 metadata on configuration changes +4/-0

Invalidate cached MP4 metadata on configuration changes

• Clears the cached header and video parameter sets when stream configuration changes. This prevents codec or dimension changes from reusing stale initialization metadata.

src/fmt/mp4.c

Refactor (1) +2 / -2

bitbuf.cReplace byte-wise buffer copying with memcpy +2/-2

Replace byte-wise buffer copying with memcpy

• Uses 'memcpy' when writing data at a bit-buffer offset, reducing per-byte processing overhead.

src/fmt/bitbuf.c

Other (2) +4 / -0

app_config.cPersist and parse the RTSP audio codec +3/-0

Persist and parse the RTSP audio codec

• Adds 'rtsp.audio_codec' to configuration defaults, parsing, and YAML serialization. MP3 remains the upstream default while deployments may select PCMA.

src/app_config.c

app_config.hStore the configured RTSP audio codec +1/-0

Store the configured RTSP audio codec

• Extends 'AppConfig' with storage for the 'pcma' or 'mp3' RTSP audio codec value.

src/app_config.h

[qodo-free-for-open-source-projects]

Code Review by Qodo

🐞 Bugs (5) 📘 Rule violations (0) 📜 Skill insights (0)

1. Audio-only TCP never flushes 🐞 Bug ≡ Correctness

Description

__tcp_flush_each() checks the video track's TCP state even when flushing an audio transfer.
Clients that SETUP only track 1 over interleaved TCP accumulate audio in tx_buf but never receive
it.

Code

src/rtsp/rtp.c[69]

+    if (!con->trans[0].is_tcp || !con->tx_buf || !con->tx_len) return SUCCESS;

Evidence

SETUP records TCP state on the requested track, and packet staging uses that track's state. The new
flush callback instead tests trans[0], so an audio-only track-1 connection is skipped before
rtp_send_pcma() or rtp_send_mp3() destroys its temporary transfer list.

src/rtsp/rtp.c[63-73]
src/rtsp/rtp.c[213-258]
src/rtsp/rtp.c[552-575]
src/rtsp/rtsp.c[411-463]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Audio-only interleaved RTSP sessions never flush staged RTP because the flush callback always checks track 0.

## Issue Context
The callback already has the current `transfer_item_t`; use its associated track or remove the incorrect video-track gate while retaining the buffer checks and locking.

## Fix Focus Areas
- src/rtsp/rtp.c[63-73]
- src/rtsp/rtp.c[213-258]
- src/rtsp/rtp.c[552-575]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Codec switch corrupts sessions 🐞 Bug ≡ Correctness

Description

The restart-only API setting immediately changes the codec branches read by aenc_thread().
Existing sessions negotiated for PT 8/8 kHz or PT 14/90 kHz consequently start receiving the other
payload type and clock without a new DESCRIBE/SETUP.

Code

src/server.c[R1223-1224]

+                else if (EQUALS(key, "audio_codec"))
+                    strncpy(app_config.rtsp_audio_codec, value, sizeof(app_config.rtsp_audio_codec) - 1);

Evidence

The endpoint writes directly to global configuration, while the running encoder thread evaluates
that field for every frame. The selected send routines mutate audioPt between 8 and 14 and use
different RTP clocks, although RTSP advertises a single codec during DESCRIBE.

src/server.c[1223-1235]
src/media.c[91-92]
src/media.c[118-126]
src/rtsp/rtp.c[552-601]
src/rtsp/rtsp.c[181-195]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Updating the restart-only codec setting changes active RTP output immediately and invalidates negotiated sessions.

## Issue Context
Keep the active RTSP codec immutable for the server lifetime, restart the RTSP/audio pipeline atomically, or reject live changes while sessions are active.

## Fix Focus Areas
- src/server.c[1223-1235]
- src/media.c[91-92]
- src/media.c[118-126]
- src/rtsp/rtp.c[552-601]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. PCMA resampling runs fast 🐞 Bug ≡ Correctness

Description

rtsp_pcma_feed() truncates audio_srate / 8000, so a supported rate such as 44.1 kHz emits 8,820
samples per second instead of 8,000. The resulting 160-byte packets represent about 18.1 ms while
RTP timestamps and SDP declare an 8 kHz clock.

Code

src/media.c[R61-65]

+    if (!dec) dec = app_config.audio_srate / 8000 > 0 ? app_config.audio_srate / 8000 : 1;
+    for (unsigned int i = 0; i < samples; i++) {
+        acc += pcm[i];
+        if (++accN < dec) continue;
+        alaw[fill++] = pcm_to_alaw((short)(acc / (int)dec));

Evidence

Configuration accepts every integer rate from 8,000 through 96,000 Hz, but the loop emits one sample
after each truncated integer dec input samples. For 44,100 Hz, dec is 5 and therefore produces
44,100/5 = 8,820 PCMA bytes per second against the advertised 8 kHz clock.

src/media.c[54-71]
src/app_config.c[457-462]
src/rtsp/rtp.c[552-575]
src/rtsp/rtsp.c[191-195]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Integer-factor decimation does not produce 8 kHz PCMA for supported non-multiple input rates.

## Issue Context
Use a fractional phase accumulator or resampler that emits exactly 8,000 samples per second for every accepted capture rate, and derive RTP timestamp progression from emitted samples.

## Fix Focus Areas
- src/media.c[54-71]
- src/rtsp/rtp.c[552-575]
- src/app_config.c[457-462]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

View action required (1)

4. Codec YAML overflows buffer 🐞 Bug ⛨ Security

Description

The new 8-byte rtsp_audio_codec array is passed to the unbounded parse_param_value(). A YAML
value longer than seven characters writes beyond the field and can corrupt adjacent application
configuration during startup.

Code

src/app_config.c[437]

+    parse_param_value(&ini, "rtsp", "audio_codec", app_config.rtsp_audio_codec);

Evidence

The destination introduced by this PR has capacity eight, while parse_param_value() uses
sprintf(param_value, "%.*s", ...) with the full matched YAML length and then writes a terminator
without knowing the destination size.

src/app_config.h[60-63]
src/app_config.c[435-437]
src/hal/config.c[58-98]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Parsing a long `rtsp.audio_codec` YAML value overflows the newly added eight-byte field.

## Issue Context
Introduce a size-aware parser or parse into a sufficiently sized temporary buffer, validate against exactly `pcma` and `mp3`, and only then copy into the configuration field.

## Fix Focus Areas
- src/app_config.c[435-437]
- src/app_config.h[60-63]
- src/hal/config.c[58-98]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

5. API emits invalid JSON 🐞 Bug ≡ Correctness

Description

The new endpoints interpolate decoded usernames and codec values directly into quoted JSON strings
without escaping them. Quotes, backslashes, or control characters therefore make the response
invalid and cause the web UI's JSON.parse() to fail.

Code

src/server.c[R1232-1235]

+            "{\"enable\":%s,\"enable_auth\":%s,\"port\":%d,\"auth_user\":\"%s\",\"audio_codec\":\"%s\","
+            "\"note\":\"port and codec changes apply after restart\"}",
+            app_config.rtsp_enable ? "true" : "false", app_config.rtsp_enable_auth ? "true" : "false",
+            app_config.rtsp_port, app_config.rtsp_auth_user, app_config.rtsp_audio_codec);

Evidence

Both handlers accept arbitrary URI-decoded strings and insert them directly between JSON quotation
marks. The UI parses each response with JSON.parse(), so an accepted value such as a username
containing " breaks subsequent configuration reads.

src/server.c[1219-1235]
src/server.c[1251-1264]
res/index.html[184-224]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
User-controlled RTSP and ONVIF values are emitted as raw JSON string contents.

## Issue Context
Serialize responses with a JSON encoder or correctly escape quotation marks, backslashes, and control characters before formatting both endpoint responses.

## Fix Focus Areas
- src/server.c[1219-1235]
- src/server.c[1251-1264]
- res/index.html[184-224]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Tip of the day

💡 Did you know, you can keep summaries lean with Finding overflow, which tucks the rest behind 'View more'

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[qodo-free-for-open-source-projects]

1. Audio-only tcp never flushes 🐞 Bug ≡ Correctness

__tcp_flush_each() checks the video track's TCP state even when flushing an audio transfer.
Clients that SETUP only track 1 over interleaved TCP accumulate audio in tx_buf but never receive
it.

Agent Prompt

## Issue description
Audio-only interleaved RTSP sessions never flush staged RTP because the flush callback always checks track 0.

## Issue Context
The callback already has the current `transfer_item_t`; use its associated track or remove the incorrect video-track gate while retaining the buffer checks and locking.

## Fix Focus Areas
- src/rtsp/rtp.c[63-73]
- src/rtsp/rtp.c[213-258]
- src/rtsp/rtp.c[552-575]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[qodo-free-for-open-source-projects]

2. Codec switch corrupts sessions 🐞 Bug ≡ Correctness

The restart-only API setting immediately changes the codec branches read by aenc_thread().
Existing sessions negotiated for PT 8/8 kHz or PT 14/90 kHz consequently start receiving the other
payload type and clock without a new DESCRIBE/SETUP.

Agent Prompt

## Issue description
Updating the restart-only codec setting changes active RTP output immediately and invalidates negotiated sessions.

## Issue Context
Keep the active RTSP codec immutable for the server lifetime, restart the RTSP/audio pipeline atomically, or reject live changes while sessions are active.

## Fix Focus Areas
- src/server.c[1223-1235]
- src/media.c[91-92]
- src/media.c[118-126]
- src/rtsp/rtp.c[552-601]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[qodo-free-for-open-source-projects]

3. Pcma resampling runs fast 🐞 Bug ≡ Correctness

rtsp_pcma_feed() truncates audio_srate / 8000, so a supported rate such as 44.1 kHz emits 8,820
samples per second instead of 8,000. The resulting 160-byte packets represent about 18.1 ms while
RTP timestamps and SDP declare an 8 kHz clock.

Agent Prompt

## Issue description
Integer-factor decimation does not produce 8 kHz PCMA for supported non-multiple input rates.

## Issue Context
Use a fractional phase accumulator or resampler that emits exactly 8,000 samples per second for every accepted capture rate, and derive RTP timestamp progression from emitted samples.

## Fix Focus Areas
- src/media.c[54-71]
- src/rtsp/rtp.c[552-575]
- src/app_config.c[457-462]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[qodo-free-for-open-source-projects]

4. Codec yaml overflows buffer 🐞 Bug ⛨ Security

The new 8-byte rtsp_audio_codec array is passed to the unbounded parse_param_value(). A YAML
value longer than seven characters writes beyond the field and can corrupt adjacent application
configuration during startup.

Agent Prompt

## Issue description
Parsing a long `rtsp.audio_codec` YAML value overflows the newly added eight-byte field.

## Issue Context
Introduce a size-aware parser or parse into a sufficiently sized temporary buffer, validate against exactly `pcma` and `mp3`, and only then copy into the configuration field.

## Fix Focus Areas
- src/app_config.c[435-437]
- src/app_config.h[60-63]
- src/hal/config.c[58-98]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[qodo-free-for-open-source-projects]

5. Api emits invalid json 🐞 Bug ≡ Correctness

The new endpoints interpolate decoded usernames and codec values directly into quoted JSON strings
without escaping them. Quotes, backslashes, or control characters therefore make the response
invalid and cause the web UI's JSON.parse() to fail.

Agent Prompt

## Issue description
User-controlled RTSP and ONVIF values are emitted as raw JSON string contents.

## Issue Context
Serialize responses with a JSON encoder or correctly escape quotation marks, backslashes, and control characters before formatting both endpoint responses.

## Fix Focus Areas
- src/server.c[1219-1235]
- src/server.c[1251-1264]
- res/index.html[184-224]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools