Skip to content

A frame-accurate video seeks one frame at a time - #511

Merged
hblee12294 merged 1 commit into
mainfrom
fix-frame-accurate-seek-reentrancy
Oct 29, 2026
Merged

hblee12294 merged 1 commit into
mainfrom
fix-frame-accurate-seek-reentrancy

Conversation

@hblee12294

Copy link
Copy Markdown
Member

The bug. FrameAccurateVideoSource.seekTo decodes through one VideoDecoder and collects that decode's frames through one handler (onFrame). Each call installs its own handler and sets it back to null when it ends, and nothing stops two calls from overlapping. A render loop that writes props.currentTime every frame (live playback, which, unlike a capture, does not await the decode) starts a seek before the previous one finishes. The two then steal or drop each other's frames, so the video shows frames out of order, and the GOP decodes pile up on the main thread until the clock crawls.

Captures never showed it: the capture loop awaits pendingDecodes before every paint, so its seeks never overlap.

The fix.

  • latestOnly (new, src/video/latestOnly.ts): runs an async job for the latest target only, one run at a time. A call made during a run records its target; when the run ends it goes straight to the newest target, skipping the ones in between.
  • Resolution guarantee: every call's promise settles once a run has completed for a target at least as new as its own. A caller that awaits a seek still sees its frame (or a newer one) on the canvas.
  • seekTo = latestOnly(decodeAt), where decodeAt is the old body.
  • No-op seeks: a seek to the frame already on the canvas decodes nothing (drawnIndex).
  • Unchanged: a capture, which awaits every seek, decodes exactly what it did before.

Tests. latestOnly.test.ts covers: one run at a time; replaced targets skipped and the latest always reached; a call settling only once its target or a newer one is done; every target run when each call is awaited (the capture path); a fresh run after the last one ended; and rejection plus recovery. The package's 162 tests, typecheck, lint and build pass.

Measured. A live program seeking two 1080p H.264 videos on every frame, played in vos preview under 4× CPU throttling, sampled every 250 ms against the frame-exact render of the same program:

  • Published 0.12.1: 5 of 40 samples were whole wrong frames (mean pixel difference 39–49).
  • This branch: none (max 15.6, the sampler's timing during fast camera moves, present in both runs).

Exports unchanged, measured. The same program rendered with vos render on the published package and on this branch: the decoded video is identical, all 632 frames (the same MD5 over the decoded picture stream, infinite PSNR on every frame).

Not in this PR. Each seek still decodes its whole GOP. A forward step through the same GOP could reuse the decoder's state or a small frame cache instead of re-decoding from the keyframe; that's a separate change with its own memory trade-off.

FrameAccurateVideoSource.seekTo decodes through one VideoDecoder and collects
its frames through one handler, and two calls could overlap: a render loop
seeking every frame started a decode before the last one ended, so they stole
or dropped each other's frames. Decodes now run one at a time through
latestOnly: a seek made during a decode replaces the next run's target, every
call resolves once its frame or a newer one is drawn, and a seek to the frame
already drawn decodes nothing. A capture, awaiting each seek, is unchanged.
@hblee12294
hblee12294 merged commit c9afc7a into main Oct 29, 2026
3 checks passed
@hblee12294
hblee12294 deleted the fix-frame-accurate-seek-reentrancy branch October 29, 2026 17:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant