Repository navigation
A frame-accurate video seeks one frame at a time - #511
Merged
Merged
Conversation
FrameAccurateVideoSource.seekTo decodes through one VideoDecoder and collects its frames through one handler, and two calls could overlap: a render loop seeking every frame started a decode before the last one ended, so they stole or dropped each other's frames. Decodes now run one at a time through latestOnly: a seek made during a decode replaces the next run's target, every call resolves once its frame or a newer one is drawn, and a seek to the frame already drawn decodes nothing. A capture, awaiting each seek, is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug.
FrameAccurateVideoSource.seekTodecodes through oneVideoDecoderand collects that decode's frames through one handler (onFrame). Each call installs its own handler and sets it back tonullwhen it ends, and nothing stops two calls from overlapping. A render loop that writesprops.currentTimeevery frame (live playback, which, unlike a capture, does not await the decode) starts a seek before the previous one finishes. The two then steal or drop each other's frames, so the video shows frames out of order, and the GOP decodes pile up on the main thread until the clock crawls.Captures never showed it: the capture loop awaits
pendingDecodesbefore every paint, so its seeks never overlap.The fix.
latestOnly(new,src/video/latestOnly.ts): runs an async job for the latest target only, one run at a time. A call made during a run records its target; when the run ends it goes straight to the newest target, skipping the ones in between.seekTo=latestOnly(decodeAt), wheredecodeAtis the old body.drawnIndex).Tests.
latestOnly.test.tscovers: one run at a time; replaced targets skipped and the latest always reached; a call settling only once its target or a newer one is done; every target run when each call is awaited (the capture path); a fresh run after the last one ended; and rejection plus recovery. The package's 162 tests, typecheck, lint and build pass.Measured. A live program seeking two 1080p H.264 videos on every frame, played in
vos previewunder 4× CPU throttling, sampled every 250 ms against the frame-exact render of the same program:Exports unchanged, measured. The same program rendered with
vos renderon the published package and on this branch: the decoded video is identical, all 632 frames (the same MD5 over the decoded picture stream, infinite PSNR on every frame).Not in this PR. Each seek still decodes its whole GOP. A forward step through the same GOP could reuse the decoder's state or a small frame cache instead of re-decoding from the keyframe; that's a separate change with its own memory trade-off.