Repository navigation
server: preserve controls and CJK boundaries in long-form speech - #5
Draft
krunkosaurus wants to merge 1 commit into
Draft
krunkosaurus wants to merge 1 commit into
krunkosaurus wants to merge 1 commit into
Conversation
Keep complete special tokens, bracketed cues and nested paired tags intact across sentence, clause, space and code-point splitting. Protect topic-shift preprocessing and recognize CJK boundaries without changing prompt or frame ceilings. Exercise exact whole and streamed prompt segments, oversized spans, malformed controls and redraws. Drain inherited socket fixtures before joining so CPU validation completes on macOS. Refs: plotarmordev/drama-tensorfold#25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Long-form speech currently splits paired directions and bracket cues across synthesis requests. For example, a 20-character target cuts
<whisper>One. Two. Three.</whisper>into unmatched pieces. This patch preserves complete special tokens, balanced brackets and nested simple tags; it also recognizes CJK sentence/clause punctuation and prevents paragraph markers inside controls from forcing cuts.Segments remain slices of the normalized input, preserving redraw offsets and join gaps. Only an indivisible control may exceed the character target; prompt, segment and frame limits still apply. This changes the CUDA NVFP4 long-form route on
drama-nvfp4. Tracks recipe issue 25.The behavioral reference is Sogni-AI/drama3-mlx chunking. Implementation and regressions extend native TensorFold independently; no Fish-derived Python implementation is copied.
Receipt
Base
b1f4b9b4830c9bd43bf7c432cffe7dea21313537; Zig 0.17.0, macOS arm64. CPU tests use synthetic code/PCM fixtures.zig build test-server-cpu: 364 passed, 4 skipped, 0 failed.zig build test-golden -Dcpu=apple_m1: 309/319 equal, 10 known differences, 0 unexpected.std.c.O.DIRECTuse, reproduced against the exact base.Draft qualification limit: no CUDA device/model inference, audio identity, acoustic listening or performance qualification was run. No quality/speed claim. The local full-model launcher admission floor was not met; its guard was not overridden.
Integration
This is the first legacy source contribution in a short chunking → sampling → dialogue-speakers stack. The protected special-token boundary is a prerequisite for the later dialogue hard-cut case. Independent review found no remaining native blocker; the recipe checksum-coverage finding is corrected in the recipe companion.
Pinned-engine packaging and user documentation: drama-tensorfold companion PR.