Preserve forked native conditioning in multi-history tokenization - #983
Draft
bradhilton wants to merge 22 commits into
Draft
bradhilton wants to merge 22 commits into
bradhilton wants to merge 22 commits into
Conversation
bradhilton
marked this pull request as draft
September 26, 2026 15:46
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Complete recorded exchanges can still fail tokenization when a later native prompt is not an extension of an earlier response's full prompt/output sequence. The existing
multi_history=Trueand internal exchange-training paths now split that history into contiguous native streams, preserving each original conditioned token sequence, logprobs and first sampled ownership. Single-linear APIs retain their strict refusal. No new option is introduced.Stacked on #977 at
02ca697e7. Existing successful native paths remain first. Refinement requires complete unchanged Chat provenance, exhaustive source-span ownership and authoritative STOP proof. Rendered request-owned ASSISTANT/STOP spans require a sufficient complete recorded prompt; later sampled native bodies retain their recorded authority. A request-owned assistant with no rendered extent adds no guessed role flags. Original and expanded histories retain consumed-evidence/context ledgers and callback authority through final validation. Explicit rendering overrides and incomplete or edited provenance do not authorize refinement.Validation:
f24d63f86has 71 passing affected controls and passes lint, formatting and types. Independent composition review verifies all 159 shared parent definitions, the full module inverse, and unchanged six stream helpers/five dispatchers. It receives the parent's decoder-admission and scalar/Enum-slot corrections exactly. Fresh exact-head Astra/Fable reviews and canonical CI are running; predecessor clearances are not current-head approval.1886e0d3verifies 31 encounters/30 unique sources, three histories, 4,139 finite first-owned terms and 29 sampled STOPs. Ef339 at1c3bade7verifies 11 original sources, 24,026 finite terms and six STOPs, with serialized result and all four masks equal to the earlier successful result. These remain predecessor invocations, not current-head captured acceptance. Their ordinary-input applicability and unchanged native emission/source-key helpers are recorded separately from the new affected public controls; no private replay was repeated.The captured normalization check has one trajectory and zero centered advantage, so it does not establish original-group gradient equivalence. Earlier responses outside a stream become request context; universal parity with every old generic
OUTPUT/SFT layout is not claimed. Extra streams may repeat context. Generic rendering overhead in the parent remains unresolved; a separate ledger optimization is not included. No production speed, GPU-memory, model-update or fresh learning result is claimed. Current experiments and frozen pins remain unchanged.