Repository navigation
lanes: a lone stream holds drafts only while its drafted rounds measure faster - #1382
Closed
BlivionIaG wants to merge 10 commits into
Closed
BlivionIaG wants to merge 10 commits into
BlivionIaG wants to merge 10 commits into
Conversation
engine.zig is 592 lines and the shared-round guard needs room in it. pipelinedRound and landInflight move unchanged to pipelined.zig as round and land; Engine.familyRound and Result become public for them. No behaviour change.
…el.shared_guard) A drafter whose hold is a pass of its own costs a shared round more than the depth rule's head steps price: the rule treats the rest of a round's time as overhead that plain rounds pay too. Facts.shared_guard (and Model.shared_guard for a backend) lets the engine compare the two modes directly. shared_guard.zig keeps, for each stream count, committed tokens over the round's wall time in plain stretches (no budgets, so no holds) and in drafted stretches. It starts plain, measures plain rounds for 16 rounds and drafted ones for 4, runs the faster mode for 512 rounds and doubles that, up to 64 times, while the same mode wins again; then it measures the other mode again. Drafted rounds must beat plain ones by 5%. A mode's first round still carries the other mode's drafts or holds and is not measured. An entry is allocated once a stream count; later rounds take no allocation. A plain stretch changes only how many drafts a window carries, never what the target draws, so replies stay the same tokens. Without the flag the engine never consults the guard. Tests: the guard's choice and re-measures, ever rarer re-measures of a losing mode, its skipped first rounds, no allocation after a stream count's first round, and replies equal to one-token decode with the guard on and off.
… stream joining one holds no drafts With the shared-round guard on, a plain round still verified the lane core's context and Recall copies, which are drafts too, priced from a prior; and a stream joining other streams held its first drafts even while the guard ran plain rounds at that stream count, a hold pass for each join. The engine now marks a shared round plain when the guard runs that stream count plain, and copyProposal proposes nothing for it, so the guard's plain mode is the drafts-off round. Open.firstDepth gives no drafts to a stream that joins live streams while the guard runs their count plus one plain. Both are behind Model.shared_guard; without it plain_round stays false and firstDepth is unchanged. Tests: a guarded plain round proposes no copies; a stream joining plain-guarded rounds holds no first drafts and, without the flag, holds them as before.
… holds follow the grants A drafter that gives each held chain draft's chance (Drafter.VTable.chances) may set Facts.chances. Drafted.facts then sets Model.draft_probabilities and Model.granted_depth, so the window allocator prices each stream's verify rows by those chances, as it does a head's node chances. With Facts.confidence_floor it instead verifies the drafts at least that likely (a fixed window when 0). With granted_depth a stream's next hold is as deep as its last grant plus a slack: one more draft when the grant took every held draft, none when it stopped short of them (Stream.grant_short), and after a grant of none a probe of draft_slack drafts at the depth rule's probe interval. Facts.slack sets one slack for all three. A shared round's granted depths are cut so the drafting streams' blocks fit one hold pass of batch_rows. When the shared-round guard starts a drafted stretch, its first round grants every stream one draft, so the stretch measures drafted rounds rather than plain ones left by grants of none. All of it is behind Facts.chances and Model.granted_depth, off by default; windows.allocate records grant_short only for granted depths. Tests: chances price the chain's rows and drafted replies equal one-token decode alone and at 1, 4 and 8 shared streams, with the scheduled verify ragged across streams and verifying fewer rows than a fixed window; the facts reach the lane config; granted depths, their slacks and the one-pass cut; holds as wide as the grants.
…py, and holds kept only while they pay The depth rule asks one head draft for a stream with a copy ahead, and a shared round's granted holds were priced only by their verify rows. A drafter whose hold runs as a pass of its own (Facts.copy_first) pays that pass for the one draft, and the copy round then drops it; Model.copy_first now asks it for none. A drafter may also give its hold pass's timed cost by blocks and depth (Facts.hold_costs, priced by hold_price.zig). With granted depths a shared round then keeps granted holds, most expected drafts a row first, only while each raises the next round's expected tokens a ms over its verify rows and the pass; a probe after a grant of none keeps its hold. A stream's last grant records the drafts it expected to keep (Stream.grant_gain). Both are off by default (copy_first false, hold_costs empty). Tests: with copy_first no hold is asked for a copy round and replies equal one-token decode at 1 and 3 streams; a hold's cost between and past its timed passes; granted holds kept while they pay, none for a dear pass, and a probe's hold kept.
…re faster With granted depths a lone stream's verify rows are priced over the shared costs and the one-stream overhead, which is measured on drafted rounds and so holds the drafter's hold time; nothing compared a lone drafted round's tokens a ms with a plain round's. lone_guard.zig measures each lone stream's rounds of held drafts in stretches of 16 against plain rounds' tokens a ms, kept from stream to stream by sampling mode and context band and taken once in a plain stretch when unknown; copy rounds are left out of both. Drafting goes on while it beats plain by 5%; else the stream runs plain rounds and measures drafting again after 16 plain rounds, doubling to 512 while plain keeps winning. A drafted stretch's first hold drafts one and its slack, and a mode's first round is not measured. It runs only with granted depths and the shared-round guard on a timed engine (Facts.chances and Facts.shared_guard); shared rounds keep their own guard. Off by default. Tests: the guard's verdicts, rarer probes and kept plain rate; through the round loop a lone stream whose drafted rounds lose runs mostly plain and one whose drafted rounds win keeps drafting, replies equal to one-token decode.
This was referenced Oct 26, 2026
BlivionIaG
marked this pull request as ready for review
October 26, 2026 12:47
ashhart
requested changes
Oct 26, 2026
ashhart
left a comment
ashhart
left a comment
Owner
There was a problem hiding this comment.
Thanks @BlivionIaG. Its lone guard correctly respects `timed`, but it depends on #1379's shared guard, which does not, and on the plain-stretch policy decision. This stack also needs alternating A/B on current main for every family the shared lane core serves.
…ule does not depend on the clock The guard planned, observed and chose with the wall clock even when timed was false. Round planning, joining a shared round and observation now skip it in that mode. A test runs the same three streams on a fast and a slow fake clock and checks equal holds and replies.
…xt stream, and probes less often A loss is counted on the stretch's own rate. After two losses in a row the plain wait grows until a probe costs at most 3% of the plain time. The band keeps its last losing verdict, so the next stream in that band starts plain. Any winning stretch clears it.
Contributor
Author
|
Thanks @ashhart. The #1379 fix is merged in, so the guards respect |
This was referenced Oct 26, 2026
Owner
|
Thanks @BlivionIaG, and thank you for the long-context numbers; they make the problem clear. We will not take the lone guard, for the same reason as #1379: it is a second mechanism that switches a lone stream's The pricing route is already there. With #1381, a grant of none holds nothing except a probe every Please close this PR, or rework it into that pricing; nothing else in the stack should depend on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Branch
series/lanes-lone-guard, head6539b72be: the lone-guard commit, the long-context change and a merge of the #1379 fix, onseries/lanes-chances. Sits on #1381. Part of #1374.What this changes
With granted depths and the shared-round guard on a timed engine, a lone stream's rounds of held drafts are measured
in stretches of 16 against plain rounds' tokens a ms (kept by sampling mode and context band, copy rounds left out).
Drafting goes on while it beats plain by 5%; else the stream runs plain rounds and measures drafting again after 16
plain rounds, doubling to 512. Off unless the drafter sets both
Facts.chancesandFacts.shared_guard. Nothing onmain does; it lands with the LFM2 DSpark drafter PR.
Receipts at this head
Carries the lanes: a shared round drafts only while drafting measures faster (Model.shared_guard) #1379
timedfix and a change to the lone guard for long contexts: a loss is counted on a stretch's own rate; after two losses in a row the plain wait grows until a probe costs at most 3% of plain time; a band remembers its last losing verdict, so the next stream in that band starts plain, and any win clears it. No flag and no environment switch.zig build test: 500 of 500 steps passed; lean 433, the same as the base's; two new tests.One stream, cold prompts, drafter against none, M5 Ultra, three replies each (decode tok/s change): 130,816 tokens -11.6%, -15.5%, -5.6% before and -9.0%, -0.7%, -0.7% after; 100,000 tokens -5.0%, -12.8%, -0.6% before and -5.1%, -4.0%, -0.8% after; 65,536 tokens +4.6%, -6.5%, -5.0% before and +4.5%, -4.9%, -1.7% after; 16,384 tokens -0.8%, +4.2%, +7.7% before and -0.9%, +5.4%, +7.6% after; 4,096 tokens -2.2%, +22.7%, +24.6% both. Short prompts are unchanged (one stream code +25% and chat +21% before and after). The first request after a server start still loses 5% to 9% at 64k and above.
Why drafting loses at long context: see LFM2.5-24B-A2B on Metal: drafts with LiquidAI's DSpark drafter (proposal and gate) #1374 (the released drafter's acceptance falls with context length; our GPU drafts equal the fp32 reference to 130,816 tokens).
Test that fails before the change (the lone guard off in the engine):
263 pass, 1 fail, "a lone stream whosedrafted rounds lose runs plain rounds and probes now and then" (
guarded.holds * 3 < guarded.rounds). With it:264/264.
A/B on current main (other families and builds)
M4 Pro, LFM2.5-24B-A2B 4-bit, no drafter, 4 streams, a different prompt on each, greedy 256 tokens, 4 alternated rounds, a fresh server per cell, a quiet machine (load under 2, no Zig build), every build merged with main
8c36359d5; aggregate tok/s by round:Every build is within 0.5% of main and every reply token-equal to main's. An earlier single-round run showed #1378 at -6% and #1379 at -4%; four rounds showed nothing, so those were session effects.
M5 Ultra, Qwen3.8-27B + DFlash2, 4 streams, distinct prompts, greedy 256 tokens, 6 rotated rounds, 3 reps a cell,
token_shaequal across all four binaries.main2is8c36359d5built a second time in a separate worktree. Median of the 3 reps, tok/s:6539b72be)1 stream: 296.0, 296.5, 296.0 and 295.8. The two builds of identical source differ as much as the lane head does, and the spread inside a cell is 304 to 333, so the gap follows the build or the session, not the code. An earlier session had reported 1.8% to 2.7% lower at this cell; I did not reproduce it.
M5 Ultra, main
8c36359d5against the lane head6539b72be, distinct prompts, greedy 256 tokens, fresh server per cell, 3 alternated rounds,token_shaequal in every cell and every alone and serial check equal (0 failed): Qwen3.8-27B + DFlash2 at 1 stream 296.1 against 296.4 tok/s; Qwen3.8-27B without a drafter 47.6 against 47.5 (1 stream) and 172.7 against 172.6 (4 streams); Qwen3.5-2B at 4 streams 431.8, 431.1, 434.4 against 433.3, 432.5, 433.2 (1 stream too noisy to read, 230 to 278 inside a cell); Nemotron-3.5-Lightning at 1 stream 532.7 against 531.6 and at 4 streams 682.6, 663.1, 675.3 against 660.1, 664.0, 669.1 (pooled -1.6%, the main-first rounds were the high ones). M4 Pro, Nemotron-3.5-Lightning, 3 clean rounds: 1 stream within 0.4% and 4 streams within 1.3% of main, ranges overlapping. Only the LFM2 DSpark drafter setsshared_guard,chances,lone_guardandheld_on_target, so for every other family these heads add branches that are not taken.Not measured
Related PRs
Sits on #1381, which sits on #1379. The DSpark scheduled PR (#1392) sits on it.