Repository navigation
Publish complete weight revisions and drain accepted rollout work - #18
Merged
kaiwenKevinn merged 4 commits intoOct 23, 2026
Merged
Conversation
Drain admitted requests before installation and reopen only after physical TP worker receipts and prefix-cache reset. Give each transfer a fresh snapshot identity and actual optimizer revision, while renewing NCCL sessions only when receiver membership changes. Keep training ranks in a common export failure protocol, fence NCCL stream dependencies, and ignore stale startup commands. Omit the unused KV reservation and migration APIs. Validation: 59 distinct targeted CPU cases passed across publication, serving, NCCL lifecycle, and two-rank collective export tests.
This was referenced Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 1 of 3. Merge this series in order: #18 → #19 → #20.
Publishing the same optimizer revision twice can leave the sender waiting while receivers skip a reused transfer identity. Restarted services can also retain an obsolete communication session. Each transfer now receives a fresh snapshot identity and carries the actual optimizer revision. NCCL sessions follow physical receiver membership, with a shared listener and separate store namespaces; superseded communicators are released after replacement succeeds.
Services close admission, drain active requests, install the complete physical TP worker set, reset the prefix cache, and then reopen. Core ranks share preparation and export failures, and the transport preserves the producer and receive-stream dependencies.
Publication pauses retain ownership of unsubmitted tasks and drain work already accepted by the runner. Admission is serialized with pause, timeout notifications do not consume successful-result credit, and publication and prefetch use completed-update indices.
Validation: 170 CPU cases passed on the standalone publication candidate, including request lifetimes, TCPStore lifecycle and a real two-rank collective. The complete three-PR series also passed a fresh 50-update Qwen3-4B run on four GPUs: 200 complete GRPO groups, 800 trajectories, 12 successful Hybrid training batches, four evaluations of the same 20 tasks, final physical-worker publication receipts, and clean audit, cleanup and Slurm exit. History consolidation preserves the validated production code. A follow-up test-only correction uses asyncio.TimeoutError so the drain-timeout assertion also works on Python 3.10.
This is the first PR in the series. The EHP lifecycle PR connects dynamic receiver selection and keeps Hybrid loans across ordinary publication.
This three-part series supersedes #16 and #17.