Skip to content

Verify complete TP publication and physical service lifecycle - #16

Closed
tcyjohn wants to merge 2 commits into
NetX-lab:mainfrom
tcyjohn:codex/fix-serving-kv-lifecycle
Closed

tcyjohn wants to merge 2 commits into
NetX-lab:mainfrom
tcyjohn:codex/fix-serving-kv-lifecycle

Conversation

@tcyjohn

@tcyjohn tcyjohn commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Superseded by #18 (publication/request lifecycle), #19 (fixed-batch GRPO and optimizer correctness), and #20 (EHP resource lifecycle). Please review and merge the replacement series in that order. This PR is closed without merging.


Weight reloads must wait for admitted requests and service reservations, then verify installation on every physical TP worker before reopening. Reject partial or stale receipts, preserve snapshot/revision and service-incarnation identities across restart, and require prefix-cache invalidation after changing weights.

Preserve timeout=0 as unlimited waiting and transfer TCPStore listener ownership so cleanup does not close the same socket twice. Register the serving helpers in the installed package and include the HTTP lifecycle test dependency in CPU CI.

This PR covers publication and request lifecycle. CPU-offload/NIXL cached-KV migration changes have been removed. The existing lease API remains because it also protects a version-checked request between reservation and dispatch.

Validation:

  • Scoped branch CPU CI: 116 passed, including cancelled-reservation cleanup, physical TP receipts, prefix reset, and TCPStore lifecycle/timeout regressions.
  • Real Qwen3-4B on four A800 GPUs: the native NCCL publication subphase sent 398 tensors (8,044,936,192 bytes) to TP1 and TP2 services in 21.33 seconds, with all three receiver workers acknowledged and parameter probes checked. The containing job 1835628 failed its separate KV-continuation phase; the publication result is reported independently.
  • Combined EHP run 1835849: after actual Hybrid training and GPU return, all three current TP1 services persisted installation and reopen for the same final revision-5 snapshot. That run exhausted its 1,800-second acceptance budget after installation, so it is not reported as a clean whole-job pass.

@tcyjohn tcyjohn changed the title Fence KV reuse and reloads by physical service and weight identity Verify complete TP publication and physical service lifecycle Oct 19, 2026
@tcyjohn tcyjohn closed this Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant