[tcyjohn]

Superseded by #18 (publication/request lifecycle), #19 (fixed-batch GRPO and optimizer correctness), and #20 (EHP resource lifecycle). Please review and merge the replacement series in that order. This PR is closed without merging.

---

Depends on #14. This branch includes its planner fix; after #14 merges, the remaining diff covers elastic training.

Elastic Hybrid replicas share the configured global batch with Core replicas. Partition complete GRPO groups across live replicas, weight gradients by their sample counts, and reject stale or duplicate deliveries. Borrowing a GPU changes who computes the batch, not how many samples the optimizer consumes.

Select the Core TP/DP layout at startup and keep it fixed during training. EHP borrowing and return, rollout repartition, GRP decisions, live optimizer-state transfer, and rollout publication remain active. Online Core TP/DP rebuilds are excluded.

Preserve model parameters, FP32 master weights, Adam moments, and independent per-parameter steps during borrowing and return. Check readiness and process ownership before changing resource roles. Count the physical GPUs stopped with a whole rollout engine against the rollout reserve, and start Hybrid replicas only from confirmed complete GPU allocations on one host. An unsuitable engine partition records a capacity shortfall while preserving its serving processes and existing Hybrid replicas.

Bound gradient reduction temporaries, own asynchronous CPU gradient snapshots, and use per-parameter temporaries for automatically selected native Torch Adam foreach behavior while retaining explicit choices and fused optimizers.

Validation:

• Scoped branch CPU CI: 361 passed, covering fixed-batch reduction, optimizer-state restoration, native Adam temporaries, physical rollout reserve, and resource lifecycle.
• Actual MCore DP-zero gather/scatter with two Gloo Core ranks restores a DP1 Hybrid lane, including different source/target padding. FP32 master weights, BF16 parameters, Adam moments/steps, and the next update match the reference exactly.
• Real Qwen3-4B on four A800 GPUs, fixed Core TP1/DP1: three Hybrid updates split the same global batch of 8 into Core 4 + Hybrid 4, then returned the GPU and completed a Core-only continuation. Model, FP32 master, Adam moments, and step comparisons completed under the existing tolerances (atol=2e-4, rtol=2e-3); the pre-Hybrid restore used an exact comparison.
• All three current TP1 rollout services persisted installation and reopen for the same final revision-5 snapshot after training and return.
• The containing Slurm job 1835849 timed out its 1,800-second acceptance budget after final installation. Full CPU state snapshots/comparisons dominated that budget; the final error-summary and clean-shutdown receipt were not flushed. These are functional subphase results, not a clean whole-job pass or a training-throughput benchmark. No unrecorded maximum errors are reported.