[Qubitium]

Builds on the P2P module merged in #3479 rather than replacing it: it keeps enable_peer_access / enable_for_devices / PeerStatus / peer_matrix and tests/test_p2p.py, and adds the pieces #3479 does not cover.

What this adds

Configuration, not environment variables

P2P behavior is a config object. The former GPTQMODEL_P2P* / GPTQMODEL_USE_TORCH_REPLICATE env switches are gone:

from gptqmodel import GPTQModel
from gptqmodel.quantization.config import QuantizeConfig
from gptqmodel.utils.p2p import P2PConfig

config = QuantizeConfig(bits=4, p2p=P2PConfig(mode="auto", probe=False))
model = GPTQModel.load("/path/to/quantized", p2p=P2PConfig(probe=True))

P2PConfig(mode=..., probe=..., probe_mib=..., min_gbps=..., min_ratio=..., use_torch_replicate=...); a plain dict is accepted anywhere a config is. p2p= is also on from_pretrained / from_quantized. The loopers install BaseQuantizeConfig.p2p before enabling peer access, and the inference loader installs load(..., p2p=...).

Peer access is an accelerator concept: P2PConfig is a no-op for CPU-only runs and for CPU<->CPU transfers. CPU devices are skipped, no warning is emitted, and the CPU move helpers keep their regular copy path.

GIL-safe runtime peer copies

peer_copy_batch(sources, targets) drives cudaMemcpyPeerAsync / hipMemcpyPeerAsync through ctypes with the GIL released. It joins the source stream to the destination stream with an event, enqueues on the destination's current stream, gates on enable_peer_access(...).can_copy(src, dst), and drains the sticky CUDA/HIP last error on failure. No custom kernel is needed: the runtime peer copy is already the native DMA path, and a hand-written kernel would be slower for the large contiguous copies we move.

broadcast_coalesced is confirmed broken on both backends under PYTHON_GIL=0:

• ROCm 4x R9700 (torch 2.16.0.dev20261025+rocm10.1, Python 3.14.10t): 6/6 realistic trials failed (4 hangs, 2 torn copies); a 4-trial rerun gave 1 pass, 2 torn copies and 1 SIGSEGV.
• NVIDIA 2x RTX 4090 (torch 2.16.0.dev+cu130, Python 3.14.10t): 6/6 trials torn; a 10-trial rerun gave 8 torn copies + 2 hangs.
• Payload dependence: two 8192-element tensors passed 8/8, but 1024x1024 (4 MiB) payloads tore 7/8 with one hang. Treat any size as unsafe.

torch.cuda.comm.reduce_add is also unsafe: SIGSEGV on ROCm, and hangs 2/3 trials on NVIDIA inside nccl.reduce.

The fast path is guarded by a regression test that makes every torch.cuda.comm.* entry point raise.

The enqueue runs with the destination device current. HIP binds a stream to the calling thread's current device context, not to the device that owns the stream: issuing the copy while the source device was current put the work on the source queue, where the destination stream and Tensor.cpu() never waited for it. Measured on 4x R9700 with PYTHON_GIL=0, a p2p_move_nested whose first leaf was 32 KiB read back uninitialised memory in 24/24 single-threaded trials and tore 3/3 concurrent processes; synchronize(destination) did not fix it, synchronize(source) did. After wrapping the enqueue in with torch.cuda.device(target_device): 24/24 tasks byte-exact in 3/3 processes, with no synchronize needed before a read on the destination device. CUDA ordered the copy correctly either way, so the defect is HIP-specific, but the explicit scope is correct on both. Regression test: test_p2p_move_nested_is_ordered_on_destination_stream.

Contiguity invariant

Kernels and the peer-copy path both assume contiguous parameters and buffers. That already holds today, so this adds a check instead of a rewrite: gptqmodel/utils/tensor_layout.py provides iter_model_tensors, find_non_contiguous_tensors and assert_model_tensors_contiguous, and the tiny MoE smoke test asserts the invariant across load, quantize and reload. broadcast_module_to_device now logs the non-contiguous fallback rather than silently skipping the peer path.

Measured: 0/26 non-contiguous after load, 0/74 after quantize, 0/74 after save, 0/106 after reload (CPU TORCH); CUDA TORCH, GPTQ_TORCH and GPTQ_TRITON are clean through load+quantize. Marlin, ExLlamaV2, Swordfish, BitBLAS and W4AFP8 could not be instantiated in this environment and remain unverified.

This also settles why coalescing needs a pack step: it is not a strided-layout problem. Contiguous leaves still live in separate allocations (4x Linear(256,256) yields 8 distinct storages with allocator gaps), which no single memcpy can span. Forcing contiguity would not remove the pack.

Lifecycle wiring

• move_module_to_device / move_nested_to_device in looper_helpers.py use peer_copy_batch; the module path refuses non-contiguous and tied leaves and waits for the destination stream before returning.
• Module replication and per-layer moves in module_looper.py / weight_only_looper.py use the new helper while keeping upstream's enable_for_devices(...), _cross_device_pool(), and quant_device state.
• Calibration input/mask/position_ids/kwarg staging in forward_executor.py and forward_batch_worker uses move_nested_to_device.
• Hessian partial reduction reuses one staging buffer only when P2P is fully enabled and free VRAM is > 1.1x the buffer; adopt_hessian_from uses non-blocking CUDA-to-CUDA copies.
• clone_module_for_devices skips torch.nn.parallel.replicate when the GIL is disabled, because it funnels into broadcast_coalesced.

One-time actionable warning

When a multi-GPU run cannot enable/validate every ordered pair, enable_peer_access logs exactly one warning explaining that the GPUs do not have usable P2P because of motherboard compatibility (PCIe wiring, ACS/IOMMU), BIOS compatibility (Above 4G decoding, Resizable BAR, PCIe generation), or hardware compatibility, and that speed may be sub-optimal. The existing per-run info summary is unchanged.

GIL=0 ledger, skill, probe

• pytorch_gil_0_compat.md records the measured GIL=0 status of every PyTorch API we use, including the both-backend broadcast_coalesced and reduce_add results, with workarounds.
• scripts/gil0_api_probe.py regenerates the table with a per-API subprocess timeout so hangs are recorded instead of wedging a run.
• .agents/skills/gptqmodel-pytorch-gil0/SKILL.md plus an AGENTS.md rule require reading and updating the ledger for threaded PyTorch work.

Performance

Measured against the pre-change tree (upstream/main at ca525580), 2x RTX 4090, within one process so both paths see identical clocks, median of 3 runs x 10 roundtrips over 12 module shapes:

• Small leaves: 3-15% faster. 256x256 modules (256 KiB per leaf) move in 11.64 ms vs 13.07 ms (-10.9%), and 64x256 in 2.80 ms vs 3.30 ms (-15.2%). At this size the per-copy dispatch cost dominates the transfer, so replacing Tensor.to with a runtime peer copy pays off.
• Large leaves: gated to the regular path. The peer copy needs an extra destination allocation and an event join per leaf, which costs 0-6% at 2 MiB and above (2x2048 was +6.3%). _PEER_COPY_MAX_AVG_LEAF_BYTES (256 KiB) now sends those modules back through move_to, so every large-leaf shape lands within 1.6% of the pre-change tree.
• Coalescing does not work here. Packing into one buffer, one peer copy, scatter was prototyped and measured 7-17% slower on every shape: it replaces N peer transactions with 2N local ones plus one. Not used.
• End-to-end: no meaningful difference. 438M-parameter synthetic Llama, 12 layers, 8x256-token calibration, 2 GPUs, dense_vram_strategy=BALANCED: medians 22.56 s (main) vs 24.58 s (this branch) with ±20% run-to-run variance. Compute dominates.

For completeness: main is not observably broken under PYTHON_GIL=0 today, because clone_module_for_devices serializes torch.nn.parallel.replicate behind a device lock, so broadcast_coalesced never runs concurrently there (8 threads x 16 clones, 3/3 processes passed). This branch removes the reliance on that lock instead of relying on it.

Verification

On 2x RTX 4090, torch 2.15.0+cu132 (GIL build):

• pytest tests/test_p2p.py tests/test_p2p_copy_utils.py tests/test_looper_helpers.py tests/test_hessian_device_reduction_order.py tests/test_looper_concurrency.py tests/test_gptq_hessian_pipeline.py tests/test_tiny_moe_quant_smoke.py -q -> 97 passed
• ruff check and py_compile pass on every changed Python file.

On 4x R9700, torch 2.16.0.dev20261025+rocm10.1, Python 3.14.10t, PYTHON_GIL=0 (sys._is_gil_enabled() is False):

• pytest tests/test_p2p.py tests/test_p2p_copy_utils.py tests/test_looper_helpers.py tests/test_tensor_layout.py -q -> 49 passed
• Concurrent module-move + nested-staging harness (8 threads x 24 tasks, byte-exact assertions) -> 24/24 in 3/3 processes, both implementations.

No test was skipped or disabled to obtain these results; P2P-dependent cases skip only when the host genuinely lacks a usable directed pair.