Skip to content

Fix vmem_chunked allocator alignment, granularity, and rollback edge cases - #633

Open
Abhishek-Varma wants to merge 1 commit into
ROCm:mainfrom
Abhishek-Varma:harden-vmem-chunked
Open

Abhishek-Varma wants to merge 1 commit into
ROCm:mainfrom
Abhishek-Varma:harden-vmem-chunked

Conversation

@Abhishek-Varma

Copy link
Copy Markdown

Summary

Fixes alignment, granularity, and rollback edge cases bugs in VMemChunkedAllocator (refer #600)

Fixes

  1. Granularity reporting - distinguish minimum vs. recommended allocation
    granularity.

    • Added BaseDriver.get_recommended_granularity() (a concrete default that
      returns the minimum) and query both minimum/recommended from the AMD
      local and fabric drivers.
    • The allocator now backs memory in units of the recommended granule
      while reporting the true minimum via get_stats()
      (recommended_granularity / min_granularity).
    • AMD behavior is byte-identical: the old get_minimum_granularity()
      already returned the recommended value.
  2. Alignment - honor a requested alignment larger than the granule.

    • allocate() computes an effective alignment (≥ min_alignment, rounded
      to a power of two) and aligns the heap offset to it, keeping allocations
      symmetric across ranks.
    • Free lists are keyed by (size, alignment) - a Bucket - so a block is
      only reused for a request with matching alignment.
  3. Use-after-close - add _check_open() guards to allocate(),
    import_external_tensor(), import_peer_chunk(), and
    get_allocation_chunks_since(). Operations after close() now raise
    RuntimeError instead of silently corrupting allocator state.

  4. Failed-allocation rollback - build the tensor before committing the
    bump pointer and bookkeeping. If tensor construction raises, a popped
    free-list block is returned to its bucket and the bump pointer is left
    untouched - no orphaned reservation, so a rank can always re-converge to
    the symmetric layout.

  5. Non-power-of-two heap sizes - drop the power-of-two chunk_size
    requirement. VA reservation is rounded up to a whole number of chunks
    arithmetically (math.ceil) instead of via a bitmask, so arbitrary
    granularity-aligned chunk sizes work for any heap size.

Testing

  • Added reproduce-first regression tests for all five fixes:
    • Use-after-close coverage for all four guarded entry points (not just
      allocate()).
    • A free-list-path rollback test (in addition to the bump-pointer path).
    • A same-offset assertion proving the alignment fix actually reuses a freed
      block, rather than coincidentally landing on an aligned bump.
    • Two GPU-free driver-level unit tests pinning that
      get_minimum_granularity() / get_recommended_granularity() route
      distinct HIP flags.
  • Full allocator suite: 37/37 passing on 2× MI300A (1- and 2-rank).
  • Driver unit tests (GPU-free): 40/40 passing.

Addresses the bug-fix bundle from the vmem_chunked hardening epic. Scope is
AMD-only: the shared driver interface change is implemented and tested on the
AMD local and fabric drivers; NVIDIA keeps its existing behavior via the
base-class default (no untested driver code is shipped).

1. Granularity reporting: distinguish minimum vs recommended allocation
   granularity. Add BaseDriver.get_recommended_granularity() (a concrete
   default returning the minimum) and query both minimum/recommended from the
   AMD local and fabric drivers. The allocator now backs memory in units of the
   recommended granule while reporting the true minimum via get_stats()
   (recommended_granularity / min_granularity). AMD behavior is byte-identical
   since the old get_minimum_granularity() already returned the recommended
   value; NVIDIA inherits the default (recommended == minimum), i.e. unchanged.

2. Alignment: honor a requested alignment larger than the granule. allocate()
   computes an effective alignment (>= min_alignment, rounded to a power of
   two) and aligns the heap offset to it, keeping allocations symmetric across
   ranks. Free lists are keyed by (size, alignment) -- a "Bucket" -- so a block
   is only reused for a request with matching alignment.

3. Use-after-close: add _check_open() guards to allocate(),
   import_external_tensor(), import_peer_chunk() and
   get_allocation_chunks_since(); operations after close() now raise
   RuntimeError instead of corrupting state.

4. Failed-allocation rollback: build the tensor before committing bump pointer
   and bookkeeping. If tensor construction raises, a popped free-list block is
   returned and the bump pointer is left untouched -- no orphaned reservation,
   so a rank can always re-converge to the symmetric layout.

5. Non-power-of-two heap sizes: drop the power-of-two chunk_size requirement.
   VA reservation is rounded up to a whole number of chunks arithmetically
   (math.ceil) instead of via a bitmask, so arbitrary granularity-aligned chunk
   sizes work for any heap size.

Adds reproduce-first unit tests for all five fixes. Full allocator suite
passes 37/37 on 2x MI300A (1- and 2-rank).

Review follow-ups folded into this commit:
- Renamed the HIP _query_granularity(flag) parameter to granularity_flag and
  documented it, instead of a bare, unexplained "flag".
- Named and documented the (size, alignment) free-list key as a module-level
  Bucket type alias, with a defining comment at every site that creates,
  stores, or reads one (previously only explained piecemeal across scattered
  comments, with no single definition).
- Reconciled _check_open()'s docstring with the one call site
  (get_allocation_chunks_since) that doesn't hold the lock.
- Closed coverage gaps found while re-reviewing the five fixes: use-after-close
  tests for import_external_tensor/import_peer_chunk/get_allocation_chunks_since
  (previously only allocate() was tested), a free-list-path rollback test for
  the failed-allocation fix (previously only the bump-pointer path was
  tested), a same-offset assertion proving the alignment fix actually reuses
  the freed block rather than coincidentally re-bumping into alignment, and
  two GPU-free driver-level tests pinning that get_minimum_granularity /
  get_recommended_granularity route distinct HIP flags.

Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant