Harden Foundations memory and concurrency contracts

[tdecroyere]

Goal

Harden the low-level Foundations runtime around explicit memory, lifetime, and concurrency contracts. This keeps the existing arena/stack design intact while fixing correctness bugs that could otherwise lead to out-of-bounds allocations, races, stale handles, corrupted free lists, or unsafe access to uncommitted memory.

MemoryArena

• remove public SystemPopMemory and keep rollback internal to the stack-memory implementation;
• keep SystemClearMemoryArena / SystemFreeMemoryArena as explicit exclusive lifetime operations;
• reject explicit clear of stack-memory handles before mutating scoped allocator state;
• guard the 8-bit stack lifetime level so the 256th simultaneously nested scope fails cleanly instead of wrapping to zero;
• separate stack-memory mechanism from C++ scope lifetime: SystemBeginStackMemoryArena() / SystemEndStackMemoryArena() operate on a plain StackMemoryArenaScope token with no automatic language-level lifetime behavior;
• keep SystemGetStackMemoryArena() / StackMemoryArena::~StackMemoryArena() as the C++ convenience layer over the explicit begin/end primitives;
• require explicit stack scopes to be ended on their creating thread and in reverse nesting order, preventing wrapper misuse from corrupting stack state;
• expose SystemReleaseThreadMemory() so language runtimes/wrappers can explicitly release the current thread's lazily-created stack-memory backing once no scopes remain active;
• keep automatic C++ thread-exit cleanup by having the internal TLS wrapper delegate to SystemReleaseThreadMemory();
• page-align the allocator header, keep it permanently committed, and make page metadata describe data pages only;
• calculate page indexes relative to the data region and remove floating-point page calculations;
• make system page-size initialization thread-safe without cross-translation-unit initialization-order dependence;
• replace the shared push check-then-add race with a bounded CAS reservation loop;
• fix StackExtraStorage capacity checks to use the actual working arena;
• serialize shared commit/decommit bookkeeping before mutating PageInfo, committed-state bits, or page counts;
• make platform reserve/commit/decommit accounting atomic and only update counters after successful OS operations;
• account for still-committed pages when an arena reservation is released;
• reject size/alignment/multiplication overflow instead of allowing wrapped reservations or array sizes;
• propagate allocation failures through zero/array/string helpers instead of dereferencing null storage;
• make SystemCommitMemory report success/failure and prevent committed pushes from returning a pointer when the platform commit fails;
• make SystemClearMemoryArena directly decommit committed data pages and reset all page tracking instead of relying on conservative range bookkeeping;
• preserve intentionally conservative per-page min/max tracking for normal sparse commit/decommit operations to avoid heavier fragmentation metadata;
• align typed array/struct allocations to alignof(T) while keeping raw byte pushes on the Foundations default alignment;
• make alignment padding explicit arena capacity consumption and include it in allocated-byte accounting;
• define Foundations memory as raw storage only, independent of any language-level ownership/construction/destruction model;
• document typed pushes as raw storage only: no constructors, destructors, copy constructors, or other hidden object-lifetime behavior;
• keep raw buffer copy/duplicate/concat helpers byte-copy based and document that element types must be safe for byte-wise copying;
• make wide-character duplication follow the same logical-length + hidden null-terminator convention as character strings;
• make POSIX decommit replace committed pages with fixed anonymous PROT_NONE mappings so physical backing is released while the virtual address range remains reserved;
• preserve the existing advanced nested stack-memory lifetime model and its regression coverage.

DataPool

• keep DataPool as a low-cost lock-free handle table for high-frequency resource and command-list lifetimes;
• replace the unsafe ABA-prone free-list head with a 64-bit tagged state combining head index and item generation;
• require lock-free 64-bit atomics on supported targets and keep lookup paths free of locks/retry loops;
• use bounded CAS allocation for new indexes so concurrent growth cannot advance past capacity;
• claim item generation with CAS on remove so concurrent removal of the same handle recycles the slot exactly once;
• reuse the item generation as the free-list ABA tag rather than introducing a second lifetime counter;
• isolate mutation-heavy free-list/count state from the read-mostly lookup high-water mark to reduce cache-line interference;
• validate generations with atomic loads and skip the reserved invalid generation value on wrap;
• propagate backing-memory commit failures before publishing/writing a new item;
• define T / TFull storage as raw-storage-compatible data only: DataPool does not construct, destroy, retain, release, or otherwise manage language-level object lifetime;
• document that lookups are thread-safe but returned raw pointers do not pin item lifetime against later remove/reuse.

Dictionary

• replace the unsafe concurrent linked-list/free-list mutation scheme with deterministic synchronized dictionary operations;
• initialize hash/value/next before publishing an entry into its bucket;
• bound entry allocation under contention and remove timing-dependent retry loops;
• prevent removed entries from being recycled while another dictionary operation is traversing them;
• fix ReadOnlySpan<T> hashing so Length is converted to bytes with sizeof(T);
• make missing SystemGetDictionaryValue return nullptr as documented while preserving the zero-value operator[] convenience;
• define dictionary values as raw-storage-compatible data only, with no language-level construction/destruction semantics in Foundations;
• document raw pointer/reference lifetime limitations after lookup.

Span and string buffers

• keep Span / ReadOnlySpan lightweight, non-owning pointer-and-length views;
• make const-buffer construction and slicing const-correct;
• define character span Length as logical characters excluding the null terminator;
• fix SystemDuplicateBuffer<char> to preserve that logical length;
• make SystemDuplicateBuffer<wchar_t> follow the same null-terminated backing-storage convention;
• make the shader serializer write its null terminator explicitly instead of depending on the previous incorrect string length.

Regression coverage

Adds focused tests for:

• arena size/push overflow and exact VM accounting after free;
• clear after partial page commitment, including clean reuse/decommit of the same page;
• explicit clear requests on stack arenas leaving scoped allocations untouched;
• maximum stack nesting rejecting the 256th active scope without lifetime-level wrap and remaining usable after unwind;
• explicit stack scope begin/end behavior, including ancestor-lifetime allocation through nested scopes and explicit thread-memory release/recreation;
• typed alignof(T) allocation after a deliberately misaligned bump position;
• concurrent typed alignment under CAS contention;
• concurrent push capacity and shared-page commit;
• StackExtraStorage ancestor allocations;
• short-lived thread cleanup of both primary stack arena storage and nested StackExtraStorage backing;
• commit result validation;
• DataPool capacity, duplicate concurrent remove, and hundreds of thousands of concurrent remove/reuse cycles;
• Dictionary capacity, concurrent remove/reuse cycles, missing lookup semantics, and non-char span hashing;
• Span/string logical length and null termination for both char and wchar_t duplication.

Intentional follow-ups outside the final SystemMemory review: replace C++ lifetime-owning payloads such as DX12 ComPtr values stored in Foundations raw containers with raw handles/pointers, and harden the platform-specific descriptor/query free lists in the graphics backends.