Repository navigation
Conversation
Lt-Flash
force-pushed
the
feature/hg-malloc-v3-final
branch
11 times, most recently
from
October 11, 2026 08:26
dd92ab9 to
9f140e2
Compare
Lt-Flash
force-pushed
the
feature/hg-malloc-v3-final
branch
5 times, most recently
from
October 11, 2026 12:42
5223184 to
6bd3352
Compare
Lt-Flash
changed the title
Lt-Flash
force-pushed
the
feature/hg-malloc-v3-final
branch
2 times, most recently
from
October 11, 2026 13:36
6642c73 to
0c21c59
Compare
Author
|
Hi, This PR is ready for review. It is a single commit on top of current master, and the description has the measurements and the design summary; full documentation and benchmarks are in HG_MALLOC is opt-in ( Thank you for your time! |
Lt-Flash
force-pushed
the
feature/hg-malloc-v3-final
branch
2 times, most recently
from
October 16, 2026 12:57
9e02cc0 to
8de237b
Compare
Author
|
Two fixes folded into the commit, both in paths the earlier rounds did not exercise:
Checked with the default build and with |
Lt-Flash
force-pushed
the
feature/hg-malloc-v3-final
branch
from
October 18, 2026 01:08
8de237b to
e749a5a
Compare
Author
|
Pushed a small portability fix, squashed into the HG_MALLOC commit. HG_MALLOC is enabled by default in There is no portable way to free shared pages, so off Linux Tested by compiling the HG sources the way a non-Linux build sees them (Linux-only |
Author
|
Updated the description with call-rate data over TLS and WSS (plus UDP) from a second setup. There one unit is a call relayed by a stateful proxy (2 transactions, 13 datagrams), not an INVITE answered by tm as in the existing tables, so its rates do not compare with theirs. 8 workers per transport, Debian 13 / Linux 6.12, no hugetlb pool, both allocators' shared memory on plain 4 KB pages (the host's THP At the same rate of relayed calls, F_MALLOC -> HG_MALLOC:
Over TLS and WSS the saving is in the TCP workers. TCP main, which does the TLS I/O, is within 3% with either allocator and sets the ceiling, so there HG_MALLOC lowers the cost per call rather than raising capacity. Over UDP with the default 512 KB socket buffer the highest sustained rate goes from 10,000 to 20,000 relayed calls/s (22,000 with Edited 2026-10-19: on that host HG_MALLOC's shared arena ran on 4 KB pages, not on THP as first stated; added the units and the 8 MB socket-buffer result. |
A new shared- and private-memory allocator, selected at runtime with -a HG_MALLOC (or -s/-k for one memory type); no behaviour change for existing configurations. - 21 size classes (64 B - 64 KB) served from per-process free lists, so the common allocation and free take no lock; cells move between processes through a shared per-class pool, in batches. Larger requests go to a coalescing large tier. - Blocks are carved from a buddy allocator over a grid of pages the size of the host's huge page (2 MB on x86-64; 256 KB for small arenas on 4 KB memory), whichever tier backs them; a block whose cells are all free returns to the buddy and can be re-cut for any class. - The arena is acquired through a four-tier ladder - MAP_HUGETLB, THP via MADV_HUGEPAGE, MADV_COLLAPSE, plain 4 KB pages - each tier verified from the kernel's counters. Huge pages are an optimisation, not a requirement. On tier 3 a grow done by a worker that ran out of memory is published on 4 KB pages at once and the maintenance process collapses it afterwards, so no worker waits for MADV_COLLAPSE. - Elastic arenas: -m INIT:CAP / -M INIT:CAP (k/m/g suffixes, bare numbers in MB) reserve the cap and commit pages on demand, in a configurable granule (shm_grow_granule / pkg_grow_granule), growing on exhaustion or ahead of it through an auto_scaling_profile (shm_/pkg_auto_scaling_profile), and releasing quiet pages. A maintenance process grows ahead of demand and runs the release with the arena lock dropped. Growth stops at the cap, the backing tier and a host free-memory floor (hg_ram_floor_mb); a sustained refusal by the backing or the floor latches GROW-BLOCKED, raised as E_CORE_SHM_GROW_BLOCKED (reaching the cap is only a notice). - Host memory is checked before anything is committed, on every tier (MemAvailable above hg_ram_floor_mb, plus the hugetlb pool for arenas on hugetlb pages): at startup for every process's -M INIT (refuse when the startup processes do not fit, warn for the auto-scaling maximum and for caps beyond MemTotal), before each fork (an auto-scaler fork that does not fit is skipped), and on each arena creation and grow. - On hugetlb pages every grow is an mlock() of the new range, and mlock() checks RLIMIT_MEMLOCK there as well: the arena warns at startup (and under -C) when the limit cannot cover its growth, and a refused grow names the resource that ran out - the limit or the hugetlb pool. On 4 KB and THP pages a limit too low to pin the arena leaves it running and growing unpinned (swappable) instead. - core:hg_stats (MI) per arena: tier, committed/reserved, per-class carve and reuse, grows/shrinks, lock hold/wait histograms per reason and the corruption counters; core:hg_advise (MI) recommends -m/-M from the peaks reached; an hgmem: statistics group, including hg_pkg_grow_refused and hg_pkg_grow_blocked summed over every process's private arena; E_CORE_HG_LOCK_STALL for a hold above hg_lock_stall_us. - A per-block free map at the pool boundary catches a double free when it reaches the shared pool and stops a free-list walk at a scribbled link; both are counted and logged and the process keeps running. - mem/mem_arena.h: a module may ask for its own HG-managed arena, independent of the core allocator choice (stubs returning NULL when HG is not built). - modules/hgstress: a soak and fault-injection test module, excluded from the default build. Every worker churns stamped shm cells across all classes and the large tier and verifies the stamps while the others churn; MI commands park and release memory to drive grow and shrink, and inject a double free or a scribbled free-list link. Builds clean with gcc and clang -Werror, with HG_MALLOC compiled out, on arm32/arm64, i386 and musl. Design, configuration, deployment and test procedures: mem/README.hg_malloc.md.
Lt-Flash
force-pushed
the
feature/hg-malloc-v3-final
branch
from
October 21, 2026 14:28
e749a5a to
31d1c97
Compare
- A process leaving for good (auto-scaling down) gives back what it holds in the shared arenas - its cached cells and the unused rest of its chunks (hg_cache_drain_self(), from dynamic_process_final_exit()); they stayed counted as live in their blocks, which never drained until restart. - TCP main's IO threads flush their caches at a job boundary when a sweep was published: the sweep RPC bumps hg_sweep_gen in the process it runs in (the counter is per process, and nothing called hg_cache_flush_if_due() before). - A double free that stays on the freeing thread's private list is refused and counted: the list head in every build, the whole list under HG_MALLOC_DBG. A large fragment is tagged when freed, and a second free of it is refused and counted. - HG_MALLOC_DBG: a freshly carved cell gets empty debug fields - on recycled memory they held old payload bytes, which the status dump at shutdown dereferenced. - A worker that ran out of memory while a trim or punch held the range token took it for a completed grow and failed its allocation; it now compares a grow generation, and grows itself otherwise. - The free-list rebuild skips pages being released; cell_block() counts and logs an address off the page grid instead of returning silently; a free the pool refuses no longer moves the statistics. - Process groups check their auto-scaling profile where they take it: a profile in k/m/g units or with 1000+ processes stops the startup again, as before memory profiles existed (it only disabled auto-scaling). - -m/-M: a negative size is refused; INIT:CAP with an allocator other than HG_MALLOC warns that only the initial size is used. - The rpm allocator error names -e (not -r); stale comments; the tier-1 README note on growing back after a shrink.
The unit tests now also run with HG_MALLOC and HG_MALLOC_DBG: the core suites, the allocator stress suite (test_malloc, disabled upstream since 2021, enabled when HG serves shm and pkg) and HG-specific cases: - pkg memory allocated before fork() realloc'ed and freed in the child - shm cells of every class freed by another process, then reused - the large tier, and growth of an elastic arena past INIT - a process leaving through dynamic_process_final_exit() gives its cells back (the arena's real_used returns to its baseline) - a cell freed twice in a row, and a large fragment freed twice, are refused, counted and never handed out twice - the corruption counters A child reports through shm with a timeout, so a crash or abort fails its test instead of hanging the run, and it starts only after internal_fork() has returned in the parent. make test takes UT_ALLOC and UT_MEM (default HP_MALLOC, as before); the CI build wrapper passes both into its container, and each HG step fails unless the log shows HG_MALLOC was the allocator under test.
…hecks and unit tests updated
Author
|
While going through the allocator again for this round, I found a number of issues and fixed them. I have kept the fixes as a separate commit on top of the original one, so what changed is easy to review on its own; the tests and the reviewer section of the README follow as two more commits. Fixes (
Tests ( For reviewers: a guide to the code ( What changes for users who do not select HG_MALLOC:
Questions:
|
…flush output before forking The growth case stopped allocating 32 MB past INIT; on a slow runner the maintenance process had already grown the arena beyond that, so nothing grew and the case failed. It now allocates until a grow (or a re-grow into holes) is seen. stdout is flushed before every test fork, or each child printed the parent's pending TAP lines again on exit.
Author
|
For reviewers: a guide to the code ( |
add_stats_group() grew the array with sizeof(stat_var) * grp->no + 1 which C reads as (sizeof(stat_var) * no) + 1: the first stat added to a group got a 1-byte array and its 8-byte pointer was written past it, and every later one over-allocated a whole stat_var per element. It went unnoticed because every allocator rounds a 1-byte request up to a cell that holds the pointer; HG_MALLOC_DBG's red zone reports it.
…ks, memory limits
- Every cell header carries the cell's state (fresh, live, free) and a
16-bit check over its address, class and state, seeded per run - a
check against accidents, not a security mechanism. A free needs an
intact live header and moves it to free in one compare-and-swap (a
large fragment under the arena lock), so a double free is refused
whatever came between the two frees, also two frees of one cell in two
processes at once; an allocation takes only an intact fresh or free
cell of its class, so a cell is never handed out twice; a damaged
header or a pointer into a cell is refused, not followed. One multiply
per operation: the check is base(address, class) ^ key(state).
- HG_MALLOC_DBG (mem/hg_dbg.{h,c}), per arena rather than global: a red
zone and trailer after the size asked for, checked at free and realloc;
freed cells poisoned and held in a per-thread quarantine (256 cells,
about a thousandth of the reservation), checked when they leave it and
reported with the site of the free; drained at exit and before an
allocation would fail; a child forgets the parent's.
- Sanitizer hooks (mem/hg_san.h): ASan, or Valgrind with -DHG_VALGRIND,
is told which bytes of a private arena a caller may touch;
ThreadSanitizer is told about the arena lock. Nothing without them.
- The default host floor is MemTotal/20, at least the smaller of 256 MB
and MemTotal/8: the flat 256 MB minimum was more than a small host has,
and OpenSIPS refused to start there.
- Commit checks also respect the memory cgroup's limit
(mem/hg_memlimit.{h,c}, cgroup v1 and v2): in a container or a unit
with MemoryMax, /proc/meminfo shows the host, and arenas grew until the
cgroup's OOM killer stepped in.
A grower that found the range token taken polled it without the lock and, once it saw it clear, took the lock back. If another range operation had taken the token in between - a grow following one that was refused, the drain's punch right after its trim - it counted a wait timeout and failed the allocation after microseconds. It now waits again, within the same cumulative budget, until the token is clear under the lock or the budget is spent. Found by model-checking the grow/trim/punch protocol.
…fuzz tests - test_hg_header: a double free with frees in between, a damaged header, a pointer into a cell, a realloc of freed memory - all refused. - test_hg_dbg (HG_MALLOC_DBG): writes past the request at free and around a realloc, small and large; a write after free, from this process and from another, found when the cell leaves quarantine and never handed out again; a child forked over a full quarantine finds its copy empty. - test_hg_san (ASan builds): pkg cells open for the bytes asked for and closed after them and once freed; shm cells left alone. - test_hg_memlimit: the floor formula, and the cgroup room read from fake cgroup v1 and v2 trees. - test_hg_fuzz: a seeded alloc/free/realloc sequence against an independent model - no address handed out twice, every block keeps its bytes, realloc keeps the prefix, the arena's counters follow the model; a capped arena fails at its cap and serves again. HG_FUZZ_SEED and HG_FUZZ_OPS replay a failure. - The child runner's verdict is published with release/acquire ordering.
…ry limits; invariant 9; test table
The cgroup files count in 64-bit bytes, and hg_memlimit.c read them into an unsigned long: a 32-bit build did not compile (1UL << 62), and with that fixed it would have misread any limit or charge above 4 GB. They are now read and compared as unsigned long long; a room of 4 GB or more is no limit to a 32-bit process. struct hg_cg_info carries MB, which an unsigned long holds on every build. test_hg_memlimit: a 6 GB limit.
/proc/meminfo counts the host in kB, and the checks turned that into bytes in an unsigned long. On a 32-bit build with 4 GB or more available the count wrapped: startup refused or let through depending on how much happened to be free, grows refused at random, and the hugetlb pool and the sums over processes (every child's -M INIT, a pkg grow times the process count) misjudged - those sums pass 4 GB even when no single process does. The room above the floor, the hugetlb pool and those sums are now 64-bit on every build (HG_ROOM_ANY: no limit known), and so is the cgroup's room. /proc/meminfo is read through a test hook; test_hg_memlimit: on an 8 GB host, a grow with 4 GB + 100 MB available passes and one with 8 MB above the floor is refused. README: a test-table row for the memory checks.
Author
|
I re-ran the benchmarks from the description against the current head ( Setup A, INVITEs answered by tm, mean of two runs:
Registrar, 1.2 M REGISTERs at 60k/s: HG_MALLOC again registers all of them with a p99 of 4 ms on 1.9 cores, against 15-24 s on 4 cores for F_MALLOC and F_PARALLEL_MALLOC and 0.53 M registrations for HP_MALLOC. The binary behind the description's tables, run alongside, is within 3.2% of the current head in CPU, in both directions. Setup B, calls relayed by a stateful proxy, F_MALLOC vs HG_MALLOC in the three receive modes of #4365:
All tables, the method and what differed between the runs: https://github.com/Lt-Flash/opensips/blob/hg_malloc/docs/benchmarks-after-review.md |
…rena
core:hg_stats gains two fields per arena:
- unpinned: 1 when the arena runs swappable - init could not mlock() it,
or a grow the memory-lock limit could not cover went on unpinned.
- last_grow_refusal: what refused the most recent grow - cap,
profile_ceiling, host_ram_floor, cgroup_limit, memlock_limit,
hugetlb_pool, mlock_failed, hole_repin_failed, grow_wait_timeout or
dry_run ("none" until the first refusal). It stays once growth works
again; grow_blocked says whether it is current. Until now the cause
was only in the line logged once per refusal episode, which names the
episode's first cause, not a later one.
Each refusal site records its cause. An mlock() refusal is classified
once per episode and cached: telling the memory-lock limit from the
hugetlb pool reads /proc, and an arena that cannot grow retries on every
exhausted request. The classification is now a pure function,
hg_mlock_refusal_why(); the warnings it selects are unchanged.
Unit tests: the classifier's cases; the host-floor and cgroup causes on
the fake /proc/meminfo and cgroup tree; the dry-run and ceiling causes
through hg_buddy_grow() under the arena lock, with every borrowed field
restored before the lock is released.
… the contract at both ends Four functions release hb->lock in the middle and take it again before returning - the grow's two-phase commit: hg_buddy_grow (around the populate and the in-flight wait), hg_buddy_shrink and hg_buddy_punch (around the release), hg_buddy_regrow_holes (around the re-commit) - and the three ticks that call them inherit it. Nothing at a call site said so: a reader of carve_chunk() or hg_cache_flush_self() saw an ordinary call under the lock, and every value read before it silently stopped being trustworthy after it. Now the lock knows when it is free - hg_lock_leave() writes HG_LK_NONE into lk_cur before releasing - and hg_lock_assert_held() can check the one thing that matters: held on entry, held again on return. The four are renamed hg_buddy_*_unlocks and wrapped so the check runs at both ends; hg_grow_tick, hg_shrink_tick and hg_grow_headroom_tick are wrapped the same way. A breach counts as corruption.internal and logs a BUG naming the function and the end. The bodies are untouched; the callers, the comments and the hg_stats unit test use the new names. Fail-first: a mutant that returns from the grow body right after the populate's lock release fires "hg_buddy_grow_unlocks return without hb->lock held" on the first grow of the exhaustion rig; the real code runs the churn, the exhaustion grows and the give-back shrinks with zero such lines.
The fast-path counters (used, fragments, cell_live) live in one slot per process so that no two processes ever write the same words. TCP main is not one process in that sense any more: since the threaded TCP main landed, its IO pool threads allocate and free pkg concurrently - the WS handshake parser alone - and all of them hit the TCP main's slot with plain read-modify-writes. Lost updates, in both directions; the TCP main's pkg "used" and "fragments" drifted under WSS load and nothing could say by how much. A thread now claims a slot of its own the moment it starts (hg_thread_stats_init() from tcp_thread_routine, under HG_MALLOC): slots are handed out from the top of the array through a counter in the shm arena, so the index is unique across every process that could start a thread, and the readers that sum the slots need no change. The main thread keeps the process slot; a thread that finds no free slot (more than three quarters of the array in use) shares the process slot as before and says so once. One __thread read on the fast path, no shared write. hg_status() prints the live figures - used, fragments, cell bytes - so a pkg dump of a process shows what its slots sum to. Measured with a WSS handshake rig (64 connections x 3 rounds repeated for 30 s against a 4-worker TCP main with an IO thread per CPU), the TCP main's pkg counters read before the storm and twelve seconds after it, everything freed: shared slot - 54,016 handshakes: used +264 B, fragments +3 (lost) slot per thread - 51,456 handshakes: used +0, fragments +0
…the rest off the request path Every drained block went straight back to the buddy (HG_GC_KEEP 0), and it went back on the donate path: a worker whose private cache tipped over its threshold pushed cells to the pool and, if that drained a block, walked the drained list and freed it to the buddy - under the arena lock, in the middle of a free. A class that then carved again a second later walked the buddy for a fresh block, split it, and stamped every cell, to get back exactly what it had just given away. Keeping: each class keeps gc_keep[c] drained blocks - carved, empty, the right class, which gpool_pop() already prefers over a carve. The keep follows demand: the class's carves in the policy cycle just ended, decaying by half per idle cycle, capped at HG_GC_KEEP_MAX (8) blocks and 1/256 of the arena per class. A class that stops carving stops keeping within a few cycles; one that carves every cycle keeps what it carved. Off the request path: the maintenance process returns drained blocks beyond keep at most HG_GC_TICK_BUDGET (32) per second, round-robin over the classes; a private arena does the same in its owner's sweep. The donate path only acts past HG_GC_EMERGENCY (32) blocks beyond keep, so the list cannot grow without bound between ticks; the pressure flush waives the keep (a kept block is a page the carve could use, and the alternative is a grow); the sweep flush and the exit drain return down to keep after each batch. hg_stats reports gc_tick_blocks, gc_inline_blocks, gc_kept_blocks and each class's gc_keep and drained count. Measured on the churn rig (A = before, B = this), two sweeps: carves 123 -> 112, GC passes 46 -> 26 with 18 of 42 blocks returned by the tick, keeps settling at 3 (64 B - 1 KB), 2 (1.5-2 KB) and 1 (3-4 KB) blocks, the donate path's hold max 68 -> 65 us; the GC itself costs 0.1 -> 0.3 ms in 80 s either way. Exhaustion rig (8:64, hold_mb 3): grows 3 and the same give-back (56 -> 32 MB) in both arms, carves 184 -> 139, GC passes 108 -> 37, refill hold max 301 -> 92 us. Kept blocks are bounded, and a shrink needs whole pages, which a keep cannot pin for long once its class goes idle.
…ething The idle-cache sweep sent its flush job to every process every interval, whether or not that process had freed anything since its last flush: each job is an IPC round trip, a wake-up and a few lock holds in a process that may have been idle for an hour - 24 of them every 30 s on an SBC, most of them for nothing. A process now says when it has something to sweep: its free fast path sets a dirty mark in its own stats slot - a store into the cache line that path is already writing, so it costs nothing on top - and its flush clears the mark first thing, so a free that lands during the flush is caught by the next sweep. The dispatcher skips a clean process and counts it (sweeps_skipped in hg_stats). The mark lives in the shared arena's slot; a process's pkg caches are its own business and invisible to the dispatcher, so every HG_SWEEP_FORCE_EVERY (4th) interval sweeps everyone regardless - pkg is at most two minutes late, which is well inside the shrink policy's own cadence. The IO-pool threads keep the generation counter as before. Measured on the churn rig over four idle intervals after the churn, A = without, B = with: sweeps 5/10/15/20 vs 5/5/5/6 (skipped 4/13/22/26), flush lock holds 145 vs 74, the busy processes' cells flushed the same, the idle MI process's pkg swept at the forced interval (1 -> 2) instead of every one (2 -> 5). An idle process's residue is collected at the forced interval, at most two minutes later, instead of the next sweep.
…a constant 16 The per-thread private-cache budget is 25% of the arena divided by a count of concurrent cache-holders, and that count was the constant 16 - "a per-CPU IO pool" guess. On a box with more cores than that each thread's share is too large (the caches can sum past the 25% the figure is meant to bound); on a smaller box it is needlessly small. The divisor now tracks the real IO pool: tcp_threads when the config sets it, otherwise the online-CPU count - the same fallback tcp_init_children() uses when tcp_threads is unset - clamped to [4, 64] so a 1-CPU box does not hand one thread the whole share and a 128-CPU box does not shrink each cache to nothing. tcp_threads is 0 until the config is parsed, which happens after the shm and pkg arenas are created, so private_caps_init() uses the online-CPU count at init and hg_autoscale_post_cfg() - the existing post-config hook, pre-fork - re-runs it once tcp_threads is known, for both the shm and the pre-fork pkg arena, so every worker inherits the result. The caps are set here and only read on the fast path; this stays the last write to them. Verified: the DBG budget line reports "over N consumers" = the online CPU count with no tcp_threads, and = tcp_threads when set (tcp_threads=48 -> 48); churn smoke clean.
…x/nproc The sweep is staggered by dispatching process i in second (i * interval / nproc) of each interval. That assumed the live processes were spread across the whole index range; they are not. The processes that actually flush - the SIP receivers and the TCP pool - hold a contiguous low block of indices, so they mapped into the first fraction of the interval, and skipping the clean processes (previous commit) tightened the cluster further: measured on a production SBC, the flushes fell in the first ~13 s of the 30 s interval instead of spanning it. A process's sweep second is now a multiplicative hash of its index (i * 2654435761 mod interval), which scatters any contiguous block of indices across the whole interval. Each index still maps to exactly one second per interval, so every process still flushes once per interval - only the phase changes. Verified: the busy indices (SIP receivers 6-17, TCP 26-29) span seconds 0..28 after the change instead of ~4..13 before; all 39 process slots still land one-per-second across the interval.
Author
|
I pushed six more commits. They come from an internal review of the allocator that I did just before opening this PR; they were still being tested then, so I left them out. They have been running in production since 17 October, on SBCs, registrars and load balancers, and I'm adding them so that this PR is the code that runs there:
Each commit message has the details and the measurements. New head: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A new allocator for shared and private memory, selected with
-a HG_MALLOC. Nothing changes unless it is selected.sched_yieldcalls than F_MALLOC, with HG_MALLOC's shared arena on plain 4 KB pages like F_MALLOC's; over UDP with the default 512 KB socket buffer the highest sustained rate rises from 10k to up to 22k relayed calls/s.-m 128:1024starts at 128 MB, grows on demand up to 1 GB and gives memory back when idle.-mand each process's-M- is checked against host memory before it is committed; an impossible combination is refused at startup, and shared-arena growth and the auto-scaler both stop at a host free-memory floor instead of crashing mid-run.MADV_HUGEPAGE, THP byMADV_COLLAPSE, plain 4 KB pages - each tier verified from the kernel's counters; huge pages are used when the host offers them, not required. On plain 4 KB pages, like the other allocators, HG_MALLOC still uses 4-14% less CPU at 60k INVITEs/s; huge pages save another 13% and halve the p99.-M 256k:2mworks), and on huge pages INIT is rounded up to one page with a notice.For reviewers: a guide to the code - reading order, the invariants and where they are enforced, known limits, and which test shows which property.
Docs, configuration and benchmarks: https://github.com/Lt-Flash/opensips/tree/hg_malloc (in this PR:
mem/README.hg_malloc.md,opensips.8); 4.1 migration and troubleshooting notes: OpenSIPS/opensips-docs#30Details
At high call rates the allocator becomes the bottleneck: at 60k INVITEs/s with 8 UDP workers, F_MALLOC, F_PARALLEL_MALLOC and HP_MALLOC drop packets while 10 or more of the host's 16 cores are idle (mpstat), and HG_MALLOC drops none. And
-m/-Mare fixed at boot: too small fails at peak, too big pins memory that is never used.Two setups that count different things, so their rates do not compare: on setup A one unit is an INVITE answered 486 by tm on the server (1 transaction, 3 datagrams); on setup B, further down, it is a call relayed to a UAS (2 transactions, 13 datagrams). What was measured where: https://github.com/Lt-Flash/opensips/blob/hg_malloc/docs/benchmarks.md#what-was-measured-where
Measurements, setup A (16-vCPU VM, Xeon E5-2699 v4, Ubuntu 24.04, gcc 13, 16-queue NIC, 8 MB UDP socket buffers; HG_MALLOC on a hugetlb pool (tier 1) except in the per-tier table, the other allocators on plain 4 KB pages; sipp from two hosts with 16-queue NICs; same binary, allocator chosen on the command line; F_PARALLEL_MALLOC as
-a F_MALLOC -s F_PARALLEL_MALLOC)INVITE answered by tm, 8 UDP workers,
-m 2048(HG-m 128:2048), and the 60k point again with-m 4096(HG-m 128:4096); difference from F_MALLOC in brackets; peak memory is the shared memory in use (for HG_MALLOC the peak carved, which slightly overstates it):-m 4096At 60k INVITEs/s with
-m 2048F_MALLOC ran out of shared memory 16 s into the run, and F_PARALLEL_MALLOC ran out in one of its pools after 11 s; tm then failed INVITEs. With-m 4096neither ran out (F_MALLOC peaked at 2.5 GB), and F_MALLOC still lost 4,412 INVITEs to drops. HG_MALLOC grew from 128 MB to 1.5 GB in both runs.The 60k point with
-m 4096twice more, back to back, with 8 MB and with 32 MB socket buffers (maxbufferand the kernel limits); difference from F_MALLOC in brackets, host idle from mpstat:The larger buffer does not stop F_MALLOC's and HP_MALLOC's drops: with about 10 of the 16 cores idle in every run (9.8 to 11.5, mpstat), their workers stall for longer than 32 MB covers.
HG_MALLOC alone at 60k INVITEs/s on each backing tier, the host set up for one tier at a time (
-m 128:4096, 8 MB buffers; the other allocators above ran on plain 4 KB pages):On THP by collapse the arena grew from 128 MB to 1.6 GB on 4 KB pages during the run and the maintenance process collapsed it in the background within about 25 s, so part of that run was measured on 4 KB pages while pages were being collapsed.
1.2 M REGISTERs at 60k/s into an empty registrar, 4 UDP workers, so 4.0 cores means saturated (
-m 4096; HG grows from 128 MB to 1.7 GB), two runs each, difference from F_MALLOC in brackets:Longest HG lock hold in all setup A runs: 1.4 ms (5 holds over 1 ms, all at 45k-60k INVITEs/s).
Measurements, setup B: calls relayed by a stateful proxy over UDP, TLS and WSS, F_MALLOC against HG_MALLOC, on another host (16-vCPU VM, Xeon E5-2699 v4, Debian 13, Linux 6.12; no hugetlb pool and THP
shmem_enabled=never, so both allocators' shared memory on plain 4 KB pages and HG_MALLOC's private arenas on THP; 8 workers per transport; HG-m 512:4096 -M 64:512, F-m 2048 -M 32; TLS and WSS over 24 long-lived connections); same binary, script and load; 30-second cells, two repetitions; ranges cover the three receive modes of #4365,sharedbeing the current code. Full tables and the per-syscall breakdown: https://github.com/Lt-Flash/opensips/blob/udp-rx-mode-measurements/README.md#6-hg_malloc-vs-f_malloc-data-for-the-hg_malloc-pull-request-4327sched_yieldper callWith the default 512 KB socket buffer the highest sustained rate rises from 10,000 relayed calls/s to 20,000 on the current receive path and 22,000 with #4365's
auto(F_MALLOC: 13,000 at best, with an 8 MB socket buffer). With an 8 MB socket buffer HG_MALLOC drops nothing up to 16,000 relayed calls/s (12 of 12 cells), starts to drop at 18,000 (3 of 6 cells), and up to 20,000 retransmissions recover every drop; F_MALLOC already drops at 12,000. F_MALLOC hits the buffer limit before the CPU limit: at 12,000/s the kernel drops datagrams while its workers are 63-84% busy. Its extra syscalls aresched_yieldfrom the shared-memory lock's adaptive wait and grow with load (57 to 92 per call from 5,000 to 12,000/s), while HG_MALLOC's stay at 28-30 per call from 12,000 to 20,000/s: its common path takes no lock. Over TLS and WSS the TCP workers, which run the script, use 16-23% less CPU; TCP main, which does the TLS I/O, is within 3% with either allocator and sets the ceiling (5,000-8,000 calls/s), so there HG_MALLOC lowers the cost per call rather than raising the ceiling.Builds: gcc and clang
-Werror, x86-64, i386, arm32/arm64, glibc and musl, and with HG compiled out; soak-tested on x86-64, i386 and musl.Solution
MADV_HUGEPAGE, THP byMADV_COLLAPSE, plain 4 KB pages. Huge pages are optional. On the collapse tier a grow done by a worker that ran out of memory is published on 4 KB pages at once and collapsed afterwards by the maintenance process, so no worker waits forMADV_COLLAPSE.-m INIT:CAP/-M INIT:CAP: the cap is reserved, pages are committed as needed and released when idle. Growth can be driven ahead of demand by the existingauto_scaling_profilegrammar (shm_auto_scaling_profile,pkg_auto_scaling_profile). If growth keeps being refused - on any tier by the host free-memory floor (hg_ram_floor_mb) or the kernel failing to back the memory, and on hugetlb pages also by the memory-lock limit or an empty pool -E_CORE_SHM_GROW_BLOCKEDis raised; reaching the cap is only logged.-mINIT is committed first, then every process's-MINIT is checked against what is left: RAM abovehg_ram_floor_mb, plus the hugetlb pool when the host has one. OpenSIPS refuses a startup that cannot fit, warns when the-mcap plus-Mcap x processes exceeds the host, and at runtime shared-arena grows and auto-scaler forks are both refused at the floor.core:hg_statsandcore:hg_advise(recommended-m/-Mfrom observed peaks) MI commands and anhgmem:statistics group (including private-arena growth refusals summed over all processes), with what to alert on: https://github.com/Lt-Flash/opensips/blob/hg_malloc/docs/configuration.md#monitoringE_CORE_SHM_GROW_BLOCKEDandE_CORE_HG_LOCK_STALL, parameters and examples: https://github.com/Lt-Flash/opensips/blob/hg_malloc/docs/configuration.md#eventsmem/mem_arena.h: modules can request their own HG arena.modules/hgstress: a soak and fault-injection test module, not built by default.mem/test/test_hg_malloc.crun with HG_MALLOC and HG_MALLOC_DBG in the unit-test workflow (make test UT_ALLOC=HG_MALLOC UT_MEM="-m 64:1024 -M 16:256").Compatibility
-m/-Mnow also accept k/m/g suffixes andINIT:CAPfor every allocator; only HG uses the cap, and the others warn that they use INIT.Q_MALLOC_DBG, leaves a build withoutDBG_MALLOCunable to start unless-ais given; an#elif define F_PARALEL_MALLOCtypo inshm_mem.c.LimitMEMLOCK=infinityrecommended: it keeps the arena pinned, so it cannot be swapped. With a lower memory-lock limit, HG runs and grows unpinned on 4 KB and THP pages, but on hugetlb pages an elastic arena cannot grow past the limit. HG warns at startup when the limit is too low.Closing issues
None.