Repository navigation
Resource governance: SIGHUP limits and a cluster cap on compute threads - #125
Conversation
Ceilings set in the configuration file that clamp the session settings at use, and a cap on compute threads across the cluster that grants the threads free and waits only while none is. Decided by the coordinator, pending the owner's review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
… threads DESIGN.md §14 item 52. warren.threads_limit, warren.compute_memory_limit, warren.cache_memory_limit, warren.build_memory_limit and warren.walk_paths_limit (SIGHUP, -1 for none) bound the session settings they are named after: a call uses the lower, and an error that names a setting names the limit where it is the lower (module ceiling). warren.cluster_threads_limit (SIGHUP, default the cores the server sees) caps the compute threads of every backend's calls together: a count in the activity table's header (format 2), which a call takes from before it runs on its pool and gives back when the pool drops, or at exit. A call gets the threads free, up to what it asks, and waits on its latch while none is, showing the wait event WarrenComputeThreads; interrupts end the wait. Without the table (PG 15 and 16 not preloaded) the cap is not enforced and the backend logs so once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
test/compute_threads.py runs four backends computing at once and measures their compute threads' CPU time from /proc, uncapped and under warren.cluster_threads_limit = 1, preloaded and not; it cancels and times out a wait for threads, terminates the cap's holder mid-kernel and checks that its thread comes back, and checks the per-call limits. In CI after the concurrent refresh check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
docs/limits.md describes the ceilings and the cluster's cap on compute threads, and suggests warren.shared_cache_memory and the memory limits per instance size; CHANGELOG opens 0.9.0; DESIGN.md §6.9 and §5.7 and CLAUDE.md's module list follow the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
docs/security.md (from #124) said the limits were planned for 0.9.0; it now lists the six sighup limits and the cluster thread cap as built. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
On by default at the server's cores, the cap turned the harness's concurrent refresh on a 4-core runner into statement timeouts: a wait for threads counts against statement_timeout, so concurrent workloads that fit their timeouts start failing with 57014. That is a change an administrator should choose, so warren.cluster_threads_limit now defaults to -1, like the other limits, and the docs suggest the server's cores as a value. DESIGN.md §14 item 52 records why. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
A new last column, compute_threads (owner decision, 2026-10-29): on a backend's row the threads its running call holds (granted, so under the cap possibly fewer than warren.threads; 0 between calls), and on the cluster row the count in the activity table's header that warren.cluster_threads_limit bounds. A managed service's administrator has no shell, so this is how they see the cap work. It follows the privacy rule of the other sizes; the cluster row's is everyone's. The count is now kept wherever the activity table exists, cap or no cap (a call counts under a cap it never reaches), so the column shows the threads in use on any instance and a cap set by a reload counts the calls already running. Each slot gains a word for its backend's threads (still 512 bytes; the table's format stays 2, unreleased). The column is last because pgrx forms a set-returning function's rows by position: with the new library and 0.8.0's definition, before ALTER EXTENSION UPDATE, an inserted column would put a bigint where the old definition reads jsonb. The upgrade script drops and creates the function again, keeping its privileges. Tests: pg_tests for a call holding threads (and the cluster's count with no cap set), a capped call showing what it was granted, and none once given back; test/sql/backends.sql asserts idle, privacy and the cluster row; test/compute_threads.py checks the column against the cap during the capped run, past it uncapped, and the holder of the cap's thread. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
Item 52 is the cluster thread cap's (governance, #125). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
|
This looks like a race in the soak's chaos thread, not something this PR changed. Chaos picks a transaction-pool backend that is I'll re-run the failed job once when the run finishes. If it fails again I'll treat it as real. Either way, the soak needs a fix in its own PR: make the cancel conditional on the same call still running, or have a client retry a 57014 on its setup statements. Generated by Claude Code |
Part of 0.9.0 (#103, "resource governance"; DESIGN.md §13). Records §14 item 52, marked "Decided (coordinator, pending owner review)".
An administrator can now bound what sessions ask for, without taking per-session tuning away from ordinary roles.
SIGHUP limits, applied at use. A call uses the lower of the session's setting and the limit.
-1means no limit;0means the same.warren.threads_limitwarren.threadswarren.compute_memory_limitwarren.max_compute_memorywarren.cache_memory_limitwarren.max_cache_memorywarren.build_memory_limitwarren.max_build_memorywarren.walk_paths_limitwarren.max_walk_pathsWhen a limit is the lower value, the error names the limit. The SQLSTATEs are the existing ones (53400 and 54000).
A cap on compute threads across the cluster.
warren.cluster_threads_limitdefaults to-1(no cap). The docs suggest setting it to the server's core count. It first defaulted to the core count, but in CI concurrent calls then waited for threads until they hitstatement_timeout, so it is now opt-in.WarrenComputeThreads.No SQL surface change. Schema drift reports none against 0.8.0.
Tests.
#[test]s for the clamp and grant arithmetic.#[pg_test]s for each limit at its use site, and for the pool's grant, its nesting and a cancelled wait.test/compute_threads.py, run in CI. Four backends atthreads = 4use about 3.7 cores without a cap and about 1.0 with a cap of 1.test/run.py66/66 on PG 18 and on PG 16 preloaded.Merge order. This branch waits until #123 (release 0.8.0) is merged. It then needs main merged in, with the "0.9.0 — in development" CHANGELOG heading unified.
Owner questions:
compute_threadscolumn towarren.backends()? That would be a schema change in 0.9.0.rds_superuserset these parameters through a parameter group? This needs checking on a real instance.Generated by Claude Code