Skip to content

Resource governance: SIGHUP limits and a cluster cap on compute threads - #125

Merged
wfaithfull merged 12 commits into
mainfrom
governance
Oct 29, 2026
Merged

wfaithfull merged 12 commits into
mainfrom
governance

Conversation

@wfaithfull

@wfaithfull wfaithfull commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Part of 0.9.0 (#103, "resource governance"; DESIGN.md §13). Records §14 item 52, marked "Decided (coordinator, pending owner review)".

An administrator can now bound what sessions ask for, without taking per-session tuning away from ordinary roles.

SIGHUP limits, applied at use. A call uses the lower of the session's setting and the limit. -1 means no limit; 0 means the same.

Limit Bounds
warren.threads_limit warren.threads
warren.compute_memory_limit warren.max_compute_memory
warren.cache_memory_limit warren.max_cache_memory
warren.build_memory_limit warren.max_build_memory
warren.walk_paths_limit warren.max_walk_paths

When a limit is the lower value, the error names the limit. The SQLSTATEs are the existing ones (53400 and 54000).

A cap on compute threads across the cluster. warren.cluster_threads_limit defaults to -1 (no cap). The docs suggest setting it to the server's core count. It first defaulted to the core count, but in CI concurrent calls then waited for threads until they hit statement_timeout, so it is now opt-in.

  • The count of threads in use is held in the activity table in shared memory. The table's format moves to 2.
  • A call runs with whichever threads are free. When none is free it waits on its latch and stays cancellable; the wait event is WarrenComputeThreads.
  • Threads are given back when the pool drops, or at backend exit.
  • The cap is not enforced where the shared table does not exist (PG 15/16 not preloaded). Each backend logs that once.

No SQL surface change. Schema drift reports none against 0.8.0.

Tests.

  • Pure #[test]s for the clamp and grant arithmetic.
  • #[pg_test]s for each limit at its use site, and for the pool's grant, its nesting and a cancelled wait.
  • New test/compute_threads.py, run in CI. Four backends at threads = 4 use about 3.7 cores without a cap and about 1.0 with a cap of 1.
  • The worker's local runs: pgrx pg16 845/845 and pg18 846/846; test/run.py 66/66 on PG 18 and on PG 16 preloaded.

Merge order. This branch waits until #123 (release 0.8.0) is merged. It then needs main merged in, with the "0.9.0 — in development" CHANGELOG heading unified.

Owner questions:

  • The cap is now off by default. Should it be on by default in a later release?
  • Add a compute_threads column to warren.backends()? That would be a schema change in 0.9.0.
  • Waiting has no queue, so a backend can wait a long time under constant load. Is a fair queue wanted?
  • Can rds_superuser set these parameters through a parameter group? This needs checking on a real instance.

Generated by Claude Code

Ceilings set in the configuration file that clamp the session settings at
use, and a cap on compute threads across the cluster that grants the
threads free and waits only while none is. Decided by the coordinator,
pending the owner's review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
… threads

DESIGN.md §14 item 52. warren.threads_limit, warren.compute_memory_limit,
warren.cache_memory_limit, warren.build_memory_limit and
warren.walk_paths_limit (SIGHUP, -1 for none) bound the session settings
they are named after: a call uses the lower, and an error that names a
setting names the limit where it is the lower (module ceiling).

warren.cluster_threads_limit (SIGHUP, default the cores the server sees)
caps the compute threads of every backend's calls together: a count in the
activity table's header (format 2), which a call takes from before it runs
on its pool and gives back when the pool drops, or at exit. A call gets the
threads free, up to what it asks, and waits on its latch while none is,
showing the wait event WarrenComputeThreads; interrupts end the wait.
Without the table (PG 15 and 16 not preloaded) the cap is not enforced and
the backend logs so once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
test/compute_threads.py runs four backends computing at once and measures
their compute threads' CPU time from /proc, uncapped and under
warren.cluster_threads_limit = 1, preloaded and not; it cancels and times
out a wait for threads, terminates the cap's holder mid-kernel and checks
that its thread comes back, and checks the per-call limits. In CI after
the concurrent refresh check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
docs/limits.md describes the ceilings and the cluster's cap on compute
threads, and suggests warren.shared_cache_memory and the memory limits per
instance size; CHANGELOG opens 0.9.0; DESIGN.md §6.9 and §5.7 and
CLAUDE.md's module list follow the code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
docs/security.md (from #124) said the limits were planned for 0.9.0; it
now lists the six sighup limits and the cluster thread cap as built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
On by default at the server's cores, the cap turned the harness's
concurrent refresh on a 4-core runner into statement timeouts: a wait for
threads counts against statement_timeout, so concurrent workloads that
fit their timeouts start failing with 57014. That is a change an
administrator should choose, so warren.cluster_threads_limit now defaults
to -1, like the other limits, and the docs suggest the server's cores as
a value. DESIGN.md §14 item 52 records why.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
A new last column, compute_threads (owner decision, 2026-10-29): on a
backend's row the threads its running call holds (granted, so under the
cap possibly fewer than warren.threads; 0 between calls), and on the
cluster row the count in the activity table's header that
warren.cluster_threads_limit bounds. A managed service's administrator
has no shell, so this is how they see the cap work. It follows the
privacy rule of the other sizes; the cluster row's is everyone's.

The count is now kept wherever the activity table exists, cap or no cap
(a call counts under a cap it never reaches), so the column shows the
threads in use on any instance and a cap set by a reload counts the calls
already running. Each slot gains a word for its backend's threads (still
512 bytes; the table's format stays 2, unreleased).

The column is last because pgrx forms a set-returning function's rows by
position: with the new library and 0.8.0's definition, before ALTER
EXTENSION UPDATE, an inserted column would put a bigint where the old
definition reads jsonb. The upgrade script drops and creates the function
again, keeping its privileges.

Tests: pg_tests for a call holding threads (and the cluster's count with
no cap set), a capped call showing what it was granted, and none once
given back; test/sql/backends.sql asserts idle, privacy and the cluster
row; test/compute_threads.py checks the column against the cap during
the capped run, past it uncapped, and the holder of the cap's thread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
wfaithfull pushed a commit that referenced this pull request Oct 29, 2026
Item 52 is the cluster thread cap's (governance, #125).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Rxfe2L3iVjSwSBPdUu1Ke

Copy link
Copy Markdown
Contributor Author

Generators + SQL harness (PG 16) failed in the pooler soak on f9dc3c0, with one client stopped by RuntimeError("SET statement_timeout = '120s': canceling statement due to user request"). Everything else in the job passed: the SQL suite with and without the preload, the concurrent refresh, compute threads, and crash recovery.

This looks like a race in the soak's chaos thread, not something this PR changed. Chaos picks a transaction-pool backend that is active in a warren. call from pg_stat_activity, then calls pg_cancel_backend(pid) (test/harness/pooler_soak/workload.py). If the call finishes in between, the cancel lands on whatever that backend runs next. Here that was another client's session setup (test/harness/session.py's SESSION_SETTINGS), which the client treats as fatal. The same job passed on this PR's previous head (ea2c6bc), which had the same soak and the same warren.backends() code. The merge from main added only additive changes from #135.

I'll re-run the failed job once when the run finishes. If it fails again I'll treat it as real. Either way, the soak needs a fix in its own PR: make the cancel conditional on the same call still running, or have a client retry a 57014 on its setup statements.


Generated by Claude Code

@wfaithfull
wfaithfull merged commit 9bad4a5 into main Oct 29, 2026
74 of 75 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants