Rate-limit df.start

CHANGELOG.md

@@ -4,6 +4,34 @@ All notable changes to this project are documented in this file. The format is b
 
 Pre-1.0 note: while `pg_durable` is in major version `0`, minor releases may include breaking changes.
 
+## [Unreleased]
+
+### Security
+
+- **Start admission quotas (#139):** `df.max_concurrent_per_user` (default
+  `100`) and `df.max_instances_per_user` (default `10000`) limit active and
+  retained instances per submitting role and origin database. Both are
+  superuser-settable; `0` disables a quota and superuser identities are exempt.
+  Active accounting uses protected engine state, not caller-writable display
+  status. Cancellation frees active capacity after the engine processes it.
+  A protected ledger commits atomically with engine enqueue, so transaction
+  and savepoint rollbacks cannot bypass quotas. Rolled-back submissions remain
+  charged until the engine processes them and reconciliation removes their history.
+  Starts serialize per role until the submitting transaction ends and reject
+  over-quota work before insert/enqueue, including on older installed schemas.
+
+> **Upgrade:** These defaults apply after a binary-only upgrade. Existing
+> workflows whose SQL starts more workflows also encounter the quotas, and an
+> over-quota SQL step can fail its enclosing workflow. Completed and cancelled
+> instances still consume retained capacity. With the default 30-day retention,
+> a role holding 10,000 recent instances may remain blocked until history ages
+> out or an administrator changes policy; a reconciliation pass alone need not
+> free capacity. See [Instance Limits](USER_GUIDE.md#instance-limits).
+> Run `ALTER EXTENSION pg_durable UPDATE TO '0.2.10'` to add the owner-leading
+> index used by quota checks. The new binary still enforces quotas on older
+> schemas; without the update their local quota queries can scan other roles'
+> history.
+
 ## [0.2.9] - 2026-10-07
 
 > **Upgrade:** Install/restart the new binary and wait for the control runtime

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "pg_durable"
-version = "0.2.9"
+version = "0.2.10"
 edition = "2021"
 license = "PostgreSQL"
 repository = "https://github.com/microsoft/pg_durable"

USER_GUIDE.md

@@ -26,10 +26,11 @@ pg_durable is a PostgreSQL extension that brings durable, fault-tolerant functio
 16. [Connection Host](#connection-host)
 17. [HTTP Security](#http-security)
 18. [HTTP Allowed Domains](#http-allowed-domains)
-19. [Connection Limits](#connection-limits)
-20. [Troubleshooting](#troubleshooting)
-21. [Quick Reference Card](#quick-reference-card)
-22. [Appendix: Test Data Setup](#appendix-test-data-setup)
+19. [Instance Limits](#instance-limits)
+20. [Connection Limits](#connection-limits)
+21. [Troubleshooting](#troubleshooting)
+22. [Quick Reference Card](#quick-reference-card)
+23. [Appendix: Test Data Setup](#appendix-test-data-setup)
 
 ---
 
@@ -2011,8 +2012,9 @@ runtime. Distinguish three roles:
     does not need the extension unless the SQL itself calls `df` APIs.
 
 Satellite installation requires a compatible control worker with readiness schema
-version `2` (including its `_origins` registry). Provider migrations run only in
-control through the worker's `ApplyAll`; satellite creation never creates control
+version `3` (including its origin registry and protected submission accounting).
+Provider migrations run only in control through the worker's `ApplyAll`;
+satellite creation never creates control
 automatically. The provider namespace is extension-owned, its contents are created
 by `pg_durable.worker_role`, and satellite objects belong to the local installer.
 
@@ -2545,6 +2547,12 @@ worker runs a **best-effort reconciliation pass** that does two things:
   age past `retention_days`. Anything the engine is still tracking with a live
   `df` row is left untouched.
 
+After the corresponding engine work, history, and origin metadata are gone,
+reconciliation removes the protected submission record and frees its retained
+quota capacity. This includes submissions whose caller transaction or savepoint
+rolled back. Failed metadata cleanup delays retained-capacity release, but does
+not make retired work consume active capacity again.
+
 Satellite maintenance visits `_origins` registrations made by activity routing for
 submitted work, in bounded batches under the shared origin connection budget.
 Registration is not synchronous with `df.start()`; unused installations are not
@@ -3003,6 +3011,99 @@ for the full syntax and default domain list.
 
 ---
 
+## Instance Limits
+
+`df.start()` checks two quotas before creating an instance or enqueueing work:
+
+| Setting | Default | Counts |
+|---------|---------|--------|
+| `df.max_concurrent_per_user` | `100` | Instances not yet terminal in the durable engine, including queued, sleeping, looping, and signal-waiting workflows |
+| `df.max_instances_per_user` | `10000` | Retained submissions, including rolled-back starts and completed, failed, and cancelled workflows |
+
+Each quota applies to the **submitting role (`current_user`) in the origin
+database**. `SET ROLE` and `SECURITY DEFINER` affect that identity just as they
+affect workflow execution. An explicit SQL target database does not move the
+quota to that database. Superuser submission identities bypass both quotas,
+but still require `pg_durable.enable_superuser_instances = on`.
+
+Both settings accept integers from `0` to `2147483647`; `0` disables that
+individual quota. They are superuser-settable (`SUSET`) and can be changed
+without a restart. For example, an administrator can set defaults for new
+connections by an application role:
+
+```sql
+ALTER ROLE app_user IN DATABASE appdb SET df.max_concurrent_per_user = 200;
+ALTER ROLE app_user IN DATABASE appdb SET df.max_instances_per_user = 25000;
+```
+
+Server settings in `postgresql.conf` or `ALTER SYSTEM` take effect on reload.
+Session `SET` overrides affect `'caller'` starts. **`transaction_mode => 'new'`
+uses the separate session's role/database/server defaults**, not the caller's
+session overrides. Configure those defaults to apply the same policy to both
+modes; `SET ROLE` alone does not load a role's connection defaults.
+
+At a quota, the call fails with SQLSTATE `53400` (`configuration_limit_exceeded`)
+and a `HINT` identifying the relevant setting. Finishing work frees concurrent
+capacity once the engine records the terminal state. Cancellation releases it
+after the engine processes the request, not immediately when `df.cancel()`
+returns. Changing the displayed instance status does not release capacity.
+Neither rollback, completion, nor cancellation frees retained-instance capacity.
+Capacity is freed when [automatic reconciliation](#automatic-reconciliation)
+removes the engine state and protected submission accounting. If reconciliation
+is disabled, or retained work has not aged out, an administrator must adjust
+retention or raise the quota.
+
+The default retained quota equals reconciliation's 10,000-terminal-instance
+retention cap. A role with 10,000 recent instances can therefore remain blocked
+until its history ages out under `pg_durable.retention_days` (30 days by default);
+running reconciliation alone need not free a slot. Tune the quota for the
+expected submission volume and retention window before deploying. Retention
+settings require a restart; the quota itself can be changed at runtime.
+
+Concurrent starts by the same role in the same database wait for the preceding
+**submitting transaction** to commit or roll back before checking capacity.
+They do not wait for workflows to finish. Keep submitting transactions short
+and use PostgreSQL `lock_timeout` to bound admission waits. Counts include
+current committed rows and the caller's own changes even under `REPEATABLE READ`
+or `SERIALIZABLE`. Rollback, including rollback to a savepoint, releases the
+transaction lock and discards the graph, but does not erase submission accounting:
+engine enqueue commits independently. Rolled-back starts consume active capacity
+until the engine records their failure and retained capacity until reconciliation
+removes their engine state and accounting. Quota counts can therefore exceed the
+number of visible `df.instances` rows.
+
+Perform business-table updates before starting workflows when possible, and
+avoid leaving submitting transactions idle. Consider PostgreSQL
+`idle_in_transaction_session_timeout` as well as `lock_timeout`. Applications
+must retry entire transactions on deadlock or serialization failures; admission
+adds another lock/read to transactions, including workflow SQL that starts
+other workflows. Both quotas must be `0` to disable admission locking.
+
+Do not follow a `'caller'` start with a `'new'` start for the same role and
+database while the caller transaction remains open: the separate session waits
+on the caller's admission lock and eventually hits its 30-second timeout.
+Commit or roll back the caller transaction first.
+
+These are instance quotas, not calls-per-second or byte limits. They do not
+bound an individual graph's size or history. Changing a quota does not cancel
+or delete existing work; it only controls admission of new instances. The
+defaults also apply after a binary-only upgrade, without `ALTER EXTENSION UPDATE`.
+Existing workflows whose SQL calls `df.start()` also encounter the new limits;
+an over-quota SQL step can fail its enclosing workflow.
+
+Fresh 0.2.10 installations include an owner-leading index for quota checks.
+Run `ALTER EXTENSION pg_durable UPDATE TO '0.2.10'` to add it on an existing
+installation. Binary-only upgrades still enforce quotas, but the local count
+can scan other roles' history until the index is installed. Active accounting
+also checks the role's retained IDs against the engine, so larger histories
+increase admission cost even with the index. Measure representative workloads
+before substantially raising retention or quotas. Disabling the active quota
+skips execution-status lookups, but retained accounting still reads the protected
+submission ledger. Disabling both quotas skips admission checks. Successful
+enqueues are recorded even while limits are disabled.
+
+---
+
 ## Connection Limits
 
 pg_durable uses separate limits for control management, provider, origin metadata,

docs/api-reference.md

@@ -527,6 +527,13 @@ database privileges. Install in `pg_durable.database` first and wait for control
 readiness before creating satellites; see
 [Multi-Database Support](../USER_GUIDE.md#multi-database-support).
 
+Non-superuser starts are subject to `df.max_concurrent_per_user` (default `100`)
+and `df.max_instances_per_user` (default `10000`) in their origin database.
+Rejection raises SQLSTATE `53400` with a tuning hint before any insert or enqueue.
+Concurrent starts by the same role wait until the earlier submitting transaction
+ends. See [Instance Limits](../USER_GUIDE.md#instance-limits) for quota scope,
+retention, transaction interactions, and configuration.
+
 #### transaction_mode
 
 Selects which transaction the *start itself* runs in. It changes nothing about
@@ -893,6 +900,30 @@ These settings are configured via `ALTER SYSTEM SET` or `postgresql.conf`. See e
 
 ---
 
+### df.max_concurrent_per_user / df.max_instances_per_user
+
+Admission quotas for `df.start()`, per submitting role and origin database.
+
+| Property | Concurrent quota | Retained-instance quota |
+|----------|------------------|-------------------------|
+| Default | `100` | `10000` |
+| Counts | Queued instances and executions not terminal in the protected engine state | Retained submissions, including starts whose caller transaction or savepoint rolled back |
+| Range | `0` to `2147483647`; `0` disables | `0` to `2147483647`; `0` disables |
+| Context | `SUSET` | `SUSET` |
+
+Superuser submission identities are exempt. Administrators can change either
+setting at runtime, including per-role/database connection defaults.
+Cancellation frees active capacity after the engine processes the request;
+changing display status alone does not release it.
+Rollback removes the graph, not the protected submission charge. It remains
+active until the engine records failure and retained until reconciliation removes
+the engine record and accounting.
+`transaction_mode => 'new'` uses its separate session's defaults and does not
+inherit the caller's session `SET` overrides. See
+[Instance Limits](../USER_GUIDE.md#instance-limits) for tuning and admission waits.
+
+---
+
 ### pg_durable.http_security
 
 Server-wide outbound HTTP policy for `df.http()` and `df.http_multipart()`.

docs/upgrade-testing.md

@@ -356,6 +356,67 @@ gate, so they never need to be added to the exclude list.
 Each schema-changing PR should add a section here documenting what changed,
 what the upgrade script handles, and any backward compatibility considerations.
 
+### v0.2.9 → v0.2.10 (unreleased)
+
+#### `df.start()` admission quotas (#139)
+
+- **DDL/B1:** The upgrade adds `idx_instances_submitted_by` on
+  `df.instances(submitted_by, id)`, matching fresh installation. No new
+  extension tables or grants are needed. Both start wrapper symbols use the same
+  admission check against existing `df.instances` columns and protected engine
+  state, including on 0.2.2+ schemas without `ALTER EXTENSION UPDATE`.
+  No status grants are revoked or used as an
+  accounting authority; queued or not-yet-materialized engine instances count
+  as active. Cancellation releases capacity only after the engine processes it.
+  B1 exercises a non-superuser reaching its quota on each supported schema.
+  Old schemas remain correct without the index but can scan other roles'
+  history. The normal index build blocks concurrent writes; set `lock_timeout`
+  and schedule the upgrade appropriately on large installations.
+- **Worker-managed accounting:** Worker schema version `3` adds protected
+  `_submissions` in the control provider schema, indexed by origin and submitting
+  role. The worker creates it before publishing readiness, including against old
+  extension schemas. Each new root's ledger row and provider enqueue commit in
+  one transaction; neither caller nor savepoint rollback removes the charge.
+  Admission unions these records with visible legacy instance IDs, avoiding
+  double counting. Bounded reconciliation removes ledger entries only after
+  no corresponding engine instance, execution, history, queue item, or lock
+  remains and origin metadata is confirmed absent. If metadata deletion fails
+  or the origin cannot be inspected, the ledger preserves trusted retirement
+  evidence rather than reclassifying stale metadata as active legacy work.
+  The enqueue WorkItem and orchestration input are unchanged.
+  ID allocation reserves the ledger key before the local instance row and retries
+  collisions in either store, including IDs retained after caller rollback.
+  The ledger transaction stays uncommitted until enqueue; local persistence
+  failures release the reservation without leaving a charge or queue item.
+  Pre-upgrade orphans that already lost their `df` identity cannot be attributed
+  retroactively; existing orphan cleanup remains responsible for them.
+- **Behavior change:** New superuser-settable GUCs
+  `df.max_concurrent_per_user` (default `100`) and `df.max_instances_per_user`
+  (default `10000`) apply immediately after loading the new binary.
+  Existing work remains intact; starts at or above either quota fail with
+  SQLSTATE `53400`. Set a quota to `0` to disable it.
+- **Concurrency:** Admission uses a database-local, per-role transaction
+  lock with a tag that SQL advisory-lock functions cannot construct, and a
+  current snapshot, including under repeatable-read and serializable isolation.
+  E2E tests cover concurrent commit, rollback, savepoint rollback, lock timeout,
+  caller-writable status, attempts to squat on public advisory-lock keys,
+  repeated transaction/savepoint rollbacks, and quota recovery after cleanup.
+- **B2/replay:** No data migration, runtime schema detection, durable payload,
+  or orchestration changes. Workflows that call `df.start()` from SQL are
+  subject to the new quotas when that SQL activity executes. The separate
+  `'new'` session uses its own configuration defaults, not caller session
+  overrides. See [Instance Limits](../USER_GUIDE.md#instance-limits).
+- **Scale validation:** Warm-cache PG17 measurements on a development server,
+  with 500,000 synthetic retained instances distributed across 100 roles,
+  reduced the median local ID/count query from about 35 ms to 19 ms with the
+  owner-leading index; retained-only counting fell from about 30 ms to 11 ms.
+  The protected-engine batch for 5,000 owned IDs took about 244 ms in that run.
+  These are query measurements, not throughput guarantees. The index reduces
+  local cross-role scanning, not the per-owner engine-lookup cost. Only retained
+  counting runs when the active quota is disabled; both disabled skips admission.
+  These pre-ledger measurements do not include the subsequent protected-ledger
+  union; ledger lookups use the origin/owner-leading index.
+
 ### v0.2.8 → v0.2.9
 
 The post-tag changes in #379, #388, #389, #390, and #380 belong to the 0.2.9

scripts/test-e2e-docker.sh

@@ -44,11 +44,13 @@ SKIP_TESTS=(
     "47_http_dsl_disabled"
     "48_http_allow_all"
     # Needs the "reconcile" phase GUCs (reconcile_interval=2, retention_days=0)
-    # so a pass acts within the 30s test window. The single fixed-config Docker
+    # so a pass acts within the test window. The single fixed-config Docker
     # container keeps the production defaults (reconcile_interval=3600,
     # retention_days=30), under which the orphan is never reclaimed in time.
     "54_reconcile_orphans"
     "76_multi_database_reconcile"
+    "95_start_limits_rollback_cleanup"
+    "96_start_limits_failed_metadata_cleanup"
     "88_multi_database_maintenance_backlog"
     # Require the force-drop phase (one execution slot, reconciliation disabled).
     "78_multi_database_force_drop"

scripts/test-e2e-local.sh

@@ -221,7 +221,7 @@ phase_for_test() {
         46_connection_limit_startup_validation)
             echo "connlimit-startup"
             ;;
-        54_reconcile_orphans|76_multi_database_reconcile)
+        54_reconcile_orphans|76_multi_database_reconcile|95_start_limits_rollback_cleanup|96_start_limits_failed_metadata_cleanup)
             echo "reconcile"
             ;;
         88_multi_database_maintenance_backlog)