feat(delivery): dispatcher in production on an isolated egress path, lanes, heartbeats, delivery gate, schema 50 (ANYM-534)

.claude/rules/migrations.md

@@ -5,8 +5,8 @@ paths:
 <!-- Sources: AGENTS.md, CONTRIBUTING.md, FILE_INDEX.md, MEDIA_STORAGE.md, messaging_service/store.py -->
 # Messaging SQL migrations
 - Applied migrations are immutable: `Store.migrate()` compares each file's sha256 with `archive_migrations` and raises `migration_checksum_mismatch`. Never edit, rename, reorder or reformat an existing file.
-- Schema changes get a new next-numbered file (current head: `049_delivery_endpoints.sql`, schema 49).
-- In the same change: register it in the `migrate()` list, raise the `version>49` runtime ceiling, update backup `TABLES`/`V*_TABLES` and the restore range in `backup.py`, and update role grants (`*_role.sql`).
+- Schema changes get a new next-numbered file (current head: `050_component_heartbeats.sql`, schema 50).
+- In the same change: register it in the `migrate()` list, raise the `version>50` runtime ceiling, update backup `TABLES`/`V*_TABLES` and the restore range in `backup.py`, and update role grants (`*_role.sql`).
 - Check roles/ACL and a repeated `migrate()` (CONTRIBUTING.md). Update the documented schema version (README, RELEASING).
 - Keep source revisions, original provider IDs, archive IDs and acknowledgement semantics stable. Older runtimes deliberately reject a newer database.
 - Expand, then contract: a migration works with both the deployed and the new code; drop, rename or narrow only in a later release, once no deployed code uses the old shape. Merging an expand migration is AUTO+review; destructive SQL is OWNER (`scripts/change_path.py`). The agent of a task (a ready card) applies its migration after the backup gate of docs/operations/production.md (AGENTS.md section 2). Reserve the next number in the PR title while you hold the `schema` lane.

ARCHITECTURE.md

@@ -27,7 +27,7 @@ principal sources.
 | AnyMsg WhatsApp Connector | Primary whatsmeow adapter `whatsapp/whatsmeow/` for new connections; whatsapp-web.js fallback and rollback adapter `whatsapp/*.mjs`; `whatsapp_collector.py`, pairing/identity modules, `whatsapp_capabilities.py`, WhatsApp deployment examples | Synthetic provider/IPC tests for both adapters, command lanes, history bounds, identity evidence, account provider identity, passive receipts and media handling |
 | AnyMsg Media Storage & Index | `blobs.py`, `storage_admission.py`, `media_policy.py`, `media_requests.py`, `attachment_metadata.py`, `uploads.py`, `file_extract.py`, `file_index.py`, `file_lifecycle.py`, indexer and file-lifecycle SQL roles, `deploy/compose.file-lifecycle.yaml` | Original integrity, scoped retrieval, bounded original requests and staged uploads, extraction safety, passages, provenance, derivative invalidation and reuse, operator-only original residency |
 | AnyMsg AI Enrichment | `enrichment.py`, `enrichment_providers.py`, `codex_enrichment.py`, `file_understanding.py`, `resource_admission.py`, enrichment SQL role, file-understanding deploy overlay and role, enrichment requirements and root `enrichment.*.example.json` | Budgets, isolated processing, profile validation, result fences, independent file descriptions and untrusted model output |
-| AnyMsg Subscriptions & Events | `subscriptions.py`, `subscription_worker.py`, `subscription_consumption.py`, `processing_events.py`, `event_delivery.py`, `delivery_worker.py`, subscription and delivery SQL roles, `packages/wake-relay/` (host receiver) | Key ownership, committed frontier, lifecycle/version checks, fair matching, atomic checkpointing, receipt-bound ACK, signed hints that never ACK, artifact references and the host relay that turns a hint into one assistant run |
+| AnyMsg Subscriptions & Events | `subscriptions.py`, `subscription_worker.py`, `subscription_consumption.py`, `processing_events.py`, `event_delivery.py`, `delivery_worker.py`, `heartbeats.py`, subscription and delivery SQL roles, `deploy/compose.subscriptions.yaml`, `deploy/compose.delivery.yaml`, `scripts/delivery_egress_check.py`, `packages/wake-relay/` (host receiver) | Key ownership, committed frontier, lifecycle/version checks, fair matching, atomic checkpointing, receipt-bound ACK, signed hints that never ACK, artifact references and the host relay that turns a hint into one assistant run |
 | AnyMsg Backup & Recovery | `backup.py`, crypto, runner, worker, receiver, R2 receiver/verifier, restore worker and recovery modules; R2 examples, verifier systemd units and R2 requirements | Consistent table/blob manifests, encrypted transfer, restore equality, quarantine and retention anchors |
 | AnyMsg Deployment & Verification | SQL migrations, `schema.sql`, `__init__.py` version, `release.py`, `telemetry.py`, root `Dockerfile`, Compose overlays, messaging packaging (`services/messaging/` Dockerfile, `pyproject.toml`, `hatch_build.py`, lockfiles), root `uv.lock`, release and branch-protection examples in `deploy/`, `services/release-router/`, `.github/`, `scripts/`, `tools/dev/` | Schema checksums, least privilege, images/packages, release routing, synthetic network drills and authentic CI evidence |
 | AnyMsg Python Client & Operator CLI | `services/messaging/anymsg/`, `__main__.py` operator entry point, root `client_check.py` and `media_benchmark.py`, packaged assets | Safe HTTP client, token-file CLI, source assets and explicit owner administration |

CLAUDE.md

@@ -9,7 +9,7 @@ compatibility meaning too. Every service keeps its own lockfile, environment,
 process and state.
 
 ## Layout
-- `services/messaging`: AnyMsg server (Python 3.13, PostgreSQL 18, schema 49)
+- `services/messaging`: AnyMsg server (Python 3.13, PostgreSQL 18, schema 50)
 - `services/gateway`: common MCP gateway (HTTP only, imports no provider code)
 - `services/calendar`: family calendar (Google API/CalDAV), has its own AGENTS.md
 - `services/calls`: voice calls; `services/calls/frontend`: Vite UI

README.md

@@ -38,7 +38,7 @@ use SHA-256 deduplication; references to every source message stay separate.
 [Conversation view and compatibility](CONVERSATION.md) explain message kinds,
 ordering uncertainty and the last known dialogue turn.
 
-Development version: **0.9.19 — experimental**, PostgreSQL schema 49.
+Development version: **0.9.19 — experimental**, PostgreSQL schema 50.
 
 The expanded live-operation catalog covers messages, registered attachments,
 reactions, groups, contacts and profile fields. Access requires explicit per-key

RELEASING.md

@@ -3,7 +3,7 @@
 The public repository is a fresh source export, not the Personal OS git history.
 Never publish correspondence, operational state, backups, session data or tokens.
 Python and npm distribution versions must match `messaging_service.__version__`.
-The current source version is 0.9.19 (experimental), with PostgreSQL schema 49.
+The current source version is 0.9.19 (experimental), with PostgreSQL schema 50.
 The published npm/PyPI release remains 0.9.9 until a separate registry release.
 Sending is implemented in the source with explicit account opt-in and per-key
 permissions; see [SENDING.md](SENDING.md).

SECURITY.md

@@ -26,7 +26,9 @@ agent's own headers; never message text, names or file content. SSRF controls: h
 on port 443, every resolved address public (no private, loopback, CGNAT, multicast,
 NAT64, 6to4, Teredo or IPv4-mapped answers), resolution again before each attempt with
 the address pinned, TLS hostname verification, no redirects, no proxies, and the
-host policy rechecked before each attempt. Signing secrets and header values are
+host policy rechecked before each attempt. The dispatcher (ANYM-534) shares a network
+only with PostgreSQL and egresses over its own bridge, which host rules limit to DNS on one configured resolver and
+public port 443 (no host, tailnet, private, link-local, metadata or IPv6 address). Signing secrets and header values are
 write-only for the API role and never returned, logged or audited. This is privilege
 separation, not encryption: database owners and anyone who can decrypt a backup can
 read them. Details: [SUBSCRIPTIONS.md](SUBSCRIPTIONS.md#self-service-webhook-endpoints-schema-49).

SUBSCRIPTIONS.md

@@ -242,9 +242,22 @@ A receiver HTTP success is not a verified host turn; all hints and probes report
 `host_turn_verified=false`. Host adapters must separately prove that integration.
 
 Run `python -m messaging_service.delivery_worker` using only the dispatcher DSN
-and private receiver secret mounts. `--once` is a bounded operator pass. Keep the
-matcher on its existing internal-only network; public HTTPS egress belongs to a
-separate dispatcher deployment. No dispatcher is started by installing the
+and private receiver secret mounts; in production `deploy/compose.delivery.yaml`
+runs it (docs/operations/production.md, migration 050). `--once` is one serial
+bounded operator pass. Otherwise it runs separate lanes (ANYM-534): one for operator
+receivers, with its own DNS lookup slots, and three for self-service endpoints. An
+endpoint lane leases at most one attempt per key at a time, counting the key's
+endpoint tests, and serves the least recently served key first. A key holds at
+most 32 queued endpoint hints across its subscriptions and hint kinds: past it a
+due hint stays held (`held_reason` `budget`) and fires on a later tick, and
+`test_subscription` answers `rate_limited` (events stay fetchable). So endpoints that hang for the full timeout never delay operator
+receivers, and a failing endpoint never stops a lane. The dispatcher heartbeat is
+written every 30 s and only while every lane finished a pass within 60 s. The
+dispatcher reaches PostgreSQL only over an internal network that holds nothing
+else, and the internet over its own bridge, which host rules restrict to DNS on
+the one configured resolver and public port 443; host-bound packets are dropped on
+both of its bridges (`scripts/delivery_egress_check.py`). Keep the matcher on its
+existing internal-only network. No dispatcher is started by installing the
 package. Operator `--disable-receiver UUID` uses the same authorization fence as
 in-flight delivery and cancels queued hints. No internal/private IP exceptions
 are enabled.
@@ -467,7 +480,11 @@ the switch, the host policy and the limits.
 both), with the same version check and fence as a receiver.
 
 **Switch.** Every endpoint tool, the `delivery` field and endpoint staging answer
-`delivery_disabled` until the operator enables the feature (S2b). With the switch
+`delivery_disabled` until the operator enables the feature with
+`python -m messaging_service set-delivery-features --enable` (ANYM-534). That refuses
+until the API, the matcher and the dispatcher report this version and schema
+through fresh heartbeats (`component_heartbeats`, under 120 s); `--disable` is the
+rollback. With the switch
 off no endpoint hint is staged, and turning it off cancels queued endpoint hints
 and tests. The webhook transport in capabilities then shows `operational: false`
 with reason `delivery_disabled`.
@@ -484,8 +501,8 @@ suffix matches on a DNS label boundary (`example.com` matches `a.example.com`, n
 `badexample.com`) and deny wins. A missing or invalid file, unknown keys or an
 empty allowlist without `allow_any_public: true` leave the policy unset:
 registration answers `delivery_policy_unset` and the webhook transport shows
-reason `delivery_policy_unset`. Set `allow_any_public: true` only after S2b's
-isolation test passes. The policy is read again before every attempt; a host that
+reason `delivery_policy_unset`. Set `allow_any_public: true` only after the
+dispatcher's egress check passes (production.md, migration 050). The policy is read again before every attempt; a host that
 leaves it cancels the hint (`endpoint_policy_refused`). After editing the file, run
 `python -m messaging_service reload-delivery-policy`: it waits for every in-flight
 POST and registration that read the old file, then cancels queued hints and tests of

deploy/compose.delivery.yaml

@@ -0,0 +1,76 @@
+# Opt-in overlay (ANYM-534), after compose.subscriptions.yaml. Install delivery_role.sql as owner and
+# configure a separate LOGIN inheriting messaging_event_delivery; never use the gateway or matcher DSN.
+# The dispatcher reaches PostgreSQL only over delivery_db (internal bridge dk-deliverydb0, no API, matcher,
+# gateway or collector) and the internet only over its own bridge dk-delivery0. The host rules from
+# `scripts/delivery_egress_check.py rules` drop host-bound packets on both bridges and restrict dk-delivery0 to
+# DNS on DELIVERY_RESOLVER and public 443 (docs/operations/production.md). An explicit `dns` makes Docker's
+# embedded resolver query DELIVERY_RESOLVER from the container namespace, so those rules apply to DNS too.
+services:
+  delivery:
+    image: anymsg-delivery:${MESSAGING_VERSION:-0.9.19}
+    build:
+      context: ..
+      target: runtime
+    command: [python, -m, messaging_service.delivery_worker]
+    init: true
+    restart: unless-stopped
+    depends_on:
+      db: {condition: service_healthy}
+    environment:
+      DATABASE_URL_FILE: /run/secrets/database_url
+      DELIVERY_POLICY_FILE: /run/secrets/delivery-policy.json
+    volumes:
+      - type: bind
+        source: ./private/delivery.dsn
+        target: /run/secrets/database_url
+        read_only: true
+        bind: {create_host_path: false}
+      - type: bind
+        source: ./private/delivery-policy.json
+        target: /run/secrets/delivery-policy.json
+        read_only: true
+        bind: {create_host_path: false}
+      # Operator receiver secret files (subscription_receivers.secret_file names /run/secrets/receivers/<file>).
+      - type: bind
+        source: ./private/delivery-receivers
+        target: /run/secrets/receivers
+        read_only: true
+        bind: {create_host_path: false}
+    networks: [delivery_db, delivery_egress]
+    dns: ["${DELIVERY_RESOLVER:-1.1.1.1}"]
+    read_only: true
+    user: "10001:10001"
+    cap_drop: [ALL]
+    security_opt: [no-new-privileges:true]
+    tmpfs: ["/tmp:size=16m,noexec,nosuid"]
+    mem_limit: 256m
+    cpu_shares: 512
+    pids_limit: 64
+    stop_grace_period: 15s
+    logging:
+      driver: json-file
+      options: {max-size: 5m, max-file: "2"}
+  db:
+    networks: [delivery_db]
+  # Self-service registration checks the same host allowlist as the dispatcher.
+  api:
+    environment:
+      DELIVERY_POLICY_FILE: /run/secrets/delivery-policy.json
+    volumes:
+      - type: bind
+        source: ./private/delivery-policy.json
+        target: /run/secrets/delivery-policy.json
+        read_only: true
+        bind: {create_host_path: false}
+
+networks:
+  delivery_db:
+    internal: true
+    enable_ipv6: false
+    driver_opts:
+      com.docker.network.bridge.name: dk-deliverydb0
+  delivery_egress:
+    driver: bridge
+    enable_ipv6: false
+    driver_opts:
+      com.docker.network.bridge.name: dk-delivery0