[egerev]

Summary

The delivery dispatcher gets a production deployment on an isolated egress path, separate operator and self-service lanes, a heartbeat, and the operator command that switches ANYM-519 self-service delivery on only when the API, matcher and dispatcher run one version. Push epic ANYM-517, story S2b.

• deploy/compose.delivery.yaml (crit 1): python -m messaging_service.delivery_worker with only private/delivery.dsn, the host policy and the operator receiver secret directory (all read-only binds, create_host_path: false), read-only root, uid 10001, cap_drop: ALL, pids 64. Networks: delivery_db (internal bridge dk-deliverydb0, no IPv6, only db and delivery) and delivery_egress (bridge dk-delivery0, no IPv6). dns: [${DELIVERY_RESOLVER}] so Docker's embedded resolver queries the one allowed resolver from the container namespace. Not on backend. The API gets the same policy mount for registration. Mounts sit under /run/secrets (promotion guard roots).
• scripts/delivery_egress_check.py (crit 2): rules --resolver IP prints the DOCKER-USER chain (DNS to the resolver, then DROP host-local, RFC 1918, tailnet 100.64/10, link-local incl. metadata, reserved; then public 443; DROP rest), INPUT drops for the host on both bridges (dk-delivery0, dk-deliverydb0), ip6tables drops; rules --undo. check --open/--blocked runs inside the container (stdlib, piped over stdin); a blocked target must be an IP literal and passes only on timeout or no route (refused, unresolved or malformed fail). --open-dns/--blocked-dns IP send one UDP DNS query straight to a resolver; listen --port N --seconds S is a bounded synthetic listener for the matcher (it has none) as a positive/negative control.
• Lanes (crit 3): run() starts one operator lane (its own DNS slots) and three endpoint lanes. The endpoint lane picks under a transaction advisory lock the due hint of the least recently served key with no live lease, so at most one attempt per key is in flight; one scheduler (next_endpoint_item) orders endpoint hints and endpoint tests together: one key order (least recently served, the lanes' shared served), the same lock and in-flight rule (KEY_IN_FLIGHT), so a key's test never runs beside its hint or ahead of a waiting key; a lane yields to a test when the next key's item is a test. A key holds at most 32 queued endpoint hints across subscriptions and kinds (under the per-key budget lock): a due events or silence hint stays held (held_reason budget, retried after 1 min), test_subscription answers rate_limited. Lane exceptions print one stdout line; the lane continues. The fence is unchanged. --once and dispatch_hints(store) keep the old serial pass.
• Heartbeat (crit 4): every 30 s, written only while every lane finished a pass within 60 s; the existing 120 s capability rule reads it.
• Promotion (crit 5): production.md, migration 050: a one-time owner-authorized first enrollment (production.py refuses a selected service with no running container and never restarts the database): enroll tags of the verified runtime image, up --no-start --no-deps, docker network connect --alias db of the running database, start, then both overlays/services in production.json and the next main SHA applied by hand: plan A prepare/apply and plan B seamless_release.py prepare/promote (the serving API gains the policy mount from plan A's documents); undo before and after that apply. The ops plan releases the 050 SHA by hand under the pause file (the SQL inventory pauses autodeploy). Later deploys include both. Card candidate: enrollment of new services in production.py.
• Activation (crit 6): migration 050 component_heartbeats (one row per component, RLS: each role writes only its own). API (lifespan task), matcher (after a successful match) and dispatcher write version + schema every 30 s. python -m messaging_service set-delivery-features --enable|--disable; enable refuses (exit 1, JSON with the component) on an absent, stale, other-version or other-schema heartbeat. Disable cancels queued endpoint hints under the ANYM-519 fence.
• Schema 49 → 50 everywhere (backup tables 79, restore range, docs).

Data egress: no new destination or datum. Hint POSTs keep their ANYM-519/218 bodies; they now leave only over dk-delivery0. Heartbeat rows (component, package version, schema, time) stay in PostgreSQL. New stdout lines delivery_lane_failed (lane name, exception class) and delivery_heartbeat_failed (exception class); no telemetry, IDs, URLs or content. DNS from the dispatcher goes only to DELIVERY_RESOLVER, set to the host's current upstream (no new third party).

Tests

• tests/test_event_delivery.py: operator hint within 15 s while 4 endpoints hang for the timeout and one raises (worker alive); one attempt per key in flight and lanes split; least recently served key first; the key-wide queue bound holds the third subscription's wake and fires it after drains, refuses a test at the bound, events stay fetchable; a key's endpoint test and hint never in flight together (both directions, a second endpoint of the key; an expired test lease does not block); endpoint tests follow the lanes' least-recently-served key order; hints and tests share one key order; heartbeat stops when a lane stalls and reports version/schema; each role writes only its own heartbeat row (real role SQL).
• tests/test_capability_availability.py: a lane pulse keeps webhook operational for 120 s, then dispatcher_heartbeat_stale.
• tests/test_delivery_endpoints.py: enable refuses for an absent (older) matcher, other version, other schema, stale; disable cancels queued endpoint hints, no POST, events fetchable.
• tests/test_coverage_subscription_deploy.py (Compose render), tests/test_production.py (overlay list + docs fragment), tests/test_delivery_egress_check.py (new: TCP/UDP-DNS probes, synthetic listener, rules and undo), tests/test_compose_runtime.py (ANYM-417 sandbox: delivery service, networks, secrets).
• Changed existing assertions (names and ids kept): schema pins 49 → 50, table count 78 → 79.
• All new tests fail on base 832744b (missing APIs, overlay, script, table). Schema pins in existing tests moved 49→50 and backup table counts 78→79 (assertion values only, names unchanged); the application set gains delivery. Local after r1 fixes: event_delivery, delivery_endpoints, delivery_policy, delivery_egress_check, coverage_subscription_deploy, subscriptions all passed; ruff clean.

Spec: ANYM-534, ANYM-519, ANYM-218
Delivers: ANYM-534
Spec impact:

• ANYM-534 (Delivers): criteria 1-6 built; criterion 7 is the post-merge operation (plan in the review packet).
• ANYM-519: the S2a gate gets its operator command; criteria unchanged. ANYM-218: dispatcher, role and fence kept.
  Lane: deploy/messaging/scripts/schema
  Review level: important (security, access, data)
  Change path: OWNER (compose overlay, host firewall step, role SQL: gateway_role.sql, subscription_role.sql, delivery_role.sql; migration 050)
  Owner note: Диспетчер вебхуков запускается в проде в своей изолированной сети (только PostgreSQL и выход наружу на 443 через правила хоста); медленные агентские эндпоинты больше не задерживают операторские. Самообслуживание включается командой только когда API, matcher и диспетчер на одной версии.
  Migration: 050 (-- rollout: manual), after 049. Role files reinstalled after 050: gateway, subscription, delivery.
  Deployed baseline: production behind 045-049; this PR goes live after 049 (ANYM-519). Exact deployed SHAs, host production.py and gateway versions are not verified here (no production access); the ops worker checks them read-only before step 1.
  Product context: read (related specs ANYM-218, 163, 135, 417, 252, 257, 517; projection in the review packet, context.md).
  Rollout: operation plan in the review packet (backup gate, migrate, role SQL, private files, owner-approved firewall step, host file list, check, enable, grok-live enablement; one live register only after the owner's exact confirmation).
  Review record: Astra r1 (gpt-6-astra): CHANGES 1 blocking, 6 major, 3 minor, fixed; Astra r2 (scoped): B1, M1, M4, M5, m1, m3 resolved; Astra r3 (scoped): M2, m2 resolved, M3, M6 major remained, fixed in this head (one scheduler over hints and tests; http-production.json overlays in the procedure); Sol r4 (gpt-6.1-sol xhigh, scoped M3 + M6): both resolved, no new findings, APPROVE.
  Model: claude-opus-5-5 high
  SP: 5

🤖 Generated with Claude Code