Axum: define a production server lifecycle with bounded graceful shutdown #392

Part of #391.

Description

Make the native server safe to start, monitor, and stop under a container supervisor. Define a production hosting contract without replacing the useful development runner by a second unrelated server implementation.

Current implementation on audited main

• Generated main.rs calls dev_server::run_app; see templates/src/main.rs.hbs, lines 1-L5.
• AxumDevServerConfig contains only the bind address and enable_ctrl_c. run_app creates its own multi-thread Tokio runtime and installs the shared request service.
• The graceful-shutdown future waits only for signal::ctrl_c(). No SIGTERM branch or explicit drain deadline exists; see dev_server.rs, lines 308-L321.
• The runner exposes no production readiness/draining contract. Store setup happens before serving, but there is no documented health-check integration.

Prerequisite PR and scope correction

PR #275 is the implementation prerequisite for the native HTTP serving path. It is open and draft as of this update; do not treat its contracts as merged or its advertised capabilities as production certification.

It replaces the old per-request block_in_place bridge with owned Hyper HTTP/1 connections on a current-thread runtime/LocalSet, and adds response completion/abort ownership. The current connection loop still selects Ctrl-C or an internal shutdown channel and exits the LocalSet; see dev_server.rs at the inspected PR head. Response-egress lifecycle is not process-wide graceful draining.

Build this issue on those connection and completion contracts. The remaining work is supervisor signals, readiness, tracked connection draining, and a bounded process shutdown policy, not another HTTP serving bridge.

Proposed solution

1. Extend PR feat: implement portable outbound HTTP and request lifecycle contracts #275's owned connection runner with explicit production startup, ready, draining, and stopped states. Reuse its service, completion ownership, and store setup.
2. Handle SIGTERM on Unix and SIGINT on supported native platforms. Retain application-provided shutdown for embedding and tests.
3. Track accepted connections and in-flight response attempts. On shutdown, stop admission, make readiness fail, and request graceful draining while the local runtime remains alive. After a configurable deadline, terminate remaining work with the correct existing completion outcome. Dropping the runtime immediately is not graceful draining.
4. Expose health/readiness integration with explicit route ownership or an operator-configured management listener. Do not silently override app routes. Dependency checks remain app-owned.
5. Preserve development convenience and document protocol support, process shutdown, and forced termination separately from response-egress completion.

Done when

• Re-audit the accepted PR feat: implement portable outbound HTTP and request lifecycle contracts #275 implementation and record which checks are already covered. Add only residual changes and missing production acceptance evidence.
• Subprocess tests show SIGTERM and SIGINT both initiate graceful shutdown.
• A finite in-flight request completes during draining; a never-ending request or stream cannot keep the process alive past the configured deadline.
• Readiness stays false during startup, becomes true after required initialization, and becomes false when draining starts.
• Invalid configuration, initialization failure, and bind failure return a non-zero exit status with a useful diagnostic.
• Health checks do not override existing application routes, and embedding callers can supply shutdown without installing signal handlers.
• State transitions, timeout defaults, and orchestrator termination-grace recommendations are documented.

Related work and boundaries

Coordinate admission and cancellation with the request-limits sub-issue. Configuration validation belongs to the configuration/storage sub-issue. TLS termination and platform-specific orchestrators are operator responsibilities. This task does not require a public API rename or broad server refactor.

Affected areas

Adapter -- Axum; Documentation; CI / Tooling.

Dependencies

Primary prerequisite: PR #275. Build residual production work on its accepted contracts, not the superseded main implementation.

Delivery priority and packaging integration

First runtime integration priority alongside #396. Build on accepted PR #275's connection/completion contracts, not a parallel server. #398 Dockerfile preparation and #399 read-only architecture CI can proceed before this closes; a successful build/HTTP smoke must not be called production shutdown certification.

Publish the actual signal, readiness, in-flight drain, forced-exit, and exit-status contract for reuse by #398/#399 and #400. Test process-wide SIGTERM with an in-flight response, not just a unit shutdown future. Trusted Server #1142 owns application health/startup readiness and should consume these hooks. Do not assume a generic process probe proves an application's stores, signing, or consent configuration is usable.

Implementation and relationship tracking

Implementation: draft PR #406, shared with #396, on issue-392-396-production-hosting-spec. Its base is PR #275 (docs/outbound-http-spec). Accepted-prerequisite integration and remaining acceptance are still required before merge. GitHub Development now links this issue to PR #406, whose description includes Closes #392. The issue remains open while the PR is draft.