Axum: make production configuration and local storage explicit #396

Part of #391.

Description

Make application startup independent of source-tree layout and current working directory. Required configuration should fail clearly before readiness, and embedded storage should have an honest production support model.

Current implementation on audited main

• run_app reads EDGEZERO__ environment configuration and baked store metadata without requiring edgezero.toml, which already supports standalone binaries; see dev_server.rs, lines 333-L384.
• KV paths are derived under cwd-relative .edgezero/; declared KV initialization is required, but embedded redb permits only one owning process per database file.
• Config stores discover a manifest root or fall back to cwd-relative .edgezero/local-config-<id>.json. Missing files produce empty stores. Malformed files cause registry entries to be dropped with a warning, not startup failure; see dev_server.rs, lines 439-L480.
• Config is an in-memory snapshot loaded at startup. Secrets read the process environment. Invalid bind environment settings can fall back with warnings.
• The KV implementation documents lazy expiry and database growth; see key_value_store.rs, lines 1-L54. A delete-and-restart troubleshooting message is not suitable as a production recovery procedure.

Prerequisite PR and scope correction

PR #275 is a prerequisite for integration because application construction, error types, configuration reads, and generated applications change. It introduces fallible Hooks::configure and core-owned App::build::(platform); the Axum runner now propagates construction failure before dispatch. See dev_server.rs at the inspected PR head.

This addresses application configuration assembly, but does not by itself supply production filesystem paths, mandatory local config policy, backup/restore, or replicated embedded storage. The inspected runner still drops malformed local-config registry entries with warnings, and KV paths remain cwd-relative. Use the accepted fallible assembly/configuration contracts for the residual work. Do not introduce a parallel startup validation API or change the new portable configuration contracts.

Proposed solution

1. Build on PR feat: implement portable outbound HTTP and request lifecycle contracts #275's fallible App::build::(platform), Hooks::configure, and bounded configuration-read contracts. Re-audit their startup/store behavior before choosing a fix location.
2. Add explicit validated production data/config locations and strictness policy. Required malformed or missing configuration, inaccessible storage, and invalid required runtime values must fail before readiness. Use existing metadata or app-owned requirements; the adapter must not guess mandatory secret names.
3. Support read-only config mounts and an explicit writable KV mount, independent of cwd or source-tree discovery. Preserve documented permissive development defaults.
4. Document startup snapshots and restart-based updates unless live reload is separately designed. Keep diagnostic redaction compatible with the new typed error contracts.
5. Keep redb support single-process/single-replica per database and provide safe backup/restore, permissions, capacity/expiry maintenance, and recovery guidance. A new distributed store remains out of scope.

Done when

• Re-audit the accepted PR feat: implement portable outbound HTTP and request lifecycle contracts #275 implementation and record which checks are already covered. Add only residual changes and missing production acceptance evidence.
• A standalone release binary starts from an unrelated working directory using only configured paths and required environment inputs.
• Missing or malformed required config, unreadable mounts, invalid bind values, and locked KV files fail before readiness with a non-zero exit and redacted diagnostic.
• Development mode retains its documented permissive behavior.
• Tests prove KV values survive a stop/restart on the same mounted data directory and that a second process cannot silently share the file.
• A backup/restore check reads stored values back; the guide never recommends deleting production data as ordinary lock recovery.
• Read-only config and writable-data mounts work with the runtime UID used by the Docker image.
• Config snapshot behavior, secret update behavior, and replica limitations are documented.

Related work and boundaries

Do not change the portable raw ConfigStore contract as part of this task; coordinate with #324. Local provisioning remains under #288, and secret-management CLI work remains under #284. Reuse path-validation rules instead of introducing another directory convention.

Affected areas

Adapter -- Axum; Core (routing, extractors, middleware) only for shared validation hooks; CLI (new, build, deploy, dev); Documentation.

Dependencies

Primary prerequisite: PR #275. Build residual production work on its accepted contracts, not the superseded main implementation.

Delivery priority and consumer contract

First runtime integration priority alongside #392. #398/#399 packaging/CI preparation can progress independently; final required-config/mount/persistence checks must use this accepted contract.

Use Trusted Server #1141 config handoff, #1139 store wiring, and #843 secret migration as concrete consumer requirements. Expose framework-level configuration/data locations and fail-fast hooks; leave the application's blob validation, mandatory fields, signing bootstrap/rotation, and feature policy downstream.

Do not infer that EdgeZero redb support wires Trusted Server's current UnavailableKvStore. Document single-writer local storage and source-independent paths for #400; multi-replica state requires a separately supported shared backend. Neither Docker nor Kubernetes should silently select unsafe storage semantics.

Implementation and relationship tracking

Implementation: draft PR #406, shared with #392, on issue-392-396-production-hosting-spec. Its base is PR #275 (docs/outbound-http-spec). Accepted-prerequisite integration and remaining storage/configuration acceptance are still required before merge. GitHub Development now links this issue to PR #406, whose description includes Closes #396. The issue remains open while the PR is draft.