Add AWS configuration and secret-store providers for self-hosted applications #410

Part of #391. Downstream consumer: Trusted Server #1236.

Description

Add reusable AWS-backed configuration and secret-store providers to EdgeZero for native self-hosted applications, initially through Axum. Applications should select their configuration and secret providers independently of their HTTP runtime and ordinary KV provider.

The preferred initial AWS services are AWS AppConfig for application configuration and AWS Secrets Manager for secret values. Evaluate DynamoDB-backed configuration as an alternative for applications that need a straightforward key-to-value store rather than AppConfig's deployment workflow. The design must settle the supported configuration provider set before implementation; this issue does not require implementing every AWS storage service.

These are store providers, not a new AWS HTTP runtime or a Trusted Server-owned storage implementation. The application should continue using EdgeZero's portable store interfaces, typed configuration envelopes and secret references.

Problem and current behavior

On main at 683202c66948146ec360f490126f1d60f920288b:

• ConfigStore is a read-only async key lookup. The Axum implementation loads per-store local JSON files into an in-memory map.
• SecretStore retrieves bytes by store name and key. The Axum implementation reads process environment variables.
• There is no reusable AWS configuration or Secrets Manager provider identified in the current adapter set. A downstream application must inject cloud-managed values into files/environment itself or add its own AWS integration.
• Axum: make production configuration and local storage explicit #396 addresses production paths, required-startup behavior and mounted local inputs. It does not provide remote AWS stores. ConfigStore::get is envelope-only on Fastly but a plain passthrough on Axum/Cloudflare #324 tracks inconsistent raw configuration lookup behavior and must not be bypassed with another provider-specific interpretation of configuration values.

Local files and environment delivery remain valid choices. They do not, by themselves, supply native access to AWS-managed versioning, authorization, configuration delivery and secret rotation. Requiring each application to implement that access repeats authentication, resource mapping, caching, retry and error-handling code.

Several mismatches need an explicit design:

1. AppConfig delivers versioned configuration documents through a session/polling protocol, while ConfigStore::get retrieves one string by key. A document cannot be treated as an ordinary map without defining that mapping.
2. Secrets Manager identifies AWS secrets and versions, while EdgeZero names logical stores and keys. The mapping must preserve namespaces and cannot silently select arbitrary JSON fields.
3. Cloud access can fail because of missing resources, credentials, authorization, decryption, throttling or connectivity. Those failures must not all become a successful missing-key result or a silent fallback to development values.
4. Provider polling and caching do not make an application hot-reload its configuration or safely rotate signing keys. Those are separate application lifecycle decisions.
5. AWS SDK/runtime dependencies must not leak into portable core or break existing WASM adapters.

AWS service roles

DynamoDB and Valkey can hold bytes, but storing secret bytes in an ordinary database is not equivalent to adopting Secrets Manager's secret lifecycle. Do not merge configuration, application state and secrets into one provider merely because all can be expressed as key/value pairs.

Proposed solution

1. Design the provider boundary first

Write and approve a design spec before implementation. Use accepted PR #275 configuration-read and fallible app-construction contracts; coordinate with #396 startup requirements and #324 raw-value semantics.

Keep existing portable ConfigStore, SecretStore, handles and typed app-config resolution as the application-facing contracts. Put AWS access in optional native provider code, isolated from core and WASM dependency graphs. Determine whether a dedicated provider crate or a smaller feature-gated module is appropriate. Do not impose new Send requirements on portable futures.

Allow AWS config plus environment secrets, or local config plus AWS secrets, as well as both AWS providers. Selecting an AWS store must not replace the application's HTTP adapter or unrelated KV store.

2. Define explicit bindings and bootstrap

Map logical store IDs and keys to reviewed AWS resource references. Configuration bindings need region and the selected AppConfig application/environment/profile identifiers, or DynamoDB table/key/value mappings if that provider is selected. Secret bindings need a store/key-to-secret-ID mapping and a documented version selector.

Bootstrap region, resource references and credential discovery outside the configuration being fetched. Do not require AWS credentials to be retrieved from the same secret provider needed to authenticate the first request.

Use supported AWS temporary-credential discovery and workload roles for production, with a documented local developer authentication path. Do not require long-lived AWS keys in application config, images or source files. Separate runtime read permissions from operator publication/mutation permissions.

3. Implement configuration delivery without changing its meaning

For AppConfig, choose a supported access path in the spec: AWS AppConfig Agent or direct AppConfig Data API. AWS recommends the Agent where practical. There is no requirement to implement both initially.

Define how a config key maps to a profile/document and how raw strings versus an explicit key/value document are represented. Preserve existing BlobEnvelope bytes and typed validation. Do not silently flatten arbitrary JSON, resolve secret values into provider blobs or require every raw config value to be an envelope.

If direct polling is selected, serialize session-token ownership, use each token once, respect the returned polling interval and preserve the previous configuration when AWS returns an unchanged empty response. An empty unchanged response is not a missing configuration. Specify recovery after token expiry and ambiguous request failure.

Load required configuration before readiness and expose a coherent validated snapshot. Start with startup-loaded application configuration and restart-based adoption unless live reload is explicitly designed. Provider refresh alone must not patch a running typed application state.

If DynamoDB config is selected, define consistency, version selection, item schema, value limits and missing-item behavior. Implementing it must not be presented as satisfying the separate EC CAS/strong-prefix-list contract.

4. Implement Secrets Manager reads and version behavior

Return string secrets as their UTF-8 bytes and binary secrets as decoded bytes, according to the chosen AWS client API. Preserve byte fidelity. Return the complete secret value by default; JSON-field extraction requires a separately explicit contract.

Document the default AWSCURRENT behavior and support or explicitly defer pinned version selection. Reject contradictory version settings. Define when a running application adopts a changed secret and how this interacts with startup snapshots and caches.

The provider retrieves secrets. It does not automatically make application-specific key rotation safe, mutate signing stores, create rotation Lambdas or grant runtime write permissions. Secret-management CLI operations remain coordinated with #284.

5. Bound work and make failures observable safely

Reuse process-owned clients rather than creating AWS clients per request. Define cache size, value-size bounds, refresh ownership, operation budgets and bounded retry behavior. Avoid one cloud lookup per incoming HTTP request when a cached startup value is sufficient.

Distinguish an actual missing value from denied access, credential failure, decryption failure, throttling and transport failure. Required inputs must fail before readiness. Never silently fall back to local files or environment values after selecting an AWS provider.

Specify stale-value policy separately for configuration and secrets, including whether a revoked or rotated secret may remain cached. Never cache an access-denied response as a successful miss. Keep secret material, credential material and provider response bodies out of logs and public errors. Preserve the accepted framework's deadline capability distinctions rather than claiming all AWS credential/network work is forcibly cancellable.

Test plan

• Run existing configuration and secret-store contract suites against deterministic provider fixtures.
• Test logical namespace isolation, existing and missing keys, raw values, BlobEnvelope roundtrip and nested secret-reference resolution.
• Test AppConfig document mapping, unchanged responses, version changes, token expiry, polling interval and concurrent reads against the selected access path.
• Test string and binary secrets, pinned/current versions, refresh or restart adoption and cache invalidation policy.
• Exercise access denied, expired credentials, decryption failure, throttling, timeout, malformed/oversized values and startup failure. Assert that captured logs and public errors do not contain fixture secrets.
• Default tests must not require AWS credentials or network access. Mock service behavior is contract evidence, not proof of AWS IAM or deployed service behavior.
• Provide a separately authorized, opt-in disposable AWS acceptance run with least-privilege credentials, cost limits and cleanup ownership. Read configuration and secrets through a generated native application, change their versions, restart or refresh as documented, and verify denied-access behavior.
• Build existing native and provider-WASM targets to prove optional AWS dependencies have not changed their supported contracts.

Done when

• An approved design settles the initial config provider set, AppConfig access path, key/resource mapping, credential bootstrap, snapshot/cache policy, version selection and runtime update behavior.
• Native applications can independently bind AWS configuration and Secrets Manager through the existing portable interfaces.
• Required AWS configuration and app-declared secret resolution complete before readiness, with typed, redacted failures and no silent development fallback.
• Existing raw config values, typed envelopes and nested secret references retain their meanings.
• String/binary secret fidelity and the chosen version/cache policy are covered by regression tests.
• Deterministic tests cover provider protocol, concurrency and failure cases without credentials or network access.
• Authorized AWS acceptance demonstrates real resource mapping, least-privilege access, version/update behavior and cleanup. Record the exact tested revision and environment.
• Documentation includes native setup, IAM/read permissions, resource bindings, local development, limits, startup/update behavior and a clear support matrix. Examples use fictional resources and contain no actual credentials.
• Existing workspace and WASM gates pass with AWS providers disabled; enabled native-provider gates pass as well.

Non-goals

• DynamoDB EC identity-graph KV, generic conditional KV/CAS, distributed rate limiting or multi-replica EC certification. Those need a separate owning issue and backend contract.
• Valkey, PostgreSQL, S3 or Parameter Store provider implementations in this task unless the approved design deliberately replaces the proposed initial config provider.
• AWS Lambda/Lambda@Edge adapters, infrastructure provisioning, managed-cloud deployments or changes to the container publisher.
• Automatic application hot reload, secret mutation, signing-key rotation or rotation Lambda implementation.
• AWS provider support on Fastly, Cloudflare or Spin WASM runtimes in the first release. Preserve their existing builds and document native-only scope.
• Trusted Server-specific AWS clients, required secret names, configuration schemas or EC policy inside EdgeZero.

Related work and ownership

• Make the Axum adapter production-ready for self-hosted deployments #391 owns self-hosted runtime support. This is an additional AWS store-provider slice, not a prerequisite for every local-store deployment.
• Axum: make production configuration and local storage explicit #396 owns local production paths and startup behavior; Axum: define a production server lifecycle with bounded graceful shutdown #392 owns lifecycle/readiness hooks.
• PR feat: implement portable outbound HTTP and request lifecycle contracts #275 is the accepted-contract integration prerequisite, not authorization to ship against an unaccepted draft branch.
• ConfigStore::get is envelope-only on Fastly but a plain passthrough on Axum/Cloudflare #324 owns raw config semantics; Add secrets CLI for typed app-config secret management #284 owns secret-management CLI workflow.
• Trusted Server #1236, #1141, #843 and #1142 own downstream binding, app validation, secret requirements and readiness policy. EdgeZero supplies reusable AWS access; downstream apps still need to adopt it.

References

• AWS AppConfig configuration retrieval
• AWS configuration-store quotas
• Secrets Manager GetSecretValue
• AWS SDK for Rust authentication
• Parameter Store, AppConfig and Secrets Manager comparison

Affected areas

Adapter -- Axum; Core store integration only where required; CLI provider selection/documentation; Documentation; CI / Tooling.