diff --git a/ORIGINAL-SPEC-DRAFT.md b/ORIGINAL-SPEC-DRAFT.md deleted file mode 100644 index b8863b9..0000000 --- a/ORIGINAL-SPEC-DRAFT.md +++ /dev/null @@ -1,213 +0,0 @@ -> **Superseded.** This is the original single-document draft the full specification at [compositecode.github.io/substratalapps.com](https://compositecode.github.io/substratalapps.com/) was elaborated from — kept here for history, in the project root rather than the docs site, specifically so it's never mistaken for current spec. It contradicts current decisions in several structural ways (no [Tenant](https://compositecode.github.io/substratalapps.com/domain-model/tenancy/) concept, single-method auth, a 3-question access algorithm instead of the current 2-axis union model) — if anything below conflicts with the live docs site or [`decisions.md`](https://compositecode.github.io/substratalapps.com/decisions/), the live site wins, always. See [README.md](README.md) and the [Changelog](https://compositecode.github.io/substratalapps.com/changelog/#2026-10-03)'s initial-publication entry for how this became that. - -# Substratal Hub — Access & Identity API - -**Status:** Draft for elaboration · **Scope:** platform API that sits behind substratalapps.com · **Audience:** engineers turning this into a full technical spec - -## 1. What this document is - -Substratal Apps is the hub a customer lands on to see every app, product, and service they've purchased from Substratal, sign into it with one identity, and manage their account across all of them. This document describes the API that powers that hub: the system of record for *who* a user is, *what* they're allowed to touch, and *how* each app they own is configured for them. - -It is deliberately a level above endpoint-by-endpoint detail. The goal is a domain model and resource map solid enough that a full OpenAPI spec, data schema, and build plan can be written from it without re-litigating the core decisions. Section 13 lists the decisions this draft deferred — resolve those before the full spec is final. - -**In scope:** user identity, roles & permissions, per-app entitlement (the purchase → access → on/off lifecycle), per-user profile data (global and per-app), per-user settings (global and per-app), the events/tokens downstream apps need to trust the hub. - -**Out of scope (adjacent systems this API talks to, but doesn't replace):** payment processing, app-specific business logic/data, marketing site content, support ticketing. - -## 2. Domain model - -| Entity | What it represents | -|---|---| -| **User** | A person with an account on Substratal. One identity, used everywhere. | -| **Organization** *(optional — see §13)* | A billing/access group of Users, e.g. a team plan. If absent, every User is implicitly its own org of one. | -| **Application** | A catalog entry for a product/service Substratal sells — "Product," not "running process." The app a user ultimately gets routed to. | -| **Role** | A named bundle of Permissions. Scoped either to the whole platform (`PlatformRole`) or to one Application (`AppRole`). | -| **Permission** | An atomic, checkable capability (`billing.manage`, `app.timetrack.export`, …). | -| **Entitlement** | The join between a User (or Org) and an Application: *do they own it, and is it currently switched on.* This is the record behind "turn an app on/off for a user." | -| **Profile** | Display/contact identity data. Global (name, avatar, email, locale) and, separately, per-app (fields only that app cares about — e.g. a display handle used only inside one product). | -| **Setting** | Configuration, not identity. Global (site-wide preferences) and per-app (that app's preferences for that user), each layered over an app-level default. | -| **Order / Subscription** | The commerce record (what was bought, when, recurring or not) that an Entitlement traces back to. Owned by billing, referenced here. | -| **Audit Event** | An immutable record of who changed what access, when. | - -### 2.1 Entity relationships - -```mermaid -erDiagram - USER ||--o{ ENTITLEMENT : holds - ORGANIZATION ||--o{ USER : contains - ORGANIZATION ||--o{ ENTITLEMENT : "holds (org-wide)" - APPLICATION ||--o{ ENTITLEMENT : "granted via" - ENTITLEMENT ||--o| ORDER : "traces to" - USER ||--|| PROFILE : "has (global)" - USER ||--o{ APP_PROFILE : "has, per app" - APPLICATION ||--o{ APP_PROFILE : scopes - USER ||--|| SETTINGS : "has (global)" - USER ||--o{ APP_SETTINGS : "has, per app" - APPLICATION ||--o{ APP_SETTINGS : scopes - USER }o--o{ PLATFORM_ROLE : assigned - USER }o--o{ APP_ROLE : "assigned, per app" - APP_ROLE }o--|| APPLICATION : scopes - PLATFORM_ROLE ||--o{ PERMISSION : grants - APP_ROLE ||--o{ PERMISSION : grants -``` - -The two relationships worth pausing on: an **Entitlement** is independent of a **Role**. Entitlement answers "can this user reach App X at all, right now" (the on/off switch). Role answers "once inside, what can they do." A support agent can hold a platform Role that lets them *see* every user's entitlements without ever holding an entitlement to a single app themselves — access-to-manage-access is a different axis from access-to-use. - -## 3. Access control model - -Three independent questions, in the order they get checked: - -1. **Is the app itself live for this user?** → `Entitlement.status == active` for (User, Application). If no, stop — nothing below matters. This is the literal on/off toggle an admin or billing event flips. -2. **Does the user hold a Role — platform or app-scoped — that grants the Permission being invoked?** → union of `PlatformRole` permissions and, for that Application, `AppRole` permissions. -3. **Org-level override, if Organizations are in play** → an Org admin's Role can widen or narrow what members of that org inherit by default. - -``` -allow(user, application, permission) := - entitlement(user, application).status == "active" - AND permission ∈ effective_permissions(user, application) -``` - -`effective_permissions` is the union of permissions from every `PlatformRole` assigned to the user, plus every `AppRole` assigned to the user scoped to that `application`. - -**Platform roles** (examples): `superadmin`, `support`, `billing_admin`, `member`. These govern the hub itself — managing other users, issuing refunds, viewing audit logs. - -**App roles** are defined per Application by whoever owns that app's catalog entry (e.g. an app might define `admin` / `editor` / `viewer`), but the hub is the registry and the enforcement point for them — a downstream app doesn't maintain its own user table, it asks the hub. - -## 4. Resource & data model - -### 4.1 User -| Field | Notes | -|---|---| -| `id` | opaque, stable | -| `email` | unique, verified flag | -| `status` | `active` \| `invited` \| `suspended` \| `deleted` | -| `auth` | password hash / SSO subject, MFA state — likely delegated to an identity provider rather than owned here (see §13) | -| `created_at`, `last_login_at` | | -| `organization_id` | nullable | - -### 4.2 Profile (global) & AppProfile -| Field | Global `Profile` | `AppProfile` (per User × Application) | -|---|---|---| -| name, avatar_url | ✓ | — (inherits unless overridden) | -| display_handle | — | ✓ app-local identity, e.g. a username only meaningful inside one product | -| contact email/phone | ✓ | — | -| custom fields | — | ✓ arbitrary JSON the owning app defines for itself | - -### 4.3 Settings (global) & AppSettings -| Field | Global `Settings` | `AppSettings` (per User × Application) | -|---|---|---| -| locale, timezone, theme | ✓ | inherits from global unless set | -| notification preferences | ✓ (site-wide) | ✓ (per-app channel overrides) | -| app-specific preferences | — | ✓ arbitrary JSON, validated against that app's declared settings schema | - -Resolution order for any setting an app reads: **AppSettings → Settings → Application's declared default.** The hub stores the override layers; it does not need to know what each key means, only enforce the schema the Application registered for itself. - -### 4.4 Application (catalog entry) -| Field | Notes | -|---|---| -| `id`, `slug`, `name`, `description`, `icon_url` | catalog/display | -| `launch_url` / `sso_redirect` | where the hub sends an authenticated user | -| `settings_schema` | JSON Schema the app declares, so AppSettings can be validated centrally | -| `available_app_roles` | roles this app defines, for role-assignment UI | -| `visibility` | `public` (anyone can acquire) \| `invite_only` \| `internal` | - -### 4.5 Entitlement -| Field | Notes | -|---|---| -| `id`, `user_id` (or `organization_id`), `application_id` | | -| `status` | `active` \| `disabled` \| `expired` \| `revoked` | -| `source` | `purchase` \| `trial` \| `admin_grant` \| `org_seat` | -| `order_id` | nullable link to commerce record | -| `starts_at`, `ends_at` | for trials/subscriptions | -| `disabled_reason` | free text / enum, set when an admin flips it off manually | - -### 4.6 Role / Permission -| Field | Notes | -|---|---| -| `Role.id`, `name`, `scope` | `scope` = `platform` or `application_id` | -| `Role.permissions[]` | list of Permission keys | -| `Permission.key` | dotted string, e.g. `users.manage`, `app..export` | -| `UserRoleAssignment` | (`user_id`, `role_id`, nullable `application_id` for app-scoped roles) | - -### 4.7 Order / Subscription (referenced, owned by billing) -Minimal fields the access API needs to react to: `id`, `user_id`/`organization_id`, `application_id`(s), `status` (`paid`, `past_due`, `cancelled`, `refunded`), `renews_at`. A billing webhook updates Entitlement status off of these — this API doesn't process payment. - -### 4.8 Audit Event -`id`, `actor_user_id`, `action` (`entitlement.disabled`, `role.assigned`, …), `target_user_id`, `application_id` (nullable), `before`/`after` snapshot, `timestamp`. - -## 5. API surface - -Resource-oriented REST, versioned (`/v1`), JSON, cursor pagination on list endpoints, idempotency keys on mutating ones. - -| Group | Endpoints | Purpose | -|---|---|---| -| **Auth** | `POST /v1/auth/login`, `POST /v1/auth/token/refresh`, `POST /v1/auth/sso/{provider}/callback`, `POST /v1/auth/logout` | Issue/refresh the session; see §7 for what downstream apps do with the result. | -| **Users** | `GET/POST /v1/users`, `GET/PATCH/DELETE /v1/users/{id}`, `POST /v1/users/{id}/suspend` | Account lifecycle. | -| **Profile** | `GET/PATCH /v1/users/{id}/profile`, `GET/PATCH /v1/users/{id}/apps/{appId}/profile` | Global vs. per-app profile. | -| **Settings** | `GET/PATCH /v1/users/{id}/settings`, `GET/PATCH /v1/users/{id}/apps/{appId}/settings` | Global vs. per-app settings, validated against the app's schema. | -| **Applications (catalog)** | `GET /v1/applications`, `GET/POST/PATCH /v1/applications/{id}` (admin-only write) | The app catalog shown on the hub. | -| **Entitlements** | `GET /v1/users/{id}/entitlements`, `POST /v1/users/{id}/entitlements`, `PATCH /v1/entitlements/{id}` (`{status: "disabled"}` is the on/off switch), `DELETE /v1/entitlements/{id}` | Grant, list, toggle, revoke access to an app. | -| **Roles** | `GET/POST /v1/roles`, `GET /v1/users/{id}/roles`, `POST/DELETE /v1/users/{id}/roles/{roleId}` | Assign/remove platform or app roles. | -| **Permissions** | `GET /v1/permissions`, `GET /v1/users/{id}/applications/{appId}/effective-permissions` | The resolved-permission check downstream apps and UI call. | -| **Organizations** *(if in scope)* | `GET/POST /v1/organizations`, `.../members`, `.../entitlements` | Team/seat management. | -| **Audit** | `GET /v1/audit-events` (filterable by user/app/actor) | Compliance and support investigation. | -| **Webhooks** | `GET/POST /v1/webhooks` (subscription management) | See §6. | - -## 6. Key workflows - -**Purchase → access.** Billing completes an Order → billing service calls (or the hub listens for) a webhook → hub creates/updates an `Entitlement(status=active, source=purchase, order_id=...)` → hub emits `entitlement.granted` → the hub UI now shows the app as available, and the downstream app (if it independently checks) sees it on next token refresh or introspection call. - -```mermaid -sequenceDiagram - participant Billing - participant Hub as Hub API - participant App as Downstream App - participant User - - Billing->>Hub: order.paid webhook (user, application) - Hub->>Hub: upsert Entitlement(status=active, source=purchase) - Hub->>Hub: emit entitlement.granted - User->>Hub: opens dashboard - Hub-->>User: app now listed as available - User->>App: launches app (SSO redirect) - App->>Hub: verify token / introspect - Hub-->>App: valid, entitlement=active, effective_permissions=[...] -``` - -**Admin turns an app off for a user.** Support/admin calls `PATCH /entitlements/{id} {status: disabled, disabled_reason}` → hub writes an Audit Event → emits `entitlement.revoked` → any live session the user has in that app is invalidated on next check (the app should re-verify on a short interval, not just at login — see §7). - -**Role change.** Admin assigns/removes a Role → `effective-permissions` for that user+app changes immediately → downstream app picks it up the next time it checks (introspection or next token refresh), not necessarily mid-session unless the app subscribes to the webhook. - -## 7. How downstream apps trust the hub - -Each Application is a largely independent product; it shouldn't maintain its own user table. Two mechanisms, meant to be used together: - -- **Short-lived JWT**, issued at login/SSO-redirect, carrying `sub` (user id), `org_id`, and claims for *that specific app* (`entitlement_status`, `roles`, `effective_permissions`), signed by the hub. Cheap to verify locally, but can go stale within its TTL — keep the TTL short (minutes) for anything access-sensitive. -- **Introspection endpoint** (`GET /v1/users/{id}/applications/{appId}/effective-permissions`) for anything that needs a live answer — e.g. before a destructive action, or to immediately honor an admin's "turn this off" click rather than waiting out a token TTL. -- **Webhooks** (`entitlement.granted`, `entitlement.revoked`, `role.assigned`, `role.removed`) for apps that want to react immediately (kill an active session) rather than poll. - -## 8. Non-functional requirements - -- **AuthN:** OAuth2/OIDC for user-facing login (own IdP or delegate to one — see §13); service-to-service calls (billing → hub, app → hub) use scoped API keys or mTLS. -- **AuthZ enforcement point:** the hub is the only writer of Role/Entitlement state. Apps read, they don't maintain parallel copies. -- **Multi-tenancy:** every record scoped by `organization_id` where orgs are in play; row-level isolation, not just application-level filtering. -- **Audit:** every write to Entitlement, Role assignment, or admin-initiated Profile/Settings change is logged immutably. -- **Rate limiting & idempotency:** standard on all mutating endpoints; idempotency key required on Entitlement and Order-triggered writes since billing webhooks retry. -- **Versioning:** `/v1` now; additive changes preferred over breaking ones; deprecation window policy to be set in the full spec. - -## 9. Phased build - -| Phase | Delivers | -|---|---| -| **MVP** | User, Profile (global only), Application catalog, Entitlement (grant/revoke/toggle), Platform Roles only, basic audit log. Enough for "see my apps, click to launch, admin can turn access on/off." | -| **Phase 2** | App-scoped Roles & Permissions, per-app Profile and Settings with schema validation, webhooks, introspection endpoint. | -| **Phase 3** | Organizations/seats, delegated admin (org admin manages their own members), richer audit/compliance views, SCIM-style provisioning for larger customers. | - -## 10. Open decisions (resolve before writing the full spec) - -1. **Identity provider:** build auth in-house, or delegate to an IdP (Auth0, Clerk, WorkOS, Cognito, …)? Changes what §4.1/`auth` and §7's JWT issuance actually look like. -2. **Organizations:** is multi-seat/team access a day-one requirement, or does every account start as a single user and orgs get bolted on later? Affects whether `organization_id` is load-bearing now or added in Phase 3. -3. **Downstream app architecture:** are the "apps" separately hosted services that need SSO + token verification (as assumed above), or more like modules/iframes inside one deployment, where access control could live entirely in the hub's own session? This is the single biggest fork in the API shape. -4. **Billing system of record:** which processor, and does it push webhooks to the hub or does the hub poll it? Determines the contract in §2's Order entity and §6's purchase workflow. -5. **Settings schema ownership:** does each Application register its own JSON Schema with the hub (as assumed in §4.3), or does the hub stay fully opaque to app settings and just store/return a blob? -6. **Session model for revocation:** how fast must "turn off this user's access" take effect inside an already-open app session — immediately (requires apps to poll/subscribe) or "by next login" (much simpler, weaker guarantee)? diff --git a/PLAN.md b/PLAN.md deleted file mode 100644 index 75cda9c..0000000 --- a/PLAN.md +++ /dev/null @@ -1,122 +0,0 @@ -# Build Plan - -How to actually build the API this project specifies, in what order, and which document governs each piece. This is a plan for *implementation* — what to build and when. The API itself is specified at **[compositecode.github.io/substratalapps.com](https://compositecode.github.io/substratalapps.com/)**; this document doesn't repeat that spec, it sequences building it. - -See [README.md](README.md) for why this file — and the others alongside it — live in the project root instead of the docs site. - -1. [Before writing code](#before-writing-code) -2. [Build order](#build-order) -3. [Phase 0 — Infrastructure skeleton](#phase-0--infrastructure-skeleton) -4. [Phase 1 — MVP](#phase-1--mvp) -5. [Phase 2 — Per-app customization & trust](#phase-2--per-app-customization--trust) -6. [Phase 3 — Marketplace & scale features](#phase-3--marketplace--scale-features) -7. [Cross-cutting, build throughout](#cross-cutting-build-throughout) -8. [Status](#status) -9. [Test-mode User expiry](#test-mode-user-expiry) - ---- - -## Before writing code - -Read, in this order — each is short, and skipping one tends to show up as a wrong assumption two phases later: - -1. **[Getting Started](https://compositecode.github.io/substratalapps.com/getting-started/)** and **[Domain Model](https://compositecode.github.io/substratalapps.com/domain-model/)** — the nine-ish entities and how they relate. Everything else assumes this. -2. **[Access Control](https://compositecode.github.io/substratalapps.com/access-control/)** — the one algorithm (`allow(user, application, permission)`) every endpoint ultimately defers to. -3. **[Database Schema](https://compositecode.github.io/substratalapps.com/domain-model/database-schema/)** — Postgres types, constraints, indexes, Row-Level Security. This is the literal starting migration. -4. **[DEPLOYMENT.md](DEPLOYMENT.md)** — where it all runs, and the infrastructure build checklist that interleaves with the phases below. - -## Build order - -The short version: **infrastructure skeleton, then the access-control spine, then everything else radiates outward from Entitlement.** Every phase below is buildable and demoable on its own — this isn't "build the whole schema, then wire up endpoints," it's vertical slices that happen to accumulate into the full spec. - -## Phase 0 — Infrastructure skeleton - -Before any endpoint handler exists. Maps directly to [DEPLOYMENT.md → Build checklist](DEPLOYMENT.md#build-checklist) steps 1–4: - -- [ ] AWS account, `us-east-1`, under Organizations — [DEPLOYMENT.md → AWS account & region](DEPLOYMENT.md#aws-account--region). -- [ ] **AWS Budgets + Cost Anomaly Detection first**, before any billable resource — [DEPLOYMENT.md → Cost guardrails](DEPLOYMENT.md#cost-guardrails). Not optional, not step 2. -- [ ] Aurora Serverless v2 cluster, Data API enabled, Secrets Manager credential. -- [x] The local-dev repository/data-access interface described in [DEPLOYMENT.md → Local development](DEPLOYMENT.md#local-development), *before* the first table is written against it — retrofitting this seam after business logic exists is real rework, building it first is not. -- [x] CI running the [Testing strategy](https://compositecode.github.io/substratalapps.com/non-functional-requirements/#testing-strategy) categories (contract tests against `openapi.yaml`, unit tests on `effective_permissions`, integration tests against local Postgres) against an empty schema — so the pipeline exists before there's much to test, not retrofitted. - -**Exit criteria:** `GET /v1/health` deployed and returning `200` through the real Lambda → API Gateway → (nothing else yet) path. Proves the skeleton, not the product. - -## Phase 1 — MVP - -Matches the [Roadmap → MVP](https://compositecode.github.io/substratalapps.com/roadmap/#mvp) feature scope, sequenced for implementation: - -1. **`users`, `user_identities`, `sessions`, `refresh_tokens`, `auth_tokens`, `profiles`, `settings` tables + native [Auth](https://compositecode.github.io/substratalapps.com/api-reference/auth/) (signup, login + MFA challenge, refresh with reuse detection, logout, TOTP, password reset/change, email verification, invitations, JWKS via KMS, SES email, and **`POST /v1/auth/app-tokens`**, without which no Application can log anyone in) and [Users](https://compositecode.github.io/substratalapps.com/api-reference/users/)/[Profiles](https://compositecode.github.io/substratalapps.com/api-reference/profiles/)/[Settings](https://compositecode.github.io/substratalapps.com/api-reference/settings/) (global only) endpoints.** `POST /v1/users` first specifically — per [Non-Functional Requirements → Transaction boundaries](https://compositecode.github.io/substratalapps.com/non-functional-requirements/#transaction-boundaries), its "also creates a Profile, Settings, and default Role" behavior is one transaction; build it as one from the start rather than three calls glued together later. Build `user_identities` as the multi-method table it's specified as from day one (see [Decisions #1](https://compositecode.github.io/substratalapps.com/api-reference/auth/#native-auth-and-per-organization-sso)) even though only `method: 'password'` rows exist until Phase 2+ — `sso_connections` and the broker integration are explicitly deferred, not this phase's problem. -2. **Platform Roles & the [permission catalog](https://compositecode.github.io/substratalapps.com/domain-model/roles-and-permissions/#platform-permission-catalog)** — seed the `superadmin`/`support`/`billing_admin`/`member` rows. This is also literally how the first real `superadmin` gets into the system — see [Non-Functional Requirements → Bootstrapping](https://compositecode.github.io/substratalapps.com/non-functional-requirements/#authentication): seeded directly, not through the API. -3. **`applications` table + [Applications](https://compositecode.github.io/substratalapps.com/api-reference/applications/) (list/fetch; writes admin-only).** -4. **`entitlements` table + [Entitlements](https://compositecode.github.io/substratalapps.com/api-reference/entitlements/), especially the `PATCH` toggle.** This is the feature the rest of the MVP exists to support — see [Access Control](https://compositecode.github.io/substratalapps.com/access-control/) for why Entitlement and Role are evaluated as independent axes, which should shape the authorization-check code from the first line, not get refactored in later. -5. **`audit_events` table, written transactionally alongside every Entitlement/Role write from step 2 onward** — not bolted on after. See [Non-Functional Requirements → Transaction boundaries](https://compositecode.github.io/substratalapps.com/non-functional-requirements/#transaction-boundaries): an Audit Event that doesn't correspond to a committed change is worse than no Audit Event, which is only true if it's written in the same transaction from day one. -6. **`api_keys` table + [API Keys](https://compositecode.github.io/substratalapps.com/api-reference/api-keys/), including app-confined permissions.** An Application's backend needs an app-scoped key with confined `entitlements.manage` to reflect its own billing outcomes. That's the developer's one required integration besides login. - -**Exit criteria:** the [Quickstart](https://compositecode.github.io/substratalapps.com/quickstart/) walkthrough runs end to end against the real deployment — create a user, grant an app, check access, turn it off. This is the MVP's own acceptance test, already written. - -## Phase 2 — Per-app customization & trust - -Matches [Roadmap → Phase 2](https://compositecode.github.io/substratalapps.com/roadmap/#phase-2): - -1. **App-scoped Roles** — extend the Phase 1 Role/permission machinery with `scope: ` and `available_app_roles` on Application. Not a new system, the same tables with a column that was already reserved for this. -2. **`app_profiles`/`app_settings` tables**, with the [typed generated-column mechanism](https://compositecode.github.io/substratalapps.com/domain-model/database-schema/#typed-fields-per-application-views-and-expression-indexes) for declared `settings_schema` fields. -3. **[Webhooks](https://compositecode.github.io/substratalapps.com/api-reference/webhooks/)** — delivery queue, signing, the retry/backoff schedule. This is also where the [Trust Model's](https://compositecode.github.io/substratalapps.com/trust-model/#how-fast-does-revocation-need-to-land) **required** immediate-revocation integration becomes real: an Application can't actually meet that requirement until this ships, so treat Phase 2 webhooks as a hard prerequisite for telling any real Application developer their integration is compliant, not a nice-to-have. -4. **Live introspection** (`effective-permissions`) — the other half of the [Trust Model](https://compositecode.github.io/substratalapps.com/trust-model/), alongside the app token from Phase 1's Auth work. -5. **[Organizations](https://compositecode.github.io/substratalapps.com/api-reference/organizations/)** — `organizations`, `organization_memberships`, org-wide Entitlements with `member_scope`/`member_overrides`. Pulled forward from a hypothetical later phase per [Domain Model → Organization](https://compositecode.github.io/substratalapps.com/domain-model/users-and-organizations/#organization) — both individual and team end users are expected from early on. -6. **[Tenancy](https://compositecode.github.io/substratalapps.com/domain-model/tenancy/)** — `tenants` table, the `shared` tier (free, automatic, every Application owner gets one). `isolated`/`dedicated_region` provisioning mechanics ([DEPLOYMENT.md → Tenancy tiers](DEPLOYMENT.md#tenancy-tiers--where-they-run)) can lag slightly behind the schema, since they're support-run and gated by an actual customer need, not a launch-day requirement. -7. **Stripe Billing** — [DEPLOYMENT.md → Stripe Billing](DEPLOYMENT.md#stripe-billing) and [API Reference → Billing](https://compositecode.github.io/substratalapps.com/api-reference/billing/) in full: the `subscriptions` table, `applications.access_tier`, the platform-managed `free`/`subscription` Entitlements and their reconcile, the catalog sync script, the User and Organization Checkout/Portal endpoints, the webhook handler, and the daily Organization quantity sync, in Stripe test mode first. This is what makes [Pricing](https://compositecode.github.io/substratalapps.com/pricing/) real rather than aspirational — build it alongside Organizations, since Organization subscriptions reuse their member scoping. -8. **The [MCP server](https://compositecode.github.io/substratalapps.com/mcp-server/)** — generated from `openapi.yaml`'s `operationId`s, per [DEPLOYMENT.md → MCP server](DEPLOYMENT.md#mcp-server). Natural to build once the REST surface it wraps is otherwise stable, late in this phase rather than early. -9. **[Push notifications](https://compositecode.github.io/substratalapps.com/api-reference/notifications/)** (specified 2026-10-17, built 2026-10-18), in this order: - - The `push_credentials`, `devices`, `notifications`, and `notification_deliveries` tables ([Database Schema](https://compositecode.github.io/substratalapps.com/domain-model/database-schema/#push_credentials-devices-notifications-notification_deliveries)), as one expand-only migration, plus `applications.receives_platform_notifications`. - - `notifications.send` in the permission catalog, app-confined. - - The credential and device endpoints, including deactivation on session revocation, suspension, and deletion. - - Sending, with deliveries written in the same transaction as the Notification, and the inbox endpoints. - - The delivery worker: APNs (HTTP/2, token auth), FCM HTTP v1, and Web Push (VAPID and RFC 8291 encryption). Retries, token invalidation, and credential failure handling. See [DEPLOYMENT.md → Push delivery](DEPLOYMENT.md#push-delivery). - - The platform alerts, the export and erasure-cascade additions, the stale-device and retention jobs, and the webhook events. - - The provider clients live in `internal/push`, and the endpoints, worker, and alerts in `internal/core`. - -**Exit criteria:** a real third-party-shaped integration test, an app that isn't Substratal's own, can sign a user in through the embedded flow, receive `access.revoked` and kill the session, call introspection, have its own `settings_schema` validated and typed on write, and register a test device and receive a push sent through its app key. A test-mode User and a test-mode Organization can subscribe to Standard through Checkout, gain the standard Applications, and lapse back to Free. The hosted PKCE flow is Phase 3, with the hosted sign-in page. - -## Phase 3 — Marketplace & scale features - -Matches [Roadmap → Phase 3](https://compositecode.github.io/substratalapps.com/roadmap/#phase-3) — explicitly **not** built speculatively ahead of the triggers named in [DEPLOYMENT.md → Scale-out](DEPLOYMENT.md#scale-out) and [Applications → The review lifecycle](https://compositecode.github.io/substratalapps.com/domain-model/applications/#the-review-lifecycle): - -- **SSO broker integration** (`sso_connections` becoming real, an actual WorkOS-or-equivalent wiring behind `POST /v1/auth/sso/{provider}/callback`) — the schema and the route exist from Phase 1 per [Auth → Native auth and per-Organization SSO](https://compositecode.github.io/substratalapps.com/api-reference/auth/#native-auth-and-per-organization-sso), deliberately built in advance precisely so this phase is additive wiring, not a migration. Triggered by a real Organization actually asking for it, not a phase-number default. -- Self-service Application registration + review queue. The *process* is already fully specified — reviewer model (no dedicated assignment, any `applications.manage` holder works the FIFO queue), a 5-business-day policy SLA, the `rejected` status with required `review_notes` and edit-to-resubmit, non-cascading suspension — see [Applications → The review lifecycle](https://compositecode.github.io/substratalapps.com/domain-model/applications/#the-review-lifecycle) and [API Reference → Applications → Reviewing a submission](https://compositecode.github.io/substratalapps.com/api-reference/applications/#reviewing-a-submission). What's actually deferred to this phase is just the self-service *submission* endpoint itself (`POST /v1/applications` opening up beyond platform-admin-only) — the review mechanics it feeds into are build-ready today. -- SCIM-style provisioning. -- Richer audit/compliance views (filtering, export) — driven by actual [Compliance](https://compositecode.github.io/substratalapps.com/compliance/) needs (a SOC 2 audit, a customer's data-export request), not built ahead of either. - -## Cross-cutting, build throughout - -Not a phase — these apply from Phase 0 onward and should never be "added later": - -- **[Access Control](https://compositecode.github.io/substratalapps.com/access-control/)'s permission checks**, on every endpoint, from the first one. Retrofitting authorization onto endpoints built "open" is how it gets missed somewhere. -- **[Conventions](https://compositecode.github.io/substratalapps.com/api-reference/conventions/)'s error shape, pagination, idempotency, and `me` self-addressing** — pick these once, in Phase 1, as shared middleware/helpers. Every later endpoint should get them for free, not reimplement them. -- **Contract tests against `openapi.yaml`** — every endpoint, the day it's built, not batched at the end of a phase. -- **The spec pages and the [Changelog](https://compositecode.github.io/substratalapps.com/changelog/)** stay live documents during implementation, the same way they were during spec-writing. An implementation detail that contradicts or settles something in the spec gets written into the page it governs, with a Changelog entry, immediately, not in an end-of-project cleanup pass. - -## Status - -**Build status (2026-10-06):** the code for every phase's API surface is implemented, push notifications included (built 2026-10-18), along with Phase 3's hosted PKCE endpoints (SSO still returns `501` until the broker integration, as specified), and the Quickstart acceptance test passes against the devserver. The Phase 0 *infrastructure* items above are written as Terraform (`infra/terraform/`) but not yet applied; see [README → One-time setup](README.md#one-time-setup-phase-0). The choices the implementation made where the spec was silent or self-contradictory are written into the spec pages they govern. - -**No product decisions are open.** Each one is written into the spec page it governs, and recorded in the [Changelog](https://compositecode.github.io/substratalapps.com/changelog/). The separate decision log was retired on 2026-10-18. - -**Push notifications were added on 2026-10-17 and built on 2026-10-18** (Phase 2 item 9 above). - -**Pricing changed on 2026-10-17**: two tiers, Free and Standard ($6 per user per month), paid by the people using Substratal's apps, individually or through an Organization. **Subscriber billing was built on 2026-10-18**: Phase 2 item 7 now implements it, replacing the earlier developer-billed Starter/Team/Enterprise code. Billing is monthly in USD, with no annual plan and no Standard trial ([Pricing → Stripe catalog](https://compositecode.github.io/substratalapps.com/pricing/#stripe-catalog)). Either can be added later as Stripe configuration. - -## Test-mode User expiry - -Specified 2026-10-20 ([Auth → Test-mode Users → Lifetime](https://compositecode.github.io/substratalapps.com/api-reference/auth/#lifetime), [Changelog](https://compositecode.github.io/substratalapps.com/changelog/)); **not yet built**. Each step can land and be tested on its own, in this order: - -1. **Migration `0004_test_user_expiry`.** Add `users.expires_at` and `users.expired_processed_at`. Backfill existing test-mode Users with `created_at + interval '30 days'`, which is when the purge would have removed them anyway, so nothing changes for them. Then add `check ((expires_at is not null) = test_mode)` and `check (expires_at <= created_at + interval '30 days')`, and the partial index `(expires_at) where test_mode and expired_processed_at is null`. Add `test_user_expired` to the `sessions.revoked_reason` and `devices.inactive_reason` checks. -2. **Creation** (`internal/core/users_create.go`, `createUser`). Test-mode Users get `expires_at` from the request, or now + 24 hours by default. Validate the range (≥ 15 minutes from now, ≤ `created_at` + 30 days) as `422 validation_failed` with `expires_at_out_of_range`, and reject it on a live create with `live_user`. This covers both `POST /v1/users` and test-mode signup. -3. **Enforcement at the edge** (`internal/core/principal.go`, the per-request User lookup). Load `expires_at` alongside `status`, and reject an expired test-mode User's tokens the same way a suspended User's are. Login, `token/refresh`, `app-tokens`, `oauth/token`, and the `mfa/verify` step return `403 test_user_expired`. This is what makes expiry exact to the second. The sweep below only tidies up. -4. **Token lifetimes** (`internal/core/auth_session.go`). Cap the access token's `exp`, the app token's `exp`, and the session's `absolute_expires_at` at the User's `expires_at`, so Applications that verify app tokens locally against JWKS stop accepting them on time without calling the API. -5. **`PATCH /v1/users/{id}` `expires_at`** (`users.manage` only; self → `403 status_change_forbidden`). Same range check. Moving it later on an expired User clears `expired_processed_at` and fires `access.granted` (reason `test_user_extended`) for every Application they still have active access to. Writes `user.updated`. -6. **Expiry sweep**, in the existing `entitlement-sweep` job (every 5 minutes, no new schedule). Select `test_mode and expires_at <= now() and expired_processed_at is null`. For each, in one transaction: revoke sessions (`test_user_expired`), deactivate devices, fire `access.revoked` (reason `test_user_expired`) per Application with active access, enqueue the `user.expired` webhook, write the `user.expired` Audit Event (actor `system`), and set `expired_processed_at`. Reuse the suspension path's revoke-and-notify code, with the reason passed in. -7. **Subscriptions.** The test-mode covered-member count excludes expired Users (`internal/core/subscriptions.go`). Live counts are unaffected, because they never include test-mode Users. -8. **OpenAPI.** Add `expires_at` to `User`, to the `POST /v1/users` and signup request bodies, and to `PATCH /v1/users/{id}`. Add `test_user_expired` to the error, revoke-reason, and device-reason enums, and `user.expired` to the webhook and audit enums. Bump `info.version`, then run `make gen`. -9. **Tests.** Integration tests for each of these: the 24-hour default, the range errors, a token rejected after `expires_at` (with a test clock), `exp` capping, the sweep's events firing exactly once, extension restoring access, a live create rejecting `expires_at`, and the backfill leaving existing test Users working. Update the Quickstart acceptance test if it relies on a test User living longer than 24 hours. -10. **Docs.** Remove the "Not yet built" line from the 2026-10-20 Changelog entry and add a short "built" entry, as was done for push notifications. diff --git a/README.md b/README.md index 7e3e15e..d17327f 100644 --- a/README.md +++ b/README.md @@ -11,14 +11,16 @@ This repository is split deliberately, by audience: | Audience | Lives | Why | |---|---|---| | **An API consumer** — a developer building an Application on this platform, or their AI coding agent | [`docs/`](docs/), published at [compositecode.github.io/substratalapps.com](https://compositecode.github.io/substratalapps.com/) | The API specification, and nothing else: domain model, access control, every endpoint and schema, workflows, the MCP server for agent callers, pricing. Everything a consumer needs to build *against* this API. | -| **Whoever is building *this* API** | The project root (this file and its siblings) | Deployment/infrastructure, the implementation build order, and the internal decision log aren't things a consumer needs to use the API — they're how it gets built and run. Kept separate so the docs site stays focused, per its own [Home page](https://compositecode.github.io/substratalapps.com/#what-this-site-is). | +| **Whoever is building *this* API** | The project root (this file and its siblings) | Deployment/infrastructure, the remaining build plan, and the manual setup steps aren't things a consumer needs to use the API — they're how it gets built and run. Kept separate so the docs site stays focused, per its own [Home page](https://compositecode.github.io/substratalapps.com/#what-this-site-is). | Root-level documents: -- **[PLAN.md](PLAN.md)** — the build order: what to implement first, which spec page governs each piece, phase-by-phase exit criteria. -- **[DEPLOYMENT.md](DEPLOYMENT.md)** — AWS infrastructure (cost-capped, Aurora Serverless v2, Lambda, no VPC/NAT at Tier 0), Stripe Billing implementation, the MCP servers recommended for building against AWS/Postgres/Stripe, and the scale-out triggers for when any of this needs to change. +- **[plan-remaining.md](plan-remaining.md)** — everything specified but not built yet: test-mode User expiry, the first-party web pages, Phase 3, dedicated tenancy infrastructure, scale-out triggers, and the rules every change follows. +- **[remaining_manual_steps.md](remaining_manual_steps.md)** — one-time setup still to do by hand to go live: the Lambda quota, the superadmin password, SES production access, Stripe, and legal. +- **[infra/README.md](infra/README.md)** — the deployment architecture as built: AWS infrastructure (cost-capped, Aurora Serverless v2, Lambda, no VPC/NAT at Tier 0), the account setup, Stripe Billing, email, webhook and push delivery, audit archival, and local development. - **[LICENSE](LICENSE)** — MIT. -- **[ORIGINAL-SPEC-DRAFT.md](ORIGINAL-SPEC-DRAFT.md)** — the original single-document draft this whole spec was elaborated from. Kept for history, moved here (out of `docs/`) specifically so it can't be mistaken for current spec — it contradicts several later decisions. + +The original single-document draft, the build plan, and the old deployment document were retired on 2026-10-20 once their content was built and documented where it belongs; they remain in git history. Not yet moved to the root, but should be, by the same reasoning above — tracked so this doesn't get silently forgotten: @@ -37,18 +39,18 @@ Decided 2026-10-06. The API is written in **Go**, deployed to **AWS Lambda**, an | Layer | Choice | Why | |---|---|---| -| Language | **Go** (latest stable release, pinned in `go.mod`) | Cold starts in the tens of milliseconds and small memory use on Lambda. That directly serves the p95 < 150 ms target for `effective-permissions`, `app-tokens`, and `oauth/token` in [Non-Functional Requirements](https://compositecode.github.io/substratalapps.com/non-functional-requirements/), and the cost-first rules in [DEPLOYMENT.md](DEPLOYMENT.md#cost-principles). Strongly typed, stable for years, and plain enough that a one-person team and an AI assistant can both read and change it safely. | +| Language | **Go** (latest stable release, pinned in `go.mod`) | Cold starts in the tens of milliseconds and small memory use on Lambda. That directly serves the p95 < 150 ms target for `effective-permissions`, `app-tokens`, and `oauth/token` in [Non-Functional Requirements](https://compositecode.github.io/substratalapps.com/non-functional-requirements/), and the cost-first rules in [infra/README.md](infra/README.md#cost-principles). Strongly typed, stable for years, and plain enough that a one-person team and an AI assistant can both read and change it safely. | | Lambda runtime | `provided.al2023` on **arm64** (Graviton) | Go compiles to one static `bootstrap` binary. arm64 is cheaper per GB-second than x86 for the same work. | | HTTP layer | Standard library `net/http`. The route table, path/query/header binding, and the `ServerInterface` every operation must implement are **generated from [`docs/openapi.yaml`](docs/openapi.yaml)** by `oapi-codegen`. Request and response bodies are hand-written in `internal/core`. | The spec is the contract: an operation without a handler doesn't compile. The generated *body* models can't express explicit JSON `null`s or PATCH field presence, both of which Conventions requires, so bodies are hand-written and the contract tests validate every response against `openapi.yaml` instead. | | Lambda adapter | `aws-lambda-go`, plus a thin adapter that turns API Gateway HTTP API events into `http.Request`s | The **same** `http.Handler` serves requests on a laptop and in Lambda. Nothing in handler code knows which one it's running in. | -| Database | **PostgreSQL 16**. In AWS: Aurora Serverless v2 through the **RDS Data API**. Locally and in CI: plain Postgres through **`pgx`**. | Both implement one small driver interface (`internal/db`), as [DEPLOYMENT.md → Local development](DEPLOYMENT.md#local-development) requires. See [The two database backends](#the-two-database-backends). | +| Database | **PostgreSQL 16**. In AWS: Aurora Serverless v2 through the **RDS Data API**. Locally and in CI: plain Postgres through **`pgx`**. | Both implement one small driver interface (`internal/db`), as [infra/README.md → Local development](infra/README.md#local-development) requires. See [The two database backends](#the-two-database-backends). | | Migrations | Plain, numbered SQL files in `migrations/`, applied by an in-repo `cmd/migrate` tool | Same tool, same files, same tracking table (`schema_migrations`) in every environment. An in-repo runner is used instead of a third-party tool because the common Go migration tools need a `database/sql` driver, and the Data API has no first-party one. | | Crypto | `golang.org/x/crypto/argon2` (Argon2id), standard-library HMAC/RSA, AWS KMS for JWT signing and secret encryption | Matches [Non-Functional Requirements → Security](https://compositecode.github.io/substratalapps.com/non-functional-requirements/) exactly, with no native add-ons. | | AWS access | AWS SDK for Go v2 (Data API, KMS, SQS, Secrets Manager, S3, SES) | | | Billing | `stripe-go`, Stripe's official Go library | | | MCP server | Go, implementing the buffered-JSON subset of MCP Streamable HTTP directly (`internal/mcp`). Tools are generated from `openapi.yaml` by `cmd/mcpgen`. | The spec requires a stateless, signed `Mcp-Session-Id` and one invocation per call; the official SDK keeps sessions in memory, which doesn't fit a stateless Lambda. It only proxies HTTPS calls to `/v1`. | | Infrastructure | **Terraform**, remote state in S3 (native S3 state locking, no DynamoDB table) | Language-neutral and widely known, and `terraform plan` output can be reviewed on the pull request before it reaches production. Works with OpenTofu too. | -| CI/CD | **GitHub Actions**, deploying to AWS through **OIDC federation** | No long-lived AWS keys anywhere, per [DEPLOYMENT.md → Build checklist](DEPLOYMENT.md#build-checklist) step 9. | +| CI/CD | **GitHub Actions**, deploying to AWS through **OIDC federation** | No long-lived AWS keys anywhere, per [infra/README.md → Cost principles](infra/README.md#cost-principles). | | Local services | **Docker Compose**: Postgres 16, LocalStack (SQS, KMS, Secrets Manager, S3), Mailpit (captures outgoing email) | The whole API runs on a laptop with no AWS account and no internet beyond Stripe test mode. | Considered and not chosen: **TypeScript on Node.js** (fastest early development and the most mature MCP SDK, but slower cold starts, types that disappear at runtime, and more dependency churn over a long-lived project), and **Java** (mature, and SnapStart fixes cold starts, but heavier memory use and more framework machinery than Tier 0's small per-request Lambdas and a one-person team need). Revisit only if the deployment model changes, for example a move from Lambda to long-running containers. @@ -120,7 +122,7 @@ Every push to `dev` (and to `feature/*`), and every pull request into `main`, ru 3. **Unit tests**, including exhaustive tests of the Access Control resolution logic (every combination of entitlement status × platform role × app role), per [Testing strategy](https://compositecode.github.io/substratalapps.com/non-functional-requirements/#testing-strategy). 4. **Integration tests** against real Postgres 16 and LocalStack running as GitHub Actions service containers, with every migration applied from zero. These run the pgx backend, including the Row-Level Security policies, which a mocked data layer would let pass while broken. 5. **Contract tests.** The devserver starts, and every response is validated against `docs/openapi.yaml`. -6. **Quickstart test.** The [Quickstart](https://compositecode.github.io/substratalapps.com/quickstart/) walkthrough runs as a script against the devserver. It's the Phase 1 acceptance test from [PLAN.md](PLAN.md#phase-1--mvp), so it runs on every change, not just at milestones. +6. **Quickstart test.** The [Quickstart](https://compositecode.github.io/substratalapps.com/quickstart/) walkthrough runs as a script against the devserver. It's the Phase 1 acceptance test ([plan-remaining.md → Verify the build in production](plan-remaining.md#verify-the-build-in-production)), so it runs on every change, not just at milestones. 7. **Build.** Every `cmd/*` function compiles for `linux/arm64` (`CGO_ENABLED=0`, `-tags lambda.norpc`) and is zipped as `bootstrap`, which proves the release artifacts can be built. 8. **Docs build.** `jekyll build` of `docs/`, so a broken docs page is caught on `dev`, not on the release. @@ -128,7 +130,7 @@ On a pull request into `main` only, one more step runs: 9. **`terraform plan` against production**, using a **read-only** plan role through OIDC. The plan is posted as a PR comment. This is the one thing to read carefully before merging a release: it is exactly what the merge will change in AWS. -There is deliberately **no deployed dev environment** at Tier 0. A second copy of the stack would roughly double the Aurora floor cost (~$43/month), and the local stack covers what it would be used for. Adding one, ideally in a separate AWS account, is the "more than one engineer shipping concurrently" trigger in [DEPLOYMENT.md → Scale-out](DEPLOYMENT.md#scale-out). When that happens, a push to `dev` gains a deploy to that account. Nothing else about this flow changes. +There is deliberately **no deployed dev environment** at Tier 0. A second copy of the stack would roughly double the Aurora floor cost (~$43/month), and the local stack covers what it would be used for. Adding one, ideally in a separate AWS account, is the "more than one engineer shipping concurrently" trigger in [infra/README.md → Scale-out](plan-remaining.md#scale-out). When that happens, a push to `dev` gains a deploy to that account. Nothing else about this flow changes. ### What a merge into `main` does: `deploy.yml` @@ -185,9 +187,9 @@ For an urgent fix while `dev` holds unreleased work: ### One-time setup (Phase 0) -The steps above assume these exist. They're part of [PLAN.md → Phase 0](PLAN.md#phase-0--infrastructure-skeleton), done once, by hand: +The steps above assume these exist. They're done once, by hand; what's still left of them is tracked in [remaining_manual_steps.md](remaining_manual_steps.md): -1. A dedicated AWS account in `us-east-1` ([DEPLOYMENT.md → AWS account & region](DEPLOYMENT.md#aws-account--region)), with its default VPC in place (Aurora's subnets come from it; Lambda never joins a VPC), and a Route 53 hosted zone for `substratalapps.com`. +1. A dedicated AWS account in `us-east-1` ([infra/README.md → AWS account & region](infra/README.md#aws-account--region)), with its default VPC in place (Aurora's subnets come from it; Lambda never joins a VPC), and a Route 53 hosted zone for `substratalapps.com`. 2. From your own machine, with admin credentials: `cd infra/terraform/bootstrap && terraform init && terraform apply -var billing_email=you@example.com`. This creates the **cost guardrails first** (warn and hard budgets with a circuit-breaker action, Cost Anomaly Detection, a billing alarm), then the state and artifact buckets, the GitHub OIDC provider, and the read-only `plan` and `deploy` roles. It's the only Terraform ever applied from a laptop. 3. In GitHub: create the `production` environment (deployments from `main` only), add the `main` ruleset, and set repository **variables** from the bootstrap outputs: `AWS_PLAN_ROLE_ARN`, `AWS_DEPLOY_ROLE_ARN`, `AWS_REGION`, `TF_STATE_BUCKET`, `ARTIFACT_BUCKET`, plus `API_DOMAIN` (`api.substratalapps.com`). None are secrets; OIDC needs no stored credentials. 4. Merge into `main`. The first deploy creates the stack and then migrates the new cluster. @@ -244,7 +246,7 @@ Configuration is entirely environment variables, documented in `.env.example`. ` ### The two database backends -[DEPLOYMENT.md → Local development](DEPLOYMENT.md#local-development) explains why the Data API can't run locally. Here's how the code handles it: +[infra/README.md → Local development](infra/README.md#local-development) explains why the Data API can't run locally. Here's how the code handles it: - Every database call goes through one small interface in `internal/db` (`Tx`, `Query`, `Exec`). Handlers never import `pgx` or the Data API client. - SQL is written once, with named parameters (`:user_id`). The pgx driver rewrites them to `$1` positional form; the Data API driver passes them as `SqlParameter`s. Parameters are only ever scalars; arrays and JSON travel as text with explicit casts (`::text[]`, `::jsonb`), because the Data API has no array parameters. diff --git a/cmd/mcp/main.go b/cmd/mcp/main.go index 1ec15c2..946d5da 100644 --- a/cmd/mcp/main.go +++ b/cmd/mcp/main.go @@ -1,7 +1,7 @@ // Command mcp is the MCP server Lambda at /mcp. It holds no data-plane // permissions at all: every tool call is an HTTPS call to the API's own // /v1 routes on the same API Gateway, with the caller's credential -// forwarded unchanged (DEPLOYMENT.md → MCP server). +// forwarded unchanged (infra/README.md → MCP server). package main import ( diff --git a/cmd/stripe-webhook/main.go b/cmd/stripe-webhook/main.go index e847b80..ab3f131 100644 --- a/cmd/stripe-webhook/main.go +++ b/cmd/stripe-webhook/main.go @@ -1,6 +1,6 @@ // Command stripe-webhook is the Lambda at /internal/stripe/webhook: the one // route whose caller isn't a Bearer holder at all, verified instead by -// Stripe's own signature (DEPLOYMENT.md → Stripe Billing). +// Stripe's own signature (infra/README.md → Stripe Billing). package main import ( diff --git a/docs/api-reference/billing.md b/docs/api-reference/billing.md index 476119a..630ffa1 100644 --- a/docs/api-reference/billing.md +++ b/docs/api-reference/billing.md @@ -25,7 +25,7 @@ The Free/Standard subscription: a User subscribing to Standard for themselves, o - **Access follows the subscription.** While a subscription covers someone, the platform keeps a `source: subscription` Entitlement for them to each standard Application. When it stops covering them, those Entitlements end. Applications hear about every change through the usual [`access.granted` / `access.revoked`](../webhooks/#event-types) webhooks. - **Business and Enterprise** plans aren't offered yet. Checkout for either returns `422 plan_requires_sales`. -Stripe tells this API about every change through its webhook. This API never takes card data, and a subscription's `status` only changes in response to Stripe events. The handler is documented in [root `DEPLOYMENT.md` → Stripe Billing](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md#stripe-billing). +Stripe tells this API about every change through its webhook. This API never takes card data, and a subscription's `status` only changes in response to Stripe events. The handler is documented in [`infra/README.md` → Stripe Billing](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md#stripe-billing). ## Endpoints diff --git a/docs/api-reference/conventions.md b/docs/api-reference/conventions.md index a49af49..076433a 100644 --- a/docs/api-reference/conventions.md +++ b/docs/api-reference/conventions.md @@ -38,7 +38,7 @@ Authorization: Bearer User-facing requests carry a user access token (issued at login, see [Auth](../auth/)). Service-to-service requests (billing, an app's backend) carry a scoped [API Key](../api-keys/). Both go in the same header — the token type is distinguishable server-side by prefix, not by a different header name. -**Test vs. live:** every API Key is created with `satk_test_…` or `satk_live_…` (see [API Keys](../api-keys/)) — there is no separate sandbox deployment to point at. A `test` key operates against the same database, but every record it creates is tagged `test_mode: true`. Test and live rows are invisible to each other: a test credential reads and writes only test rows, a live credential only live rows, and no query parameter crosses that line. Test-mode events only reach test-mode webhook subscriptions, and test traffic is excluded from analytics counters. A User created in test mode signs in with `"test_mode": true` on the auth endpoints (see [Auth → Test-mode Users](../auth/#test-mode-users)), and every token issued to them carries `test_mode: true`, which confines it to test rows the same way. This is cheaper to build and run than a parallel environment, and it's the right call at the current scale (see [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md)) — a true isolated sandbox is a Scale-out-trigger-shaped decision, not a day-one one. +**Test vs. live:** every API Key is created with `satk_test_…` or `satk_live_…` (see [API Keys](../api-keys/)) — there is no separate sandbox deployment to point at. A `test` key operates against the same database, but every record it creates is tagged `test_mode: true`. Test and live rows are invisible to each other: a test credential reads and writes only test rows, a live credential only live rows, and no query parameter crosses that line. Test-mode events only reach test-mode webhook subscriptions, and test traffic is excluded from analytics counters. A User created in test mode signs in with `"test_mode": true` on the auth endpoints (see [Auth → Test-mode Users](../auth/#test-mode-users)), and every token issued to them carries `test_mode: true`, which confines it to test rows the same way. This is cheaper to build and run than a parallel environment, and it's the right call at the current scale (see [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md)) — a true isolated sandbox is a Scale-out-trigger-shaped decision, not a day-one one. ## IDs diff --git a/docs/api-reference/tenancy.md b/docs/api-reference/tenancy.md index c619bf2..6a860dd 100644 --- a/docs/api-reference/tenancy.md +++ b/docs/api-reference/tenancy.md @@ -131,7 +131,7 @@ Creating a request doesn't start a migration. It's the record and the queue entr | `pending`/`scheduled → cancelled` | No Tenant change. `notes` is required. | | `in_progress → cancelled` | A rollback: the Tenant's `status` returns to `active` on its *original* tier/region. `notes` is required. | -Any other transition returns `409 invalid_request_transition`. Every change writes `tenant.tier_change_request_updated`. The physical migration steps are ops runbook work. See [root `DEPLOYMENT.md` → Migration mechanics](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md). This endpoint records and gates those steps; it doesn't perform them. +Any other transition returns `409 invalid_request_transition`. Every change writes `tenant.tier_change_request_updated`. The physical migration steps are ops runbook work. See [`infra/README.md` → Migration mechanics](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md). This endpoint records and gates those steps; it doesn't perform them. ## Errors specific to this resource diff --git a/docs/changelog.md b/docs/changelog.md index 748c492..0a5871a 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -20,7 +20,7 @@ Test mode already never counted toward billing (decision #36, confirmed on 2026- - **At expiry** sign-in, refresh, and app-token issuance return the new `403 test_user_expired`, and every token already issued stops working. No token's `exp` is ever later than the User's `expires_at`. The 5-minute entitlement-expiry sweep then revokes sessions and devices (reason `test_user_expired`), fires `access.revoked` (reason `test_user_expired`) and the new `user.expired` webhook, and writes the `user.expired` Audit Event. `status`, Entitlements, Roles, and memberships are unchanged. An expired test User isn't a covered member of a test-mode subscription. - Test-mode API Keys, Organizations, and Applications don't expire. The nightly 30-day purge is unchanged. -Not yet built: until it is, test-mode Users last until the 30-day purge, as before. `openapi.yaml` gains `expires_at` in the same change as the code. The build steps are in the repository's `PLAN.md` → Test-mode User expiry. +Not yet built: until it is, test-mode Users last until the 30-day purge, as before. `openapi.yaml` gains `expires_at` in the same change as the code. The build steps are in the repository's `plan-remaining.md` → Test-mode User expiry. ## 2026-10-18 — Push notifications implemented @@ -32,7 +32,7 @@ Not yet built: until it is, test-mode Users last until the 30-day purge, as befo - A push-credential `POST` validates its fields before checking whether the provider is already configured, so an invalid body is `422` even when a credential exists. - [Database Schema](../domain-model/database-schema/#push_credentials-devices-notifications-notification_deliveries) records each notification's push counts on the notification itself, since deliveries are deleted after 30 days. -Delivery shares the webhook queue, worker, and 1-minute sweep (see the repository's `DEPLOYMENT.md` → Push delivery), and a daily `push-maintenance` job runs the device and inbox retention rules. Locally, `PUSH_BACKEND=log` logs pushes instead of sending them. +Delivery shares the webhook queue, worker, and 1-minute sweep (see the repository's `infra/README.md` → Push delivery), and a daily `push-maintenance` job runs the device and inbox retention rules. Locally, `PUSH_BACKEND=log` logs pushes instead of sending them. ## 2026-10-18 — Billing options settled; decision log retired @@ -171,7 +171,7 @@ A full consistency/UX review surfaced drift between the prose docs, `openapi.yam - **Fixed a self-contradiction in [Pricing](../pricing/):** Starter's own "For" cell claimed "3+ Applications" while its own Applications row capped at 1. Reworded; "3+ Applications" is Team's case, not Starter's. - **Added [Pricing → Enforcement](../pricing/):** the Applications/custom-AppRole/webhook-subscription caps now have a real mechanism — checked server-side at creation time against the owning Tenant's plan, independent of the caller's own permission standing (the same "two independent gates" pattern `restrict_destructive` already uses), returning `409`/`code: "plan_limit_reached"`. New database constraint: `check (plan != 'starter' or owner_type = 'user')` on `tenants`. -- **Resolved the audit-retention contradiction** between [Pricing](../pricing/)'s plan-tiered "retention" numbers and [Non-Functional Requirements](../non-functional-requirements/)/[Compliance](../compliance/)'s "retained indefinitely": those numbers were never a deletion period, they're a **hot-storage window** — added [Non-Functional Requirements → Audit log lifecycle](../non-functional-requirements/#audit-log-lifecycle) specifying the archive-to-cold-storage mechanism (shape-only, `before`/`after` dropped at archive time, same redaction the hard-delete cascade already does) that makes "indefinite" literally true without every plan paying for the same hot-storage size. Root [`DEPLOYMENT.md` → Audit log archival](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md#audit-log-archival) specifies the AWS mechanics (EventBridge Scheduler → Lambda → S3 Glacier Deep Archive). +- **Resolved the audit-retention contradiction** between [Pricing](../pricing/)'s plan-tiered "retention" numbers and [Non-Functional Requirements](../non-functional-requirements/)/[Compliance](../compliance/)'s "retained indefinitely": those numbers were never a deletion period, they're a **hot-storage window** — added [Non-Functional Requirements → Audit log lifecycle](../non-functional-requirements/#audit-log-lifecycle) specifying the archive-to-cold-storage mechanism (shape-only, `before`/`after` dropped at archive time, same redaction the hard-delete cascade already does) that makes "indefinite" literally true without every plan paying for the same hot-storage size. Root [`DEPLOYMENT.md` → Audit log archival](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md#audit-log-archival) specifies the AWS mechanics (EventBridge Scheduler → Lambda → S3 Glacier Deep Archive). - **Fixed [Orders & Audit → Action catalog](../domain-model/orders-and-audit/#action-catalog):** added `tenant.tier_change_requested` and `tenant.tier_changed`, both already live in `openapi.yaml`'s `AuditAction` enum but missing from this page's own "authoritative source" table. - **Added the MFA enrollment endpoint to `openapi.yaml`** (`POST /v1/users/{id}/mfa/totp`) — fully specified in [Auth](../api-reference/auth/) since its own addition, never added to the machine-readable spec. - **Added `mode` (live/test) to `ApiKeyWrite`/`ApiKey` in `openapi.yaml`** — present in the database schema and every prose example since [API Keys](../api-reference/api-keys/) was written, absent from the YAML. @@ -185,8 +185,8 @@ A full consistency/UX review surfaced drift between the prose docs, `openapi.yam - **Elaborated [Conventions → Errors](../api-reference/conventions/#errors):** fixed the illustrative example (`entitlement_not_active`, which no page actually defines) to a real code (`entitlement_required`); added an explicit HTTP-status-to-error-class mapping table and the `409`-vs-`422` distinction, for implementation; clarified there's no single global error-code catalog — each resource's own table is authoritative, the same pattern [Orders & Audit](../domain-model/orders-and-audit/#action-catalog) already uses for `action` values. - **Fixed [MCP Server](../mcp-server/)'s hardcoded operation count** ("all 55 operations") — already stale (56+ since this was written), and guaranteed to drift again. Reworded to not state a number at all. - **Fixed [Domain Model](../domain-model/)'s "Ten entities" framing** — added a Supporting entities table listing `UserIdentity`, `SSOConnection`, `OrganizationMembership`, and `UserRoleAssignment`, which the original count silently omitted. -- **Moved the original single-document draft** from `docs/specs/Substratal-Hub-Access-API.md` to root [`ORIGINAL-SPEC-DRAFT.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/ORIGINAL-SPEC-DRAFT.md), with a superseded banner — it contradicts several current decisions structurally (no Tenant concept, single-method auth) and sitting inside `docs/` risked it being mistaken for current spec despite being excluded from the build. -- **Updated root [`README.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/README.md) and [`PLAN.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/PLAN.md)**, both of which had gone stale against decisions resolved in the prior round: README's Status section still named Decision #1 as the one open item (all 15 are resolved); PLAN's Phase 3 section still described the Application review process as unspecified (fully specified by Decision #9). +- **Moved the original single-document draft** from `docs/specs/Substratal-Hub-Access-API.md` to root [`ORIGINAL-SPEC-DRAFT.md`](https://github.com/CompositeCode/substratalapps.com/blob/e3826b8cbc306f2a0bfc428d8e3a61a8f757dd3d/ORIGINAL-SPEC-DRAFT.md), with a superseded banner — it contradicts several current decisions structurally (no Tenant concept, single-method auth) and sitting inside `docs/` risked it being mistaken for current spec despite being excluded from the build. +- **Updated root [`README.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/README.md) and [`PLAN.md`](https://github.com/CompositeCode/substratalapps.com/blob/e3826b8cbc306f2a0bfc428d8e3a61a8f757dd3d/PLAN.md)**, both of which had gone stale against decisions resolved in the prior round: README's Status section still named Decision #1 as the one open item (all 15 are resolved); PLAN's Phase 3 section still described the Application review process as unspecified (fully specified by Decision #9). ## 2026-10-04 @@ -198,7 +198,7 @@ A full consistency/UX review surfaced drift between the prose docs, `openapi.yam - Added the `me` self-addressing convention and a credential-vs-resource-ID distinction to [Conventions](../api-reference/conventions/). - Added [Quickstart](../quickstart/) (a runnable walkthrough) and this Changelog. - Fixed: the Audit endpoint's documented filter name (`user_id` → `target_user_id`) didn't match its own example. -- Added [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md): first-deployment and scale-out plans on AWS, replacing an earlier Vercel-based plan, designed around predictable, capped cost rather than deploy-and-see. +- Added [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md): first-deployment and scale-out plans on AWS, replacing an earlier Vercel-based plan, designed around predictable, capped cost rather than deploy-and-see. - **Reframed the product.** Substratal Apps is the user/organization/tenancy/settings/storage API layer an app developer builds on — not a storefront a customer browses. See [Home](../) and [What Substratal Apps actually is](../#what-substratal-apps-actually-is). Every page that described the platform as "the hub a customer lands on to reach apps they purchased" was describing the eventual onboarding UI, not the API's own purpose. - Added `owner_user_id`/`owner_organization_id` and `review_status` to [Application](../domain-model/applications/) — a real gap once outside developers can register their own apps: there was no concept of who owns an Application's catalog entry, only platform-admin control. [API Reference → Applications](../api-reference/applications/) now distinguishes an owner's self-service rights from platform moderation. - Added [Trust Model → Trust runs the other direction too](../trust-model/#trust-runs-the-other-direction-too): the existing API Key scoping and webhook signing already made third-party apps safe to onboard later — now stated explicitly instead of left implicit. @@ -207,13 +207,13 @@ A full consistency/UX review surfaced drift between the prose docs, `openapi.yam - Added [Database Schema](../domain-model/database-schema/): Postgres types, constraints, and indexes for every entity — the implementer-facing counterpart to the API-caller-facing field tables. - Added implementer-readiness gaps across [Non-Functional Requirements](../non-functional-requirements/) (concurrency control via `ETag`/`If-Match`, transaction boundaries, the concrete Postgres Row-Level Security multi-tenancy mechanism, enum forward-compatibility policy, request tracing, testing strategy) and [Conventions](../api-reference/conventions/) (test-vs-live API keys, multi-field validation error shape, default exclusion of soft-deleted/terminal records from lists, idempotency-key storage detail). - Added [Webhooks → Verifying the signature](../api-reference/webhooks/#verifying-the-signature): the concrete `Substratal-Signature: t=…,v1=…` header format and replay-window guidance — "HMAC-SHA256 over the raw body" wasn't enough to actually implement against. -- Added [Deployment Architecture → Local development](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) (the RDS Data API choice has no clean local emulator — addressed via a repository-pattern seam, not by changing the AWS choice) and [→ Growth trajectory](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) (the stated dozens → 10x → 100x plan mapped against Tier 0 and the Scale-out triggers). -- **Added [Tenant](../domain-model/tenancy/)**: the infrastructure-placement and data-isolation entity "tenancy" named in the product pitch but previously unspecified. Tied to the subscription — the owner of an Application's catalog entry (`owner_user_id`/`owner_organization_id`), not to Organization, which stays a pure grouping/domain object with no infrastructure meaning. Three tiers (`shared` default, `isolated`, `dedicated_region`), support-gated tier changes via `tenants.manage` and `POST /v1/tenants/{id}/tier-change-requests`, a scheduled-downtime migration process, and deliberately no reach into a downstream Application's own infrastructure. See [Decisions #11](../domain-model/tenancy/#tenant-vs-organization) and [#12](../domain-model/tenancy/#how-a-tier-change-happens-today), [API Reference → Tenancy](../api-reference/tenancy/), [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md), and the new `tenants`/`organization_memberships` tables in [Database Schema](../domain-model/database-schema/). Directly resolves [Compliance](../compliance/#gdpr-and-ccpa--build-for-it-now)'s previously-open data-residency gap. +- Added [Deployment Architecture → Local development](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md) (the RDS Data API choice has no clean local emulator — addressed via a repository-pattern seam, not by changing the AWS choice) and [→ Growth trajectory](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md) (the stated dozens → 10x → 100x plan mapped against Tier 0 and the Scale-out triggers). +- **Added [Tenant](../domain-model/tenancy/)**: the infrastructure-placement and data-isolation entity "tenancy" named in the product pitch but previously unspecified. Tied to the subscription — the owner of an Application's catalog entry (`owner_user_id`/`owner_organization_id`), not to Organization, which stays a pure grouping/domain object with no infrastructure meaning. Three tiers (`shared` default, `isolated`, `dedicated_region`), support-gated tier changes via `tenants.manage` and `POST /v1/tenants/{id}/tier-change-requests`, a scheduled-downtime migration process, and deliberately no reach into a downstream Application's own infrastructure. See [Decisions #11](../domain-model/tenancy/#tenant-vs-organization) and [#12](../domain-model/tenancy/#how-a-tier-change-happens-today), [API Reference → Tenancy](../api-reference/tenancy/), [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md), and the new `tenants`/`organization_memberships` tables in [Database Schema](../domain-model/database-schema/). Directly resolves [Compliance](../compliance/#gdpr-and-ccpa--build-for-it-now)'s previously-open data-residency gap. - **Replaced `User.organization_id` with [OrganizationMembership](../domain-model/users-and-organizations/#organizationmembership)**, a many-to-many join — a User can belong to more than one Organization at once (even across different Tenants), which a single column couldn't express. Carries `role: org_admin | member`, which also resolves [Organizations → Delegated admin](../api-reference/organizations/#delegated-admin)'s previously-unresolved question: an org admin is this field, not a third Role scope. - **Added `member_scope`/`member_overrides` to org-wide Entitlements** and resolved Decision #13: an Organization's own grant is authoritative over its own members (an `allowlist`/`denylist` an org admin controls), but never reaches into a member's separate, personally-sourced Entitlement to the same app. [Access Control → Organization vs. User precedence](../access-control/#organization-vs-user-precedence) has the formal resolution rule and a worked multi-org example; [Entitlements → Attribution](../domain-model/entitlements/#attribution) shows the `granted_via` shape that surfaces which Organization is responsible for a given member's resolved access. - **Resolved Decision #8:** storage stays Postgres `jsonb` as the source of truth, with a Postgres [generated column](../domain-model/database-schema/#typed-fields-per-application-views-and-expression-indexes) backing every field an Application has declared in its `settings_schema` — typed and indexed, without giving up the blob's flexibility for undeclared fields. -- Added [Deployment Architecture → Database engine: AWS options compared](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md): RDS vs. Aurora Provisioned vs. Aurora Serverless v2, evaluated for lowest starting price. Aurora Serverless v2 stays the pick — not the cheapest floor in isolation (~$30/month more than bare RDS), but the only option that keeps one connection mechanism across every [Tenancy](../domain-model/tenancy/) tier and avoids a VPC/NAT Gateway entirely; the comparison and the reasoning are now explicit rather than asserted. -- **Added [MCP Server](../mcp-server/)**: a Model Context Protocol server letting an AI agent call the Platform API as typed tools instead of hand-rolled HTTP calls. Introduces no new authorization model — every tool call forwards the caller's own Bearer credential (user token or [API Key](../api-reference/api-keys/)) into the same REST call and the same [Access Control](../access-control/) check. The tool list is generated from `openapi.yaml`'s `operationId`s rather than hand-authored, so it tracks the REST API automatically — see [Roadmap → The MCP server tracks the REST API automatically](../roadmap/#the-mcp-server-tracks-the-rest-api-automatically). Added `operationId` to all 55 operations in [openapi.yaml](../openapi.yaml) (none had one before) specifically so this generation scheme is buildable today, not aspirational. [Deployment Architecture → MCP server](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) specifies the AWS build-out: a second, dedicated Lambda behind the existing API Gateway at `/mcp` (no new domain, no VPC, no database — fully stateless at Tier 0), with named scale-out triggers (Provisioned Concurrency, a Lambda Function URL with streaming responses for server-push/resumability, a small DynamoDB table if session state ever outgrows a signed token). Added [Decision #14](../api-reference/api-keys/#agent-keys--restrict_destructive), still open: whether agent callers should get a recommended (or eventually required) narrower API Key scoping convention, given an LLM's tool-call decision is a different risk shape from deterministic service code. +- Added [Deployment Architecture → Database engine: AWS options compared](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md): RDS vs. Aurora Provisioned vs. Aurora Serverless v2, evaluated for lowest starting price. Aurora Serverless v2 stays the pick — not the cheapest floor in isolation (~$30/month more than bare RDS), but the only option that keeps one connection mechanism across every [Tenancy](../domain-model/tenancy/) tier and avoids a VPC/NAT Gateway entirely; the comparison and the reasoning are now explicit rather than asserted. +- **Added [MCP Server](../mcp-server/)**: a Model Context Protocol server letting an AI agent call the Platform API as typed tools instead of hand-rolled HTTP calls. Introduces no new authorization model — every tool call forwards the caller's own Bearer credential (user token or [API Key](../api-reference/api-keys/)) into the same REST call and the same [Access Control](../access-control/) check. The tool list is generated from `openapi.yaml`'s `operationId`s rather than hand-authored, so it tracks the REST API automatically — see [Roadmap → The MCP server tracks the REST API automatically](../roadmap/#the-mcp-server-tracks-the-rest-api-automatically). Added `operationId` to all 55 operations in [openapi.yaml](../openapi.yaml) (none had one before) specifically so this generation scheme is buildable today, not aspirational. [Deployment Architecture → MCP server](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md) specifies the AWS build-out: a second, dedicated Lambda behind the existing API Gateway at `/mcp` (no new domain, no VPC, no database — fully stateless at Tier 0), with named scale-out triggers (Provisioned Concurrency, a Lambda Function URL with streaming responses for server-push/resumability, a small DynamoDB table if session state ever outgrows a signed token). Added [Decision #14](../api-reference/api-keys/#agent-keys--restrict_destructive), still open: whether agent callers should get a recommended (or eventually required) narrower API Key scoping convention, given an LLM's tool-call decision is a different risk shape from deterministic service code. - **Added [Pricing](../pricing/)**: three subscription tiers for what Substratal itself charges the Application-owner developer (Starter free, Team per-seat, Enterprise with a choice of [Tenant](../domain-model/tenancy/) `tier` as a priced option — `shared` included, `isolated` +$750/month, `dedicated_region` +$1,500/month) — distinct from what that developer charges their own end users, per [Decision #4](../domain-model/orders-and-audit/#billing-system-of-record). Added `plan` (`starter`\|`team`\|`enterprise`) to [Tenant](../domain-model/tenancy/#fields), independent of `tier`, with a database constraint tying `isolated`/`dedicated_region` to `enterprise` only. Added [Decision #15](../pricing/#how-the-subscription-is-charged): how Substratal bills this subscription is still open (leans Stripe Billing), with no corresponding API surface built yet. Cites the AWS Pricing, Billing and Cost Management, and Postgres MCP servers as the tooling to keep these numbers grounded in real infrastructure cost over time. diff --git a/docs/compliance.md b/docs/compliance.md index d3dd8e2..f925b27 100644 --- a/docs/compliance.md +++ b/docs/compliance.md @@ -42,7 +42,7 @@ The mechanisms below cover both regimes; CCPA/CPRA's individual-rights requireme | Right to erasure | **Specified.** A request arrives via `POST /v1/users/{id}/erasure-requests` (self-service, or support on the user's behalf; see [API Reference → Users](../api-reference/users/#post-v1usersiderasure-requests)). [Non-Functional Requirements → Hard-delete cascade](../non-functional-requirements/#hard-delete-cascade) names the actual cascade: soft-delete is immediate on request, and the hard delete runs 7 days later, well inside GDPR Article 12(3)'s 30-day ceiling (not "eventually") and, in order, deletes `password_hash`/`mfa_secret` from [UserIdentity](../domain-model/users-and-organizations/#useridentity), the User's [Profile](../domain-model/profiles/), scrubs PII out of every [AppProfile](../domain-model/profiles/#appprofile)/[AppSettings](../domain-model/settings/#appsettings) `custom`/`overrides` blob the User holds, and redacts (not deletes — see below) the `before`/`after` snapshots on their [Audit Events](../domain-model/orders-and-audit/#audit-event). Entitlement and Order rows are retained as pseudonymized business/financial records, which GDPR permits under a legitimate-interest basis distinct from the erasure right. | | Right to access / data portability | **Specified.** `GET /v1/users/{id}/export` — see [API Reference → Users](../api-reference/users/) — returns everything this API holds about one User: Profile, Settings, every AppProfile/AppSettings, Entitlements, Role assignments, Organization memberships, UserIdentity methods (never `password_hash`/`mfa_secret`), push devices (never the token), notifications, and their own Audit trail, as one JSON bundle. Satisfies GDPR Article 20 portability and the equivalent CCPA/CPRA right-to-know in one endpoint rather than two. | | Lawful basis & consent records | **Missing from the spec on purpose** — not a schema problem, a product/legal one: each Application's developer is the one collecting consent from their own end users (Substratal doesn't run the signup form), so this is as much about what the *developer* agrees to via Substratal's terms of service as what the API stores. Unchanged by this round of decisions. | -| Data Processing Agreement with sub-processors | **Operational, not code** — AWS (the infrastructure in [root `DEPLOYMENT.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md), including Amazon SES for transactional email, so no separate email vendor), **Stripe** (Free/Standard subscription billing, per [Pricing → How the subscription is charged](../pricing/#how-the-subscription-is-charged)), and the push providers that carry [push notifications](../api-reference/notifications/#what-a-device-receives): **Apple** (APNs), **Google** (FCM), and the push service of whichever browser a User subscribes from (Google, Mozilla, Apple, or Microsoft). A push payload is the only data they receive, which is why senders are told to keep sensitive content out of it. Both publish standard DPAs (AWS Artifact; Stripe's own, plus Stripe's existing PCI-DSS Level 1 status, which is part of why [the billing system of record](../domain-model/orders-and-audit/#billing-system-of-record) keeps this API's own PCI scope minimal). Executing either is still a step to actually take, not code to write. | +| Data Processing Agreement with sub-processors | **Operational, not code** — AWS (the infrastructure in [`infra/README.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md), including Amazon SES for transactional email, so no separate email vendor), **Stripe** (Free/Standard subscription billing, per [Pricing → How the subscription is charged](../pricing/#how-the-subscription-is-charged)), and the push providers that carry [push notifications](../api-reference/notifications/#what-a-device-receives): **Apple** (APNs), **Google** (FCM), and the push service of whichever browser a User subscribes from (Google, Mozilla, Apple, or Microsoft). A push payload is the only data they receive, which is why senders are told to keep sensitive content out of it. Both publish standard DPAs (AWS Artifact; Stripe's own, plus Stripe's existing PCI-DSS Level 1 status, which is part of why [the billing system of record](../domain-model/orders-and-audit/#billing-system-of-record) keeps this API's own PCI scope minimal). Executing either is still a step to actually take, not code to write. | | Data residency | **Resolved.** [Tenancy](../domain-model/tenancy/)'s `dedicated_region` tier is the mechanism — a customer with a hard EU-residency requirement gets a dedicated Tenant in an EU region, not a platform-wide region change. Still support-gated today, not self-service — see [Tenancy → How a tier change happens today](../domain-model/tenancy/#how-a-tier-change-happens-today). | | Breach notification (72-hour clock under GDPR) | **Commitment stated, runbook still operational.** The 72-hour clock to notify affected supervisory authorities (and, where risk is high, the data subjects themselves) starts at confirmed awareness of a breach — [Audit Events](../domain-model/orders-and-audit/#audit-event) are the forensic source for scoping what was accessed and when. The specific internal runbook (who's paged, who approves the notification, the template) is operational process, deliberately not specified in an API doc — but the 72-hour commitment itself is now a stated requirement, not an unnamed gap. | | Audit trail vs. erasure tension | **Resolved in the data model.** [Audit Events](../domain-model/orders-and-audit/#audit-event) are retained after a user is hard-deleted, but only the *shape* of what happened (`action`, `timestamp`, which fields changed) — never a copy of the erased personal data itself. The hard-delete cascade above redacts `before`/`after` snapshot values at the same time it deletes the source data, specifically so an Audit Event never becomes the one remaining copy of something erasure was supposed to remove. This is the same redaction every Audit Event eventually gets on its own, by age rather than by request — see [Non-Functional Requirements → Audit log lifecycle](../non-functional-requirements/#audit-log-lifecycle) — so "retained indefinitely" and [Pricing](../pricing/#the-tiers)'s plan-tiered hot-storage window describe different things, not a contradiction. | @@ -53,7 +53,7 @@ Everything in this list is good engineering practice independent of SOC 2, which - **Access control** — already the spine of this entire spec: [Entitlements](../domain-model/entitlements/) + [Roles & Permissions](../domain-model/roles-and-permissions/), least-privilege [API Keys](../api-reference/api-keys/). - **Audit logging** — already specified: the [Action catalog](../domain-model/orders-and-audit/#action-catalog). -- **Encryption in transit and at rest** — HTTPS everywhere (already assumed throughout the [API Reference](../api-reference/)); Aurora's encryption-at-rest in [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) should be turned on by default, not left as a checkbox for later. +- **Encryption in transit and at rest** — HTTPS everywhere (already assumed throughout the [API Reference](../api-reference/)); Aurora's encryption-at-rest in [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md) should be turned on by default, not left as a checkbox for later. - **Change management** — the [Changelog](../changelog/) convention already *is* this, applied to the spec; the same discipline (what changed, when, why) needs to carry into infrastructure and code changes once implementation starts. - **Vendor risk management** — AWS, Stripe, and the push providers (see the DPA row above). Keep the list explicit as it grows — each Application developer's *own* payment processor choice for their own end users is their vendor risk to own, per [Orders & Audit → Billing system of record](../domain-model/orders-and-audit/#billing-system-of-record), not added here. - **Incident response plan** — the one item on this list that's pure process, no code. Write it down before it's needed, not during an actual incident. diff --git a/docs/domain-model/database-schema.md b/docs/domain-model/database-schema.md index d7c6293..9698ee4 100644 --- a/docs/domain-model/database-schema.md +++ b/docs/domain-model/database-schema.md @@ -8,7 +8,7 @@ nav_order: 8 # Database Schema {: .no_toc } -The entity pages describe *what* each field means to an API caller. This page is for an implementer: concrete Postgres types, constraints, and the indexes the documented query patterns actually need. See [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) for why Postgres (Aurora Serverless v2) is the chosen engine. +The entity pages describe *what* each field means to an API caller. This page is for an implementer: concrete Postgres types, constraints, and the indexes the documented query patterns actually need. See [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md) for why Postgres (Aurora Serverless v2) is the chosen engine. {: .fs-6 .fw-300 } 1. TOC @@ -20,7 +20,7 @@ The entity pages describe *what* each field means to an API caller. This page is - Every table's primary key is the entity's own prefixed id (`usr_…`, `ent_…`, …) stored as `text`, not a surrogate `bigint` — the prefix convention in [Conventions → IDs](../../api-reference/conventions/#ids) *is* the primary key, not a display layer on top of one. - Every end-user-scoped table carries `organization_id text null references organizations(id)`, per [Non-Functional Requirements → Multi-tenancy](../../non-functional-requirements/#multi-tenancy), scoped in the data-access layer (see [Non-Functional Requirements → Multi-tenancy](../../non-functional-requirements/#multi-tenancy)) — not repeated per-table below. This is **team grouping**, not infrastructure placement. -- Every table scoped to one Application also carries `tenant_id text not null references tenants(id)`, denormalized from `applications.tenant_id` at write time — a **second, independent** scoping dimension, enforced in the data-access layer and used to route a row to the right physical cluster (see [Tenancy](../tenancy/) and [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md)). Don't conflate this with `organization_id` above — a row can carry both, either, or neither. +- Every table scoped to one Application also carries `tenant_id text not null references tenants(id)`, denormalized from `applications.tenant_id` at write time — a **second, independent** scoping dimension, enforced in the data-access layer and used to route a row to the right physical cluster (see [Tenancy](../tenancy/) and [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md)). Don't conflate this with `organization_id` above — a row can carry both, either, or neither. - Every table carries `created_at timestamptz not null default now()` (the two pure join tables name it for what it means: `organization_memberships.invited_at` and `user_role_assignments.assigned_at`); tables with mutable fields also carry `updated_at timestamptz not null default now()`, maintained by a trigger, not application code (so it's correct even for a direct `UPDATE` run by a migration or a support script). - Soft-deletable tables carry `deleted_at timestamptz null` rather than a boolean — `null` means active, a timestamp means both *that* it's deleted and *when*, which a boolean throws away. - Every table with a `PATCH` endpoint carries `version integer not null default 1`, incremented by the same `updated_at` trigger. It's the source of the `ETag` (`W/""`) and the `If-Match` comparison. See [Conventions → Concurrency](../../api-reference/conventions/#concurrency-etag--if-match). @@ -409,7 +409,7 @@ This only applies to fields an Application has declared. An arbitrary, undeclare | `test_mode` | `boolean` | not null, default `false` — copied from the change that produced it, so test traffic never shows in a live audit query | | `timestamp` | `timestamptz` | not null, default `now()` | -No `updated_at`, no soft-delete column — this table is append-only by design (see [Non-Functional Requirements → Audit](../../non-functional-requirements/#audit)). A trigger (`audit_events_append_only`) rejects every `UPDATE` and `DELETE` unless the transaction has set `app.audit_maintenance = 'on'`, so neither an application bug nor a direct SQL session can violate the "never edited" guarantee. Only two scheduled jobs set it: the archival job (see [Non-Functional Requirements → Audit log lifecycle](../../non-functional-requirements/#audit-log-lifecycle) and [root `DEPLOYMENT.md` → Audit log archival](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md#audit-log-archival)), which deletes rows it has already archived shape-only to cold storage, and the erasure cascade, which redacts `before`/`after` for an erased User. **Indexes:** `(target_user_id, timestamp desc)`, `(application_id, timestamp desc)`, `(actor_id, timestamp desc)`, `(target_type, target_id, timestamp desc)`, `(organization_id, timestamp desc)`, `(tenant_id, timestamp desc)` — one per documented filter in [API Reference → Audit](../../api-reference/audit/), the last one being what a tier-change migration (and the archival job itself) uses to pull "every event for this Tenant" without scanning the whole table. +No `updated_at`, no soft-delete column — this table is append-only by design (see [Non-Functional Requirements → Audit](../../non-functional-requirements/#audit)). A trigger (`audit_events_append_only`) rejects every `UPDATE` and `DELETE` unless the transaction has set `app.audit_maintenance = 'on'`, so neither an application bug nor a direct SQL session can violate the "never edited" guarantee. Only two scheduled jobs set it: the archival job (see [Non-Functional Requirements → Audit log lifecycle](../../non-functional-requirements/#audit-log-lifecycle) and [`infra/README.md` → Audit log archival](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md#audit-log-archival)), which deletes rows it has already archived shape-only to cold storage, and the erasure cascade, which redacts `before`/`after` for an erased User. **Indexes:** `(target_user_id, timestamp desc)`, `(application_id, timestamp desc)`, `(actor_id, timestamp desc)`, `(target_type, target_id, timestamp desc)`, `(organization_id, timestamp desc)`, `(tenant_id, timestamp desc)` — one per documented filter in [API Reference → Audit](../../api-reference/audit/), the last one being what a tier-change migration (and the archival job itself) uses to pull "every event for this Tenant" without scanning the whole table. ## api_keys @@ -482,7 +482,7 @@ webhook_deliveries ( ); ``` -A `webhook_events` row is inserted **in the same transaction** as the change it describes. A dispatcher (SQS-fed, see [DEPLOYMENT.md](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md)) fans it out to matching subscriptions. That makes "every committed change produces its event" structural rather than best-effort. Events and deliveries are deleted after 30 days. **Indexes:** `(dispatched_at) where dispatched_at is null`; `(subscription_id, created_at desc)` and `(next_retry_at) where status = 'pending'` on deliveries. +A `webhook_events` row is inserted **in the same transaction** as the change it describes. A dispatcher (SQS-fed, see [infra/README.md](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md)) fans it out to matching subscriptions. That makes "every committed change produces its event" structural rather than best-effort. Events and deliveries are deleted after 30 days. **Indexes:** `(dispatched_at) where dispatched_at is null`; `(subscription_id, created_at desc)` and `(next_retry_at) where status = 'pending'` on deliveries. ## push_credentials, devices, notifications, notification_deliveries @@ -639,7 +639,7 @@ stripe_events ( ); ``` -Kept 90 days. See [root `DEPLOYMENT.md` → Stripe Billing](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md#stripe-billing). +Kept 90 days. See [`infra/README.md` → Stripe Billing](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md#stripe-billing). ## idempotency_keys @@ -656,4 +656,4 @@ idempotency_keys ( ); ``` -Per [Conventions → Idempotency](../../api-reference/conventions/#idempotency)'s implementation note — a scheduled job (see [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md)) deletes expired rows rather than relying on unbounded table growth. +Per [Conventions → Idempotency](../../api-reference/conventions/#idempotency)'s implementation note — a scheduled job (see [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md)) deletes expired rows rather than relying on unbounded table growth. diff --git a/docs/domain-model/tenancy.md b/docs/domain-model/tenancy.md index 7a7e8e9..cc73432 100644 --- a/docs/domain-model/tenancy.md +++ b/docs/domain-model/tenancy.md @@ -36,8 +36,8 @@ Every Application owner gets a Tenant automatically, the moment they register th | `owner_user_id` | string, nullable | Set iff `owner_type: user`. | | `owner_organization_id` | string, nullable | Set iff `owner_type: organization`. | | `tier` | enum | `shared` (default) \| `isolated` \| `dedicated_region`. See [Tiers](#tiers). **Infrastructure placement** — where the data lives. | -| `region` | string | The AWS region the data lives in, always set: `us-east-1` for `shared` and `isolated` (same region as [Tier 0](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md), with `isolated` just a dedicated cluster within it), and the chosen region for `dedicated_region`. Supported regions: `us-east-1`, `us-west-2`, `ca-central-1`, `eu-west-1`, `eu-central-1`, `ap-southeast-2`. | -| `status` | enum | `active` \| `migrating` \| `suspended`. `migrating` is the transitional state during a tier-change maintenance window — see [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md). | +| `region` | string | The AWS region the data lives in, always set: `us-east-1` for `shared` and `isolated` (same region as [Tier 0](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md), with `isolated` just a dedicated cluster within it), and the chosen region for `dedicated_region`. Supported regions: `us-east-1`, `us-west-2`, `ca-central-1`, `eu-west-1`, `eu-central-1`, `ap-southeast-2`. | +| `status` | enum | `active` \| `migrating` \| `suspended`. `migrating` is the transitional state during a tier-change maintenance window — see [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md). | | `application_count` | integer, read-only | Computed: how many Applications this Tenant owns. | | `created_at`, `updated_at` | timestamp | | @@ -66,8 +66,8 @@ Every Application owner gets a Tenant automatically, the moment they register th | Tier | What changes physically | Cost | Who it's for | |---|---|---|---| -| `shared` | Rows live in the shared Aurora Serverless v2 cluster from [Tier 0](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md), logically isolated by `tenant_id`, enforced in the data-access layer (see [NFR → Multi-tenancy](../../non-functional-requirements/#multi-tenancy)). | $0 incremental — the default. | Every customer, from day one, with no action taken. | -| `isolated` | A dedicated Aurora Serverless v2 cluster (own Secrets Manager secret), same AWS account and region as Tier 0 — stronger blast-radius and noisy-neighbor separation, same Lambda codebase routed by a tenant→cluster lookup. | A full extra Aurora floor (~$45–55/month, see [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md)) — priced by business and enterprise pricing, at least covering it. | A customer with a real compliance or isolation requirement, not just a preference. | +| `shared` | Rows live in the shared Aurora Serverless v2 cluster from [Tier 0](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md), logically isolated by `tenant_id`, enforced in the data-access layer (see [NFR → Multi-tenancy](../../non-functional-requirements/#multi-tenancy)). | $0 incremental — the default. | Every customer, from day one, with no action taken. | +| `isolated` | A dedicated Aurora Serverless v2 cluster (own Secrets Manager secret), same AWS account and region as Tier 0 — stronger blast-radius and noisy-neighbor separation, same Lambda codebase routed by a tenant→cluster lookup. | A full extra Aurora floor (~$45–55/month, see [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md)) — priced by business and enterprise pricing, at least covering it. | A customer with a real compliance or isolation requirement, not just a preference. | | `dedicated_region` | Like `isolated`, but the dedicated cluster sits in the customer's chosen AWS region instead of the default one. | The `isolated` floor again, in a second region — the most expensive tier. | A customer with a genuine data-residency requirement (e.g. "our EU users' data must stay in the EU"). | This is a strict ladder — `dedicated_region` implies everything `isolated` provides, plus region choice; there's no "dedicated region, still logically shared" combination. @@ -78,7 +78,7 @@ There is no self-service API for this. `tenants.manage` (support/superadmin only 1. A customer asks (support is the intake, not a self-serve button) for a tier upgrade. 2. Support calls `POST /v1/tenants/{id}/tier-change-requests` (see [API Reference → Tenancy](../../api-reference/tenancy/)), then moves the request through `pending → scheduled → in_progress → completed` (or `cancelled`) with `PATCH` as the work happens. -3. Support schedules a brief maintenance window, flips `status` to `migrating`, and runs the snapshot/restore cutover into the new infrastructure (new cluster, and for `dedicated_region`, a new region) described in [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md). +3. Support schedules a brief maintenance window, flips `status` to `migrating`, and runs the snapshot/restore cutover into the new infrastructure (new cluster, and for `dedicated_region`, a new region) described in [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md). 4. `tier`, `region`, and `status` are updated back to `active`; an [Audit Event](../orders-and-audit/#audit-event) (`tenant.tier_changed`) records the change. A brief, scheduled downtime window during step 3 is an accepted tradeoff. Tier changes are support-run and infrequent at current scale, so a snapshot/restore cutover is enough. Zero-downtime migration (logical replication) is a legitimate future upgrade once tier changes are self-serve and frequent, not a day-one requirement. diff --git a/docs/domain-model/users-and-organizations.md b/docs/domain-model/users-and-organizations.md index c4b0cc2..23c25b1 100644 --- a/docs/domain-model/users-and-organizations.md +++ b/docs/domain-model/users-and-organizations.md @@ -124,7 +124,7 @@ This table is deliberately thin — scoped to *that* a User can authenticate via A domain/grouping object for Users — a company, or a group within a company (a department, a team) — used to organize who shares admin standing over a set of Users and which Applications a group is granted as a whole, rather than one admin re-granting access per person. -Both individual and team/company end users are expected, so this isn't a someday-maybe feature. It's pulled forward to [Phase 2](../../roadmap/#phase-2) rather than [Phase 3](../../roadmap/#phase-3). It isn't in the MVP itself, because the first few dozen users are expected to be mostly individual early adopters. But it's needed well before the 10x/100x growth horizon in [Deployment Architecture → Growth trajectory](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md), where team accounts are assumed to matter. Making `organization_id` load-bearing in the schema from day one (see [below](#why-it-matters-even-before-organizations-are-customer-facing-everywhere)) was never a hedge against a hypothetical. +Both individual and team/company end users are expected, so this isn't a someday-maybe feature. It's pulled forward to [Phase 2](../../roadmap/#phase-2) rather than [Phase 3](../../roadmap/#phase-3). It isn't in the MVP itself, because the first few dozen users are expected to be mostly individual early adopters. But it's needed well before the 10x/100x growth horizon in [Deployment Architecture → Growth trajectory](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md), where team accounts are assumed to matter. Making `organization_id` load-bearing in the schema from day one (see [below](#why-it-matters-even-before-organizations-are-customer-facing-everywhere)) was never a hedge against a hypothetical. {: .note } Organization carries no infrastructure meaning. Where a group's data physically lives is [Tenant](../tenancy/)'s job, not this entity's — see [Tenancy → Tenant vs. Organization](../tenancy/#tenant-vs-organization) for why these are deliberately separate. An Organization can incidentally *own* a Tenant (if it also owns an Application — see [Applications](../applications/#fields)), but most Organizations, most of the time, don't own one at all. diff --git a/docs/index.md b/docs/index.md index ed5442a..06b7801 100644 --- a/docs/index.md +++ b/docs/index.md @@ -59,4 +59,4 @@ Two questions get asked, separately, every time a user touches an app: Everything else in this spec (profiles, settings, orders, audit) hangs off of that split. See [Access Control](access-control/) for the full model. {: .note } -This site is self-hosting its own context: it's generated from Markdown committed to this same repository, under `docs/`, via GitHub Pages. See the repo root's [`ORIGINAL-SPEC-DRAFT.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/ORIGINAL-SPEC-DRAFT.md) for the original single-document draft this site was built out from — kept out of `docs/` itself so it's never mistaken for current spec. This site is deliberately scoped to **the API specification itself** — how a developer or their AI agent builds against this API. Deployment/infrastructure lives as plain Markdown in the [repository root](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) instead, not here — see the repo's `README.md` for the full map of what's where, including which other project-planning material (the decision log, the build plan, the roadmap) is slated to join it there. +This site is self-hosting its own context: it's generated from Markdown committed to this same repository, under `docs/`, via GitHub Pages. It's deliberately scoped to **the API specification itself**: how a developer or their AI agent builds against this API. How it's deployed is in the repository's [`infra/README.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md), and what's left to build is in [`plan-remaining.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/plan-remaining.md); the repo's `README.md` maps what's where. diff --git a/docs/mcp-server.md b/docs/mcp-server.md index 62468b5..0611e53 100644 --- a/docs/mcp-server.md +++ b/docs/mcp-server.md @@ -34,9 +34,9 @@ This is a new **surface**, not a new **capability**. Every tool call ultimately POST https://api.substratalapps.com/mcp ``` -Implements MCP's **Streamable HTTP** transport: a single endpoint, JSON-RPC 2.0 request/response bodies, with the client's `Mcp-Session-Id` header (issued on `initialize`) tying together the calls in one session. Deliberately on the existing `api.substratalapps.com` domain rather than a new `mcp.` subdomain — no new Route 53 zone or ACM certificate needed, consistent with [Deployment Architecture → Cost principles](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md). Deliberately **outside** `/v1` — this endpoint isn't a REST resource, and MCP has its own protocol-version negotiation (a date-based version string exchanged during `initialize`) that's independent of this API's own `/v1` URL versioning — the same "versioned independently" pattern [Non-Functional Requirements → Versioning](../non-functional-requirements/#versioning) already applies to webhook payloads. +Implements MCP's **Streamable HTTP** transport: a single endpoint, JSON-RPC 2.0 request/response bodies, with the client's `Mcp-Session-Id` header (issued on `initialize`) tying together the calls in one session. Deliberately on the existing `api.substratalapps.com` domain rather than a new `mcp.` subdomain — no new Route 53 zone or ACM certificate needed, consistent with [Deployment Architecture → Cost principles](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md). Deliberately **outside** `/v1` — this endpoint isn't a REST resource, and MCP has its own protocol-version negotiation (a date-based version string exchanged during `initialize`) that's independent of this API's own `/v1` URL versioning — the same "versioned independently" pattern [Non-Functional Requirements → Versioning](../non-functional-requirements/#versioning) already applies to webhook payloads. -Tier 0's implementation is deliberately the simplest conforming option: every tool call returns one buffered JSON response, no server-initiated push, no mid-call progress notifications. See [Deployment Architecture → MCP server](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) for why that's also the cheapest option, and what upgrading past it looks like. +Tier 0's implementation is deliberately the simplest conforming option: every tool call returns one buffered JSON response, no server-initiated push, no mid-call progress notifications. See [Deployment Architecture → MCP server](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md) for why that's also the cheapest option, and what upgrading past it looks like. ## Authentication — no new model @@ -96,7 +96,7 @@ The same classification now has a second, server-side consumer, not just this cl ## Statelessness -An MCP session's state (negotiated protocol version, declared client capabilities) is kept entirely in the `Mcp-Session-Id` the server hands back on `initialize` — a short-lived, signed, self-contained token, not a database row. Tier 0 needs **no new table and no new database** for this: every individual tool call is a single stateless Lambda invocation that decodes the session token, forwards the caller's own Bearer credential to the matching REST call, and returns the result. See [Deployment Architecture → MCP server](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) for when (and why) this stops being enough. +An MCP session's state (negotiated protocol version, declared client capabilities) is kept entirely in the `Mcp-Session-Id` the server hands back on `initialize` — a short-lived, signed, self-contained token, not a database row. Tier 0 needs **no new table and no new database** for this: every individual tool call is a single stateless Lambda invocation that decodes the session token, forwards the caller's own Bearer credential to the matching REST call, and returns the result. See [Deployment Architecture → MCP server](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md) for when (and why) this stops being enough. ## Resources @@ -104,4 +104,4 @@ Alongside tools, the server exposes a small, fixed set of MCP **Resources** — ## Where this runs -Build-out and AWS deployment — Tier 0 cost, and what triggers an upgrade — is specified alongside the rest of the platform's infrastructure, not duplicated here: see [Deployment Architecture → MCP server](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md). +Build-out and AWS deployment — Tier 0 cost, and what triggers an upgrade — is specified alongside the rest of the platform's infrastructure, not duplicated here: see [Deployment Architecture → MCP server](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md). diff --git a/docs/non-functional-requirements.md b/docs/non-functional-requirements.md index bfe6232..6e1d56c 100644 --- a/docs/non-functional-requirements.md +++ b/docs/non-functional-requirements.md @@ -93,7 +93,7 @@ Targets for the API itself, measured monthly at the API Gateway. They're publish | All other endpoints | p95 < 400 ms | | Webhook dispatch (commit → first attempt) | p95 < 10 s | | Push notification dispatch (commit → first provider attempt) | p95 < 10 s | -| JWKS availability | 99.99%. JWKS is served by the API with `Cache-Control: public, max-age=3600`, and Applications cache it for that hour, so a short API outage doesn't stop token verification. CloudFront goes in front of the whole API domain at the [Scale-out](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md#scale-out) trigger, not at Tier 0. | +| JWKS availability | 99.99%. JWKS is served by the API with `Cache-Control: public, max-age=3600`, and Applications cache it for that hour, so a short API outage doesn't stop token verification. CloudFront goes in front of the whole API domain at the [Scale-out](https://github.com/CompositeCode/substratalapps.com/blob/main/plan-remaining.md#scale-out) trigger, not at Tier 0. | ## Versioning @@ -111,7 +111,7 @@ Every response carries an `X-Request-Id` (server-generated if the caller didn't - **Contract tests** validate the implementation's actual responses against [openapi.yaml](../openapi.yaml) — this is what keeps the machine-readable spec from silently drifting from reality, which is the normal failure mode for hand-maintained API docs. - **Unit tests** on the [Access Control](../access-control/) resolution logic (`effective_permissions`) specifically — it's the one piece of logic every single request depends on, and it's pure/deterministic enough to be cheap to test exhaustively (every combination of entitlement status × platform role × app role). -- **Integration tests** run against a real local Postgres (see [Deployment Architecture → Local development](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md)), not a mocked data layer — the Row-Level Security policies in [Multi-tenancy](#multi-tenancy) are exactly the kind of thing a mock would let silently pass while actually being broken. +- **Integration tests** run against a real local Postgres (see [Deployment Architecture → Local development](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md)), not a mocked data layer — the Row-Level Security policies in [Multi-tenancy](#multi-tenancy) are exactly the kind of thing a mock would let silently pass while actually being broken. - **Seed/fixture data** for local and test environments should be generated from the same request/response examples already in the [API Reference](../api-reference/) — one source of realistic data, not a second hand-maintained copy that drifts from the docs. ## Data retention @@ -126,8 +126,8 @@ Every response carries an `X-Request-Id` (server-generated if the caller didn't "Retained indefinitely" and the audit log's **hot-storage window** are two different axes, not a contradiction. The hot window is one year for every Tenant; business and enterprise pricing may negotiate a longer one when it ships. 1. **Hot storage.** A freshly-written Audit Event lives in the primary `audit_events` table (Aurora Postgres) with its full `before`/`after` snapshot — queryable via [API Reference → Audit](../api-reference/audit/) exactly as specified elsewhere on this page. -2. **Archival, at the end of the hot window.** A scheduled job (see [Deployment Architecture → Audit log archival](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md)) moves every Audit Event older than the hot window out of the hot table: `action`, `timestamp`, `actor_type`/`actor_id`, `target_type`/`target_id`, and every other shape field move to cheap, indefinite cold storage; the `before`/`after` snapshot values — the one place the row could hold personal data — are dropped at this point rather than carried forward, the same redaction [Hard-delete cascade](#hard-delete-cascade) step 4 already performs for a deleted User, just applied uniformly by age instead of by erasure request. -3. **Cold storage, forever.** The shape-only archive is never deleted, satisfying "retained indefinitely" literally — what's indefinite is the record that something happened, not a guarantee that its full payload stays warm (or stays at all) past the hot window. Retrieving an archived event for a dispute or investigation is a deliberately manual, support-mediated process, not a live API path — see [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) for the actual retrieval SLA. +2. **Archival, at the end of the hot window.** A scheduled job (see [Deployment Architecture → Audit log archival](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md)) moves every Audit Event older than the hot window out of the hot table: `action`, `timestamp`, `actor_type`/`actor_id`, `target_type`/`target_id`, and every other shape field move to cheap, indefinite cold storage; the `before`/`after` snapshot values — the one place the row could hold personal data — are dropped at this point rather than carried forward, the same redaction [Hard-delete cascade](#hard-delete-cascade) step 4 already performs for a deleted User, just applied uniformly by age instead of by erasure request. +3. **Cold storage, forever.** The shape-only archive is never deleted, satisfying "retained indefinitely" literally — what's indefinite is the record that something happened, not a guarantee that its full payload stays warm (or stays at all) past the hot window. Retrieving an archived event for a dispute or investigation is a deliberately manual, support-mediated process, not a live API path — see [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md) for the actual retrieval SLA. This is also why the hot/cold split doesn't weaken [Audit trail vs. erasure tension](../compliance/#gdpr-and-ccpa--build-for-it-now): a snapshot value is gone (not just hidden) the moment it leaves hot storage, for every Tenant, regardless of whether a User ever requests erasure. diff --git a/docs/pricing.md b/docs/pricing.md index 2f1c604..0bf793a 100644 --- a/docs/pricing.md +++ b/docs/pricing.md @@ -96,7 +96,7 @@ The Products and Prices configured in Stripe. Lookup keys are what the code refe ## Why two tiers -- **Free is free, not just cheap.** The first horizon in [Growth trajectory](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md#growth-trajectory) is dozens of users, and the shared-tier cost of serving free Applications is near zero against the infrastructure floor that's paid regardless. Charging for it would tax exactly the adoption the product needs most right now. +- **Free is free, not just cheap.** The first horizon in [Growth trajectory](https://github.com/CompositeCode/substratalapps.com/blob/main/plan-remaining.md#growth-trajectory) is dozens of users, and the shared-tier cost of serving free Applications is near zero against the infrastructure floor that's paid regardless. Charging for it would tax exactly the adoption the product needs most right now. - **One paid price, for the whole catalog.** A single per-user price is easy to understand and budget for, and works the same for one person or a whole Organization. Per-Application pricing would make every new Application a new purchasing decision. - **Infrastructure choice waits for business pricing.** `isolated` and `dedicated_region` tenancy carry a real, named infrastructure cost, driven by a customer's own compliance or data-residency requirement. That's what business and enterprise pricing is for, so neither is offered on Free or Standard. @@ -104,7 +104,7 @@ The Products and Prices configured in Stripe. Lookup keys are what the code refe The subscription is processed through **Stripe Billing**. Card data never touches this API, and Stripe is PCI-DSS Level 1, which keeps PCI scope minimal (see [Compliance](../compliance/#what-this-doesnt-cover)). Stripe is one of the platform's two sub-processors, alongside AWS (see [Compliance → GDPR (and CCPA)](../compliance/#gdpr-and-ccpa--build-for-it-now)). -The implementation (object mapping, webhook event handling, the daily quantity sync) is specified in [root `DEPLOYMENT.md` → Stripe Billing](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md#stripe-billing). It's a build and operations concern, not part of the API Applications call. +The implementation (object mapping, webhook event handling, the daily quantity sync) is specified in [`infra/README.md` → Stripe Billing](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md#stripe-billing). It's a build and operations concern, not part of the API Applications call. ## Keeping these numbers honest diff --git a/docs/roadmap.md b/docs/roadmap.md index db9afad..b6719ec 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -68,4 +68,4 @@ Self-service, customer-triggered [Tenancy](../domain-model/tenancy/) tier change ## Where this runs -These phases are about feature scope, not infrastructure — the MVP and Phase 2/3 all deploy onto the same [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md), which has its own, independent phasing (first deployment vs. scale-out) driven by traffic and cost, not by which of these feature phases is live. +These phases are about feature scope, not infrastructure — the MVP and Phase 2/3 all deploy onto the same [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md), which has its own, independent phasing (first deployment vs. scale-out) driven by traffic and cost, not by which of these feature phases is live. diff --git a/docs/workflows.md b/docs/workflows.md index cff6192..e44d206 100644 --- a/docs/workflows.md +++ b/docs/workflows.md @@ -139,7 +139,7 @@ Support-run today. See [Tenancy → How a tier change happens today](../domain-m 1. A customer (an Application owner, on business or enterprise terms arranged with Substratal) asks support for dedicated infrastructure, or a specific region for data residency. 2. Support calls `POST /v1/tenants/{id}/tier-change-requests`. See [API Reference → Tenancy](../api-reference/tenancy/). -3. Support agrees a window with the customer and `PATCH`es the request to `scheduled`. At the window, it `PATCH`es to `in_progress`, which flips the [Tenant](../domain-model/tenancy/)'s `status` to `migrating` and makes writes return `503` with `Retry-After`. Then it runs the snapshot/restore cutover described in [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md). Every [Entitlement](../domain-model/entitlements/), [AppProfile](../domain-model/profiles/#appprofile), [AppSettings](../domain-model/settings/#appsettings), and Audit Event row carrying that `tenant_id` moves to the new infrastructure. +3. Support agrees a window with the customer and `PATCH`es the request to `scheduled`. At the window, it `PATCH`es to `in_progress`, which flips the [Tenant](../domain-model/tenancy/)'s `status` to `migrating` and makes writes return `503` with `Retry-After`. Then it runs the snapshot/restore cutover described in [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md). Every [Entitlement](../domain-model/entitlements/), [AppProfile](../domain-model/profiles/#appprofile), [AppSettings](../domain-model/settings/#appsettings), and Audit Event row carrying that `tenant_id` moves to the new infrastructure. 4. `PATCH` to `completed` updates `tier`/`region` and returns `status` to `active`, and an [Audit Event](../domain-model/orders-and-audit/#audit-event) (`tenant.tier_changed`) records it. 5. Nothing about the customer's Applications, Entitlements, or any end user's access changes shape. This workflow only ever moves *where* the same rows live, never *what* they say. diff --git a/DEPLOYMENT.md b/infra/README.md similarity index 74% rename from DEPLOYMENT.md rename to infra/README.md index 804938c..28aabec 100644 --- a/DEPLOYMENT.md +++ b/infra/README.md @@ -1,6 +1,6 @@ # Deployment Architecture -Where the platform API actually runs, and how it's billed. This is a deployment/operations document, not part of the [API specification](https://compositecode.github.io/substratalapps.com/) — it lives here, in the project root, rather than on the docs site, because a prospective API consumer (a developer, or their AI agent) never needs to know how this is hosted to build against it. See [README.md](README.md) for why the project draws that line here. +Where the platform API runs, as built, and how it's billed. This is a deployment and operations document, not part of the [API specification](https://compositecode.github.io/substratalapps.com/), so it lives next to the Terraform that implements it (`infra/terraform/`) rather than on the docs site: an API consumer never needs to know how this is hosted to build against it. Work that isn't built yet (scale-out, dedicated tenancy infrastructure, MCP streaming) is in [plan-remaining.md](../plan-remaining.md); one-time setup steps still to do by hand are in [remaining_manual_steps.md](../remaining_manual_steps.md). Chosen and configured around one constraint above all others: **predictable, capped cost, with no surprises.** @@ -8,17 +8,14 @@ Chosen and configured around one constraint above all others: **predictable, cap 2. [First deployment (Tier 0)](#first-deployment-tier-0) 3. [AWS account & region](#aws-account--region) 4. [Cost guardrails](#cost-guardrails) -5. [Growth trajectory](#growth-trajectory) -6. [Tenancy tiers & where they run](#tenancy-tiers--where-they-run) -7. [MCP server](#mcp-server) -8. [Stripe Billing](#stripe-billing) -9. [Transactional email](#transactional-email) -10. [Webhook dispatch](#webhook-dispatch) -11. [Push delivery](#push-delivery) -12. [Audit log archival](#audit-log-archival) -13. [Scale-out](#scale-out) -14. [Build checklist](#build-checklist) -15. [Local development](#local-development) +5. [Tenancy tiers & where they run](#tenancy-tiers--where-they-run) +6. [MCP server](#mcp-server) +7. [Stripe Billing](#stripe-billing) +8. [Transactional email](#transactional-email) +9. [Webhook dispatch](#webhook-dispatch) +10. [Push delivery](#push-delivery) +11. [Audit log archival](#audit-log-archival) +12. [Local development](#local-development) --- @@ -29,7 +26,7 @@ Four rules this architecture is built around, in priority order: 1. **No resource bills by the hour regardless of traffic.** No always-on EC2 instance, no Fargate task, no NAT Gateway. If nobody's calling the API, the API costs close to nothing. 2. **Every component with a cost has a known floor and a known ceiling.** "Serverless" isn't automatically cheaper — some AWS serverless products (Aurora Serverless v2's minimum capacity, for one) have a *higher* floor than the plain fixed-price alternative, and that trade is only worth it when something else about the product earns it back. See [Database engine: AWS options compared](#database-engine-aws-options-compared). 3. **A hard budget alarm exists before the first resource does.** Guardrails are step one of the build checklist below, not a follow-up task. -4. **Scale-out is a later, deliberate decision, not a default.** Nothing in Tier 0 auto-scales into real money without a human decision — see [Scale-out](#scale-out) for what actually triggers each upgrade. +4. **Scale-out is a later, deliberate decision, not a default.** Nothing in Tier 0 auto-scales into real money without a human decision — see [Scale-out](../plan-remaining.md#scale-out) for what actually triggers each upgrade. ## First deployment (Tier 0) @@ -80,7 +77,7 @@ Postgres is the storage engine — the domain model is relational, and the schem | **Aurora Provisioned (PostgreSQL-compatible)**, smallest instance | ~$55–60/month single-AZ — Aurora's smallest instance class is larger than RDS's smallest | Pricier than Serverless v2's floor for a low-traffic start, with none of Serverless v2's scale-with-load benefit. Not competitive against either other option at Tier 0 volume. | | **Aurora Serverless v2 (PostgreSQL-compatible)**, 0.5 ACU minimum — **chosen** | ~$43/month floor, scales up with load | Costs more than bare RDS at idle. What it buys back: the **Data API**, which is why Tier 0 has no VPC at all (see [below](#why-no-vpc-and-specifically-no-nat-gateway)), and why [tenancy tiers](#tenancy-tiers--where-they-run) can route a request to any of several clusters — shared, or a specific customer's `isolated`/`dedicated_region` one — via one lookup-table value, with every tier using the identical connection mechanism. | -**The ~$30/month gap between RDS and Aurora Serverless v2 is a real, named cost** — not hand-waved away — paid specifically for not having to build and maintain two different database-connection code paths (one VPC-bound for a cheap shared tier, one Data-API-based for dedicated tenants) or a NAT Gateway. At Tier 0's actual scale (see [Growth trajectory](#growth-trajectory)) that $30/month is a smaller cost than the engineering time either alternative would take to build and keep correct. +**The ~$30/month gap between RDS and Aurora Serverless v2 is a real, named cost** — not hand-waved away — paid specifically for not having to build and maintain two different database-connection code paths (one VPC-bound for a cheap shared tier, one Data-API-based for dedicated tenants) or a NAT Gateway. At Tier 0's actual scale (see [Growth trajectory](../plan-remaining.md#growth-trajectory)) that $30/month is a smaller cost than the engineering time either alternative would take to build and keep correct. ### Why no VPC, and specifically no NAT Gateway @@ -89,13 +86,13 @@ A NAT Gateway (~$32/month plus per-GB data processing, easy to forget it's runni - The API Lambda talks to Aurora via the **Data API** (HTTPS, IAM-signed) instead of a direct Postgres connection — so it never needs to be placed inside the VPC in the first place. - The webhook-delivery worker needs genuine outbound internet access (POSTing to arbitrary downstream URLs) — it gets that for free by **not** being placed in a VPC either; a Lambda outside a VPC has normal internet access by default, at no extra cost. -Net effect: zero VPC, zero NAT, for the entire Tier 0 stack. This is revisited in [Scale-out](#scale-out) once throughput, not cost, becomes the binding constraint. +Net effect: zero VPC, zero NAT, for the entire Tier 0 stack. This is revisited in [Scale-out](../plan-remaining.md#scale-out) once throughput, not cost, becomes the binding constraint. ## AWS account & region Resolved: a **dedicated AWS account**, in **`us-east-1`**. -**Dedicated account**, under an AWS Organization rather than reusing an existing personal/other-project account, specifically so [Cost guardrails](#cost-guardrails) below — the Budgets, the Cost Anomaly Detection baseline, the billing alarm — are scoped to exactly this system's spend from day one. A shared account makes every one of those guardrails a cost-allocation-tag exercise instead of an account-level default; at the current team size (one person plus an AI assistant, per the product's own stated growth plan) that's avoidable overhead, not a safety net worth the complexity. AWS Organizations (free) is still the right wrapper even for a single member account — it's what makes adding a second account (the [Scale-out](#scale-out) trigger for a separate staging environment) a few-minutes operation later instead of a migration. +**Dedicated account**, under an AWS Organization rather than reusing an existing personal/other-project account, specifically so [Cost guardrails](#cost-guardrails) below — the Budgets, the Cost Anomaly Detection baseline, the billing alarm — are scoped to exactly this system's spend from day one. A shared account makes every one of those guardrails a cost-allocation-tag exercise instead of an account-level default; at the current team size (one person plus an AI assistant, per the product's own stated growth plan) that's avoidable overhead, not a safety net worth the complexity. AWS Organizations (free) is still the right wrapper even for a single member account — it's what makes adding a second account (the [Scale-out](../plan-remaining.md#scale-out) trigger for a separate staging environment) a few-minutes operation later instead of a migration. **`us-east-1`**, over other candidates, for three concrete reasons rather than "it's the default": @@ -103,6 +100,11 @@ Resolved: a **dedicated AWS account**, in **`us-east-1`**. - **Pricing.** `us-east-1` is at or near the lowest-cost tier for every service in the [Tier 0 component table](#first-deployment-tier-0) above (Lambda, Aurora Serverless v2, API Gateway, S3) — a handful of regions tie it, none meaningfully undercut it for this stack. - **No data-residency constraint pulling the other way.** Nothing in the [Compliance](https://compositecode.github.io/substratalapps.com/compliance/) posture or the current customer base requires EU or other non-US placement for the *shared* tier specifically — that requirement, when it exists, is what the `dedicated_region` [tenancy tier](#tenancy-tiers--where-they-run) is *for*, scoped to the one customer who needs it rather than forcing a region choice for everyone. +**As built (2026-10-20):** production is the member account `substratal-prod` in an AWS Organization created through the AWS console's Projects experience, with a separate management account (billing and policies) and an Identity delegated-admin account. Two consequences worth knowing: + +- That Organization's region-restriction SCP (`AdvancedModeRegionRestrictionSecurityControlPolicy`) allows only global services in `us-east-1` by default. Its `UsEast1Partitional` statement carries an exception for `substratal-prod` (`StringNotEquals aws:PrincipalAccount`) so the stack can run there. The Projects feature might restore the original; if production calls suddenly fail with an explicit SCP deny, check that exception first. +- Console and CLI access go through the Projects role (`AccountFullAccessRole`), and the CLI signs in with `aws login`. The account's identity source (AWS Builder ID) doesn't offer IAM Identity Center permission sets. + This is a Tier 0 default, not a permanent commitment — nothing above prevents a `dedicated_region` Tenant from landing anywhere else, and nothing prevents revisiting the *shared* region later if a majority of the customer base ends up needing otherwise. Revisit if that happens; don't pre-build for it. ## Cost guardrails @@ -130,38 +132,18 @@ Rough, region-dependent, **not a quote**: | MCP Lambda, Stripe webhook Lambda, low call volume each | Low single digits combined, same free-tier coverage as the API handlers' Lambda | | **Floor total** | **~$45–55/month**, before any real traffic | -## Growth trajectory - -The stated plan: dozens of users and a handful of Applications in the first 6–12 months, roughly 10x from there over the following 6–12 months, and another 10x (100x from today) beyond that. - -| Horizon | Rough scale | What this means for the architecture | -|---|---|---| -| 0–6/12 months | Dozens of users, a few Applications | **Tier 0 as specified, unchanged.** The floor cost table above is sized for exactly this. | -| 6–12 months out (~10x) | Low hundreds of users | Watch rate-limit defaults and the Aurora ACU ceiling (not just the floor) — this is when the *first* [Scale-out](#scale-out) triggers are plausible, not guaranteed. Check real metrics rather than upgrading preemptively. | -| 12+ months (~100x) | Low thousands of users | The horizon [Scale-out](#scale-out)'s read-replica, Provisioned Concurrency, and RDS Proxy rows are realistically aimed at. Also the point multi-tenancy query performance (the test/live Row-Level Security policy plus the `tenant_id` filters) is worth profiling specifically, not just trusting by design. | - ## Tenancy tiers & where they run +Only `shared` is built. Every Tenant lives in the Tier 0 cluster today; the `isolated` and `dedicated_region` rows below are the design, and their provisioning, request routing, and migration runbook are in [plan-remaining.md → Dedicated tenancy infrastructure](../plan-remaining.md#dedicated-tenancy-infrastructure). + [Tenancy](https://compositecode.github.io/substratalapps.com/domain-model/tenancy/) is a second, independent scale-out axis — triggered by one customer's requirement, not by aggregate volume, and can happen on day one for a single large customer well before the traffic-driven triggers above are anywhere close. | Tier | Infrastructure | Relationship to Tier 0 above | |---|---|---| -| `shared` | Exactly [Tier 0](#first-deployment-tier-0) as specified — the same Aurora cluster every other `shared`-tier Tenant uses, isolated by `tenant_id` in the data-access layer (see [NFR → Multi-tenancy](https://compositecode.github.io/substratalapps.com/non-functional-requirements/#multi-tenancy)). | Is Tier 0. No separate infrastructure exists for this tier. | +| `shared` | **Built.** Exactly [Tier 0](#first-deployment-tier-0) as specified — the same Aurora cluster every other `shared`-tier Tenant uses, isolated by `tenant_id` in the data-access layer (see [NFR → Multi-tenancy](https://compositecode.github.io/substratalapps.com/non-functional-requirements/#multi-tenancy)). | Is Tier 0. No separate infrastructure exists for this tier. | | `isolated` | A second (third, fourth, …) Aurora Serverless v2 cluster, its own Secrets Manager secret, same AWS account and region as Tier 0. The API Lambda's Data API calls are routed to the right cluster via a small `tenants` lookup table kept in the primary/shared cluster — no new Lambda functions, no code fork. | One extra Aurora floor (~$45–55/month) per `isolated` Tenant — priced per [Pricing](https://compositecode.github.io/substratalapps.com/pricing/#business-and-enterprise). | | `dedicated_region` | Like `isolated`, but the dedicated cluster — and, if latency to that region matters, a regional API Gateway + Lambda deployment in front of it — sits in the customer's chosen AWS region. | The `isolated` floor again, in a second region. | -### Migration mechanics - -A tier change (`shared → isolated`, or either `→ dedicated_region`) is support-run, during a brief scheduled maintenance window: - -1. Provision the destination cluster (new Aurora Serverless v2 cluster; for `dedicated_region`, in the target region) via the same [Build checklist](#build-checklist) steps used for Tier 0 itself. -2. Set the source Tenant's `status` to `migrating` — the API layer treats this as read-only for that Tenant's rows specifically, not a global outage. -3. Snapshot and restore (RDS snapshot copy, or `pg_dump`/`pg_restore` at this data volume) every row carrying that `tenant_id` into the destination cluster. -4. Update the `tenants` lookup row's cluster endpoint, flip `tier`/`region`/`status` back to `active`, decommission the old rows once the cutover is confirmed. -5. Record an Audit Event (`tenant.tier_changed`). - -Because Entitlement, AppProfile, and AppSettings rows all denormalize `tenant_id`, step 3 is a `WHERE tenant_id = ...` export per table — no cross-table join needed to find everything that has to move. - ## MCP server See the [API spec's MCP Server page](https://compositecode.github.io/substratalapps.com/mcp-server/) for what this component is and why it exists; this section is the build-out and cost side of it specifically. @@ -176,15 +158,6 @@ A dedicated Lambda (shown in the [Tier 0 diagram](#first-deployment-tier-0) abov This Lambda never talks to Aurora, Secrets Manager, or SQS directly — it only ever calls the existing API handlers over HTTPS, so its IAM execution role needs no data-plane permissions at all. It's the cheapest possible shape for this component: pure compute, pay-per-call, the same free-tier coverage as the main API handlers' Lambda. -### Scaling the MCP server - -| Trigger | Change | -|---|---| -| Sustained high-frequency tool-call traffic on hot tools | Provisioned Concurrency on the MCP Lambda specifically — same pattern and trigger as the [Scale-out](#scale-out) table's cold-start row. | -| A real need for server-initiated push — mid-call progress notifications, or resumable streams | Move `/mcp` from a plain Lambda-behind-API-Gateway integration to a **Lambda Function URL with streaming responses enabled** (`InvokeMode: RESPONSE_STREAM`) — API Gateway's HTTP API integration buffers the full Lambda response, so it can't carry a chunked `text/event-stream`; a Function URL can. | -| Session/stream-resumption state that no longer fits in a short-lived signed token | A small **DynamoDB** table, keyed by session ID, TTL'd like `idempotency_keys` — ephemeral, high-churn, single-key lookup: the textbook case Aurora is the wrong tool for and DynamoDB is the right one. | -| More than a handful of concurrent agent integrations with genuinely different trust levels | Per-integration API Keys scoped as narrowly as each integration actually needs — an operational practice, not an infrastructure change. | - ## Stripe Billing [Pricing](https://compositecode.github.io/substratalapps.com/pricing/) specifies the tiers and the [Stripe catalog](https://compositecode.github.io/substratalapps.com/pricing/#stripe-catalog). [API Reference → Billing](https://compositecode.github.io/substratalapps.com/api-reference/billing/) specifies the endpoints. This section is how the integration is built and run. The subscriber is a User (an individual subscription) or an Organization (for the members it covers), never a Tenant. @@ -282,38 +255,6 @@ Outbound webhooks use a **transactional outbox**. Every state-changing transacti Cost is negligible at Tier 0 volume (Glacier Deep Archive is priced for exactly this shape of rarely-read, kept-forever data) — this is about bounding Aurora's hot-table size and honoring the redaction-on-age guarantee, not about saving money on S3 itself. -## Scale-out - -Nothing below is pre-built into Tier 0 — each is a deliberate upgrade, triggered by a specific, named condition, not a default growth path. - -| Trigger | Change | -|---|---| -| Sustained read load on `effective-permissions` / catalog reads | Add an Aurora read replica; consider API Gateway response caching for read-heavy, low-churn paths. | -| Lambda cold-start latency becomes user-visible on hot paths | Provisioned Concurrency on the `effective-permissions` and entitlement-toggle handlers specifically — not a blanket setting. | -| Sustained high-throughput DB access, where Data API's per-call overhead starts to matter | Move those paths from Data API to direct VPC connections through **RDS Proxy** — this is the point where a NAT Gateway (or a cheaper NAT instance) becomes worth its cost, because throughput now justifies it rather than convenience. | -| Geographically distributed callers, latency-sensitive | CloudFront in front of API Gateway. | -| Hub availability becomes a real incident risk for every downstream app, not a theoretical one | Multi-region active-passive — deliberately not a Tier 0/1 concern; premature multi-region is itself a cost and complexity risk. | -| Webhook delivery volume grows enough to need isolation from the main API's blast radius | Split the worker Lambda's concurrency/alerting from the API handlers' (it already runs as a separate function). | -| Push volume grows enough that APNs connection reuse, or isolation from webhook delivery, matters | Give push delivery its own worker function with reserved concurrency, so long-lived HTTP/2 connections to APNs survive between batches and a burst of pushes can't delay webhooks. | -| More than one engineer shipping concurrently | A second AWS account (via AWS Organizations) for staging, separate from production — Tier 0 runs on a single account with a `dev`/`prod` naming convention, which is enough until this trigger. | -| A specific customer requires dedicated or regional infrastructure | Run the [Tenancy tiers](#tenancy-tiers--where-they-run) migration for that one customer's Tenant — the one scale-out trigger in this table that's customer-specific rather than aggregate-volume-driven. | -| Agent/MCP tool-call traffic needs server-push, resumable streams, or dedicated concurrency | See [MCP server → Scaling the MCP server](#scaling-the-mcp-server) — its own trigger table. | -| Stripe billing volume or complexity outgrows a single webhook Lambda (e.g. real invoice-dunning logic, usage-based billing beyond simple per-seat metering) | Revisit as its own build, not folded into this table speculatively — nothing here is blocked on it. | - -## Build checklist - -The order this gets stood up in, once API implementation begins: - -1. AWS account (dedicated member account under Organizations, `us-east-1` — see [AWS account & region](#aws-account--region)) + billing contact + **Budgets and Cost Anomaly Detection from step 1**, before any other resource. -2. Route 53 hosted zone + ACM certificate for the chosen API domain. SES domain identity for `mail.substratalapps.com` (DKIM/SPF/DMARC) and an SES production-access request, which takes about a day, so start it here. -3. KMS keys: one asymmetric RSA-2048 signing key for JWTs (JWKS is published from its public half), and one symmetric key for encrypting TOTP and webhook secrets. Secrets Manager secrets: the Aurora master credential, and Stripe's secret key + webhook-signing secret (test-mode keys first — see [Stripe Billing](#stripe-billing)). Aurora Serverless v2 cluster with Data API enabled, minimum 0.5 / maximum capacity set deliberately. -4. IAM: one execution role per Lambda function (API handlers, MCP server, Stripe webhook handler, webhook worker, scheduled-jobs), each scoped to only the resources it actually needs — no shared mega-role. -5. API Gateway HTTP API + custom domain mapping; Lambda handlers deployed behind it, implementing the [API Reference](https://compositecode.github.io/substratalapps.com/api-reference/) / [openapi.yaml](https://compositecode.github.io/substratalapps.com/openapi.yaml) contract. -6. SQS queues (webhook fan-out/delivery, push delivery, email, Stripe events) and their worker Lambdas; EventBridge Scheduler rules for the entitlement-expiry sweep (every 5 minutes, which also expires test-mode Users), the webhook outbox dispatcher, the erasure cascade (hourly), the Organization quantity sync (daily 00:15 UTC), the subscription-access reconcile (hourly), test-mode purge, expired-token and idempotency-key cleanup (nightly), the push stale-device sweep (daily) and notification retention purge (nightly), and [audit log archival](#audit-log-archival). JWKS is served by the API with a 1-hour `Cache-Control`; CloudFront in front of the whole domain waits for the [Scale-out](#scale-out) trigger, since caching only that path would mean fronting the entire API. -7. Stripe account: run the catalog sync script (test mode first), configure the Customer Portal (no plan switching), enable Stripe Tax, register the `/internal/stripe/webhook` endpoint for the events in [Webhook handling](#webhook-handling). -8. CloudWatch Logs with explicit retention on every log group; a small set of alarms (error rate, Lambda throttling, Aurora ACU near max) in addition to the billing guardrails from step 1. -9. CI/CD via OIDC federation (GitHub Actions → an AWS deploy role) — no long-lived IAM user access keys committed anywhere. - ## Local development The RDS Data API choice that keeps Tier 0 NAT-free has a real cost: it doesn't have a clean open-source local emulator, so a developer can't just point the production data-access code at "Data API, but local" the way they could with a plain Postgres driver. @@ -322,4 +263,4 @@ The fix is to not let that choice leak into the application code in the first pl Stripe's own test mode (plus the Stripe MCP server referenced in [Stripe Billing](#stripe-billing)) covers the equivalent problem on the billing side — no local Stripe emulator is needed, since Stripe's hosted test mode is free and already the standard way to develop against it. -The full local setup (Docker Compose services, the devserver, `make` targets) and the `dev` → `main` → production deployment flow are in [README.md → Local development](README.md#local-development) and [README.md → Branches, CI, and deployment](README.md#branches-ci-and-deployment). +The full local setup (Docker Compose services, the devserver, `make` targets) and the `dev` → `main` → production deployment flow are in [README.md → Local development](../README.md#local-development) and [README.md → Branches, CI, and deployment](../README.md#branches-ci-and-deployment). diff --git a/infra/terraform/bootstrap/main.tf b/infra/terraform/bootstrap/main.tf index aaff311..a548842 100644 --- a/infra/terraform/bootstrap/main.tf +++ b/infra/terraform/bootstrap/main.tf @@ -1,6 +1,6 @@ # Applied once, by hand, with admin credentials (README → One-time setup). # It's the only Terraform ever applied from a laptop. In order: -# 1. Cost guardrails, before any other billable resource (DEPLOYMENT.md → +# 1. Cost guardrails, before any other billable resource (infra/README.md → # Cost guardrails: "a hard budget alarm exists before the first # resource does"). # 2. The Terraform state bucket and the artifacts bucket. diff --git a/infra/terraform/modules/lambda_function/main.tf b/infra/terraform/modules/lambda_function/main.tf index ce80495..83445ad 100644 --- a/infra/terraform/modules/lambda_function/main.tf +++ b/infra/terraform/modules/lambda_function/main.tf @@ -1,6 +1,6 @@ # One Lambda function the way every Substratal function is built: Go on # provided.al2023 / arm64, its own execution role scoped to exactly what it -# needs (DEPLOYMENT.md → Build checklist, step 4: no shared mega-role), +# needs (infra/README.md → Build checklist, step 4: no shared mega-role), # explicit log retention, and a "live" alias that API Gateway, SQS, and # the scheduler always invoke. Each deploy publishes a new version and # moves live to it; rollback.yml moves it back. @@ -70,7 +70,7 @@ resource "aws_iam_role_policy" "this" { } # Explicit retention: logs left at "never expire" are the classic growing -# CloudWatch bill (DEPLOYMENT.md → Tier 0 components). +# CloudWatch bill (infra/README.md → Tier 0 components). resource "aws_cloudwatch_log_group" "this" { name = "/aws/lambda/${var.name}" retention_in_days = var.log_retention_days diff --git a/infra/terraform/prod/main.tf b/infra/terraform/prod/main.tf index 8e4059b..d888e6c 100644 --- a/infra/terraform/prod/main.tf +++ b/infra/terraform/prod/main.tf @@ -1,4 +1,4 @@ -# Production: the Tier 0 stack from DEPLOYMENT.md → First deployment. +# Production: the Tier 0 stack from infra/README.md → First deployment. # No VPC for Lambda and no NAT Gateway: every function reaches Aurora over # the RDS Data API (IAM-signed HTTPS), and the webhook worker gets outbound # internet access for free by staying outside a VPC. @@ -13,7 +13,7 @@ locals { data "aws_caller_identity" "me" {} data "aws_route53_zone" "main" { name = var.hosted_zone } -# ── Keys (DEPLOYMENT.md → Build checklist, step 3) ───────────────────── +# ── Keys (infra/README.md → Build checklist, step 3) ───────────────────── # JWT signing: the private key never leaves KMS; JWKS publishes its public half. resource "aws_kms_key" "signing" { @@ -462,7 +462,7 @@ resource "aws_scheduler_schedule" "job" { } } -# ── Alarms (DEPLOYMENT.md → Build checklist, step 8) ─────────────────── +# ── Alarms (infra/README.md → Build checklist, step 8) ─────────────────── resource "aws_sns_topic" "alarms" { name = "${local.prefix}-alarms" diff --git a/internal/billing/billing.go b/internal/billing/billing.go index a6e9022..d815186 100644 --- a/internal/billing/billing.go +++ b/internal/billing/billing.go @@ -1,5 +1,5 @@ // Package billing is the Stripe side of the Free/Standard subscription -// (DEPLOYMENT.md → Stripe Billing): Customers for subscribing Users and +// (infra/README.md → Stripe Billing): Customers for subscribing Users and // Organizations, Standard Checkout, the Customer Portal, the inbound // webhook, subscription sync, and the daily Organization quantity sync. // Third-party developers' own billing never touches it. diff --git a/internal/billing/webhook.go b/internal/billing/webhook.go index 2645ad4..f53d5d1 100644 --- a/internal/billing/webhook.go +++ b/internal/billing/webhook.go @@ -14,7 +14,7 @@ import ( "github.com/CompositeCode/substratalapps.com/internal/core" ) -// Webhook is POST /internal/stripe/webhook (DEPLOYMENT.md → Webhook +// Webhook is POST /internal/stripe/webhook (infra/README.md → Webhook // handling). It isn't Bearer-authenticated: it verifies Stripe-Signature // against the raw body. Every event id is recorded first (insert-or-skip), // so a duplicate returns 200 immediately; processing re-fetches the diff --git a/internal/core/quickstart_test.go b/internal/core/quickstart_test.go index e25ead6..27dd5a0 100644 --- a/internal/core/quickstart_test.go +++ b/internal/core/quickstart_test.go @@ -14,7 +14,7 @@ func TestMain(m *testing.M) { os.Exit(code) } -// TestQuickstart runs docs/quickstart.md end to end: PLAN.md's Phase 1 +// TestQuickstart runs docs/quickstart.md end to end: the Phase 1 // acceptance test. func TestQuickstart(t *testing.T) { e := testenv.New(t) diff --git a/internal/core/subscriptions.go b/internal/core/subscriptions.go index 8ccedc8..b12ba2c 100644 --- a/internal/core/subscriptions.go +++ b/internal/core/subscriptions.go @@ -874,7 +874,7 @@ func (s *Server) BillingCreateOrganizationPortalSession(w http.ResponseWriter, r }) } -// ── Stripe sync (DEPLOYMENT.md → Webhook handling) ───────────────────── +// ── Stripe sync (infra/README.md → Webhook handling) ───────────────────── // StripeSubscription is the subscription state the sync applies, read // fresh from Stripe by the billing package. diff --git a/internal/db/dataapi/dataapi.go b/internal/db/dataapi/dataapi.go index 8728e1d..c2df7e9 100644 --- a/internal/db/dataapi/dataapi.go +++ b/internal/db/dataapi/dataapi.go @@ -1,5 +1,5 @@ // Package dataapi implements db.DB over the RDS Data API: IAM-signed HTTPS -// to Aurora Serverless v2, with no VPC attachment (see DEPLOYMENT.md → Why +// to Aurora Serverless v2, with no VPC attachment (see infra/README.md → Why // no VPC). It's what every Lambda runs in AWS. package dataapi diff --git a/internal/jobs/jobs.go b/internal/jobs/jobs.go index 69682eb..f1ac082 100644 --- a/internal/jobs/jobs.go +++ b/internal/jobs/jobs.go @@ -1,4 +1,4 @@ -// Package jobs is every scheduled job (DEPLOYMENT.md → Build checklist, +// Package jobs is every scheduled job (infra/README.md → Build checklist, // step 6), one entry point each. EventBridge Scheduler invokes the jobs // Lambda with {"job": ""}; locally, `make job name=` runs one. package jobs diff --git a/plan-remaining.md b/plan-remaining.md new file mode 100644 index 0000000..2477dd4 --- /dev/null +++ b/plan-remaining.md @@ -0,0 +1,132 @@ +# Plan: what's left to build + +Everything specified for the Platform API that isn't built yet, in one place. What *is* built is described by the [API specification](https://compositecode.github.io/substratalapps.com/) (the system of record), [README.md](README.md) (code, CI/CD, local development), and [infra/README.md](infra/README.md) (deployment architecture as built). One-time setup that has to be done by hand, rather than built, is in [remaining_manual_steps.md](remaining_manual_steps.md). + +This replaces `PLAN.md`, `DEPLOYMENT.md`, and `ORIGINAL-SPEC-DRAFT.md`, which were retired on 2026-10-20 once everything else in them was built and documented where it belongs. They remain in git history. + +1. [Status](#status) +2. [Verify the build in production](#verify-the-build-in-production) +3. [Test-mode User expiry](#test-mode-user-expiry) +4. [First-party web pages](#first-party-web-pages) +5. [Phase 3: marketplace and scale features](#phase-3-marketplace-and-scale-features) +6. [Dedicated tenancy infrastructure](#dedicated-tenancy-infrastructure) +7. [Growth trajectory](#growth-trajectory) +8. [Scale-out](#scale-out) +9. [Scaling the MCP server](#scaling-the-mcp-server) +10. [Rules for every change](#rules-for-every-change) + +--- + +## Status + +As of 2026-10-20, every operation in `openapi.yaml` is implemented, including subscriber billing, push notifications, and the hosted PKCE endpoints. SSO returns `501 not_implemented` until the broker integration, as specified. Production infrastructure exists in AWS (`substratal-prod`, `us-east-1`); the first deploy is finishing (see [remaining_manual_steps.md](remaining_manual_steps.md)). No product decisions are open: each is written into the spec page it governs and recorded in the [Changelog](https://compositecode.github.io/substratalapps.com/changelog/). + +## Verify the build in production + +The phase exit criteria have only been checked against the local devserver. Once production is up, check them there: + +- **Phase 0:** `GET /v1/health` returns `200` through the real Lambda and API Gateway path. (Done: it returned `200` on 2026-10-20.) +- **Phase 1:** the [Quickstart](https://compositecode.github.io/substratalapps.com/quickstart/) runs end to end against production: create a user, grant an app, check access, turn it off. Run `scripts/quickstart.sh` with `SUBSTRATAL_API=https://api.substratalapps.com/v1` and a test-mode key. +- **Phase 2:** a third-party-shaped app signs a user in through the embedded flow, receives `access.revoked` and kills the session, calls introspection, has its `settings_schema` validated and typed on write, and receives a push sent through its app key. A test-mode User and a test-mode Organization subscribe to Standard through Checkout (needs a Stripe test-mode key in a non-production environment, or temporarily in production), gain the standard Applications, and lapse back to Free. + +## Test-mode User expiry + +Specified 2026-10-20 ([Auth → Test-mode Users → Lifetime](https://compositecode.github.io/substratalapps.com/api-reference/auth/#lifetime), [Changelog](https://compositecode.github.io/substratalapps.com/changelog/)); **not yet built**. Each step can land and be tested on its own, in this order: + +1. **Migration `0005_test_user_expiry`** (`0004` is taken by `0004_protected_superadmin`). Add `users.expires_at` and `users.expired_processed_at`. Backfill existing test-mode Users with `created_at + interval '30 days'`, which is when the purge would have removed them anyway, so nothing changes for them. Then add `check ((expires_at is not null) = test_mode)` and `check (expires_at <= created_at + interval '30 days')`, and the partial index `(expires_at) where test_mode and expired_processed_at is null`. Add `test_user_expired` to the `sessions.revoked_reason` and `devices.inactive_reason` checks. +2. **Creation** (`internal/core/users_create.go`, `createUser`). Test-mode Users get `expires_at` from the request, or now + 24 hours by default. Validate the range (≥ 15 minutes from now, ≤ `created_at` + 30 days) as `422 validation_failed` with `expires_at_out_of_range`, and reject it on a live create with `live_user`. This covers both `POST /v1/users` and test-mode signup. +3. **Enforcement at the edge** (`internal/core/principal.go`, the per-request User lookup). Load `expires_at` alongside `status`, and reject an expired test-mode User's tokens the same way a suspended User's are. Login, `token/refresh`, `app-tokens`, `oauth/token`, and the `mfa/verify` step return `403 test_user_expired`. This is what makes expiry exact to the second. The sweep below only tidies up. +4. **Token lifetimes** (`internal/core/auth_session.go`). Cap the access token's `exp`, the app token's `exp`, and the session's `absolute_expires_at` at the User's `expires_at`, so Applications that verify app tokens locally against JWKS stop accepting them on time without calling the API. +5. **`PATCH /v1/users/{id}` `expires_at`** (`users.manage` only; self → `403 status_change_forbidden`). Same range check. Moving it later on an expired User clears `expired_processed_at` and fires `access.granted` (reason `test_user_extended`) for every Application they still have active access to. Writes `user.updated`. +6. **Expiry sweep**, in the existing `entitlement-sweep` job (every 5 minutes, no new schedule). Select `test_mode and expires_at <= now() and expired_processed_at is null`. For each, in one transaction: revoke sessions (`test_user_expired`), deactivate devices, fire `access.revoked` (reason `test_user_expired`) per Application with active access, enqueue the `user.expired` webhook, write the `user.expired` Audit Event (actor `system`), and set `expired_processed_at`. Reuse the suspension path's revoke-and-notify code, with the reason passed in. +7. **Subscriptions.** The test-mode covered-member count excludes expired Users (`internal/core/subscriptions.go`). Live counts are unaffected, because they never include test-mode Users. +8. **OpenAPI.** Add `expires_at` to `User`, to the `POST /v1/users` and signup request bodies, and to `PATCH /v1/users/{id}`. Add `test_user_expired` to the error, revoke-reason, and device-reason enums, and `user.expired` to the webhook and audit enums. Bump `info.version`, then run `make gen`. +9. **Tests.** Integration tests for each of these: the 24-hour default, the range errors, a token rejected after `expires_at` (with a test clock), `exp` capping, the sweep's events firing exactly once, extension restoring access, a live create rejecting `expires_at`, and the backfill leaving existing test Users working. Update the Quickstart acceptance test if it relies on a test User living longer than 24 hours. +10. **Docs.** Remove the "Not yet built" line from the 2026-10-20 Changelog entry and add a short "built" entry, as was done for push notifications. + +## First-party web pages + +The API is complete, but the user-facing pages its emails and redirects point to don't exist yet. Both are clients of the API, not part of it: + +- **Dashboard** (`dashboard.substratalapps.com`, `DASHBOARD_URL`): the email-verification, password-reset, invitation-acceptance, and email-change links land here. Also where a User subscribes to Standard and manages their account, devices, and inbox, and where org admins manage members and the Organization's subscription. It's the first-party browser origin in the API's CORS allow-list. **At minimum, the verify-email and reset-password pages are needed before email-based signup works for real users.** +- **Hosted sign-in** (`auth.substratalapps.com/authorize`, `AUTHORIZE_URL`): the page behind the hosted authorization-code + PKCE flow ([Auth → Getting an app token](https://compositecode.github.io/substratalapps.com/api-reference/auth/#getting-an-app-token)). The endpoints are built; only the page is missing. Needed before any third-party Application, and by any first-party app that prefers it to embedded login. + +## Phase 3: marketplace and scale features + +Matches [Roadmap → Phase 3](https://compositecode.github.io/substratalapps.com/roadmap/#phase-3) — explicitly **not** built speculatively ahead of the triggers named in [Scale-out](#scale-out) and [Applications → The review lifecycle](https://compositecode.github.io/substratalapps.com/domain-model/applications/#the-review-lifecycle): + +- **SSO broker integration** (`sso_connections` becoming real, an actual WorkOS-or-equivalent wiring behind `POST /v1/auth/sso/{provider}/callback`) — the schema and the route exist from Phase 1 per [Auth → Native auth and per-Organization SSO](https://compositecode.github.io/substratalapps.com/api-reference/auth/#native-auth-and-per-organization-sso), deliberately built in advance precisely so this phase is additive wiring, not a migration. Triggered by a real Organization actually asking for it, not a phase-number default. +- Self-service Application registration + review queue. The *process* is already fully specified — reviewer model (no dedicated assignment, any `applications.manage` holder works the FIFO queue), a 5-business-day policy SLA, the `rejected` status with required `review_notes` and edit-to-resubmit, non-cascading suspension — see [Applications → The review lifecycle](https://compositecode.github.io/substratalapps.com/domain-model/applications/#the-review-lifecycle) and [API Reference → Applications → Reviewing a submission](https://compositecode.github.io/substratalapps.com/api-reference/applications/#reviewing-a-submission). What's actually deferred to this phase is just the self-service *submission* endpoint itself (`POST /v1/applications` opening up beyond platform-admin-only) — the review mechanics it feeds into are build-ready today. +- SCIM-style provisioning. +- Richer audit/compliance views (filtering, export) — driven by actual [Compliance](https://compositecode.github.io/substratalapps.com/compliance/) needs (a SOC 2 audit, a customer's data-export request), not built ahead of either. +- **The hosted sign-in page** above, if no first-party app has needed it sooner. + +## Dedicated tenancy infrastructure + +Only the `shared` tier is built: the schema, `tenants` table, tier-change requests, and `migrating` status all exist, but no `isolated` or `dedicated_region` cluster can be provisioned yet, and requests aren't routed per Tenant. Build this when the first customer actually needs it, alongside the business and enterprise pricing that charges for it ([Pricing → Business and Enterprise](https://compositecode.github.io/substratalapps.com/pricing/#business-and-enterprise)). The tier design is in [infra/README.md → Tenancy tiers](infra/README.md#tenancy-tiers--where-they-run). + +To build: + +- Terraform for an additional Aurora Serverless v2 cluster and secret per dedicated Tenant (and, for `dedicated_region`, optionally a regional API Gateway and Lambda deployment). +- Per-Tenant request routing: a lookup from `tenant_id` to cluster ARN and secret, read on each request by the data-access layer. +- Per-request allowed-set RLS policies (`app.allowed_tenant_ids`, with an explicit platform bypass) as a second line of defense inside a dedicated cluster ([NFR → Multi-tenancy](https://compositecode.github.io/substratalapps.com/non-functional-requirements/#multi-tenancy)). +- Tooling for the runbook below. + +### Migration runbook + +A tier change (`shared → isolated`, or either `→ dedicated_region`) is support-run, during a brief scheduled maintenance window: + +1. Provision the destination cluster (new Aurora Serverless v2 cluster; for `dedicated_region`, in the target region) via the same [Build checklist](README.md#one-time-setup-phase-0) steps used for Tier 0 itself. +2. Set the source Tenant's `status` to `migrating` — the API layer treats this as read-only for that Tenant's rows specifically, not a global outage. +3. Snapshot and restore (RDS snapshot copy, or `pg_dump`/`pg_restore` at this data volume) every row carrying that `tenant_id` into the destination cluster. +4. Update the `tenants` lookup row's cluster endpoint, flip `tier`/`region`/`status` back to `active`, decommission the old rows once the cutover is confirmed. +5. Record an Audit Event (`tenant.tier_changed`). + +Because Entitlement, AppProfile, and AppSettings rows all denormalize `tenant_id`, step 3 is a `WHERE tenant_id = ...` export per table — no cross-table join needed to find everything that has to move. + +## Growth trajectory + +The stated plan: dozens of users and a handful of Applications in the first 6–12 months, roughly 10x from there over the following 6–12 months, and another 10x (100x from today) beyond that. + +| Horizon | Rough scale | What this means for the architecture | +|---|---|---| +| 0–6/12 months | Dozens of users, a few Applications | **Tier 0 as specified, unchanged.** The [floor cost table](infra/README.md#illustrative-tier-0-floor-cost) is sized for exactly this. | +| 6–12 months out (~10x) | Low hundreds of users | Watch rate-limit defaults and the Aurora ACU ceiling (not just the floor) — this is when the *first* [Scale-out](#scale-out) triggers are plausible, not guaranteed. Check real metrics rather than upgrading preemptively. | +| 12+ months (~100x) | Low thousands of users | The horizon [Scale-out](#scale-out)'s read-replica, Provisioned Concurrency, and RDS Proxy rows are realistically aimed at. Also the point multi-tenancy query performance (the test/live Row-Level Security policy plus the `tenant_id` filters) is worth profiling specifically, not just trusting by design. | + +## Scale-out + +Nothing below is pre-built into Tier 0 — each is a deliberate upgrade, triggered by a specific, named condition, not a default growth path. + +| Trigger | Change | +|---|---| +| Sustained read load on `effective-permissions` / catalog reads | Add an Aurora read replica; consider API Gateway response caching for read-heavy, low-churn paths. | +| Lambda cold-start latency becomes user-visible on hot paths | Provisioned Concurrency on the `effective-permissions` and entitlement-toggle handlers specifically — not a blanket setting. | +| Sustained high-throughput DB access, where Data API's per-call overhead starts to matter | Move those paths from Data API to direct VPC connections through **RDS Proxy** — this is the point where a NAT Gateway (or a cheaper NAT instance) becomes worth its cost, because throughput now justifies it rather than convenience. | +| Geographically distributed callers, latency-sensitive | CloudFront in front of API Gateway. | +| Hub availability becomes a real incident risk for every downstream app, not a theoretical one | Multi-region active-passive — deliberately not a Tier 0/1 concern; premature multi-region is itself a cost and complexity risk. | +| Webhook delivery volume grows enough to need isolation from the main API's blast radius | Split the worker Lambda's concurrency/alerting from the API handlers' (it already runs as a separate function). | +| Push volume grows enough that APNs connection reuse, or isolation from webhook delivery, matters | Give push delivery its own worker function with reserved concurrency, so long-lived HTTP/2 connections to APNs survive between batches and a burst of pushes can't delay webhooks. | +| More than one engineer shipping concurrently | A second AWS account (via AWS Organizations) for staging, separate from production — Tier 0 runs on a single account with a `dev`/`prod` naming convention, which is enough until this trigger. | +| A specific customer requires dedicated or regional infrastructure | Run the [Tenancy tiers](infra/README.md#tenancy-tiers--where-they-run) migration for that one customer's Tenant — the one scale-out trigger in this table that's customer-specific rather than aggregate-volume-driven. | +| Agent/MCP tool-call traffic needs server-push, resumable streams, or dedicated concurrency | See [MCP server → Scaling the MCP server](#scaling-the-mcp-server) — its own trigger table. | +| Stripe billing volume or complexity outgrows a single webhook Lambda (e.g. real invoice-dunning logic, usage-based billing beyond simple per-seat metering) | Revisit as its own build, not folded into this table speculatively — nothing here is blocked on it. | + +## Scaling the MCP server + +| Trigger | Change | +|---|---| +| Sustained high-frequency tool-call traffic on hot tools | Provisioned Concurrency on the MCP Lambda specifically — same pattern and trigger as the [Scale-out](#scale-out) table's cold-start row. | +| A real need for server-initiated push — mid-call progress notifications, or resumable streams | Move `/mcp` from a plain Lambda-behind-API-Gateway integration to a **Lambda Function URL with streaming responses enabled** (`InvokeMode: RESPONSE_STREAM`) — API Gateway's HTTP API integration buffers the full Lambda response, so it can't carry a chunked `text/event-stream`; a Function URL can. | +| Session/stream-resumption state that no longer fits in a short-lived signed token | A small **DynamoDB** table, keyed by session ID, TTL'd like `idempotency_keys` — ephemeral, high-churn, single-key lookup: the textbook case Aurora is the wrong tool for and DynamoDB is the right one. | +| More than a handful of concurrent agent integrations with genuinely different trust levels | Per-integration API Keys scoped as narrowly as each integration actually needs — an operational practice, not an infrastructure change. | + +## Rules for every change + +These applied from the first line of code and still apply to everything above: + +- **Access Control's permission checks on every endpoint**, from the first commit of it. Retrofitting authorization onto endpoints built "open" is how it gets missed somewhere. +- **Conventions' error shape, pagination, idempotency, and `me` self-addressing** come from the shared helpers. New endpoints get them for free and don't reimplement them. +- **Contract tests against `openapi.yaml`** for every endpoint, the day it's built. +- **The spec pages and the [Changelog](https://compositecode.github.io/substratalapps.com/changelog/) stay current.** An implementation detail that contradicts or settles something in the spec gets written into the page it governs, with a Changelog entry, immediately. +- **Migrations are expand-only per release**, and numbered in the order they'll reach production ([README → Migrations](README.md#migrations-expand-then-contract)). diff --git a/remaining_manual_steps.md b/remaining_manual_steps.md new file mode 100644 index 0000000..697eea3 --- /dev/null +++ b/remaining_manual_steps.md @@ -0,0 +1,116 @@ +# Remaining manual steps + +One-time setup that has to be done by hand, in a console or from a laptop, to take the Platform API live. Code that still has to be built is in [plan-remaining.md](plan-remaining.md); how the deployment works is in [infra/README.md](infra/README.md). + +Already done (2026-10-20): the AWS account (`substratal-prod`, `196936048600`) and its management account; CLI access with `aws login`; DNS moved from Vercel to Route 53; the Organization SCP exception for `substratal-prod`; the bootstrap Terraform (budgets, anomaly detection, state and artifact buckets, GitHub OIDC roles); GitHub variables, the `production` environment limited to `main`, and the `main` ruleset; and a first partial deploy (Aurora, all five Lambdas, API Gateway, the certificate, SES domain). `https://api.substratalapps.com/v1/health` returns `200`. + +Work through these in order. The first four make the API usable; the rest make it ready for real customers. + +1. [Raise the Lambda concurrency quota](#1-raise-the-lambda-concurrency-quota) +2. [Re-run the production deploy](#2-re-run-the-production-deploy) +3. [Set the superadmin's password](#3-set-the-superadmins-password) +4. [Confirm the alert emails](#4-confirm-the-alert-emails) +5. [Get SES out of the sandbox](#5-get-ses-out-of-the-sandbox) +6. [Set up Stripe](#6-set-up-stripe) +7. [Merge push notifications and add push credentials](#7-merge-push-notifications-and-add-push-credentials) +8. [Register Substratal's Applications](#8-register-substratals-applications) +9. [Fix the pull-request plan job](#9-fix-the-pull-request-plan-job) +10. [Legal and compliance](#10-legal-and-compliance) +11. [Optional: transfer the domain registration](#11-optional-transfer-the-domain-registration) + +--- + +## 1. Raise the Lambda concurrency quota + +New AWS accounts can run only **10** concurrent Lambda executions, and AWS keeps 10 unreserved, so the webhook worker's reserved concurrency of 5 fails the deploy (`decreases account's UnreservedConcurrentExecution below its minimum value of [10]`). It would also cap the whole API at 10 simultaneous requests. + +1. Sign in to the **substratal-prod** console session, region **US East (N. Virginia)**. +2. Open https://us-east-1.console.aws.amazon.com/servicequotas/home/services/lambda/quotas/L-B99A9384 (**Service Quotas → AWS services → AWS Lambda → Concurrent executions**). +3. **Request increase at account level**, enter **1000**, and submit. +4. Watch it under **Service Quotas → Quota request history**. It's usually approved within hours, sometimes a couple of days, and AWS emails the account contact. + +From the CLI instead: `aws service-quotas request-service-quota-increase --service-code lambda --quota-code L-B99A9384 --desired-value 1000 --profile substratal-prod --region us-east-1`. + +## 2. Re-run the production deploy + +After the quota shows **1000** (`aws lambda get-account-settings --profile substratal-prod --query AccountLimit.ConcurrentExecutions`): + +- **Easiest:** GitHub → **Actions → Deploy → Run workflow**, branch `main`, **Run workflow**. +- Or from a terminal: `gh workflow run deploy.yml -R CompositeCode/substratalapps.com --ref main`. + +The run builds the Lambdas, applies Terraform (it picks up where the first run stopped), runs the migrations on the new cluster, makes sure the protected superadmin exists (`SUPERADMIN_EMAIL`), and smoke-tests `/v1/health` and JWKS. It takes about 5–10 minutes. Every later merge into `main` deploys the same way, automatically. + +## 3. Set the superadmin's password + +The deploy creates `adronhall@proton.me` as the **protected superadmin**: active, verified, holding `superadmin`, and impossible to suspend, delete, erase, or demote ([Users → The protected superadmin](https://compositecode.github.io/substratalapps.com/api-reference/users/#the-protected-superadmin)). It has no password until you set one. From a clone of the repository, signed in with `aws login --profile substratal-prod`: + +```bash +export AWS_PROFILE=substratal-prod AWS_REGION=us-east-1 +export DATABASE_BACKEND=dataapi \ + DB_CLUSTER_ARN=arn:aws:rds:us-east-1:196936048600:cluster:substratal-prod-db \ + DB_SECRET_ARN=$(aws rds describe-db-clusters --db-cluster-identifier substratal-prod-db \ + --query 'DBClusters[0].MasterUserSecret.SecretArn' --output text) +go run ./cmd/admin bootstrap -email adronhall@proton.me # prompts for the password (12+ characters) +``` + +Then sign in: `curl -s https://api.substratalapps.com/v1/auth/login -H 'content-type: application/json' -d '{"email":"adronhall@proton.me","password":"…"}'`. Turn on MFA right away ([Auth → MFA](https://compositecode.github.io/substratalapps.com/api-reference/auth/#mfa-totp)). + +## 4. Confirm the alert emails + +- **Budget and anomaly alerts:** SNS sent a subscription confirmation ("AWS Notification - Subscription Confirmation") for the `substratal-billing-alerts` topic to `adronhall@proton.me`. Click **Confirm subscription**; until then the $75 warning, the $150 hard stop, and anomaly alerts reach no one. +- **Operational alarms** (5xx rate, Lambda errors and throttles, Aurora near capacity, webhook DLQ, security log): the `alarm_email` Terraform variable is empty, so these alarms have no recipient. Decide the address; setting it is a one-line change to `infra/terraform/prod/variables.tf` and a confirmation click after the next deploy. + +## 5. Get SES out of the sandbox + +SES is still in the **sandbox**: it can only send to verified addresses, so signup verification, password resets, and invitations don't reach real users. + +1. **substratal-prod** console → **Amazon SES → Account dashboard → Request production access**. +2. Mail type **Transactional**; website `https://compositecode.github.io/substratalapps.com/`; describe the use: account verification, password resets, invitations, and security notices for users of Substratal's Applications, sent from `mail.substratalapps.com` (DKIM/SPF/DMARC configured), with bounces and complaints suppressed. +3. AWS usually answers within a day. + +Until then, verify any address you need to test with under **SES → Identities → Create identity → Email address**. + +## 6. Set up Stripe + +The billing code (Free and Standard at $6 per user per month) is built and deployed; Stripe itself isn't set up. See [Pricing](https://compositecode.github.io/substratalapps.com/pricing/) and [infra/README.md → Stripe Billing](infra/README.md#stripe-billing). + +1. **Activate the Stripe account:** business details, bank account, and the public statement descriptor. Live mode needs a privacy policy and terms of service URL (step 10). +2. **Enable Stripe Tax** and add your tax registrations (billing is tax-exclusive). +3. **Create the catalog**, test mode first: `STRIPE_SECRET_KEY=sk_test_… make stripe-catalog` creates the `substratal_standard` Product and the `standard_monthly_usd` Price ($6.00 per user, monthly). Repeat with the live key (`CONFIRM_LIVE=yes`) when going live. +4. **Configure the Customer Portal** (Settings → Billing → Customer portal): allow updating payment methods, viewing invoices, and canceling at period end. **Turn plan switching off.** +5. **Register the webhook endpoint** `https://api.substratalapps.com/internal/stripe/webhook` with these events: `checkout.session.completed`, `customer.subscription.created`, `customer.subscription.updated`, `customer.subscription.deleted`, `invoice.paid`, `invoice.payment_failed`. Copy its signing secret. +6. **Put the keys into the app secret** `substratal-prod/app` (Secrets Manager → `substratal-prod/app` → Retrieve secret value → Edit): set `stripe_secret_key` (a restricted key with Customers, Subscriptions, Checkout Sessions, Customer Portal, and Prices read access) and `stripe_webhook_secret`. Terraform never overwrites these. The Lambdas pick them up on their next cold start; re-running the deploy forces it. +7. **Decide test or live.** Production holds one key. With a test key, the whole flow can be tried end to end with Stripe's test cards; with the live key, test-mode Users can't check out (`409 test_mode_billing_unavailable`, by design). Start with test, then switch to live. + +## 7. Merge push notifications and add push credentials + +Push notifications are built in PR #12 (`dev`), not yet on `main`; until it merges, the 17 push operations return `501` in production. Merging it runs migration `0003_push.sql`, which is independent of `0004_protected_superadmin.sql` that will already be applied. + +After that, for each Application that sends pushes, its owner uploads credentials with `POST /v1/applications/{id}/push-credentials` (and rotates them with `PATCH …/push-credentials/{provider}`): an APNs key (`.p8`, key id, team id, bundle id), an FCM service account, or Web Push VAPID keys ([Push Notifications → Push credentials](https://compositecode.github.io/substratalapps.com/api-reference/notifications/#push-credentials)). + +## 8. Register Substratal's Applications + +For each of Substratal's apps, with the superadmin's token ([Applications → Owner onboarding](https://compositecode.github.io/substratalapps.com/api-reference/applications/#owner-onboarding-end-to-end)): + +1. `POST /v1/applications` with its slug, name, `launch_url`, `redirect_uris`, `permissions`, `available_app_roles`, and **`access_tier`** (`free` or `standard`, which decides who gets it under [Pricing](https://compositecode.github.io/substratalapps.com/pricing/)). +2. Give each seeded Role its permissions (`PATCH /v1/roles/role__`). +3. Create the app backend's key (`POST /v1/api-keys`, scope the app, `entitlements.manage` and `users.list`), plus a test-mode key. +4. Subscribe the backend to `access.revoked` and `role.removed` (`POST /v1/webhooks`): the required revocation integration. + +## 9. Fix the pull-request plan job + +The `plan` job on pull requests into `main` fails: the read-only plan role can't read the app secret's value (`secretsmanager:GetSecretValue`), which `terraform plan` needs to refresh `aws_secretsmanager_secret_version.app`. It isn't a required check, so it doesn't block merges, but its plan comment is the best preview of what a release changes in AWS. Choose one: + +- Allow the plan role `secretsmanager:GetSecretValue` on `substratal-prod/app` only (it then can read the Stripe keys, as anyone who can open a PR into `main` could), or +- Stop Terraform reading the value: keep the secret's initial value out of Terraform state and plan with the secret version untracked. This is a small Terraform change. + +## 10. Legal and compliance + +- **Privacy policy and terms of service**, published at stable URLs. Stripe live mode and SES production access both expect them. +- **Data processing agreements** with the sub-processors listed on [Compliance](https://compositecode.github.io/substratalapps.com/compliance/): AWS (AWS Artifact → Agreements), Stripe, and the push providers (Apple, Google, and the browser push services). + +## 11. Optional: transfer the domain registration + +DNS is on Route 53; the registration is still at Vercel (reseller for Tucows), renewing 2027-08-07. To consolidate: in Vercel, **Domains → substratalapps.com → Transfer out** to get the authorization code, then `aws route53domains transfer-domain --domain-name substratalapps.com --duration-in-years 1 --auth-code … --region us-east-1` with the registrant contact details (about $15, adds a year). Keep the Route 53 nameservers during the transfer. + +Also keep an eye on the Organization SCP exception for `substratal-prod` ([infra/README.md → AWS account & region](infra/README.md#aws-account--region)): if production calls suddenly fail with an explicit SCP deny in `us-east-1`, that exception was removed. diff --git a/scripts/quickstart.sh b/scripts/quickstart.sh index 320a6ae..e8ee227 100755 --- a/scripts/quickstart.sh +++ b/scripts/quickstart.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash # The docs/quickstart.md walkthrough as a script, against a running API # (default: the local devserver). It's the Phase 1 acceptance test from -# PLAN.md, so CI and `make quickstart` run it on every change. +# plan-remaining.md, so CI and `make quickstart` run it on every change. # # SUBSTRATAL_API=http://localhost:8080/v1 TOKEN= scripts/quickstart.sh #