diff --git a/.env.example b/.env.example index 7777442..c00eee7 100644 --- a/.env.example +++ b/.env.example @@ -31,8 +31,14 @@ SMTP_ADDR=localhost:51025 EMAIL_FROM=Substratal # Webhooks: allow http://localhost receivers locally (never in production). +# Also lets live push delivery reach a Web Push fake on http://localhost. WEBHOOKS_ALLOW_PRIVATE=true +# Push notifications: "log" records every push in the devserver's log and +# sends nothing; "live" sends to APNs, FCM, and Web Push with the +# credentials each Application configures (production's default). +PUSH_BACKEND=log + # Stripe test mode (optional; billing endpoints return 503 without it). STRIPE_SECRET_KEY= STRIPE_WEBHOOK_SECRET= diff --git a/.gitignore b/.gitignore index 4c78664..e718b15 100644 --- a/.gitignore +++ b/.gitignore @@ -18,3 +18,4 @@ dist/ infra/terraform/**/.terraform/ infra/terraform/**/*.tfstate* infra/terraform/**/.terraform.lock.hcl.bak +.claude/settings.local.json diff --git a/ORIGINAL-SPEC-DRAFT.md b/ORIGINAL-SPEC-DRAFT.md deleted file mode 100644 index b8863b9..0000000 --- a/ORIGINAL-SPEC-DRAFT.md +++ /dev/null @@ -1,213 +0,0 @@ -> **Superseded.** This is the original single-document draft the full specification at [compositecode.github.io/substratalapps.com](https://compositecode.github.io/substratalapps.com/) was elaborated from — kept here for history, in the project root rather than the docs site, specifically so it's never mistaken for current spec. It contradicts current decisions in several structural ways (no [Tenant](https://compositecode.github.io/substratalapps.com/domain-model/tenancy/) concept, single-method auth, a 3-question access algorithm instead of the current 2-axis union model) — if anything below conflicts with the live docs site or [`decisions.md`](https://compositecode.github.io/substratalapps.com/decisions/), the live site wins, always. See [README.md](README.md) and the [Changelog](https://compositecode.github.io/substratalapps.com/changelog/#2026-10-03)'s initial-publication entry for how this became that. - -# Substratal Hub — Access & Identity API - -**Status:** Draft for elaboration · **Scope:** platform API that sits behind substratalapps.com · **Audience:** engineers turning this into a full technical spec - -## 1. What this document is - -Substratal Apps is the hub a customer lands on to see every app, product, and service they've purchased from Substratal, sign into it with one identity, and manage their account across all of them. This document describes the API that powers that hub: the system of record for *who* a user is, *what* they're allowed to touch, and *how* each app they own is configured for them. - -It is deliberately a level above endpoint-by-endpoint detail. The goal is a domain model and resource map solid enough that a full OpenAPI spec, data schema, and build plan can be written from it without re-litigating the core decisions. Section 13 lists the decisions this draft deferred — resolve those before the full spec is final. - -**In scope:** user identity, roles & permissions, per-app entitlement (the purchase → access → on/off lifecycle), per-user profile data (global and per-app), per-user settings (global and per-app), the events/tokens downstream apps need to trust the hub. - -**Out of scope (adjacent systems this API talks to, but doesn't replace):** payment processing, app-specific business logic/data, marketing site content, support ticketing. - -## 2. Domain model - -| Entity | What it represents | -|---|---| -| **User** | A person with an account on Substratal. One identity, used everywhere. | -| **Organization** *(optional — see §13)* | A billing/access group of Users, e.g. a team plan. If absent, every User is implicitly its own org of one. | -| **Application** | A catalog entry for a product/service Substratal sells — "Product," not "running process." The app a user ultimately gets routed to. | -| **Role** | A named bundle of Permissions. Scoped either to the whole platform (`PlatformRole`) or to one Application (`AppRole`). | -| **Permission** | An atomic, checkable capability (`billing.manage`, `app.timetrack.export`, …). | -| **Entitlement** | The join between a User (or Org) and an Application: *do they own it, and is it currently switched on.* This is the record behind "turn an app on/off for a user." | -| **Profile** | Display/contact identity data. Global (name, avatar, email, locale) and, separately, per-app (fields only that app cares about — e.g. a display handle used only inside one product). | -| **Setting** | Configuration, not identity. Global (site-wide preferences) and per-app (that app's preferences for that user), each layered over an app-level default. | -| **Order / Subscription** | The commerce record (what was bought, when, recurring or not) that an Entitlement traces back to. Owned by billing, referenced here. | -| **Audit Event** | An immutable record of who changed what access, when. | - -### 2.1 Entity relationships - -```mermaid -erDiagram - USER ||--o{ ENTITLEMENT : holds - ORGANIZATION ||--o{ USER : contains - ORGANIZATION ||--o{ ENTITLEMENT : "holds (org-wide)" - APPLICATION ||--o{ ENTITLEMENT : "granted via" - ENTITLEMENT ||--o| ORDER : "traces to" - USER ||--|| PROFILE : "has (global)" - USER ||--o{ APP_PROFILE : "has, per app" - APPLICATION ||--o{ APP_PROFILE : scopes - USER ||--|| SETTINGS : "has (global)" - USER ||--o{ APP_SETTINGS : "has, per app" - APPLICATION ||--o{ APP_SETTINGS : scopes - USER }o--o{ PLATFORM_ROLE : assigned - USER }o--o{ APP_ROLE : "assigned, per app" - APP_ROLE }o--|| APPLICATION : scopes - PLATFORM_ROLE ||--o{ PERMISSION : grants - APP_ROLE ||--o{ PERMISSION : grants -``` - -The two relationships worth pausing on: an **Entitlement** is independent of a **Role**. Entitlement answers "can this user reach App X at all, right now" (the on/off switch). Role answers "once inside, what can they do." A support agent can hold a platform Role that lets them *see* every user's entitlements without ever holding an entitlement to a single app themselves — access-to-manage-access is a different axis from access-to-use. - -## 3. Access control model - -Three independent questions, in the order they get checked: - -1. **Is the app itself live for this user?** → `Entitlement.status == active` for (User, Application). If no, stop — nothing below matters. This is the literal on/off toggle an admin or billing event flips. -2. **Does the user hold a Role — platform or app-scoped — that grants the Permission being invoked?** → union of `PlatformRole` permissions and, for that Application, `AppRole` permissions. -3. **Org-level override, if Organizations are in play** → an Org admin's Role can widen or narrow what members of that org inherit by default. - -``` -allow(user, application, permission) := - entitlement(user, application).status == "active" - AND permission ∈ effective_permissions(user, application) -``` - -`effective_permissions` is the union of permissions from every `PlatformRole` assigned to the user, plus every `AppRole` assigned to the user scoped to that `application`. - -**Platform roles** (examples): `superadmin`, `support`, `billing_admin`, `member`. These govern the hub itself — managing other users, issuing refunds, viewing audit logs. - -**App roles** are defined per Application by whoever owns that app's catalog entry (e.g. an app might define `admin` / `editor` / `viewer`), but the hub is the registry and the enforcement point for them — a downstream app doesn't maintain its own user table, it asks the hub. - -## 4. Resource & data model - -### 4.1 User -| Field | Notes | -|---|---| -| `id` | opaque, stable | -| `email` | unique, verified flag | -| `status` | `active` \| `invited` \| `suspended` \| `deleted` | -| `auth` | password hash / SSO subject, MFA state — likely delegated to an identity provider rather than owned here (see §13) | -| `created_at`, `last_login_at` | | -| `organization_id` | nullable | - -### 4.2 Profile (global) & AppProfile -| Field | Global `Profile` | `AppProfile` (per User × Application) | -|---|---|---| -| name, avatar_url | ✓ | — (inherits unless overridden) | -| display_handle | — | ✓ app-local identity, e.g. a username only meaningful inside one product | -| contact email/phone | ✓ | — | -| custom fields | — | ✓ arbitrary JSON the owning app defines for itself | - -### 4.3 Settings (global) & AppSettings -| Field | Global `Settings` | `AppSettings` (per User × Application) | -|---|---|---| -| locale, timezone, theme | ✓ | inherits from global unless set | -| notification preferences | ✓ (site-wide) | ✓ (per-app channel overrides) | -| app-specific preferences | — | ✓ arbitrary JSON, validated against that app's declared settings schema | - -Resolution order for any setting an app reads: **AppSettings → Settings → Application's declared default.** The hub stores the override layers; it does not need to know what each key means, only enforce the schema the Application registered for itself. - -### 4.4 Application (catalog entry) -| Field | Notes | -|---|---| -| `id`, `slug`, `name`, `description`, `icon_url` | catalog/display | -| `launch_url` / `sso_redirect` | where the hub sends an authenticated user | -| `settings_schema` | JSON Schema the app declares, so AppSettings can be validated centrally | -| `available_app_roles` | roles this app defines, for role-assignment UI | -| `visibility` | `public` (anyone can acquire) \| `invite_only` \| `internal` | - -### 4.5 Entitlement -| Field | Notes | -|---|---| -| `id`, `user_id` (or `organization_id`), `application_id` | | -| `status` | `active` \| `disabled` \| `expired` \| `revoked` | -| `source` | `purchase` \| `trial` \| `admin_grant` \| `org_seat` | -| `order_id` | nullable link to commerce record | -| `starts_at`, `ends_at` | for trials/subscriptions | -| `disabled_reason` | free text / enum, set when an admin flips it off manually | - -### 4.6 Role / Permission -| Field | Notes | -|---|---| -| `Role.id`, `name`, `scope` | `scope` = `platform` or `application_id` | -| `Role.permissions[]` | list of Permission keys | -| `Permission.key` | dotted string, e.g. `users.manage`, `app..export` | -| `UserRoleAssignment` | (`user_id`, `role_id`, nullable `application_id` for app-scoped roles) | - -### 4.7 Order / Subscription (referenced, owned by billing) -Minimal fields the access API needs to react to: `id`, `user_id`/`organization_id`, `application_id`(s), `status` (`paid`, `past_due`, `cancelled`, `refunded`), `renews_at`. A billing webhook updates Entitlement status off of these — this API doesn't process payment. - -### 4.8 Audit Event -`id`, `actor_user_id`, `action` (`entitlement.disabled`, `role.assigned`, …), `target_user_id`, `application_id` (nullable), `before`/`after` snapshot, `timestamp`. - -## 5. API surface - -Resource-oriented REST, versioned (`/v1`), JSON, cursor pagination on list endpoints, idempotency keys on mutating ones. - -| Group | Endpoints | Purpose | -|---|---|---| -| **Auth** | `POST /v1/auth/login`, `POST /v1/auth/token/refresh`, `POST /v1/auth/sso/{provider}/callback`, `POST /v1/auth/logout` | Issue/refresh the session; see §7 for what downstream apps do with the result. | -| **Users** | `GET/POST /v1/users`, `GET/PATCH/DELETE /v1/users/{id}`, `POST /v1/users/{id}/suspend` | Account lifecycle. | -| **Profile** | `GET/PATCH /v1/users/{id}/profile`, `GET/PATCH /v1/users/{id}/apps/{appId}/profile` | Global vs. per-app profile. | -| **Settings** | `GET/PATCH /v1/users/{id}/settings`, `GET/PATCH /v1/users/{id}/apps/{appId}/settings` | Global vs. per-app settings, validated against the app's schema. | -| **Applications (catalog)** | `GET /v1/applications`, `GET/POST/PATCH /v1/applications/{id}` (admin-only write) | The app catalog shown on the hub. | -| **Entitlements** | `GET /v1/users/{id}/entitlements`, `POST /v1/users/{id}/entitlements`, `PATCH /v1/entitlements/{id}` (`{status: "disabled"}` is the on/off switch), `DELETE /v1/entitlements/{id}` | Grant, list, toggle, revoke access to an app. | -| **Roles** | `GET/POST /v1/roles`, `GET /v1/users/{id}/roles`, `POST/DELETE /v1/users/{id}/roles/{roleId}` | Assign/remove platform or app roles. | -| **Permissions** | `GET /v1/permissions`, `GET /v1/users/{id}/applications/{appId}/effective-permissions` | The resolved-permission check downstream apps and UI call. | -| **Organizations** *(if in scope)* | `GET/POST /v1/organizations`, `.../members`, `.../entitlements` | Team/seat management. | -| **Audit** | `GET /v1/audit-events` (filterable by user/app/actor) | Compliance and support investigation. | -| **Webhooks** | `GET/POST /v1/webhooks` (subscription management) | See §6. | - -## 6. Key workflows - -**Purchase → access.** Billing completes an Order → billing service calls (or the hub listens for) a webhook → hub creates/updates an `Entitlement(status=active, source=purchase, order_id=...)` → hub emits `entitlement.granted` → the hub UI now shows the app as available, and the downstream app (if it independently checks) sees it on next token refresh or introspection call. - -```mermaid -sequenceDiagram - participant Billing - participant Hub as Hub API - participant App as Downstream App - participant User - - Billing->>Hub: order.paid webhook (user, application) - Hub->>Hub: upsert Entitlement(status=active, source=purchase) - Hub->>Hub: emit entitlement.granted - User->>Hub: opens dashboard - Hub-->>User: app now listed as available - User->>App: launches app (SSO redirect) - App->>Hub: verify token / introspect - Hub-->>App: valid, entitlement=active, effective_permissions=[...] -``` - -**Admin turns an app off for a user.** Support/admin calls `PATCH /entitlements/{id} {status: disabled, disabled_reason}` → hub writes an Audit Event → emits `entitlement.revoked` → any live session the user has in that app is invalidated on next check (the app should re-verify on a short interval, not just at login — see §7). - -**Role change.** Admin assigns/removes a Role → `effective-permissions` for that user+app changes immediately → downstream app picks it up the next time it checks (introspection or next token refresh), not necessarily mid-session unless the app subscribes to the webhook. - -## 7. How downstream apps trust the hub - -Each Application is a largely independent product; it shouldn't maintain its own user table. Two mechanisms, meant to be used together: - -- **Short-lived JWT**, issued at login/SSO-redirect, carrying `sub` (user id), `org_id`, and claims for *that specific app* (`entitlement_status`, `roles`, `effective_permissions`), signed by the hub. Cheap to verify locally, but can go stale within its TTL — keep the TTL short (minutes) for anything access-sensitive. -- **Introspection endpoint** (`GET /v1/users/{id}/applications/{appId}/effective-permissions`) for anything that needs a live answer — e.g. before a destructive action, or to immediately honor an admin's "turn this off" click rather than waiting out a token TTL. -- **Webhooks** (`entitlement.granted`, `entitlement.revoked`, `role.assigned`, `role.removed`) for apps that want to react immediately (kill an active session) rather than poll. - -## 8. Non-functional requirements - -- **AuthN:** OAuth2/OIDC for user-facing login (own IdP or delegate to one — see §13); service-to-service calls (billing → hub, app → hub) use scoped API keys or mTLS. -- **AuthZ enforcement point:** the hub is the only writer of Role/Entitlement state. Apps read, they don't maintain parallel copies. -- **Multi-tenancy:** every record scoped by `organization_id` where orgs are in play; row-level isolation, not just application-level filtering. -- **Audit:** every write to Entitlement, Role assignment, or admin-initiated Profile/Settings change is logged immutably. -- **Rate limiting & idempotency:** standard on all mutating endpoints; idempotency key required on Entitlement and Order-triggered writes since billing webhooks retry. -- **Versioning:** `/v1` now; additive changes preferred over breaking ones; deprecation window policy to be set in the full spec. - -## 9. Phased build - -| Phase | Delivers | -|---|---| -| **MVP** | User, Profile (global only), Application catalog, Entitlement (grant/revoke/toggle), Platform Roles only, basic audit log. Enough for "see my apps, click to launch, admin can turn access on/off." | -| **Phase 2** | App-scoped Roles & Permissions, per-app Profile and Settings with schema validation, webhooks, introspection endpoint. | -| **Phase 3** | Organizations/seats, delegated admin (org admin manages their own members), richer audit/compliance views, SCIM-style provisioning for larger customers. | - -## 10. Open decisions (resolve before writing the full spec) - -1. **Identity provider:** build auth in-house, or delegate to an IdP (Auth0, Clerk, WorkOS, Cognito, …)? Changes what §4.1/`auth` and §7's JWT issuance actually look like. -2. **Organizations:** is multi-seat/team access a day-one requirement, or does every account start as a single user and orgs get bolted on later? Affects whether `organization_id` is load-bearing now or added in Phase 3. -3. **Downstream app architecture:** are the "apps" separately hosted services that need SSO + token verification (as assumed above), or more like modules/iframes inside one deployment, where access control could live entirely in the hub's own session? This is the single biggest fork in the API shape. -4. **Billing system of record:** which processor, and does it push webhooks to the hub or does the hub poll it? Determines the contract in §2's Order entity and §6's purchase workflow. -5. **Settings schema ownership:** does each Application register its own JSON Schema with the hub (as assumed in §4.3), or does the hub stay fully opaque to app settings and just store/return a blob? -6. **Session model for revocation:** how fast must "turn off this user's access" take effect inside an already-open app session — immediately (requires apps to poll/subscribe) or "by next login" (much simpler, weaker guarantee)? diff --git a/PLAN.md b/PLAN.md deleted file mode 100644 index 304f84f..0000000 --- a/PLAN.md +++ /dev/null @@ -1,106 +0,0 @@ -# Build Plan - -How to actually build the API this project specifies, in what order, and which document governs each piece. This is a plan for *implementation* — what to build and when. The API itself is specified at **[compositecode.github.io/substratalapps.com](https://compositecode.github.io/substratalapps.com/)**; this document doesn't repeat that spec, it sequences building it. - -See [README.md](README.md) for why this file — and the others alongside it — live in the project root instead of the docs site. - -1. [Before writing code](#before-writing-code) -2. [Build order](#build-order) -3. [Phase 0 — Infrastructure skeleton](#phase-0--infrastructure-skeleton) -4. [Phase 1 — MVP](#phase-1--mvp) -5. [Phase 2 — Per-app customization & trust](#phase-2--per-app-customization--trust) -6. [Phase 3 — Marketplace & scale features](#phase-3--marketplace--scale-features) -7. [Cross-cutting, build throughout](#cross-cutting-build-throughout) -8. [Status](#status) - ---- - -## Before writing code - -Read, in this order — each is short, and skipping one tends to show up as a wrong assumption two phases later: - -1. **[Getting Started](https://compositecode.github.io/substratalapps.com/getting-started/)** and **[Domain Model](https://compositecode.github.io/substratalapps.com/domain-model/)** — the nine-ish entities and how they relate. Everything else assumes this. -2. **[Access Control](https://compositecode.github.io/substratalapps.com/access-control/)** — the one algorithm (`allow(user, application, permission)`) every endpoint ultimately defers to. -3. **[Database Schema](https://compositecode.github.io/substratalapps.com/domain-model/database-schema/)** — Postgres types, constraints, indexes, Row-Level Security. This is the literal starting migration. -4. **[DEPLOYMENT.md](DEPLOYMENT.md)** — where it all runs, and the infrastructure build checklist that interleaves with the phases below. - -## Build order - -The short version: **infrastructure skeleton, then the access-control spine, then everything else radiates outward from Entitlement.** Every phase below is buildable and demoable on its own — this isn't "build the whole schema, then wire up endpoints," it's vertical slices that happen to accumulate into the full spec. - -## Phase 0 — Infrastructure skeleton - -Before any endpoint handler exists. Maps directly to [DEPLOYMENT.md → Build checklist](DEPLOYMENT.md#build-checklist) steps 1–4: - -- [ ] AWS account, `us-east-1`, under Organizations — [DEPLOYMENT.md → AWS account & region](DEPLOYMENT.md#aws-account--region). -- [ ] **AWS Budgets + Cost Anomaly Detection first**, before any billable resource — [DEPLOYMENT.md → Cost guardrails](DEPLOYMENT.md#cost-guardrails). Not optional, not step 2. -- [ ] Aurora Serverless v2 cluster, Data API enabled, Secrets Manager credential. -- [x] The local-dev repository/data-access interface described in [DEPLOYMENT.md → Local development](DEPLOYMENT.md#local-development), *before* the first table is written against it — retrofitting this seam after business logic exists is real rework, building it first is not. -- [x] CI running the [Testing strategy](https://compositecode.github.io/substratalapps.com/non-functional-requirements/#testing-strategy) categories (contract tests against `openapi.yaml`, unit tests on `effective_permissions`, integration tests against local Postgres) against an empty schema — so the pipeline exists before there's much to test, not retrofitted. - -**Exit criteria:** `GET /v1/health` deployed and returning `200` through the real Lambda → API Gateway → (nothing else yet) path. Proves the skeleton, not the product. - -## Phase 1 — MVP - -Matches the [Roadmap → MVP](https://compositecode.github.io/substratalapps.com/roadmap/#mvp) feature scope, sequenced for implementation: - -1. **`users`, `user_identities`, `sessions`, `refresh_tokens`, `auth_tokens`, `profiles`, `settings` tables + native [Auth](https://compositecode.github.io/substratalapps.com/api-reference/auth/) (signup, login + MFA challenge, refresh with reuse detection, logout, TOTP, password reset/change, email verification, invitations, JWKS via KMS, SES email, and **`POST /v1/auth/app-tokens`**, without which no Application can log anyone in) and [Users](https://compositecode.github.io/substratalapps.com/api-reference/users/)/[Profiles](https://compositecode.github.io/substratalapps.com/api-reference/profiles/)/[Settings](https://compositecode.github.io/substratalapps.com/api-reference/settings/) (global only) endpoints.** `POST /v1/users` first specifically — per [Non-Functional Requirements → Transaction boundaries](https://compositecode.github.io/substratalapps.com/non-functional-requirements/#transaction-boundaries), its "also creates a Profile, Settings, and default Role" behavior is one transaction; build it as one from the start rather than three calls glued together later. Build `user_identities` as the multi-method table it's specified as from day one (see [Decisions #1](https://compositecode.github.io/substratalapps.com/api-reference/auth/#native-auth-and-per-organization-sso)) even though only `method: 'password'` rows exist until Phase 2+ — `sso_connections` and the broker integration are explicitly deferred, not this phase's problem. -2. **Platform Roles & the [permission catalog](https://compositecode.github.io/substratalapps.com/domain-model/roles-and-permissions/#platform-permission-catalog)** — seed the `superadmin`/`support`/`billing_admin`/`member` rows. This is also literally how the first real `superadmin` gets into the system — see [Non-Functional Requirements → Bootstrapping](https://compositecode.github.io/substratalapps.com/non-functional-requirements/#authentication): seeded directly, not through the API. -3. **`applications` table + [Applications](https://compositecode.github.io/substratalapps.com/api-reference/applications/) (list/fetch; writes admin-only).** -4. **`entitlements` table + [Entitlements](https://compositecode.github.io/substratalapps.com/api-reference/entitlements/), especially the `PATCH` toggle.** This is the feature the rest of the MVP exists to support — see [Access Control](https://compositecode.github.io/substratalapps.com/access-control/) for why Entitlement and Role are evaluated as independent axes, which should shape the authorization-check code from the first line, not get refactored in later. -5. **`audit_events` table, written transactionally alongside every Entitlement/Role write from step 2 onward** — not bolted on after. See [Non-Functional Requirements → Transaction boundaries](https://compositecode.github.io/substratalapps.com/non-functional-requirements/#transaction-boundaries): an Audit Event that doesn't correspond to a committed change is worse than no Audit Event, which is only true if it's written in the same transaction from day one. -6. **`api_keys` table + [API Keys](https://compositecode.github.io/substratalapps.com/api-reference/api-keys/), including app-confined permissions.** An Application's backend needs an app-scoped key with confined `entitlements.manage` to reflect its own billing outcomes. That's the developer's one required integration besides login. - -**Exit criteria:** the [Quickstart](https://compositecode.github.io/substratalapps.com/quickstart/) walkthrough runs end to end against the real deployment — create a user, grant an app, check access, turn it off. This is the MVP's own acceptance test, already written. - -## Phase 2 — Per-app customization & trust - -Matches [Roadmap → Phase 2](https://compositecode.github.io/substratalapps.com/roadmap/#phase-2): - -1. **App-scoped Roles** — extend the Phase 1 Role/permission machinery with `scope: ` and `available_app_roles` on Application. Not a new system, the same tables with a column that was already reserved for this. -2. **`app_profiles`/`app_settings` tables**, with the [typed generated-column mechanism](https://compositecode.github.io/substratalapps.com/domain-model/database-schema/#typed-fields-per-application-views-and-expression-indexes) for declared `settings_schema` fields. -3. **[Webhooks](https://compositecode.github.io/substratalapps.com/api-reference/webhooks/)** — delivery queue, signing, the retry/backoff schedule. This is also where the [Trust Model's](https://compositecode.github.io/substratalapps.com/trust-model/#how-fast-does-revocation-need-to-land) **required** immediate-revocation integration becomes real: an Application can't actually meet that requirement until this ships, so treat Phase 2 webhooks as a hard prerequisite for telling any real Application developer their integration is compliant, not a nice-to-have. -4. **Live introspection** (`effective-permissions`) — the other half of the [Trust Model](https://compositecode.github.io/substratalapps.com/trust-model/), alongside the app token from Phase 1's Auth work. -5. **[Organizations](https://compositecode.github.io/substratalapps.com/api-reference/organizations/)** — `organizations`, `organization_memberships`, org-wide Entitlements with `member_scope`/`member_overrides`. Pulled forward from a hypothetical later phase per [Domain Model → Organization](https://compositecode.github.io/substratalapps.com/domain-model/users-and-organizations/#organization) — both individual and team end users are expected from early on. -6. **[Tenancy](https://compositecode.github.io/substratalapps.com/domain-model/tenancy/)** — `tenants` table, the `shared` tier (free, automatic, every Application owner gets one). `isolated`/`dedicated_region` provisioning mechanics ([DEPLOYMENT.md → Tenancy tiers](DEPLOYMENT.md#tenancy-tiers--where-they-run)) can lag slightly behind the schema, since they're support-run and gated by an actual customer need, not a launch-day requirement. -7. **Stripe Billing** — [DEPLOYMENT.md → Stripe Billing](DEPLOYMENT.md#stripe-billing) and [API Reference → Billing](https://compositecode.github.io/substratalapps.com/api-reference/billing/) in full: the `subscriptions` table, `applications.access_tier`, the platform-managed `free`/`subscription` Entitlements and their reconcile, the catalog sync script, the User and Organization Checkout/Portal endpoints, the webhook handler, and the daily Organization quantity sync, in Stripe test mode first. This is what makes [Pricing](https://compositecode.github.io/substratalapps.com/pricing/) real rather than aspirational — build it alongside Organizations, since Organization subscriptions reuse their member scoping. -8. **The [MCP server](https://compositecode.github.io/substratalapps.com/mcp-server/)** — generated from `openapi.yaml`'s `operationId`s, per [DEPLOYMENT.md → MCP server](DEPLOYMENT.md#mcp-server). Natural to build once the REST surface it wraps is otherwise stable, late in this phase rather than early. -9. **[Push notifications](https://compositecode.github.io/substratalapps.com/api-reference/notifications/)** (specified 2026-10-17, not built). Every route exists and returns `501 not_implemented` (`internal/core/notifications.go`). Build, in order: - - The `push_credentials`, `devices`, `notifications`, and `notification_deliveries` tables ([Database Schema](https://compositecode.github.io/substratalapps.com/domain-model/database-schema/#push_credentials-devices-notifications-notification_deliveries)), as one expand-only migration, plus `applications.receives_platform_notifications`. - - `notifications.send` in the permission catalog, app-confined. - - The credential and device endpoints, including deactivation on session revocation, suspension, and deletion. - - Sending, with deliveries written in the same transaction as the Notification, and the inbox endpoints. - - The delivery worker: APNs (HTTP/2, token auth), FCM HTTP v1, and Web Push (VAPID and RFC 8291 encryption). Retries, token invalidation, and credential failure handling. See [DEPLOYMENT.md → Push delivery](DEPLOYMENT.md#push-delivery). - - The platform alerts, the export and erasure-cascade additions, the stale-device and retention jobs, and the webhook events. - - Then remove the `501` stubs. The contract tests already cover every response shape. - -**Exit criteria:** a real third-party-shaped integration test, an app that isn't Substratal's own, can sign a user in through the embedded flow, receive `access.revoked` and kill the session, call introspection, have its own `settings_schema` validated and typed on write, and register a test device and receive a push sent through its app key. A test-mode User and a test-mode Organization can subscribe to Standard through Checkout, gain the standard Applications, and lapse back to Free. The hosted PKCE flow is Phase 3, with the hosted sign-in page. - -## Phase 3 — Marketplace & scale features - -Matches [Roadmap → Phase 3](https://compositecode.github.io/substratalapps.com/roadmap/#phase-3) — explicitly **not** built speculatively ahead of the triggers named in [DEPLOYMENT.md → Scale-out](DEPLOYMENT.md#scale-out) and [Applications → The review lifecycle](https://compositecode.github.io/substratalapps.com/domain-model/applications/#the-review-lifecycle): - -- **SSO broker integration** (`sso_connections` becoming real, an actual WorkOS-or-equivalent wiring behind `POST /v1/auth/sso/{provider}/callback`) — the schema and the route exist from Phase 1 per [Auth → Native auth and per-Organization SSO](https://compositecode.github.io/substratalapps.com/api-reference/auth/#native-auth-and-per-organization-sso), deliberately built in advance precisely so this phase is additive wiring, not a migration. Triggered by a real Organization actually asking for it, not a phase-number default. -- Self-service Application registration + review queue. The *process* is already fully specified — reviewer model (no dedicated assignment, any `applications.manage` holder works the FIFO queue), a 5-business-day policy SLA, the `rejected` status with required `review_notes` and edit-to-resubmit, non-cascading suspension — see [Applications → The review lifecycle](https://compositecode.github.io/substratalapps.com/domain-model/applications/#the-review-lifecycle) and [API Reference → Applications → Reviewing a submission](https://compositecode.github.io/substratalapps.com/api-reference/applications/#reviewing-a-submission). What's actually deferred to this phase is just the self-service *submission* endpoint itself (`POST /v1/applications` opening up beyond platform-admin-only) — the review mechanics it feeds into are build-ready today. -- SCIM-style provisioning. -- Richer audit/compliance views (filtering, export) — driven by actual [Compliance](https://compositecode.github.io/substratalapps.com/compliance/) needs (a SOC 2 audit, a customer's data-export request), not built ahead of either. - -## Cross-cutting, build throughout - -Not a phase — these apply from Phase 0 onward and should never be "added later": - -- **[Access Control](https://compositecode.github.io/substratalapps.com/access-control/)'s permission checks**, on every endpoint, from the first one. Retrofitting authorization onto endpoints built "open" is how it gets missed somewhere. -- **[Conventions](https://compositecode.github.io/substratalapps.com/api-reference/conventions/)'s error shape, pagination, idempotency, and `me` self-addressing** — pick these once, in Phase 1, as shared middleware/helpers. Every later endpoint should get them for free, not reimplement them. -- **Contract tests against `openapi.yaml`** — every endpoint, the day it's built, not batched at the end of a phase. -- **The spec pages and the [Changelog](https://compositecode.github.io/substratalapps.com/changelog/)** stay live documents during implementation, the same way they were during spec-writing. An implementation detail that contradicts or settles something in the spec gets written into the page it governs, with a Changelog entry, immediately, not in an end-of-project cleanup pass. - -## Status - -**Build status (2026-10-06):** the code for every phase's API surface is implemented, except push notifications (added to the spec on 2026-10-17, Phase 2 item 9, whose routes return `501` until built), including Phase 3's hosted PKCE endpoints (SSO still returns `501` until the broker integration, as specified), and the Quickstart acceptance test passes against the devserver. The Phase 0 *infrastructure* items above are written as Terraform (`infra/terraform/`) but not yet applied; see [README → One-time setup](README.md#one-time-setup-phase-0). The choices the implementation made where the spec was silent or self-contradictory are written into the spec pages they govern. - -**No product decisions are open.** Each one is written into the spec page it governs, and recorded in the [Changelog](https://compositecode.github.io/substratalapps.com/changelog/). The separate decision log was retired on 2026-10-18. - -**Push notifications were added on 2026-10-17** (Phase 2 item 9 above), with their defaults settled in the spec. - -**Pricing changed on 2026-10-17**: two tiers, Free and Standard ($6 per user per month), paid by the people using Substratal's apps, individually or through an Organization. **Subscriber billing was built on 2026-10-18**: Phase 2 item 7 now implements it, replacing the earlier developer-billed Starter/Team/Enterprise code. Billing is monthly in USD, with no annual plan and no Standard trial ([Pricing → Stripe catalog](https://compositecode.github.io/substratalapps.com/pricing/#stripe-catalog)). Either can be added later as Stripe configuration. diff --git a/README.md b/README.md index de22931..d17327f 100644 --- a/README.md +++ b/README.md @@ -11,14 +11,16 @@ This repository is split deliberately, by audience: | Audience | Lives | Why | |---|---|---| | **An API consumer** — a developer building an Application on this platform, or their AI coding agent | [`docs/`](docs/), published at [compositecode.github.io/substratalapps.com](https://compositecode.github.io/substratalapps.com/) | The API specification, and nothing else: domain model, access control, every endpoint and schema, workflows, the MCP server for agent callers, pricing. Everything a consumer needs to build *against* this API. | -| **Whoever is building *this* API** | The project root (this file and its siblings) | Deployment/infrastructure, the implementation build order, and the internal decision log aren't things a consumer needs to use the API — they're how it gets built and run. Kept separate so the docs site stays focused, per its own [Home page](https://compositecode.github.io/substratalapps.com/#what-this-site-is). | +| **Whoever is building *this* API** | The project root (this file and its siblings) | Deployment/infrastructure, the remaining build plan, and the manual setup steps aren't things a consumer needs to use the API — they're how it gets built and run. Kept separate so the docs site stays focused, per its own [Home page](https://compositecode.github.io/substratalapps.com/#what-this-site-is). | Root-level documents: -- **[PLAN.md](PLAN.md)** — the build order: what to implement first, which spec page governs each piece, phase-by-phase exit criteria. -- **[DEPLOYMENT.md](DEPLOYMENT.md)** — AWS infrastructure (cost-capped, Aurora Serverless v2, Lambda, no VPC/NAT at Tier 0), Stripe Billing implementation, the MCP servers recommended for building against AWS/Postgres/Stripe, and the scale-out triggers for when any of this needs to change. +- **[plan-remaining.md](plan-remaining.md)** — everything specified but not built yet: test-mode User expiry, the first-party web pages, Phase 3, dedicated tenancy infrastructure, scale-out triggers, and the rules every change follows. +- **[remaining_manual_steps.md](remaining_manual_steps.md)** — one-time setup still to do by hand to go live: the Lambda quota, the superadmin password, SES production access, Stripe, and legal. +- **[infra/README.md](infra/README.md)** — the deployment architecture as built: AWS infrastructure (cost-capped, Aurora Serverless v2, Lambda, no VPC/NAT at Tier 0), the account setup, Stripe Billing, email, webhook and push delivery, audit archival, and local development. - **[LICENSE](LICENSE)** — MIT. -- **[ORIGINAL-SPEC-DRAFT.md](ORIGINAL-SPEC-DRAFT.md)** — the original single-document draft this whole spec was elaborated from. Kept for history, moved here (out of `docs/`) specifically so it can't be mistaken for current spec — it contradicts several later decisions. + +The original single-document draft, the build plan, and the old deployment document were retired on 2026-10-20 once their content was built and documented where it belongs; they remain in git history. Not yet moved to the root, but should be, by the same reasoning above — tracked so this doesn't get silently forgotten: @@ -37,18 +39,18 @@ Decided 2026-10-06. The API is written in **Go**, deployed to **AWS Lambda**, an | Layer | Choice | Why | |---|---|---| -| Language | **Go** (latest stable release, pinned in `go.mod`) | Cold starts in the tens of milliseconds and small memory use on Lambda. That directly serves the p95 < 150 ms target for `effective-permissions`, `app-tokens`, and `oauth/token` in [Non-Functional Requirements](https://compositecode.github.io/substratalapps.com/non-functional-requirements/), and the cost-first rules in [DEPLOYMENT.md](DEPLOYMENT.md#cost-principles). Strongly typed, stable for years, and plain enough that a one-person team and an AI assistant can both read and change it safely. | +| Language | **Go** (latest stable release, pinned in `go.mod`) | Cold starts in the tens of milliseconds and small memory use on Lambda. That directly serves the p95 < 150 ms target for `effective-permissions`, `app-tokens`, and `oauth/token` in [Non-Functional Requirements](https://compositecode.github.io/substratalapps.com/non-functional-requirements/), and the cost-first rules in [infra/README.md](infra/README.md#cost-principles). Strongly typed, stable for years, and plain enough that a one-person team and an AI assistant can both read and change it safely. | | Lambda runtime | `provided.al2023` on **arm64** (Graviton) | Go compiles to one static `bootstrap` binary. arm64 is cheaper per GB-second than x86 for the same work. | | HTTP layer | Standard library `net/http`. The route table, path/query/header binding, and the `ServerInterface` every operation must implement are **generated from [`docs/openapi.yaml`](docs/openapi.yaml)** by `oapi-codegen`. Request and response bodies are hand-written in `internal/core`. | The spec is the contract: an operation without a handler doesn't compile. The generated *body* models can't express explicit JSON `null`s or PATCH field presence, both of which Conventions requires, so bodies are hand-written and the contract tests validate every response against `openapi.yaml` instead. | | Lambda adapter | `aws-lambda-go`, plus a thin adapter that turns API Gateway HTTP API events into `http.Request`s | The **same** `http.Handler` serves requests on a laptop and in Lambda. Nothing in handler code knows which one it's running in. | -| Database | **PostgreSQL 16**. In AWS: Aurora Serverless v2 through the **RDS Data API**. Locally and in CI: plain Postgres through **`pgx`**. | Both implement one small driver interface (`internal/db`), as [DEPLOYMENT.md → Local development](DEPLOYMENT.md#local-development) requires. See [The two database backends](#the-two-database-backends). | +| Database | **PostgreSQL 16**. In AWS: Aurora Serverless v2 through the **RDS Data API**. Locally and in CI: plain Postgres through **`pgx`**. | Both implement one small driver interface (`internal/db`), as [infra/README.md → Local development](infra/README.md#local-development) requires. See [The two database backends](#the-two-database-backends). | | Migrations | Plain, numbered SQL files in `migrations/`, applied by an in-repo `cmd/migrate` tool | Same tool, same files, same tracking table (`schema_migrations`) in every environment. An in-repo runner is used instead of a third-party tool because the common Go migration tools need a `database/sql` driver, and the Data API has no first-party one. | | Crypto | `golang.org/x/crypto/argon2` (Argon2id), standard-library HMAC/RSA, AWS KMS for JWT signing and secret encryption | Matches [Non-Functional Requirements → Security](https://compositecode.github.io/substratalapps.com/non-functional-requirements/) exactly, with no native add-ons. | | AWS access | AWS SDK for Go v2 (Data API, KMS, SQS, Secrets Manager, S3, SES) | | | Billing | `stripe-go`, Stripe's official Go library | | | MCP server | Go, implementing the buffered-JSON subset of MCP Streamable HTTP directly (`internal/mcp`). Tools are generated from `openapi.yaml` by `cmd/mcpgen`. | The spec requires a stateless, signed `Mcp-Session-Id` and one invocation per call; the official SDK keeps sessions in memory, which doesn't fit a stateless Lambda. It only proxies HTTPS calls to `/v1`. | | Infrastructure | **Terraform**, remote state in S3 (native S3 state locking, no DynamoDB table) | Language-neutral and widely known, and `terraform plan` output can be reviewed on the pull request before it reaches production. Works with OpenTofu too. | -| CI/CD | **GitHub Actions**, deploying to AWS through **OIDC federation** | No long-lived AWS keys anywhere, per [DEPLOYMENT.md → Build checklist](DEPLOYMENT.md#build-checklist) step 9. | +| CI/CD | **GitHub Actions**, deploying to AWS through **OIDC federation** | No long-lived AWS keys anywhere, per [infra/README.md → Cost principles](infra/README.md#cost-principles). | | Local services | **Docker Compose**: Postgres 16, LocalStack (SQS, KMS, Secrets Manager, S3), Mailpit (captures outgoing email) | The whole API runs on a laptop with no AWS account and no internet beyond Stripe test mode. | Considered and not chosen: **TypeScript on Node.js** (fastest early development and the most mature MCP SDK, but slower cold starts, types that disappear at runtime, and more dependency churn over a long-lived project), and **Java** (mature, and SnapStart fixes cold starts, but heavier memory use and more framework machinery than Tier 0's small per-request Lambdas and a one-person team need). Revisit only if the deployment model changes, for example a move from Lambda to long-running containers. @@ -120,7 +122,7 @@ Every push to `dev` (and to `feature/*`), and every pull request into `main`, ru 3. **Unit tests**, including exhaustive tests of the Access Control resolution logic (every combination of entitlement status × platform role × app role), per [Testing strategy](https://compositecode.github.io/substratalapps.com/non-functional-requirements/#testing-strategy). 4. **Integration tests** against real Postgres 16 and LocalStack running as GitHub Actions service containers, with every migration applied from zero. These run the pgx backend, including the Row-Level Security policies, which a mocked data layer would let pass while broken. 5. **Contract tests.** The devserver starts, and every response is validated against `docs/openapi.yaml`. -6. **Quickstart test.** The [Quickstart](https://compositecode.github.io/substratalapps.com/quickstart/) walkthrough runs as a script against the devserver. It's the Phase 1 acceptance test from [PLAN.md](PLAN.md#phase-1--mvp), so it runs on every change, not just at milestones. +6. **Quickstart test.** The [Quickstart](https://compositecode.github.io/substratalapps.com/quickstart/) walkthrough runs as a script against the devserver. It's the Phase 1 acceptance test ([plan-remaining.md → Verify the build in production](plan-remaining.md#verify-the-build-in-production)), so it runs on every change, not just at milestones. 7. **Build.** Every `cmd/*` function compiles for `linux/arm64` (`CGO_ENABLED=0`, `-tags lambda.norpc`) and is zipped as `bootstrap`, which proves the release artifacts can be built. 8. **Docs build.** `jekyll build` of `docs/`, so a broken docs page is caught on `dev`, not on the release. @@ -128,7 +130,7 @@ On a pull request into `main` only, one more step runs: 9. **`terraform plan` against production**, using a **read-only** plan role through OIDC. The plan is posted as a PR comment. This is the one thing to read carefully before merging a release: it is exactly what the merge will change in AWS. -There is deliberately **no deployed dev environment** at Tier 0. A second copy of the stack would roughly double the Aurora floor cost (~$43/month), and the local stack covers what it would be used for. Adding one, ideally in a separate AWS account, is the "more than one engineer shipping concurrently" trigger in [DEPLOYMENT.md → Scale-out](DEPLOYMENT.md#scale-out). When that happens, a push to `dev` gains a deploy to that account. Nothing else about this flow changes. +There is deliberately **no deployed dev environment** at Tier 0. A second copy of the stack would roughly double the Aurora floor cost (~$43/month), and the local stack covers what it would be used for. Adding one, ideally in a separate AWS account, is the "more than one engineer shipping concurrently" trigger in [infra/README.md → Scale-out](plan-remaining.md#scale-out). When that happens, a push to `dev` gains a deploy to that account. Nothing else about this flow changes. ### What a merge into `main` does: `deploy.yml` @@ -185,9 +187,9 @@ For an urgent fix while `dev` holds unreleased work: ### One-time setup (Phase 0) -The steps above assume these exist. They're part of [PLAN.md → Phase 0](PLAN.md#phase-0--infrastructure-skeleton), done once, by hand: +The steps above assume these exist. They're done once, by hand; what's still left of them is tracked in [remaining_manual_steps.md](remaining_manual_steps.md): -1. A dedicated AWS account in `us-east-1` ([DEPLOYMENT.md → AWS account & region](DEPLOYMENT.md#aws-account--region)), with its default VPC in place (Aurora's subnets come from it; Lambda never joins a VPC), and a Route 53 hosted zone for `substratalapps.com`. +1. A dedicated AWS account in `us-east-1` ([infra/README.md → AWS account & region](infra/README.md#aws-account--region)), with its default VPC in place (Aurora's subnets come from it; Lambda never joins a VPC), and a Route 53 hosted zone for `substratalapps.com`. 2. From your own machine, with admin credentials: `cd infra/terraform/bootstrap && terraform init && terraform apply -var billing_email=you@example.com`. This creates the **cost guardrails first** (warn and hard budgets with a circuit-breaker action, Cost Anomaly Detection, a billing alarm), then the state and artifact buckets, the GitHub OIDC provider, and the read-only `plan` and `deploy` roles. It's the only Terraform ever applied from a laptop. 3. In GitHub: create the `production` environment (deployments from `main` only), add the `main` ruleset, and set repository **variables** from the bootstrap outputs: `AWS_PLAN_ROLE_ARN`, `AWS_DEPLOY_ROLE_ARN`, `AWS_REGION`, `TF_STATE_BUCKET`, `ARTIFACT_BUCKET`, plus `API_DOMAIN` (`api.substratalapps.com`). None are secrets; OIDC needs no stored credentials. 4. Merge into `main`. The first deploy creates the stack and then migrates the new cluster. @@ -232,6 +234,7 @@ Then `make quickstart` runs the [Quickstart](https://compositecode.github.io/sub | REST API, `/mcp`, `/internal/stripe/webhook` | `cmd/devserver`: every route on `:8080`, the same handlers the Lambdas use | One Lambda per function behind API Gateway | | Database | Postgres 16 container on host port `55432` through pgx, connecting as the non-superuser `substratal` role so Row-Level Security applies | Aurora Serverless v2 through the Data API | | Webhook delivery | In-process: the devserver nudges its own worker after each commit and sweeps every 2 seconds. Set `WEBHOOKS_ALLOW_PRIVATE=true` (the default in `.env.example`) to deliver to `http://localhost` receivers. | SQS nudge → worker Lambda, plus a 1-minute sweep | +| Push delivery | The same in-process worker, with `PUSH_BACKEND=log` (the default in `.env.example`): every push is logged, nothing is sent, and the notification still settles as sent. Set `PUSH_BACKEND=live` to send to APNs, FCM, and Web Push with an Application's real credentials. | The same queue, worker Lambda, and sweep as webhooks, with `PUSH_BACKEND=live` | | Scheduled jobs | The entitlement sweep runs in the devserver every 5 minutes; run any job on demand with `make job name=…` (`make jobs` lists them) | EventBridge Scheduler → `jobs` Lambda | | JWT signing, secret encryption | A local RSA-2048 key, generated into `.dev/signing-key.pem` on first run, and a fixed development AES key (set `KMS_*_KEY_ID` to use KMS instead, including LocalStack's) | AWS KMS | | Secrets | `.env` | Secrets Manager (`APP_SECRET_ARN`) | @@ -243,7 +246,7 @@ Configuration is entirely environment variables, documented in `.env.example`. ` ### The two database backends -[DEPLOYMENT.md → Local development](DEPLOYMENT.md#local-development) explains why the Data API can't run locally. Here's how the code handles it: +[infra/README.md → Local development](infra/README.md#local-development) explains why the Data API can't run locally. Here's how the code handles it: - Every database call goes through one small interface in `internal/db` (`Tx`, `Query`, `Exec`). Handlers never import `pgx` or the Data API client. - SQL is written once, with named parameters (`:user_id`). The pgx driver rewrites them to `$1` positional form; the Data API driver passes them as `SqlParameter`s. Parameters are only ever scalars; arrays and JSON travel as text with explicit casts (`::text[]`, `::jsonb`), because the Data API has no array parameters. diff --git a/cmd/devserver/main.go b/cmd/devserver/main.go index 622680f..2dc1f3c 100644 --- a/cmd/devserver/main.go +++ b/cmd/devserver/main.go @@ -11,10 +11,12 @@ import ( "net/http" "os" "os/signal" + "sync/atomic" "syscall" "time" "github.com/CompositeCode/substratalapps.com/internal/billing" + "github.com/CompositeCode/substratalapps.com/internal/core" "github.com/CompositeCode/substratalapps.com/internal/jobs" "github.com/CompositeCode/substratalapps.com/internal/mcp" "github.com/CompositeCode/substratalapps.com/internal/platform/wire" @@ -28,9 +30,21 @@ func main() { if err != nil { log.Fatal(err) } - notifier := webhooks.NewLocal(ctx, base.Deliverer(), 2*time.Second) - api := base.Server(notifier) - apiHandler := api.Handler() + // One in-process worker for push and webhook delivery; pushes first, + // since their outcomes can emit webhook events. + var api atomic.Pointer[core.Server] + deliverer := base.Deliverer() + notifier := webhooks.NewLocalRunner(ctx, base.Log, 2*time.Second, func(ctx context.Context) error { + if s := api.Load(); s != nil { + if err := s.RunPush(ctx); err != nil { + return err + } + } + return deliverer.Run(ctx) + }) + server := base.Server(notifier) + api.Store(server) + apiHandler := server.Handler() mux := http.NewServeMux() mux.Handle("/", apiHandler) @@ -39,7 +53,7 @@ func main() { if base.Config.StripeSecretKey != "" && base.Config.StripeWebhookSecret != "" { stripe := billing.New(base.Config.StripeSecretKey, base.Config.StripePrices, base.Log) mux.Handle("/internal/stripe/webhook", &billing.Webhook{Secret: base.Config.StripeWebhookSecret, - Stripe: stripe, Core: api, Log: base.Log}) + Stripe: stripe, Core: server, Log: base.Log}) } // The 5-minute entitlement sweep, in-process. diff --git a/cmd/mcp/main.go b/cmd/mcp/main.go index 1ec15c2..946d5da 100644 --- a/cmd/mcp/main.go +++ b/cmd/mcp/main.go @@ -1,7 +1,7 @@ // Command mcp is the MCP server Lambda at /mcp. It holds no data-plane // permissions at all: every tool call is an HTTPS call to the API's own // /v1 routes on the same API Gateway, with the caller's credential -// forwarded unchanged (DEPLOYMENT.md → MCP server). +// forwarded unchanged (infra/README.md → MCP server). package main import ( diff --git a/cmd/stripe-webhook/main.go b/cmd/stripe-webhook/main.go index e847b80..ab3f131 100644 --- a/cmd/stripe-webhook/main.go +++ b/cmd/stripe-webhook/main.go @@ -1,6 +1,6 @@ // Command stripe-webhook is the Lambda at /internal/stripe/webhook: the one // route whose caller isn't a Bearer holder at all, verified instead by -// Stripe's own signature (DEPLOYMENT.md → Stripe Billing). +// Stripe's own signature (infra/README.md → Stripe Billing). package main import ( diff --git a/cmd/webhook-worker/main.go b/cmd/webhook-worker/main.go index a250971..ef5a993 100644 --- a/cmd/webhook-worker/main.go +++ b/cmd/webhook-worker/main.go @@ -1,6 +1,8 @@ // Command webhook-worker is the SQS-triggered Lambda that delivers -// outbound webhooks. A message only nudges it; it always drains the -// outbox and every due retry, so a lost or duplicate message is harmless. +// outbound webhooks and push notifications. A message only nudges it; it +// always drains both outboxes and every due retry, so a lost or duplicate +// message is harmless. Pushes go first, because their outcomes can emit +// webhook events (device.deactivated, notification.push_failed). package main import ( @@ -20,7 +22,11 @@ func main() { log.Fatal(err) } d := base.Deliverer() + api := base.Server(nil) lambda.Start(func(ctx context.Context, ev events.SQSEvent) error { + if err := api.RunPush(ctx); err != nil { + return err + } return d.Run(ctx) }) } diff --git a/docs/api-reference/api-keys.md b/docs/api-reference/api-keys.md index 79cac47..647e2ea 100644 --- a/docs/api-reference/api-keys.md +++ b/docs/api-reference/api-keys.md @@ -148,7 +148,7 @@ The result: an agent integration is safer by default without being less capable ## Test vs. live -Uniqueness constraints include `test_mode` (e.g. `unique (email, test_mode) where deleted_at is null`), so a test signup can't collide with a live one. A test-mode credential sees only test-mode rows and a live credential only live rows, enforced by the same Row-Level Security mechanism as `tenant_id`. Test-mode rows older than 30 days are purged nightly. +Uniqueness constraints include `test_mode` (e.g. `unique (email, test_mode) where deleted_at is null`), so a test signup can't collide with a live one. A test-mode credential sees only test-mode rows and a live credential only live rows, enforced by the same Row-Level Security mechanism as `tenant_id`. Test-mode rows older than 30 days are purged nightly. Test-mode **Users** are also time-limited: each stops working at its `expires_at`, 24 hours after creation unless the caller chooses otherwise (up to 30 days). See [Auth → Test-mode Users → Lifetime](../auth/#lifetime). `mode: "test"` produces a `satk_test_…` secret instead of `satk_live_…` — same permissions and scope, but every resource it creates (Users, Entitlements, anything) is tagged `test_mode: true`: @@ -161,7 +161,7 @@ This is how an Application's developer integration-tests against the real API wi ### Walkthrough: a test run end to end 1. **Create a test key** (as the Application's owner): `POST /v1/api-keys` with `"scope": "app_timetrack"`, `"mode": "test"`, `"permissions": ["entitlements.manage", "users.list"]`. The response's `secret` starts `satk_test_`. Applications are shared catalog rows, visible to test and live credentials alike (an Application's own `test_mode` flag only marks one created for testing, which is what allows `localhost` redirect URIs). Everything the test key *writes* is tagged `test_mode: true`. -2. **Create a test User** with that key: `POST /v1/users` `{"email": "qa+1@example.com", "status": "active", "send_invitation": false}`. The response has `"test_mode": true`. A live User with the same email can exist alongside it without conflict. +2. **Create a test User** with that key: `POST /v1/users` `{"email": "qa+1@example.com", "status": "active", "send_invitation": false}`. The response has `"test_mode": true` and an `expires_at` 24 hours away; add `"expires_at"` to the request for a shorter or longer run. A live User with the same email can exist alongside it without conflict. 3. **Grant access**: `POST /v1/users/{id}/entitlements` with an `Idempotency-Key` and `{"application_id": "app_timetrack", "source": "trial", "ends_at": "…"}`. The Entitlement is `test_mode: true`, and the `entitlement.granted`/`access.granted` webhooks go only to subscriptions created with a test key. 4. **Sign in as that User**: set a password through `POST /v1/auth/password/forgot` `{"email": "qa+1@example.com", "test_mode": true}`, then `POST /v1/auth/login` with `"test_mode": true` (see [Auth → Test-mode Users](../auth/#test-mode-users)). The access token and every app token carry `"test_mode": true`. 5. **Check isolation**: `GET /v1/users?email=qa+1@example.com` with a *live* key returns an empty list. The test User doesn't exist as far as live credentials are concerned. diff --git a/docs/api-reference/auth.md b/docs/api-reference/auth.md index 2d331bc..39544a2 100644 --- a/docs/api-reference/auth.md +++ b/docs/api-reference/auth.md @@ -136,7 +136,7 @@ The platform access token's claims: - Sends a verification email (token TTL 24 hours). An unverified User **can** log in; `email_verified` is carried in every token so an Application can decide for itself whether to gate anything on it. - `application_id` is optional and **grants nothing** — it records which Application the signup came from (`signup_application_id` on the User, used to brand the verification email with that app's `email_from_name`) and is otherwise informational. Access to an Application still requires an [Entitlement](../entitlements/), which the developer's own backend grants (see [Workflows → Purchase → access](../../workflows/#purchase--access)). - If the email belongs to an `invited` User, signup is **rejected** with `409 email_taken` (`details.reason: "invitation_pending"`), and a fresh invitation email is sent to that address (this counts against the same 3-per-email-per-hour limit as `email/verify/resend`). An invitation can only be completed with its `inv_` token, through [`invitations/accept`](#post-v1authinvitationsaccept), because whatever was already attached to the invited account (Entitlements, Roles, Organization memberships) must only go to someone who controls the mailbox. Completing it by signup would hand all of that to anyone who knew the address. -- `test_mode` (optional, default `false`): `true` creates a test-mode User instead. See [Test-mode Users](#test-mode-users). +- `test_mode` (optional, default `false`): `true` creates a test-mode User instead. A test-mode signup may also send `expires_at` to choose how long the account lasts (default 24 hours). See [Test-mode Users](#test-mode-users). - **Password policy:** 12–128 characters; rejected if it appears in the bundled breached-password list (top 100,000) or equals the email address. No composition rules. Stored as Argon2id (`m=19456 KiB, t=2, p=1`). | Error | Status | When | @@ -172,6 +172,7 @@ Clients must branch on `mfa_required` before reading `access_token`. Add `"test_ |---|---|---| | `invalid_credentials` | 401 | Unknown email, wrong password, an `invited` User with no password yet, or an SSO-only User (no `password` identity). Deliberately one code for all four — no account enumeration. | | `account_suspended` | 403 | Correct credentials, but `status: suspended`. | +| `test_user_expired` | 403 | Correct credentials, but the test-mode User's `expires_at` has passed. See [Test-mode Users → Lifetime](#lifetime). | | `too_many_attempts` | 429 | 5 failed attempts against this account's `password` identity in 15 minutes locks that identity for 15 minutes; `Retry-After` is set. Successful login resets the counter. (The counter lives on the [UserIdentity](../../domain-model/users-and-organizations/#useridentity), so a lockout never blocks the same User's SSO login.) | Updates `last_login_at` on the User and `last_used_at` on the matching UserIdentity, when the session is actually issued (for an MFA login, at `mfa/verify`). @@ -188,6 +189,18 @@ A User created by a test-mode API Key (or by signup with `"test_mode": true`) is Endpoints that take a token (`mfa/verify`, `token/refresh`, `password/reset`, `email/verify`, `invitations/accept`, `oauth/token`) don't need it, because the token already belongs to exactly one account. Every token issued to a test-mode User carries `"test_mode": true`, and the API confines that token to test-mode rows exactly as it does a `satk_test_` key. Applications themselves are shared catalog rows, visible in both modes, so a test-mode User gets app tokens for the same Applications a live one does, as long as they hold a (test-mode) Entitlement to it. +### Lifetime + +Every test-mode User is **time-limited**. Its `expires_at` is set when it's created, and at that moment the account stops working. Live Users never expire (`expires_at` is always `null`). + +- **Default: 24 hours** after creation. A caller creating a test-mode User, through [`POST /v1/users`](../users/#post-v1users) with a `satk_test_` key or through signup with `"test_mode": true`, may send `expires_at` to choose a different lifetime, to suit what's being tested: a short-lived account for one CI run, or one that lasts a week for a manual QA pass. +- **Range:** at least 15 minutes from now, and no later than **30 days after the User's `created_at`**, which is when its test data is purged. Anything outside that range is `422 validation_failed` (`details.reason: "expires_at_out_of_range"`). Sending `expires_at` for a live User is `422 validation_failed` (`details.reason: "live_user"`). +- **Changing it:** `users.manage` can move `expires_at` with [`PATCH /v1/users/{id}`](../users/#patch-v1usersid), within the same range, to extend a run, end it early, or bring back an expired User that hasn't been purged yet. +- **At expiry:** login returns `403 test_user_expired`, and every token issued to the User stops working, whatever its own `exp`. That covers access, refresh, and app tokens. No token is issued with an `exp` later than the User's `expires_at`, so an Application that verifies app tokens locally against JWKS stops accepting them at the same moment. Within 5 minutes, the expiry sweep revokes the User's sessions (`revoked_reason: test_user_expired`), deactivates their push devices, fires `access.revoked` (reason `test_user_expired`) for every Application they had active access to, and writes the `user.expired` Audit Event. Like suspension, expiry leaves `status` and the User's Entitlements, Roles, and memberships unchanged, so an extension restores exactly what was there and fires `access.granted` again. +- **Purge:** the nightly purge of test-mode rows older than 30 days is unchanged. The 30-day ceiling on `expires_at` means a test User has always expired before its data is purged. + +Test-mode **API Keys**, Organizations, and Applications don't expire. Only the accounts people or test suites sign in as do. + ## `POST /v1/auth/mfa/verify` ```json @@ -256,6 +269,7 @@ Both paths produce the identical app token. Either way, issuance **fails** if th | `application_not_available` | 409 | The Application's `review_status` is `suspended`/`rejected`/`pending_review` — see [Applications → review lifecycle](../../domain-model/applications/#the-review-lifecycle). A state conflict, not a permission failure, so it's the same `409` wherever this code appears. | | `tenant_suspended` | 403 | The Application's Tenant is `suspended` by the platform. See [Tenancy](../../domain-model/tenancy/#fields). | | `account_suspended` | 403 | The User is suspended. | +| `test_user_expired` | 403 | The test-mode User's `expires_at` has passed. See [Test-mode Users → Lifetime](#lifetime). | ### The app token @@ -574,6 +588,7 @@ GET https://api.substratalapps.com/.well-known/jwks.json |---|---|---| | `invalid_credentials` | 401 | Login or password change with a wrong password/unknown email. | | `account_suspended` | 403 | The User is suspended. | +| `test_user_expired` | 403 | A test-mode User whose `expires_at` has passed tried to sign in, refresh, or get an app token. | | `too_many_attempts` | 429 | Account lockout after repeated failures. | | `email_taken` | 409 | Signup with an email already in use. | | `weak_password` | 422 | Password fails the policy. | diff --git a/docs/api-reference/billing.md b/docs/api-reference/billing.md index 476119a..630ffa1 100644 --- a/docs/api-reference/billing.md +++ b/docs/api-reference/billing.md @@ -25,7 +25,7 @@ The Free/Standard subscription: a User subscribing to Standard for themselves, o - **Access follows the subscription.** While a subscription covers someone, the platform keeps a `source: subscription` Entitlement for them to each standard Application. When it stops covering them, those Entitlements end. Applications hear about every change through the usual [`access.granted` / `access.revoked`](../webhooks/#event-types) webhooks. - **Business and Enterprise** plans aren't offered yet. Checkout for either returns `422 plan_requires_sales`. -Stripe tells this API about every change through its webhook. This API never takes card data, and a subscription's `status` only changes in response to Stripe events. The handler is documented in [root `DEPLOYMENT.md` → Stripe Billing](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md#stripe-billing). +Stripe tells this API about every change through its webhook. This API never takes card data, and a subscription's `status` only changes in response to Stripe events. The handler is documented in [`infra/README.md` → Stripe Billing](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md#stripe-billing). ## Endpoints diff --git a/docs/api-reference/conventions.md b/docs/api-reference/conventions.md index a49af49..076433a 100644 --- a/docs/api-reference/conventions.md +++ b/docs/api-reference/conventions.md @@ -38,7 +38,7 @@ Authorization: Bearer User-facing requests carry a user access token (issued at login, see [Auth](../auth/)). Service-to-service requests (billing, an app's backend) carry a scoped [API Key](../api-keys/). Both go in the same header — the token type is distinguishable server-side by prefix, not by a different header name. -**Test vs. live:** every API Key is created with `satk_test_…` or `satk_live_…` (see [API Keys](../api-keys/)) — there is no separate sandbox deployment to point at. A `test` key operates against the same database, but every record it creates is tagged `test_mode: true`. Test and live rows are invisible to each other: a test credential reads and writes only test rows, a live credential only live rows, and no query parameter crosses that line. Test-mode events only reach test-mode webhook subscriptions, and test traffic is excluded from analytics counters. A User created in test mode signs in with `"test_mode": true` on the auth endpoints (see [Auth → Test-mode Users](../auth/#test-mode-users)), and every token issued to them carries `test_mode: true`, which confines it to test rows the same way. This is cheaper to build and run than a parallel environment, and it's the right call at the current scale (see [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md)) — a true isolated sandbox is a Scale-out-trigger-shaped decision, not a day-one one. +**Test vs. live:** every API Key is created with `satk_test_…` or `satk_live_…` (see [API Keys](../api-keys/)) — there is no separate sandbox deployment to point at. A `test` key operates against the same database, but every record it creates is tagged `test_mode: true`. Test and live rows are invisible to each other: a test credential reads and writes only test rows, a live credential only live rows, and no query parameter crosses that line. Test-mode events only reach test-mode webhook subscriptions, and test traffic is excluded from analytics counters. A User created in test mode signs in with `"test_mode": true` on the auth endpoints (see [Auth → Test-mode Users](../auth/#test-mode-users)), and every token issued to them carries `test_mode: true`, which confines it to test rows the same way. This is cheaper to build and run than a parallel environment, and it's the right call at the current scale (see [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md)) — a true isolated sandbox is a Scale-out-trigger-shaped decision, not a day-one one. ## IDs diff --git a/docs/api-reference/notifications.md b/docs/api-reference/notifications.md index 401fc9d..84370fb 100644 --- a/docs/api-reference/notifications.md +++ b/docs/api-reference/notifications.md @@ -17,7 +17,7 @@ How alerts and other updates reach a User: on their devices through Apple Push N --- {: .note } -**Implementation status.** Specified 2026-10-17 ([Changelog](../../changelog/)). Until the reference implementation builds it, every operation on this page returns `501 not_implemented`. Its defaults are settled: platform alerts push only through Applications that opt in, sending is one User per call (no broadcast yet), each recipient gets at most 60 notifications per Application per hour, the inbox keeps 90 days, the `notifications.push` channel is the only preference, and the push providers are sub-processors. +Specified 2026-10-17 and implemented 2026-10-18 ([Changelog](../../changelog/)). Its defaults are settled: platform alerts push only through Applications that opt in, sending is one User per call (no broadcast yet), each recipient gets at most 60 notifications per Application per hour, the inbox keeps 90 days, the `notifications.push` channel is the only preference, and the push providers are sub-processors. ## How it fits together @@ -213,7 +213,7 @@ A malformed `token` or `subscription` is `422 invalid_device_token`. **What makes a registration "the same" device** is the token: one Device per (`application_id`, `provider`, token) and mode, where a `web_push` device's token is its `endpoint`. Registering an existing token: -- **for the same User** refreshes it and returns `200`. `last_registered_at` moves forward, `name`, `app_version`, and `environment` take the new values, and `session_id` becomes the caller's current session. An `inactive` device becomes `active` again and fires `device.registered`. +- **for the same User** refreshes it and returns `200`. `last_registered_at` moves forward, `name`, `app_version`, and `environment` take the new values, and `session_id` becomes the caller's current session (`null` when an app key registers it). An `inactive` device becomes `active` again and fires `device.registered`. - **for a different User** (someone else signed in on that install) deletes the previous registration and creates a new Device for the caller (`201`). A device never carries one User's history over to another. The previous owner's Device is gone, and `device.deleted` fires for it. **The token itself is never returned.** It's stored KMS-encrypted, and responses carry only `token_hint`, its last four characters. Device tokens aren't bearer credentials, but they are personal data, and a client never needs to read one back. @@ -482,7 +482,7 @@ The platform raises a small, fixed set of notifications itself, about the User's | `category` | `priority` | Raised when | `data` | |---|---|---|---| -| `security.new_sign_in` | `high` | A login creates a session from a user agent not seen in the User's sessions during the last 90 days. | `session_id`, `user_agent`, `ip_address` | +| `security.new_sign_in` | `high` | A login creates a session from a user agent not seen in the User's sessions during the last 90 days. A User's first session, at signup, raises nothing. | `session_id`, `user_agent`, `ip_address` | | `security.password_changed` | `high` | `POST /v1/users/{id}/password`, or a password reset. | — | | `security.email_changed` | `high` | An email change completes. | — | | `security.mfa_disabled` | `high` | TOTP is turned off by the User, or reset by support. | — | @@ -548,7 +548,7 @@ A device a User registers with their own token is tied to the session that regis |---|---| | Logout, or [revoking that session](../auth/#delete-v1usersidsessionssessionid) | Devices registered under that session become `inactive` (`session_revoked`). Logout from every session deactivates all of them. | | Password reset, refresh-token reuse, or MFA reset (each revokes sessions) | The same, after the [`security.*` alert](#platform-alerts) is raised. | -| The User is suspended or soft-deleted | Every device they hold becomes `inactive` (`user_suspended` or `user_deleted`), including devices an app key registered. | +| The User is suspended or soft-deleted, or a test-mode User expires | Every device they hold becomes `inactive` (`user_suspended`, `user_deleted`, or `test_user_expired`), including devices an app key registered. | | The User signs in again on the same install | The client re-registers its token, which reactivates the device under the new session. | | An app-key registration (`session_id: null`) | Not affected by logout, because the platform can't see the app's own sessions. The Application's backend should `DELETE` the device when its user signs out. | | No re-registration for 270 days | `inactive` (`stale`). Providers expire tokens from installs that are never opened, so a client should re-register on launch, at most once a day. | @@ -593,4 +593,3 @@ Five events tell an Application's backend about its devices, credentials, and fa | `read_only_field` | 422 | A `PATCH` touched a read-only field, such as a device's `provider` or a credential's `application_id`. | | `invalid_status_transition` | 409 | A credential `PATCH status` other than `failed → active`. | | `rate_limited` | 429 | Over the per-sender or per-recipient limit. `details.scope` is `sender` or `recipient`. | -| `not_implemented` | 501 | Every operation on this page, until it's built. See the status note at the top. | diff --git a/docs/api-reference/tenancy.md b/docs/api-reference/tenancy.md index c619bf2..6a860dd 100644 --- a/docs/api-reference/tenancy.md +++ b/docs/api-reference/tenancy.md @@ -131,7 +131,7 @@ Creating a request doesn't start a migration. It's the record and the queue entr | `pending`/`scheduled → cancelled` | No Tenant change. `notes` is required. | | `in_progress → cancelled` | A rollback: the Tenant's `status` returns to `active` on its *original* tier/region. `notes` is required. | -Any other transition returns `409 invalid_request_transition`. Every change writes `tenant.tier_change_request_updated`. The physical migration steps are ops runbook work. See [root `DEPLOYMENT.md` → Migration mechanics](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md). This endpoint records and gates those steps; it doesn't perform them. +Any other transition returns `409 invalid_request_transition`. Every change writes `tenant.tier_change_request_updated`. The physical migration steps are ops runbook work. See [`infra/README.md` → Migration mechanics](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md). This endpoint records and gates those steps; it doesn't perform them. ## Errors specific to this resource diff --git a/docs/api-reference/users.md b/docs/api-reference/users.md index af0df69..eff7cf8 100644 --- a/docs/api-reference/users.md +++ b/docs/api-reference/users.md @@ -45,6 +45,7 @@ Password, MFA, and session endpoints under `/v1/users/{id}/…` are specified on "mfa_enabled": false, "signup_application_id": "app_timetrack", "test_mode": false, + "expires_at": null, "created_at": "2026-01-14T18:02:11Z", "updated_at": "2026-09-20T14:12:00Z", "last_login_at": "2026-10-02T09:41:03Z" @@ -57,6 +58,7 @@ Password, MFA, and session endpoints under `/v1/users/{id}/…` are specified on | `mfa_enabled` | `true` if any `password` identity has confirmed TOTP. Read-only here — see [Auth → MFA](../auth/#mfa-totp). | | `signup_application_id` | Which Application the User signed up through, if any. Read-only, informational; grants nothing. | | `test_mode` | `true` if created by a `satk_test_` key — see [Conventions → Authentication](../conventions/#authentication). | +| `expires_at` | When a test-mode User stops working; `null` for every live User. Defaults to 24 hours after creation, at most 30 days. See [Auth → Test-mode Users → Lifetime](../auth/#lifetime). | ## `GET /v1/users` @@ -113,6 +115,7 @@ GET /v1/users?status=active&email=jordan@example.com - `status` may be `invited` (default) or `active`. An `invited` user gets an invitation email (unless `send_invitation: false`) and activates via [`POST /v1/auth/invitations/accept`](../auth/#post-v1authinvitationsaccept). An `active` user created this way has no password; they sign in by using [`password/forgot`](../auth/#post-v1authpasswordforgot) (or SSO, once live). An admin can never set a user's password directly. - Creating a user also creates a default [Profile](../../domain-model/profiles/) (with `display_name` if given) and [Settings](../../domain-model/settings/), and assigns the `member` platform role — one transaction (see [Non-Functional Requirements → Transaction boundaries](../../non-functional-requirements/#transaction-boundaries)). +- With a `satk_test_` key, the User is test-mode and may take `expires_at` (default 24 hours from now, at least 15 minutes from now, at most 30 days); see [Auth → Test-mode Users → Lifetime](../auth/#lifetime). A live credential sending `expires_at` gets `422 validation_failed`. - Writes Audit Event `user.created`. ## `GET /v1/users/{id}` @@ -146,6 +149,8 @@ A User's Organization memberships aren't a field here — see `GET /v1/organizat Rejected transitions return `409 invalid_status_transition`. A self-service caller sending `status` gets `403 status_change_forbidden`. +**`users.manage`** may also set a test-mode User's `expires_at`, within the same range as at creation: no earlier than 15 minutes from now, and no later than 30 days after `created_at`. Moving it later on a User that has already expired brings them back (until the 30-day purge): login works again, and `access.granted` fires for every Application they still have active access to. Moving it earlier doesn't end the account at once; to cut a test run off immediately, suspend the User. Writes `user.updated`. A self-service caller sending `expires_at` gets `403 status_change_forbidden`. + ## `DELETE /v1/users/{id}` ```json diff --git a/docs/api-reference/webhooks.md b/docs/api-reference/webhooks.md index 843f46c..2537fd6 100644 --- a/docs/api-reference/webhooks.md +++ b/docs/api-reference/webhooks.md @@ -183,6 +183,7 @@ Why the derived, per-user `access.granted`/`access.revoked` events exist, and wh - `org_grant_changed` (an org grant's status or `member_scope` changed) - `org_member_added`, `org_member_removed` - `user_suspended`, `user_reactivated`, `user_deleted` +- `test_user_expired`, `test_user_extended` (a test-mode User's [`expires_at`](../auth/#lifetime) passed, or was moved later after it had) A change that doesn't flip someone's resolved access fires nothing for them. For example, an org grant is disabled, but the member also holds a personal Entitlement. @@ -199,6 +200,7 @@ A change that doesn't flip someone's resolved access fires nothing for them. For | `entitlement.deleted` | Hard-deleted as an error correction. | `entitlement` (last state) | | `role.assigned` / `role.removed` | A Role assignment is created or deleted. | `user_id`, `role_id`, `application_id` | | `user.suspended` / `user.reactivated` / `user.deleted` | User status changes. | `user_id` | +| `user.expired` | A test-mode User's [`expires_at`](../auth/#lifetime) passed. Only ever delivered to test-mode subscriptions. | `user_id`, `expires_at` | | `organization.member_added` / `organization.member_removed` | A membership becomes `active` (on acceptance, or a direct add), or an active membership is removed. Pending invitations don't fire either. | `organization_id`, `user_id`, `role` | | `application.review_status_changed` | The Application is approved, rejected, suspended, or reinstated. | `application_id`, `review_status`, `review_notes` | | `device.registered` / `device.deactivated` / `device.deleted` | A push [Device](../notifications/#devices) is registered or reactivated, becomes `inactive`, or is deleted. | `device` (never the token) | diff --git a/docs/changelog.md b/docs/changelog.md index 95fae92..0a5871a 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -12,6 +12,28 @@ What's changed in this specification over time. Every decision is written into t --- +## 2026-10-20 — Test-mode Users are time-limited + +Test mode already never counted toward billing (decision #36, confirmed on 2026-10-18). Now every **test-mode User expires**, so a test account can't stand in for a free production account. See [Auth → Test-mode Users → Lifetime](../api-reference/auth/#lifetime). + +- **Users gain `expires_at`.** It's set on every test-mode User and is `null` on every live one. The default is **24 hours** after creation. `POST /v1/users` with a `satk_test_` key, and signup with `"test_mode": true`, accept `expires_at` for a shorter or longer run: at least 15 minutes away, and at most 30 days after creation, the existing test-data purge horizon. `users.manage` can move it with `PATCH /v1/users/{id}`, which also brings back an expired User that hasn't been purged yet. +- **At expiry** sign-in, refresh, and app-token issuance return the new `403 test_user_expired`, and every token already issued stops working. No token's `exp` is ever later than the User's `expires_at`. The 5-minute entitlement-expiry sweep then revokes sessions and devices (reason `test_user_expired`), fires `access.revoked` (reason `test_user_expired`) and the new `user.expired` webhook, and writes the `user.expired` Audit Event. `status`, Entitlements, Roles, and memberships are unchanged. An expired test User isn't a covered member of a test-mode subscription. +- Test-mode API Keys, Organizations, and Applications don't expire. The nightly 30-day purge is unchanged. + +Not yet built: until it is, test-mode Users last until the 30-day purge, as before. `openapi.yaml` gains `expires_at` in the same change as the code. The build steps are in the repository's `plan-remaining.md` → Test-mode User expiry. + +## 2026-10-18 — Push notifications implemented + +[Push Notifications](../api-reference/notifications/) is built: all 17 operations work, and none returns `501` any more. Nothing about the API's shape changed. A few behaviors the spec left implicit are now stated: + +- A User's first session (at signup) never raises `security.new_sign_in`; there's nothing yet to compare it with. +- A device an app key registers or refreshes has `session_id: null`. +- An unknown `{provider}` in a push-credential path is `404 push_credential_not_found`. +- A push-credential `POST` validates its fields before checking whether the provider is already configured, so an invalid body is `422` even when a credential exists. +- [Database Schema](../domain-model/database-schema/#push_credentials-devices-notifications-notification_deliveries) records each notification's push counts on the notification itself, since deliveries are deleted after 30 days. + +Delivery shares the webhook queue, worker, and 1-minute sweep (see the repository's `infra/README.md` → Push delivery), and a daily `push-maintenance` job runs the device and inbox retention rules. Locally, `PUSH_BACKEND=log` logs pushes instead of sending them. + ## 2026-10-18 — Billing options settled; decision log retired - **Billing options** (decision #31, the last open question): Standard is billed **monthly in USD only**, with **no annual plan and no trial**. Free is the way to try Substratal's apps, and never needs a card. See [Pricing → Stripe catalog](../pricing/#stripe-catalog). Either option can be added later as Stripe configuration, without an API change. @@ -149,7 +171,7 @@ A full consistency/UX review surfaced drift between the prose docs, `openapi.yam - **Fixed a self-contradiction in [Pricing](../pricing/):** Starter's own "For" cell claimed "3+ Applications" while its own Applications row capped at 1. Reworded; "3+ Applications" is Team's case, not Starter's. - **Added [Pricing → Enforcement](../pricing/):** the Applications/custom-AppRole/webhook-subscription caps now have a real mechanism — checked server-side at creation time against the owning Tenant's plan, independent of the caller's own permission standing (the same "two independent gates" pattern `restrict_destructive` already uses), returning `409`/`code: "plan_limit_reached"`. New database constraint: `check (plan != 'starter' or owner_type = 'user')` on `tenants`. -- **Resolved the audit-retention contradiction** between [Pricing](../pricing/)'s plan-tiered "retention" numbers and [Non-Functional Requirements](../non-functional-requirements/)/[Compliance](../compliance/)'s "retained indefinitely": those numbers were never a deletion period, they're a **hot-storage window** — added [Non-Functional Requirements → Audit log lifecycle](../non-functional-requirements/#audit-log-lifecycle) specifying the archive-to-cold-storage mechanism (shape-only, `before`/`after` dropped at archive time, same redaction the hard-delete cascade already does) that makes "indefinite" literally true without every plan paying for the same hot-storage size. Root [`DEPLOYMENT.md` → Audit log archival](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md#audit-log-archival) specifies the AWS mechanics (EventBridge Scheduler → Lambda → S3 Glacier Deep Archive). +- **Resolved the audit-retention contradiction** between [Pricing](../pricing/)'s plan-tiered "retention" numbers and [Non-Functional Requirements](../non-functional-requirements/)/[Compliance](../compliance/)'s "retained indefinitely": those numbers were never a deletion period, they're a **hot-storage window** — added [Non-Functional Requirements → Audit log lifecycle](../non-functional-requirements/#audit-log-lifecycle) specifying the archive-to-cold-storage mechanism (shape-only, `before`/`after` dropped at archive time, same redaction the hard-delete cascade already does) that makes "indefinite" literally true without every plan paying for the same hot-storage size. Root [`DEPLOYMENT.md` → Audit log archival](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md#audit-log-archival) specifies the AWS mechanics (EventBridge Scheduler → Lambda → S3 Glacier Deep Archive). - **Fixed [Orders & Audit → Action catalog](../domain-model/orders-and-audit/#action-catalog):** added `tenant.tier_change_requested` and `tenant.tier_changed`, both already live in `openapi.yaml`'s `AuditAction` enum but missing from this page's own "authoritative source" table. - **Added the MFA enrollment endpoint to `openapi.yaml`** (`POST /v1/users/{id}/mfa/totp`) — fully specified in [Auth](../api-reference/auth/) since its own addition, never added to the machine-readable spec. - **Added `mode` (live/test) to `ApiKeyWrite`/`ApiKey` in `openapi.yaml`** — present in the database schema and every prose example since [API Keys](../api-reference/api-keys/) was written, absent from the YAML. @@ -163,8 +185,8 @@ A full consistency/UX review surfaced drift between the prose docs, `openapi.yam - **Elaborated [Conventions → Errors](../api-reference/conventions/#errors):** fixed the illustrative example (`entitlement_not_active`, which no page actually defines) to a real code (`entitlement_required`); added an explicit HTTP-status-to-error-class mapping table and the `409`-vs-`422` distinction, for implementation; clarified there's no single global error-code catalog — each resource's own table is authoritative, the same pattern [Orders & Audit](../domain-model/orders-and-audit/#action-catalog) already uses for `action` values. - **Fixed [MCP Server](../mcp-server/)'s hardcoded operation count** ("all 55 operations") — already stale (56+ since this was written), and guaranteed to drift again. Reworded to not state a number at all. - **Fixed [Domain Model](../domain-model/)'s "Ten entities" framing** — added a Supporting entities table listing `UserIdentity`, `SSOConnection`, `OrganizationMembership`, and `UserRoleAssignment`, which the original count silently omitted. -- **Moved the original single-document draft** from `docs/specs/Substratal-Hub-Access-API.md` to root [`ORIGINAL-SPEC-DRAFT.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/ORIGINAL-SPEC-DRAFT.md), with a superseded banner — it contradicts several current decisions structurally (no Tenant concept, single-method auth) and sitting inside `docs/` risked it being mistaken for current spec despite being excluded from the build. -- **Updated root [`README.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/README.md) and [`PLAN.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/PLAN.md)**, both of which had gone stale against decisions resolved in the prior round: README's Status section still named Decision #1 as the one open item (all 15 are resolved); PLAN's Phase 3 section still described the Application review process as unspecified (fully specified by Decision #9). +- **Moved the original single-document draft** from `docs/specs/Substratal-Hub-Access-API.md` to root [`ORIGINAL-SPEC-DRAFT.md`](https://github.com/CompositeCode/substratalapps.com/blob/e3826b8cbc306f2a0bfc428d8e3a61a8f757dd3d/ORIGINAL-SPEC-DRAFT.md), with a superseded banner — it contradicts several current decisions structurally (no Tenant concept, single-method auth) and sitting inside `docs/` risked it being mistaken for current spec despite being excluded from the build. +- **Updated root [`README.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/README.md) and [`PLAN.md`](https://github.com/CompositeCode/substratalapps.com/blob/e3826b8cbc306f2a0bfc428d8e3a61a8f757dd3d/PLAN.md)**, both of which had gone stale against decisions resolved in the prior round: README's Status section still named Decision #1 as the one open item (all 15 are resolved); PLAN's Phase 3 section still described the Application review process as unspecified (fully specified by Decision #9). ## 2026-10-04 @@ -176,7 +198,7 @@ A full consistency/UX review surfaced drift between the prose docs, `openapi.yam - Added the `me` self-addressing convention and a credential-vs-resource-ID distinction to [Conventions](../api-reference/conventions/). - Added [Quickstart](../quickstart/) (a runnable walkthrough) and this Changelog. - Fixed: the Audit endpoint's documented filter name (`user_id` → `target_user_id`) didn't match its own example. -- Added [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md): first-deployment and scale-out plans on AWS, replacing an earlier Vercel-based plan, designed around predictable, capped cost rather than deploy-and-see. +- Added [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md): first-deployment and scale-out plans on AWS, replacing an earlier Vercel-based plan, designed around predictable, capped cost rather than deploy-and-see. - **Reframed the product.** Substratal Apps is the user/organization/tenancy/settings/storage API layer an app developer builds on — not a storefront a customer browses. See [Home](../) and [What Substratal Apps actually is](../#what-substratal-apps-actually-is). Every page that described the platform as "the hub a customer lands on to reach apps they purchased" was describing the eventual onboarding UI, not the API's own purpose. - Added `owner_user_id`/`owner_organization_id` and `review_status` to [Application](../domain-model/applications/) — a real gap once outside developers can register their own apps: there was no concept of who owns an Application's catalog entry, only platform-admin control. [API Reference → Applications](../api-reference/applications/) now distinguishes an owner's self-service rights from platform moderation. - Added [Trust Model → Trust runs the other direction too](../trust-model/#trust-runs-the-other-direction-too): the existing API Key scoping and webhook signing already made third-party apps safe to onboard later — now stated explicitly instead of left implicit. @@ -185,13 +207,13 @@ A full consistency/UX review surfaced drift between the prose docs, `openapi.yam - Added [Database Schema](../domain-model/database-schema/): Postgres types, constraints, and indexes for every entity — the implementer-facing counterpart to the API-caller-facing field tables. - Added implementer-readiness gaps across [Non-Functional Requirements](../non-functional-requirements/) (concurrency control via `ETag`/`If-Match`, transaction boundaries, the concrete Postgres Row-Level Security multi-tenancy mechanism, enum forward-compatibility policy, request tracing, testing strategy) and [Conventions](../api-reference/conventions/) (test-vs-live API keys, multi-field validation error shape, default exclusion of soft-deleted/terminal records from lists, idempotency-key storage detail). - Added [Webhooks → Verifying the signature](../api-reference/webhooks/#verifying-the-signature): the concrete `Substratal-Signature: t=…,v1=…` header format and replay-window guidance — "HMAC-SHA256 over the raw body" wasn't enough to actually implement against. -- Added [Deployment Architecture → Local development](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) (the RDS Data API choice has no clean local emulator — addressed via a repository-pattern seam, not by changing the AWS choice) and [→ Growth trajectory](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) (the stated dozens → 10x → 100x plan mapped against Tier 0 and the Scale-out triggers). -- **Added [Tenant](../domain-model/tenancy/)**: the infrastructure-placement and data-isolation entity "tenancy" named in the product pitch but previously unspecified. Tied to the subscription — the owner of an Application's catalog entry (`owner_user_id`/`owner_organization_id`), not to Organization, which stays a pure grouping/domain object with no infrastructure meaning. Three tiers (`shared` default, `isolated`, `dedicated_region`), support-gated tier changes via `tenants.manage` and `POST /v1/tenants/{id}/tier-change-requests`, a scheduled-downtime migration process, and deliberately no reach into a downstream Application's own infrastructure. See [Decisions #11](../domain-model/tenancy/#tenant-vs-organization) and [#12](../domain-model/tenancy/#how-a-tier-change-happens-today), [API Reference → Tenancy](../api-reference/tenancy/), [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md), and the new `tenants`/`organization_memberships` tables in [Database Schema](../domain-model/database-schema/). Directly resolves [Compliance](../compliance/#gdpr-and-ccpa--build-for-it-now)'s previously-open data-residency gap. +- Added [Deployment Architecture → Local development](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md) (the RDS Data API choice has no clean local emulator — addressed via a repository-pattern seam, not by changing the AWS choice) and [→ Growth trajectory](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md) (the stated dozens → 10x → 100x plan mapped against Tier 0 and the Scale-out triggers). +- **Added [Tenant](../domain-model/tenancy/)**: the infrastructure-placement and data-isolation entity "tenancy" named in the product pitch but previously unspecified. Tied to the subscription — the owner of an Application's catalog entry (`owner_user_id`/`owner_organization_id`), not to Organization, which stays a pure grouping/domain object with no infrastructure meaning. Three tiers (`shared` default, `isolated`, `dedicated_region`), support-gated tier changes via `tenants.manage` and `POST /v1/tenants/{id}/tier-change-requests`, a scheduled-downtime migration process, and deliberately no reach into a downstream Application's own infrastructure. See [Decisions #11](../domain-model/tenancy/#tenant-vs-organization) and [#12](../domain-model/tenancy/#how-a-tier-change-happens-today), [API Reference → Tenancy](../api-reference/tenancy/), [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md), and the new `tenants`/`organization_memberships` tables in [Database Schema](../domain-model/database-schema/). Directly resolves [Compliance](../compliance/#gdpr-and-ccpa--build-for-it-now)'s previously-open data-residency gap. - **Replaced `User.organization_id` with [OrganizationMembership](../domain-model/users-and-organizations/#organizationmembership)**, a many-to-many join — a User can belong to more than one Organization at once (even across different Tenants), which a single column couldn't express. Carries `role: org_admin | member`, which also resolves [Organizations → Delegated admin](../api-reference/organizations/#delegated-admin)'s previously-unresolved question: an org admin is this field, not a third Role scope. - **Added `member_scope`/`member_overrides` to org-wide Entitlements** and resolved Decision #13: an Organization's own grant is authoritative over its own members (an `allowlist`/`denylist` an org admin controls), but never reaches into a member's separate, personally-sourced Entitlement to the same app. [Access Control → Organization vs. User precedence](../access-control/#organization-vs-user-precedence) has the formal resolution rule and a worked multi-org example; [Entitlements → Attribution](../domain-model/entitlements/#attribution) shows the `granted_via` shape that surfaces which Organization is responsible for a given member's resolved access. - **Resolved Decision #8:** storage stays Postgres `jsonb` as the source of truth, with a Postgres [generated column](../domain-model/database-schema/#typed-fields-per-application-views-and-expression-indexes) backing every field an Application has declared in its `settings_schema` — typed and indexed, without giving up the blob's flexibility for undeclared fields. -- Added [Deployment Architecture → Database engine: AWS options compared](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md): RDS vs. Aurora Provisioned vs. Aurora Serverless v2, evaluated for lowest starting price. Aurora Serverless v2 stays the pick — not the cheapest floor in isolation (~$30/month more than bare RDS), but the only option that keeps one connection mechanism across every [Tenancy](../domain-model/tenancy/) tier and avoids a VPC/NAT Gateway entirely; the comparison and the reasoning are now explicit rather than asserted. -- **Added [MCP Server](../mcp-server/)**: a Model Context Protocol server letting an AI agent call the Platform API as typed tools instead of hand-rolled HTTP calls. Introduces no new authorization model — every tool call forwards the caller's own Bearer credential (user token or [API Key](../api-reference/api-keys/)) into the same REST call and the same [Access Control](../access-control/) check. The tool list is generated from `openapi.yaml`'s `operationId`s rather than hand-authored, so it tracks the REST API automatically — see [Roadmap → The MCP server tracks the REST API automatically](../roadmap/#the-mcp-server-tracks-the-rest-api-automatically). Added `operationId` to all 55 operations in [openapi.yaml](../openapi.yaml) (none had one before) specifically so this generation scheme is buildable today, not aspirational. [Deployment Architecture → MCP server](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) specifies the AWS build-out: a second, dedicated Lambda behind the existing API Gateway at `/mcp` (no new domain, no VPC, no database — fully stateless at Tier 0), with named scale-out triggers (Provisioned Concurrency, a Lambda Function URL with streaming responses for server-push/resumability, a small DynamoDB table if session state ever outgrows a signed token). Added [Decision #14](../api-reference/api-keys/#agent-keys--restrict_destructive), still open: whether agent callers should get a recommended (or eventually required) narrower API Key scoping convention, given an LLM's tool-call decision is a different risk shape from deterministic service code. +- Added [Deployment Architecture → Database engine: AWS options compared](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md): RDS vs. Aurora Provisioned vs. Aurora Serverless v2, evaluated for lowest starting price. Aurora Serverless v2 stays the pick — not the cheapest floor in isolation (~$30/month more than bare RDS), but the only option that keeps one connection mechanism across every [Tenancy](../domain-model/tenancy/) tier and avoids a VPC/NAT Gateway entirely; the comparison and the reasoning are now explicit rather than asserted. +- **Added [MCP Server](../mcp-server/)**: a Model Context Protocol server letting an AI agent call the Platform API as typed tools instead of hand-rolled HTTP calls. Introduces no new authorization model — every tool call forwards the caller's own Bearer credential (user token or [API Key](../api-reference/api-keys/)) into the same REST call and the same [Access Control](../access-control/) check. The tool list is generated from `openapi.yaml`'s `operationId`s rather than hand-authored, so it tracks the REST API automatically — see [Roadmap → The MCP server tracks the REST API automatically](../roadmap/#the-mcp-server-tracks-the-rest-api-automatically). Added `operationId` to all 55 operations in [openapi.yaml](../openapi.yaml) (none had one before) specifically so this generation scheme is buildable today, not aspirational. [Deployment Architecture → MCP server](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md) specifies the AWS build-out: a second, dedicated Lambda behind the existing API Gateway at `/mcp` (no new domain, no VPC, no database — fully stateless at Tier 0), with named scale-out triggers (Provisioned Concurrency, a Lambda Function URL with streaming responses for server-push/resumability, a small DynamoDB table if session state ever outgrows a signed token). Added [Decision #14](../api-reference/api-keys/#agent-keys--restrict_destructive), still open: whether agent callers should get a recommended (or eventually required) narrower API Key scoping convention, given an LLM's tool-call decision is a different risk shape from deterministic service code. - **Added [Pricing](../pricing/)**: three subscription tiers for what Substratal itself charges the Application-owner developer (Starter free, Team per-seat, Enterprise with a choice of [Tenant](../domain-model/tenancy/) `tier` as a priced option — `shared` included, `isolated` +$750/month, `dedicated_region` +$1,500/month) — distinct from what that developer charges their own end users, per [Decision #4](../domain-model/orders-and-audit/#billing-system-of-record). Added `plan` (`starter`\|`team`\|`enterprise`) to [Tenant](../domain-model/tenancy/#fields), independent of `tier`, with a database constraint tying `isolated`/`dedicated_region` to `enterprise` only. Added [Decision #15](../pricing/#how-the-subscription-is-charged): how Substratal bills this subscription is still open (leans Stripe Billing), with no corresponding API surface built yet. Cites the AWS Pricing, Billing and Cost Management, and Postgres MCP servers as the tooling to keep these numbers grounded in real infrastructure cost over time. diff --git a/docs/compliance.md b/docs/compliance.md index d3dd8e2..f925b27 100644 --- a/docs/compliance.md +++ b/docs/compliance.md @@ -42,7 +42,7 @@ The mechanisms below cover both regimes; CCPA/CPRA's individual-rights requireme | Right to erasure | **Specified.** A request arrives via `POST /v1/users/{id}/erasure-requests` (self-service, or support on the user's behalf; see [API Reference → Users](../api-reference/users/#post-v1usersiderasure-requests)). [Non-Functional Requirements → Hard-delete cascade](../non-functional-requirements/#hard-delete-cascade) names the actual cascade: soft-delete is immediate on request, and the hard delete runs 7 days later, well inside GDPR Article 12(3)'s 30-day ceiling (not "eventually") and, in order, deletes `password_hash`/`mfa_secret` from [UserIdentity](../domain-model/users-and-organizations/#useridentity), the User's [Profile](../domain-model/profiles/), scrubs PII out of every [AppProfile](../domain-model/profiles/#appprofile)/[AppSettings](../domain-model/settings/#appsettings) `custom`/`overrides` blob the User holds, and redacts (not deletes — see below) the `before`/`after` snapshots on their [Audit Events](../domain-model/orders-and-audit/#audit-event). Entitlement and Order rows are retained as pseudonymized business/financial records, which GDPR permits under a legitimate-interest basis distinct from the erasure right. | | Right to access / data portability | **Specified.** `GET /v1/users/{id}/export` — see [API Reference → Users](../api-reference/users/) — returns everything this API holds about one User: Profile, Settings, every AppProfile/AppSettings, Entitlements, Role assignments, Organization memberships, UserIdentity methods (never `password_hash`/`mfa_secret`), push devices (never the token), notifications, and their own Audit trail, as one JSON bundle. Satisfies GDPR Article 20 portability and the equivalent CCPA/CPRA right-to-know in one endpoint rather than two. | | Lawful basis & consent records | **Missing from the spec on purpose** — not a schema problem, a product/legal one: each Application's developer is the one collecting consent from their own end users (Substratal doesn't run the signup form), so this is as much about what the *developer* agrees to via Substratal's terms of service as what the API stores. Unchanged by this round of decisions. | -| Data Processing Agreement with sub-processors | **Operational, not code** — AWS (the infrastructure in [root `DEPLOYMENT.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md), including Amazon SES for transactional email, so no separate email vendor), **Stripe** (Free/Standard subscription billing, per [Pricing → How the subscription is charged](../pricing/#how-the-subscription-is-charged)), and the push providers that carry [push notifications](../api-reference/notifications/#what-a-device-receives): **Apple** (APNs), **Google** (FCM), and the push service of whichever browser a User subscribes from (Google, Mozilla, Apple, or Microsoft). A push payload is the only data they receive, which is why senders are told to keep sensitive content out of it. Both publish standard DPAs (AWS Artifact; Stripe's own, plus Stripe's existing PCI-DSS Level 1 status, which is part of why [the billing system of record](../domain-model/orders-and-audit/#billing-system-of-record) keeps this API's own PCI scope minimal). Executing either is still a step to actually take, not code to write. | +| Data Processing Agreement with sub-processors | **Operational, not code** — AWS (the infrastructure in [`infra/README.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md), including Amazon SES for transactional email, so no separate email vendor), **Stripe** (Free/Standard subscription billing, per [Pricing → How the subscription is charged](../pricing/#how-the-subscription-is-charged)), and the push providers that carry [push notifications](../api-reference/notifications/#what-a-device-receives): **Apple** (APNs), **Google** (FCM), and the push service of whichever browser a User subscribes from (Google, Mozilla, Apple, or Microsoft). A push payload is the only data they receive, which is why senders are told to keep sensitive content out of it. Both publish standard DPAs (AWS Artifact; Stripe's own, plus Stripe's existing PCI-DSS Level 1 status, which is part of why [the billing system of record](../domain-model/orders-and-audit/#billing-system-of-record) keeps this API's own PCI scope minimal). Executing either is still a step to actually take, not code to write. | | Data residency | **Resolved.** [Tenancy](../domain-model/tenancy/)'s `dedicated_region` tier is the mechanism — a customer with a hard EU-residency requirement gets a dedicated Tenant in an EU region, not a platform-wide region change. Still support-gated today, not self-service — see [Tenancy → How a tier change happens today](../domain-model/tenancy/#how-a-tier-change-happens-today). | | Breach notification (72-hour clock under GDPR) | **Commitment stated, runbook still operational.** The 72-hour clock to notify affected supervisory authorities (and, where risk is high, the data subjects themselves) starts at confirmed awareness of a breach — [Audit Events](../domain-model/orders-and-audit/#audit-event) are the forensic source for scoping what was accessed and when. The specific internal runbook (who's paged, who approves the notification, the template) is operational process, deliberately not specified in an API doc — but the 72-hour commitment itself is now a stated requirement, not an unnamed gap. | | Audit trail vs. erasure tension | **Resolved in the data model.** [Audit Events](../domain-model/orders-and-audit/#audit-event) are retained after a user is hard-deleted, but only the *shape* of what happened (`action`, `timestamp`, which fields changed) — never a copy of the erased personal data itself. The hard-delete cascade above redacts `before`/`after` snapshot values at the same time it deletes the source data, specifically so an Audit Event never becomes the one remaining copy of something erasure was supposed to remove. This is the same redaction every Audit Event eventually gets on its own, by age rather than by request — see [Non-Functional Requirements → Audit log lifecycle](../non-functional-requirements/#audit-log-lifecycle) — so "retained indefinitely" and [Pricing](../pricing/#the-tiers)'s plan-tiered hot-storage window describe different things, not a contradiction. | @@ -53,7 +53,7 @@ Everything in this list is good engineering practice independent of SOC 2, which - **Access control** — already the spine of this entire spec: [Entitlements](../domain-model/entitlements/) + [Roles & Permissions](../domain-model/roles-and-permissions/), least-privilege [API Keys](../api-reference/api-keys/). - **Audit logging** — already specified: the [Action catalog](../domain-model/orders-and-audit/#action-catalog). -- **Encryption in transit and at rest** — HTTPS everywhere (already assumed throughout the [API Reference](../api-reference/)); Aurora's encryption-at-rest in [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) should be turned on by default, not left as a checkbox for later. +- **Encryption in transit and at rest** — HTTPS everywhere (already assumed throughout the [API Reference](../api-reference/)); Aurora's encryption-at-rest in [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md) should be turned on by default, not left as a checkbox for later. - **Change management** — the [Changelog](../changelog/) convention already *is* this, applied to the spec; the same discipline (what changed, when, why) needs to carry into infrastructure and code changes once implementation starts. - **Vendor risk management** — AWS, Stripe, and the push providers (see the DPA row above). Keep the list explicit as it grows — each Application developer's *own* payment processor choice for their own end users is their vendor risk to own, per [Orders & Audit → Billing system of record](../domain-model/orders-and-audit/#billing-system-of-record), not added here. - **Incident response plan** — the one item on this list that's pure process, no code. Write it down before it's needed, not during an actual incident. diff --git a/docs/domain-model/database-schema.md b/docs/domain-model/database-schema.md index edc8dd8..9698ee4 100644 --- a/docs/domain-model/database-schema.md +++ b/docs/domain-model/database-schema.md @@ -8,7 +8,7 @@ nav_order: 8 # Database Schema {: .no_toc } -The entity pages describe *what* each field means to an API caller. This page is for an implementer: concrete Postgres types, constraints, and the indexes the documented query patterns actually need. See [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) for why Postgres (Aurora Serverless v2) is the chosen engine. +The entity pages describe *what* each field means to an API caller. This page is for an implementer: concrete Postgres types, constraints, and the indexes the documented query patterns actually need. See [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md) for why Postgres (Aurora Serverless v2) is the chosen engine. {: .fs-6 .fw-300 } 1. TOC @@ -20,7 +20,7 @@ The entity pages describe *what* each field means to an API caller. This page is - Every table's primary key is the entity's own prefixed id (`usr_…`, `ent_…`, …) stored as `text`, not a surrogate `bigint` — the prefix convention in [Conventions → IDs](../../api-reference/conventions/#ids) *is* the primary key, not a display layer on top of one. - Every end-user-scoped table carries `organization_id text null references organizations(id)`, per [Non-Functional Requirements → Multi-tenancy](../../non-functional-requirements/#multi-tenancy), scoped in the data-access layer (see [Non-Functional Requirements → Multi-tenancy](../../non-functional-requirements/#multi-tenancy)) — not repeated per-table below. This is **team grouping**, not infrastructure placement. -- Every table scoped to one Application also carries `tenant_id text not null references tenants(id)`, denormalized from `applications.tenant_id` at write time — a **second, independent** scoping dimension, enforced in the data-access layer and used to route a row to the right physical cluster (see [Tenancy](../tenancy/) and [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md)). Don't conflate this with `organization_id` above — a row can carry both, either, or neither. +- Every table scoped to one Application also carries `tenant_id text not null references tenants(id)`, denormalized from `applications.tenant_id` at write time — a **second, independent** scoping dimension, enforced in the data-access layer and used to route a row to the right physical cluster (see [Tenancy](../tenancy/) and [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md)). Don't conflate this with `organization_id` above — a row can carry both, either, or neither. - Every table carries `created_at timestamptz not null default now()` (the two pure join tables name it for what it means: `organization_memberships.invited_at` and `user_role_assignments.assigned_at`); tables with mutable fields also carry `updated_at timestamptz not null default now()`, maintained by a trigger, not application code (so it's correct even for a direct `UPDATE` run by a migration or a support script). - Soft-deletable tables carry `deleted_at timestamptz null` rather than a boolean — `null` means active, a timestamp means both *that* it's deleted and *when*, which a boolean throws away. - Every table with a `PATCH` endpoint carries `version integer not null default 1`, incremented by the same `updated_at` trigger. It's the source of the `ETag` (`W/""`) and the `If-Match` comparison. See [Conventions → Concurrency](../../api-reference/conventions/#concurrency-etag--if-match). @@ -39,11 +39,13 @@ The entity pages describe *what* each field means to an API caller. This page is | `signup_application_id` | `text` | nullable, `references applications(id)` | | `stripe_customer_id` | `text` | nullable, `unique` — set at the User's first Checkout; internal, never returned by the API | | `test_mode` | `boolean` | not null, default `false` | +| `expires_at` | `timestamptz` | nullable, `check ((expires_at is not null) = test_mode)`, `check (expires_at <= created_at + interval '30 days')` — set on every test-mode User, never on a live one | +| `expired_processed_at` | `timestamptz` | nullable — internal, never returned. Set when the expiry sweep has revoked the User's sessions and fired its events; cleared when `expires_at` moves later | | `created_at`, `updated_at`, `last_login_at` | `timestamptz` | `last_login_at` nullable | | `version` | `integer` | not null, default `1` | | `deleted_at` | `timestamptz` | nullable — soft-delete | -**Indexes:** `unique (email, test_mode) where deleted_at is null` (a deleted user's email should be reusable by a new signup, which a plain unique index would block, and test-mode signups mustn't collide with live ones); `(lower(email) text_pattern_ops)` for the `q` prefix search; `(status)` for admin filtering. No `organization_id` here — see `organization_memberships` below; a single column on `users` would cap a User at one Organization. No `auth`/password column here either — see `user_identities` below; a User can hold more than one login method, per [Auth → Native auth and per-Organization SSO](../../api-reference/auth/#native-auth-and-per-organization-sso). +**Indexes:** `(expires_at) where test_mode and expired_processed_at is null` for the expiry sweep; `unique (email, test_mode) where deleted_at is null` (a deleted user's email should be reusable by a new signup, which a plain unique index would block, and test-mode signups mustn't collide with live ones); `(lower(email) text_pattern_ops)` for the `q` prefix search; `(status)` for admin filtering. No `organization_id` here — see `organization_memberships` below; a single column on `users` would cap a User at one Organization. No `auth`/password column here either — see `user_identities` below; a User can hold more than one login method, per [Auth → Native auth and per-Organization SSO](../../api-reference/auth/#native-auth-and-per-organization-sso). ## user_identities, sso_connections @@ -407,7 +409,7 @@ This only applies to fields an Application has declared. An arbitrary, undeclare | `test_mode` | `boolean` | not null, default `false` — copied from the change that produced it, so test traffic never shows in a live audit query | | `timestamp` | `timestamptz` | not null, default `now()` | -No `updated_at`, no soft-delete column — this table is append-only by design (see [Non-Functional Requirements → Audit](../../non-functional-requirements/#audit)). A trigger (`audit_events_append_only`) rejects every `UPDATE` and `DELETE` unless the transaction has set `app.audit_maintenance = 'on'`, so neither an application bug nor a direct SQL session can violate the "never edited" guarantee. Only two scheduled jobs set it: the archival job (see [Non-Functional Requirements → Audit log lifecycle](../../non-functional-requirements/#audit-log-lifecycle) and [root `DEPLOYMENT.md` → Audit log archival](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md#audit-log-archival)), which deletes rows it has already archived shape-only to cold storage, and the erasure cascade, which redacts `before`/`after` for an erased User. **Indexes:** `(target_user_id, timestamp desc)`, `(application_id, timestamp desc)`, `(actor_id, timestamp desc)`, `(target_type, target_id, timestamp desc)`, `(organization_id, timestamp desc)`, `(tenant_id, timestamp desc)` — one per documented filter in [API Reference → Audit](../../api-reference/audit/), the last one being what a tier-change migration (and the archival job itself) uses to pull "every event for this Tenant" without scanning the whole table. +No `updated_at`, no soft-delete column — this table is append-only by design (see [Non-Functional Requirements → Audit](../../non-functional-requirements/#audit)). A trigger (`audit_events_append_only`) rejects every `UPDATE` and `DELETE` unless the transaction has set `app.audit_maintenance = 'on'`, so neither an application bug nor a direct SQL session can violate the "never edited" guarantee. Only two scheduled jobs set it: the archival job (see [Non-Functional Requirements → Audit log lifecycle](../../non-functional-requirements/#audit-log-lifecycle) and [`infra/README.md` → Audit log archival](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md#audit-log-archival)), which deletes rows it has already archived shape-only to cold storage, and the erasure cascade, which redacts `before`/`after` for an erased User. **Indexes:** `(target_user_id, timestamp desc)`, `(application_id, timestamp desc)`, `(actor_id, timestamp desc)`, `(target_type, target_id, timestamp desc)`, `(organization_id, timestamp desc)`, `(tenant_id, timestamp desc)` — one per documented filter in [API Reference → Audit](../../api-reference/audit/), the last one being what a tier-change migration (and the archival job itself) uses to pull "every event for this Tenant" without scanning the whole table. ## api_keys @@ -480,7 +482,7 @@ webhook_deliveries ( ); ``` -A `webhook_events` row is inserted **in the same transaction** as the change it describes. A dispatcher (SQS-fed, see [DEPLOYMENT.md](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md)) fans it out to matching subscriptions. That makes "every committed change produces its event" structural rather than best-effort. Events and deliveries are deleted after 30 days. **Indexes:** `(dispatched_at) where dispatched_at is null`; `(subscription_id, created_at desc)` and `(next_retry_at) where status = 'pending'` on deliveries. +A `webhook_events` row is inserted **in the same transaction** as the change it describes. A dispatcher (SQS-fed, see [infra/README.md](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md)) fans it out to matching subscriptions. That makes "every committed change produces its event" structural rather than best-effort. Events and deliveries are deleted after 30 days. **Indexes:** `(dispatched_at) where dispatched_at is null`; `(subscription_id, created_at desc)` and `(next_retry_at) where status = 'pending'` on deliveries. ## push_credentials, devices, notifications, notification_deliveries @@ -520,7 +522,7 @@ devices ( app_version text check (char_length(app_version) <= 50), status text not null default 'active' check (status in ('active','inactive')), inactive_reason text check (inactive_reason in - ('unregistered','session_revoked','user_suspended','user_deleted','stale','limit','credential_removed')), + ('unregistered','session_revoked','user_suspended','user_deleted','test_user_expired','stale','limit','credential_removed')), deactivated_at timestamptz, last_registered_at timestamptz not null default now(), last_delivery_at timestamptz, @@ -553,6 +555,9 @@ notifications ( push_status text not null check (push_status in ('queued','sent','failed','skipped')), skip_reason text check (skip_reason in ('push_not_requested','push_disabled','no_devices','no_credentials','credential_failed','expired')), + push_devices_targeted integer not null default 0, -- the push summary; kept here because deliveries + push_devices_sent integer not null default 0, -- are deleted after 30 days and notifications + push_devices_failed integer not null default 0, -- after 90 sender_type text not null check (sender_type in ('api_key','user','system')), sender_id text not null, -- key_..., usr_..., or 'system'; not a foreign key, since it spans tables read_at timestamptz, @@ -634,7 +639,7 @@ stripe_events ( ); ``` -Kept 90 days. See [root `DEPLOYMENT.md` → Stripe Billing](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md#stripe-billing). +Kept 90 days. See [`infra/README.md` → Stripe Billing](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md#stripe-billing). ## idempotency_keys @@ -651,4 +656,4 @@ idempotency_keys ( ); ``` -Per [Conventions → Idempotency](../../api-reference/conventions/#idempotency)'s implementation note — a scheduled job (see [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md)) deletes expired rows rather than relying on unbounded table growth. +Per [Conventions → Idempotency](../../api-reference/conventions/#idempotency)'s implementation note — a scheduled job (see [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md)) deletes expired rows rather than relying on unbounded table growth. diff --git a/docs/domain-model/notifications.md b/docs/domain-model/notifications.md index e1b6935..adf7190 100644 --- a/docs/domain-model/notifications.md +++ b/docs/domain-model/notifications.md @@ -37,7 +37,7 @@ One install of one Application, on one device, signed in as one User, with the p | `name`, `app_version` | string, nullable | Labels the client supplies, for the User's own device list and for support. | | `session_id` | string, nullable | The [Session](../users-and-organizations/#session) that registered the device. Revoking that session deactivates it. `null` when an app key registered it. | | `status` | enum | `active` \| `inactive`. Only `active` devices are sent to. | -| `inactive_reason` | enum, nullable | Set exactly when `status` is `inactive`: `unregistered` (the provider reported the token gone), `session_revoked`, `user_suspended`, `user_deleted`, `stale` (not re-registered for 270 days), `limit` (displaced by a 21st device), or `credential_removed` (a `web_push` device whose VAPID key was deleted). | +| `inactive_reason` | enum, nullable | Set exactly when `status` is `inactive`: `unregistered` (the provider reported the token gone), `session_revoked`, `user_suspended`, `user_deleted`, `test_user_expired` (a test-mode User's `expires_at` passed), `stale` (not re-registered for 270 days), `limit` (displaced by a 21st device), or `credential_removed` (a `web_push` device whose VAPID key was deleted). | | `last_registered_at` | timestamp | The latest registration or refresh. | | `last_delivery_at` | timestamp, nullable | The latest delivery a provider accepted. | | `test_mode` | boolean | | diff --git a/docs/domain-model/orders-and-audit.md b/docs/domain-model/orders-and-audit.md index 9f1ede0..baa5311 100644 --- a/docs/domain-model/orders-and-audit.md +++ b/docs/domain-model/orders-and-audit.md @@ -81,11 +81,12 @@ Every value `action` can take. It's mirrored exactly by the `AuditAction` enum i | Action | Fires when | `target.type` | |---|---|---| | `user.created` | `POST /v1/users`, `POST /v1/auth/signup`, or an invite via org membership | `user` | -| `user.updated` | `users.manage` activates an `invited` User without a password (`PATCH status: active`). Suspension, reactivation, restore, email, and deletion each have their own action — **admin-only** | `user` | +| `user.updated` | `users.manage` activates an `invited` User without a password (`PATCH status: active`), or changes a test-mode User's `expires_at`. Suspension, reactivation, restore, email, and deletion each have their own action — **admin-only** | `user` | | `user.email_changed` | An email change completes (self, after verification; or admin, immediately) | `user` | | `user.suspended` | `POST /v1/users/{id}/suspend`, or `PATCH status: suspended` — **admin-only** | `user` | | `user.reactivated` | `PATCH status: active` on a `suspended` User, or on a soft-deleted one (a restore) — **admin-only** | `user` | | `user.deleted` | `DELETE /v1/users/{id}`, or the soft-delete step of an erasure request | `user` | +| `user.expired` | A test-mode User's `expires_at` passes; written by the expiry sweep (actor `system`). See [Auth → Test-mode Users → Lifetime](../../api-reference/auth/#lifetime) | `user` | | `user.erasure_requested` | `POST /v1/users/{id}/erasure-requests` | `user` | | `user.erasure_cancelled` | `DELETE /v1/users/{id}/erasure-requests/current` — **admin-only** | `user` | | `user.erased` | The hard-delete cascade completes (actor `system`) | `user` | diff --git a/docs/domain-model/tenancy.md b/docs/domain-model/tenancy.md index 7a7e8e9..cc73432 100644 --- a/docs/domain-model/tenancy.md +++ b/docs/domain-model/tenancy.md @@ -36,8 +36,8 @@ Every Application owner gets a Tenant automatically, the moment they register th | `owner_user_id` | string, nullable | Set iff `owner_type: user`. | | `owner_organization_id` | string, nullable | Set iff `owner_type: organization`. | | `tier` | enum | `shared` (default) \| `isolated` \| `dedicated_region`. See [Tiers](#tiers). **Infrastructure placement** — where the data lives. | -| `region` | string | The AWS region the data lives in, always set: `us-east-1` for `shared` and `isolated` (same region as [Tier 0](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md), with `isolated` just a dedicated cluster within it), and the chosen region for `dedicated_region`. Supported regions: `us-east-1`, `us-west-2`, `ca-central-1`, `eu-west-1`, `eu-central-1`, `ap-southeast-2`. | -| `status` | enum | `active` \| `migrating` \| `suspended`. `migrating` is the transitional state during a tier-change maintenance window — see [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md). | +| `region` | string | The AWS region the data lives in, always set: `us-east-1` for `shared` and `isolated` (same region as [Tier 0](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md), with `isolated` just a dedicated cluster within it), and the chosen region for `dedicated_region`. Supported regions: `us-east-1`, `us-west-2`, `ca-central-1`, `eu-west-1`, `eu-central-1`, `ap-southeast-2`. | +| `status` | enum | `active` \| `migrating` \| `suspended`. `migrating` is the transitional state during a tier-change maintenance window — see [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md). | | `application_count` | integer, read-only | Computed: how many Applications this Tenant owns. | | `created_at`, `updated_at` | timestamp | | @@ -66,8 +66,8 @@ Every Application owner gets a Tenant automatically, the moment they register th | Tier | What changes physically | Cost | Who it's for | |---|---|---|---| -| `shared` | Rows live in the shared Aurora Serverless v2 cluster from [Tier 0](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md), logically isolated by `tenant_id`, enforced in the data-access layer (see [NFR → Multi-tenancy](../../non-functional-requirements/#multi-tenancy)). | $0 incremental — the default. | Every customer, from day one, with no action taken. | -| `isolated` | A dedicated Aurora Serverless v2 cluster (own Secrets Manager secret), same AWS account and region as Tier 0 — stronger blast-radius and noisy-neighbor separation, same Lambda codebase routed by a tenant→cluster lookup. | A full extra Aurora floor (~$45–55/month, see [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md)) — priced by business and enterprise pricing, at least covering it. | A customer with a real compliance or isolation requirement, not just a preference. | +| `shared` | Rows live in the shared Aurora Serverless v2 cluster from [Tier 0](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md), logically isolated by `tenant_id`, enforced in the data-access layer (see [NFR → Multi-tenancy](../../non-functional-requirements/#multi-tenancy)). | $0 incremental — the default. | Every customer, from day one, with no action taken. | +| `isolated` | A dedicated Aurora Serverless v2 cluster (own Secrets Manager secret), same AWS account and region as Tier 0 — stronger blast-radius and noisy-neighbor separation, same Lambda codebase routed by a tenant→cluster lookup. | A full extra Aurora floor (~$45–55/month, see [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md)) — priced by business and enterprise pricing, at least covering it. | A customer with a real compliance or isolation requirement, not just a preference. | | `dedicated_region` | Like `isolated`, but the dedicated cluster sits in the customer's chosen AWS region instead of the default one. | The `isolated` floor again, in a second region — the most expensive tier. | A customer with a genuine data-residency requirement (e.g. "our EU users' data must stay in the EU"). | This is a strict ladder — `dedicated_region` implies everything `isolated` provides, plus region choice; there's no "dedicated region, still logically shared" combination. @@ -78,7 +78,7 @@ There is no self-service API for this. `tenants.manage` (support/superadmin only 1. A customer asks (support is the intake, not a self-serve button) for a tier upgrade. 2. Support calls `POST /v1/tenants/{id}/tier-change-requests` (see [API Reference → Tenancy](../../api-reference/tenancy/)), then moves the request through `pending → scheduled → in_progress → completed` (or `cancelled`) with `PATCH` as the work happens. -3. Support schedules a brief maintenance window, flips `status` to `migrating`, and runs the snapshot/restore cutover into the new infrastructure (new cluster, and for `dedicated_region`, a new region) described in [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md). +3. Support schedules a brief maintenance window, flips `status` to `migrating`, and runs the snapshot/restore cutover into the new infrastructure (new cluster, and for `dedicated_region`, a new region) described in [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md). 4. `tier`, `region`, and `status` are updated back to `active`; an [Audit Event](../orders-and-audit/#audit-event) (`tenant.tier_changed`) records the change. A brief, scheduled downtime window during step 3 is an accepted tradeoff. Tier changes are support-run and infrequent at current scale, so a snapshot/restore cutover is enough. Zero-downtime migration (logical replication) is a legitimate future upgrade once tier changes are self-serve and frequent, not a day-one requirement. diff --git a/docs/domain-model/users-and-organizations.md b/docs/domain-model/users-and-organizations.md index 26e0ba6..23c25b1 100644 --- a/docs/domain-model/users-and-organizations.md +++ b/docs/domain-model/users-and-organizations.md @@ -27,6 +27,7 @@ A person with an account on Substratal. One identity, used everywhere in the hub | `mfa_enabled` | boolean | Read-only, derived: `true` if any of the User's `password` [identities](#useridentity) has confirmed TOTP. | | `signup_application_id` | string, nullable | The Application the user signed up through, if any. Informational only; used for email branding. | | `test_mode` | boolean | Created by a test-mode key. | +| `expires_at` | timestamp, nullable | Set on every test-mode User (default 24 hours after creation, at most 30 days) and never on a live one. The account stops working when it passes. See [Auth → Test-mode Users → Lifetime](../../api-reference/auth/#lifetime). | | `created_at`, `updated_at` | timestamp | | | `last_login_at` | timestamp, nullable | | @@ -47,7 +48,7 @@ The database row also carries `version` (the source of the `ETag`) and `deleted_ hard-deleted (cascade, `user.erased`) ``` -Signup (`POST /v1/auth/signup`) creates a User directly in `active`. An admin invite (`POST /v1/users`) or an org-membership invite by email creates an `invited` one. Suspension and deletion revoke every session, deactivate every push [Device](../notifications/#device), and fire `access.revoked` for every Application the user had active access to. +Signup (`POST /v1/auth/signup`) creates a User directly in `active`. An admin invite (`POST /v1/users`) or an org-membership invite by email creates an `invited` one. Suspension and deletion revoke every session, deactivate every push [Device](../notifications/#device), and fire `access.revoked` for every Application the user had active access to. A test-mode User's expiry does the same, without changing `status`; see [Auth → Test-mode Users → Lifetime](../../api-reference/auth/#lifetime). An `invited` user has an account shell (so an Entitlement or Role can be assigned before they've logged in once) but can't authenticate until they complete signup. `suspended` blocks login but preserves every record — Entitlements, Roles, Profile, Settings — unlike `deleted`, which is the start of actual data removal. A soft-deleted User can still be **restored** by `users.manage` (`PATCH status: active`) as long as no erasure request is scheduled and their email hasn't been taken by a new signup; once the erasure cascade completes, there's nothing to restore. See [Users → `PATCH`](../../api-reference/users/#patch-v1usersid). See [Non-Functional Requirements → Data retention](../../non-functional-requirements/#data-retention). @@ -100,7 +101,7 @@ One successful login. Refresh tokens, app refresh tokens, and app-token `sid` cl | `ip_address`, `user_agent` | string | Recorded at login. They're personal data: included in export, deleted by erasure. | | `created_at`, `last_seen_at` | timestamp | `last_seen_at` is updated at most once a minute. | | `idle_expires_at`, `absolute_expires_at` | timestamp | 30 days from last refresh; 90 days from creation. | -| `revoked_at`, `revoked_reason` | timestamp/string, nullable | `logout`, `logout_all`, `password_reset`, `password_changed`, `user_suspended`, `user_deleted`, `mfa_reset`, `refresh_token_reused`, or `admin`. | +| `revoked_at`, `revoked_reason` | timestamp/string, nullable | `logout`, `logout_all`, `password_reset`, `password_changed`, `user_suspended`, `user_deleted`, `test_user_expired`, `mfa_reset`, `refresh_token_reused`, or `admin`. | ### SSOConnection @@ -123,7 +124,7 @@ This table is deliberately thin — scoped to *that* a User can authenticate via A domain/grouping object for Users — a company, or a group within a company (a department, a team) — used to organize who shares admin standing over a set of Users and which Applications a group is granted as a whole, rather than one admin re-granting access per person. -Both individual and team/company end users are expected, so this isn't a someday-maybe feature. It's pulled forward to [Phase 2](../../roadmap/#phase-2) rather than [Phase 3](../../roadmap/#phase-3). It isn't in the MVP itself, because the first few dozen users are expected to be mostly individual early adopters. But it's needed well before the 10x/100x growth horizon in [Deployment Architecture → Growth trajectory](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md), where team accounts are assumed to matter. Making `organization_id` load-bearing in the schema from day one (see [below](#why-it-matters-even-before-organizations-are-customer-facing-everywhere)) was never a hedge against a hypothetical. +Both individual and team/company end users are expected, so this isn't a someday-maybe feature. It's pulled forward to [Phase 2](../../roadmap/#phase-2) rather than [Phase 3](../../roadmap/#phase-3). It isn't in the MVP itself, because the first few dozen users are expected to be mostly individual early adopters. But it's needed well before the 10x/100x growth horizon in [Deployment Architecture → Growth trajectory](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md), where team accounts are assumed to matter. Making `organization_id` load-bearing in the schema from day one (see [below](#why-it-matters-even-before-organizations-are-customer-facing-everywhere)) was never a hedge against a hypothetical. {: .note } Organization carries no infrastructure meaning. Where a group's data physically lives is [Tenant](../tenancy/)'s job, not this entity's — see [Tenancy → Tenant vs. Organization](../tenancy/#tenant-vs-organization) for why these are deliberately separate. An Organization can incidentally *own* a Tenant (if it also owns an Application — see [Applications](../applications/#fields)), but most Organizations, most of the time, don't own one at all. diff --git a/docs/index.md b/docs/index.md index ed5442a..06b7801 100644 --- a/docs/index.md +++ b/docs/index.md @@ -59,4 +59,4 @@ Two questions get asked, separately, every time a user touches an app: Everything else in this spec (profiles, settings, orders, audit) hangs off of that split. See [Access Control](access-control/) for the full model. {: .note } -This site is self-hosting its own context: it's generated from Markdown committed to this same repository, under `docs/`, via GitHub Pages. See the repo root's [`ORIGINAL-SPEC-DRAFT.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/ORIGINAL-SPEC-DRAFT.md) for the original single-document draft this site was built out from — kept out of `docs/` itself so it's never mistaken for current spec. This site is deliberately scoped to **the API specification itself** — how a developer or their AI agent builds against this API. Deployment/infrastructure lives as plain Markdown in the [repository root](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) instead, not here — see the repo's `README.md` for the full map of what's where, including which other project-planning material (the decision log, the build plan, the roadmap) is slated to join it there. +This site is self-hosting its own context: it's generated from Markdown committed to this same repository, under `docs/`, via GitHub Pages. It's deliberately scoped to **the API specification itself**: how a developer or their AI agent builds against this API. How it's deployed is in the repository's [`infra/README.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md), and what's left to build is in [`plan-remaining.md`](https://github.com/CompositeCode/substratalapps.com/blob/main/plan-remaining.md); the repo's `README.md` maps what's where. diff --git a/docs/mcp-server.md b/docs/mcp-server.md index 62468b5..0611e53 100644 --- a/docs/mcp-server.md +++ b/docs/mcp-server.md @@ -34,9 +34,9 @@ This is a new **surface**, not a new **capability**. Every tool call ultimately POST https://api.substratalapps.com/mcp ``` -Implements MCP's **Streamable HTTP** transport: a single endpoint, JSON-RPC 2.0 request/response bodies, with the client's `Mcp-Session-Id` header (issued on `initialize`) tying together the calls in one session. Deliberately on the existing `api.substratalapps.com` domain rather than a new `mcp.` subdomain — no new Route 53 zone or ACM certificate needed, consistent with [Deployment Architecture → Cost principles](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md). Deliberately **outside** `/v1` — this endpoint isn't a REST resource, and MCP has its own protocol-version negotiation (a date-based version string exchanged during `initialize`) that's independent of this API's own `/v1` URL versioning — the same "versioned independently" pattern [Non-Functional Requirements → Versioning](../non-functional-requirements/#versioning) already applies to webhook payloads. +Implements MCP's **Streamable HTTP** transport: a single endpoint, JSON-RPC 2.0 request/response bodies, with the client's `Mcp-Session-Id` header (issued on `initialize`) tying together the calls in one session. Deliberately on the existing `api.substratalapps.com` domain rather than a new `mcp.` subdomain — no new Route 53 zone or ACM certificate needed, consistent with [Deployment Architecture → Cost principles](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md). Deliberately **outside** `/v1` — this endpoint isn't a REST resource, and MCP has its own protocol-version negotiation (a date-based version string exchanged during `initialize`) that's independent of this API's own `/v1` URL versioning — the same "versioned independently" pattern [Non-Functional Requirements → Versioning](../non-functional-requirements/#versioning) already applies to webhook payloads. -Tier 0's implementation is deliberately the simplest conforming option: every tool call returns one buffered JSON response, no server-initiated push, no mid-call progress notifications. See [Deployment Architecture → MCP server](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) for why that's also the cheapest option, and what upgrading past it looks like. +Tier 0's implementation is deliberately the simplest conforming option: every tool call returns one buffered JSON response, no server-initiated push, no mid-call progress notifications. See [Deployment Architecture → MCP server](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md) for why that's also the cheapest option, and what upgrading past it looks like. ## Authentication — no new model @@ -96,7 +96,7 @@ The same classification now has a second, server-side consumer, not just this cl ## Statelessness -An MCP session's state (negotiated protocol version, declared client capabilities) is kept entirely in the `Mcp-Session-Id` the server hands back on `initialize` — a short-lived, signed, self-contained token, not a database row. Tier 0 needs **no new table and no new database** for this: every individual tool call is a single stateless Lambda invocation that decodes the session token, forwards the caller's own Bearer credential to the matching REST call, and returns the result. See [Deployment Architecture → MCP server](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) for when (and why) this stops being enough. +An MCP session's state (negotiated protocol version, declared client capabilities) is kept entirely in the `Mcp-Session-Id` the server hands back on `initialize` — a short-lived, signed, self-contained token, not a database row. Tier 0 needs **no new table and no new database** for this: every individual tool call is a single stateless Lambda invocation that decodes the session token, forwards the caller's own Bearer credential to the matching REST call, and returns the result. See [Deployment Architecture → MCP server](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md) for when (and why) this stops being enough. ## Resources @@ -104,4 +104,4 @@ Alongside tools, the server exposes a small, fixed set of MCP **Resources** — ## Where this runs -Build-out and AWS deployment — Tier 0 cost, and what triggers an upgrade — is specified alongside the rest of the platform's infrastructure, not duplicated here: see [Deployment Architecture → MCP server](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md). +Build-out and AWS deployment — Tier 0 cost, and what triggers an upgrade — is specified alongside the rest of the platform's infrastructure, not duplicated here: see [Deployment Architecture → MCP server](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md). diff --git a/docs/non-functional-requirements.md b/docs/non-functional-requirements.md index bfe6232..6e1d56c 100644 --- a/docs/non-functional-requirements.md +++ b/docs/non-functional-requirements.md @@ -93,7 +93,7 @@ Targets for the API itself, measured monthly at the API Gateway. They're publish | All other endpoints | p95 < 400 ms | | Webhook dispatch (commit → first attempt) | p95 < 10 s | | Push notification dispatch (commit → first provider attempt) | p95 < 10 s | -| JWKS availability | 99.99%. JWKS is served by the API with `Cache-Control: public, max-age=3600`, and Applications cache it for that hour, so a short API outage doesn't stop token verification. CloudFront goes in front of the whole API domain at the [Scale-out](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md#scale-out) trigger, not at Tier 0. | +| JWKS availability | 99.99%. JWKS is served by the API with `Cache-Control: public, max-age=3600`, and Applications cache it for that hour, so a short API outage doesn't stop token verification. CloudFront goes in front of the whole API domain at the [Scale-out](https://github.com/CompositeCode/substratalapps.com/blob/main/plan-remaining.md#scale-out) trigger, not at Tier 0. | ## Versioning @@ -111,7 +111,7 @@ Every response carries an `X-Request-Id` (server-generated if the caller didn't - **Contract tests** validate the implementation's actual responses against [openapi.yaml](../openapi.yaml) — this is what keeps the machine-readable spec from silently drifting from reality, which is the normal failure mode for hand-maintained API docs. - **Unit tests** on the [Access Control](../access-control/) resolution logic (`effective_permissions`) specifically — it's the one piece of logic every single request depends on, and it's pure/deterministic enough to be cheap to test exhaustively (every combination of entitlement status × platform role × app role). -- **Integration tests** run against a real local Postgres (see [Deployment Architecture → Local development](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md)), not a mocked data layer — the Row-Level Security policies in [Multi-tenancy](#multi-tenancy) are exactly the kind of thing a mock would let silently pass while actually being broken. +- **Integration tests** run against a real local Postgres (see [Deployment Architecture → Local development](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md)), not a mocked data layer — the Row-Level Security policies in [Multi-tenancy](#multi-tenancy) are exactly the kind of thing a mock would let silently pass while actually being broken. - **Seed/fixture data** for local and test environments should be generated from the same request/response examples already in the [API Reference](../api-reference/) — one source of realistic data, not a second hand-maintained copy that drifts from the docs. ## Data retention @@ -126,8 +126,8 @@ Every response carries an `X-Request-Id` (server-generated if the caller didn't "Retained indefinitely" and the audit log's **hot-storage window** are two different axes, not a contradiction. The hot window is one year for every Tenant; business and enterprise pricing may negotiate a longer one when it ships. 1. **Hot storage.** A freshly-written Audit Event lives in the primary `audit_events` table (Aurora Postgres) with its full `before`/`after` snapshot — queryable via [API Reference → Audit](../api-reference/audit/) exactly as specified elsewhere on this page. -2. **Archival, at the end of the hot window.** A scheduled job (see [Deployment Architecture → Audit log archival](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md)) moves every Audit Event older than the hot window out of the hot table: `action`, `timestamp`, `actor_type`/`actor_id`, `target_type`/`target_id`, and every other shape field move to cheap, indefinite cold storage; the `before`/`after` snapshot values — the one place the row could hold personal data — are dropped at this point rather than carried forward, the same redaction [Hard-delete cascade](#hard-delete-cascade) step 4 already performs for a deleted User, just applied uniformly by age instead of by erasure request. -3. **Cold storage, forever.** The shape-only archive is never deleted, satisfying "retained indefinitely" literally — what's indefinite is the record that something happened, not a guarantee that its full payload stays warm (or stays at all) past the hot window. Retrieving an archived event for a dispute or investigation is a deliberately manual, support-mediated process, not a live API path — see [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md) for the actual retrieval SLA. +2. **Archival, at the end of the hot window.** A scheduled job (see [Deployment Architecture → Audit log archival](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md)) moves every Audit Event older than the hot window out of the hot table: `action`, `timestamp`, `actor_type`/`actor_id`, `target_type`/`target_id`, and every other shape field move to cheap, indefinite cold storage; the `before`/`after` snapshot values — the one place the row could hold personal data — are dropped at this point rather than carried forward, the same redaction [Hard-delete cascade](#hard-delete-cascade) step 4 already performs for a deleted User, just applied uniformly by age instead of by erasure request. +3. **Cold storage, forever.** The shape-only archive is never deleted, satisfying "retained indefinitely" literally — what's indefinite is the record that something happened, not a guarantee that its full payload stays warm (or stays at all) past the hot window. Retrieving an archived event for a dispute or investigation is a deliberately manual, support-mediated process, not a live API path — see [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md) for the actual retrieval SLA. This is also why the hot/cold split doesn't weaken [Audit trail vs. erasure tension](../compliance/#gdpr-and-ccpa--build-for-it-now): a snapshot value is gone (not just hidden) the moment it leaves hot storage, for every Tenant, regardless of whether a User ever requests erasure. diff --git a/docs/openapi.yaml b/docs/openapi.yaml index 2e6608e..cc95c0d 100644 --- a/docs/openapi.yaml +++ b/docs/openapi.yaml @@ -49,8 +49,6 @@ tags: - name: Notifications description: >- Push notifications (APNs, FCM, Web Push) and the notification inbox. - Specified 2026-10-17; every operation returns 501 not_implemented until - the reference implementation builds it. - name: API Keys - name: Meta diff --git a/docs/openapi/index.md b/docs/openapi/index.md index c95d5f4..f4a3992 100644 --- a/docs/openapi/index.md +++ b/docs/openapi/index.md @@ -62,9 +62,6 @@ Operations are grouped by tag, and each tag matches an API Reference page that e | Notifications | 17 | [Push Notifications](../api-reference/notifications/) | | API Keys | 6 | [API Keys](../api-reference/api-keys/) | -{: .note } -**Notifications are specified, not built yet.** The 17 operations under the Notifications tag are the target contract for [Push Notifications](../api-reference/notifications/). Until the reference implementation builds them, each one returns `501 not_implemented`, so build clients against the shapes, not against a live server. - Some endpoints live outside `/v1` and aren't in the document's `paths`, because they follow external standards instead: - `GET /.well-known/jwks.json` and `GET /.well-known/openid-configuration`: signing keys and discovery, for verifying app tokens with any standard JWT library. See [Auth → Signing keys](../api-reference/auth/#signing-keys-jwks). diff --git a/docs/pricing.md b/docs/pricing.md index 3933ae8..0bf793a 100644 --- a/docs/pricing.md +++ b/docs/pricing.md @@ -51,7 +51,7 @@ The count is **provisioned, not activity-based**. It counts who is covered, not The count syncs to Stripe once a day with no mid-period proration. Each invoice charges for the count on the day before it's issued. Real-time sync with prorations was considered and not chosen: it's more accurate, but noisier for customers and means more Stripe API calls. -A live subscription never counts test-mode Users, so integration tests can't consume paid seats. A test-mode subscription runs in Stripe test mode and counts only test-mode members. +A live subscription never counts test-mode Users, so integration tests can't consume paid seats. A test-mode subscription runs in Stripe test mode and counts only test-mode members whose [`expires_at`](../api-reference/auth/#lifetime) hasn't passed. Test-mode Users last 24 hours by default and never more than 30 days, so test mode can't stand in for a free production tier. ## How access follows the tier @@ -96,7 +96,7 @@ The Products and Prices configured in Stripe. Lookup keys are what the code refe ## Why two tiers -- **Free is free, not just cheap.** The first horizon in [Growth trajectory](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md#growth-trajectory) is dozens of users, and the shared-tier cost of serving free Applications is near zero against the infrastructure floor that's paid regardless. Charging for it would tax exactly the adoption the product needs most right now. +- **Free is free, not just cheap.** The first horizon in [Growth trajectory](https://github.com/CompositeCode/substratalapps.com/blob/main/plan-remaining.md#growth-trajectory) is dozens of users, and the shared-tier cost of serving free Applications is near zero against the infrastructure floor that's paid regardless. Charging for it would tax exactly the adoption the product needs most right now. - **One paid price, for the whole catalog.** A single per-user price is easy to understand and budget for, and works the same for one person or a whole Organization. Per-Application pricing would make every new Application a new purchasing decision. - **Infrastructure choice waits for business pricing.** `isolated` and `dedicated_region` tenancy carry a real, named infrastructure cost, driven by a customer's own compliance or data-residency requirement. That's what business and enterprise pricing is for, so neither is offered on Free or Standard. @@ -104,7 +104,7 @@ The Products and Prices configured in Stripe. Lookup keys are what the code refe The subscription is processed through **Stripe Billing**. Card data never touches this API, and Stripe is PCI-DSS Level 1, which keeps PCI scope minimal (see [Compliance](../compliance/#what-this-doesnt-cover)). Stripe is one of the platform's two sub-processors, alongside AWS (see [Compliance → GDPR (and CCPA)](../compliance/#gdpr-and-ccpa--build-for-it-now)). -The implementation (object mapping, webhook event handling, the daily quantity sync) is specified in [root `DEPLOYMENT.md` → Stripe Billing](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md#stripe-billing). It's a build and operations concern, not part of the API Applications call. +The implementation (object mapping, webhook event handling, the daily quantity sync) is specified in [`infra/README.md` → Stripe Billing](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md#stripe-billing). It's a build and operations concern, not part of the API Applications call. ## Keeping these numbers honest diff --git a/docs/roadmap.md b/docs/roadmap.md index db9afad..b6719ec 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -68,4 +68,4 @@ Self-service, customer-triggered [Tenancy](../domain-model/tenancy/) tier change ## Where this runs -These phases are about feature scope, not infrastructure — the MVP and Phase 2/3 all deploy onto the same [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md), which has its own, independent phasing (first deployment vs. scale-out) driven by traffic and cost, not by which of these feature phases is live. +These phases are about feature scope, not infrastructure — the MVP and Phase 2/3 all deploy onto the same [Deployment Architecture](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md), which has its own, independent phasing (first deployment vs. scale-out) driven by traffic and cost, not by which of these feature phases is live. diff --git a/docs/workflows.md b/docs/workflows.md index cff6192..e44d206 100644 --- a/docs/workflows.md +++ b/docs/workflows.md @@ -139,7 +139,7 @@ Support-run today. See [Tenancy → How a tier change happens today](../domain-m 1. A customer (an Application owner, on business or enterprise terms arranged with Substratal) asks support for dedicated infrastructure, or a specific region for data residency. 2. Support calls `POST /v1/tenants/{id}/tier-change-requests`. See [API Reference → Tenancy](../api-reference/tenancy/). -3. Support agrees a window with the customer and `PATCH`es the request to `scheduled`. At the window, it `PATCH`es to `in_progress`, which flips the [Tenant](../domain-model/tenancy/)'s `status` to `migrating` and makes writes return `503` with `Retry-After`. Then it runs the snapshot/restore cutover described in [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/DEPLOYMENT.md). Every [Entitlement](../domain-model/entitlements/), [AppProfile](../domain-model/profiles/#appprofile), [AppSettings](../domain-model/settings/#appsettings), and Audit Event row carrying that `tenant_id` moves to the new infrastructure. +3. Support agrees a window with the customer and `PATCH`es the request to `scheduled`. At the window, it `PATCH`es to `in_progress`, which flips the [Tenant](../domain-model/tenancy/)'s `status` to `migrating` and makes writes return `503` with `Retry-After`. Then it runs the snapshot/restore cutover described in [Deployment Architecture → Tenancy tiers](https://github.com/CompositeCode/substratalapps.com/blob/main/infra/README.md). Every [Entitlement](../domain-model/entitlements/), [AppProfile](../domain-model/profiles/#appprofile), [AppSettings](../domain-model/settings/#appsettings), and Audit Event row carrying that `tenant_id` moves to the new infrastructure. 4. `PATCH` to `completed` updates `tier`/`region` and returns `status` to `active`, and an [Audit Event](../domain-model/orders-and-audit/#audit-event) (`tenant.tier_changed`) records it. 5. Nothing about the customer's Applications, Entitlements, or any end user's access changes shape. This workflow only ever moves *where* the same rows live, never *what* they say. diff --git a/DEPLOYMENT.md b/infra/README.md similarity index 69% rename from DEPLOYMENT.md rename to infra/README.md index ec81571..28aabec 100644 --- a/DEPLOYMENT.md +++ b/infra/README.md @@ -1,6 +1,6 @@ # Deployment Architecture -Where the platform API actually runs, and how it's billed. This is a deployment/operations document, not part of the [API specification](https://compositecode.github.io/substratalapps.com/) — it lives here, in the project root, rather than on the docs site, because a prospective API consumer (a developer, or their AI agent) never needs to know how this is hosted to build against it. See [README.md](README.md) for why the project draws that line here. +Where the platform API runs, as built, and how it's billed. This is a deployment and operations document, not part of the [API specification](https://compositecode.github.io/substratalapps.com/), so it lives next to the Terraform that implements it (`infra/terraform/`) rather than on the docs site: an API consumer never needs to know how this is hosted to build against it. Work that isn't built yet (scale-out, dedicated tenancy infrastructure, MCP streaming) is in [plan-remaining.md](../plan-remaining.md); one-time setup steps still to do by hand are in [remaining_manual_steps.md](../remaining_manual_steps.md). Chosen and configured around one constraint above all others: **predictable, capped cost, with no surprises.** @@ -8,17 +8,14 @@ Chosen and configured around one constraint above all others: **predictable, cap 2. [First deployment (Tier 0)](#first-deployment-tier-0) 3. [AWS account & region](#aws-account--region) 4. [Cost guardrails](#cost-guardrails) -5. [Growth trajectory](#growth-trajectory) -6. [Tenancy tiers & where they run](#tenancy-tiers--where-they-run) -7. [MCP server](#mcp-server) -8. [Stripe Billing](#stripe-billing) -9. [Transactional email](#transactional-email) -10. [Webhook dispatch](#webhook-dispatch) -11. [Push delivery](#push-delivery) -12. [Audit log archival](#audit-log-archival) -13. [Scale-out](#scale-out) -14. [Build checklist](#build-checklist) -15. [Local development](#local-development) +5. [Tenancy tiers & where they run](#tenancy-tiers--where-they-run) +6. [MCP server](#mcp-server) +7. [Stripe Billing](#stripe-billing) +8. [Transactional email](#transactional-email) +9. [Webhook dispatch](#webhook-dispatch) +10. [Push delivery](#push-delivery) +11. [Audit log archival](#audit-log-archival) +12. [Local development](#local-development) --- @@ -29,7 +26,7 @@ Four rules this architecture is built around, in priority order: 1. **No resource bills by the hour regardless of traffic.** No always-on EC2 instance, no Fargate task, no NAT Gateway. If nobody's calling the API, the API costs close to nothing. 2. **Every component with a cost has a known floor and a known ceiling.** "Serverless" isn't automatically cheaper — some AWS serverless products (Aurora Serverless v2's minimum capacity, for one) have a *higher* floor than the plain fixed-price alternative, and that trade is only worth it when something else about the product earns it back. See [Database engine: AWS options compared](#database-engine-aws-options-compared). 3. **A hard budget alarm exists before the first resource does.** Guardrails are step one of the build checklist below, not a follow-up task. -4. **Scale-out is a later, deliberate decision, not a default.** Nothing in Tier 0 auto-scales into real money without a human decision — see [Scale-out](#scale-out) for what actually triggers each upgrade. +4. **Scale-out is a later, deliberate decision, not a default.** Nothing in Tier 0 auto-scales into real money without a human decision — see [Scale-out](../plan-remaining.md#scale-out) for what actually triggers each upgrade. ## First deployment (Tier 0) @@ -62,7 +59,7 @@ flowchart LR | Stripe webhook compute | A **third, dedicated Lambda**, at `/internal/stripe/webhook` | Isolated from the API handlers' Lambda for the same reason the MCP Lambda is — plus a real security reason: it's the one route whose caller isn't an API-key/Bearer-token holder at all, verified instead by Stripe's own signature scheme (see [Stripe Billing](#stripe-billing)), so keeping it a separate function keeps that distinct trust boundary legible in the IAM/routing layer, not just in code. | | Database | **Aurora Serverless v2 (PostgreSQL)**, accessed via the **RDS Data API** | The domain model is relational (joins, foreign keys, the Audit log) — Postgres fits it directly. Data API means Lambda calls the database over signed HTTPS with **no VPC attachment** — which is what avoids a NAT Gateway entirely (see below), not a minor detail. | | Webhook delivery | **SQS** queue + a dedicated worker **Lambda** | Matches the retry/backoff schedule in the [Webhooks](https://compositecode.github.io/substratalapps.com/api-reference/webhooks/) spec — SQS visibility timeouts drive the delay between attempts for free, no extra scheduler needed for that part. | -| Push delivery | A second **SQS** queue on the **same worker Lambda** | [Push notifications](https://compositecode.github.io/substratalapps.com/api-reference/notifications/) to APNs, FCM, and Web Push. The same outbox-and-queue shape as webhooks, so no new function. See [Push delivery](#push-delivery). | +| Push delivery | The webhook **SQS** queue and **worker Lambda** | [Push notifications](https://compositecode.github.io/substratalapps.com/api-reference/notifications/) to APNs, FCM, and Web Push. The same outbox-and-nudge shape as webhooks, so no new queue or function. See [Push delivery](#push-delivery). | | Scheduled jobs | **EventBridge Scheduler** → **Lambda** | Trial-expiry checks, idempotency-key cleanup — pay-per-invocation, no cron server to run. | | Secrets | **Secrets Manager** — the Aurora master credential and the Stripe secret/webhook-signing keys | Everything else non-secret goes in Lambda environment variables rather than paying per-secret for config that isn't sensitive. | | DNS / TLS | **Route 53** + **ACM** (free) + API Gateway custom domain | Backs the `api.substratalapps.com` base URL from the [API spec's Conventions](https://compositecode.github.io/substratalapps.com/api-reference/conventions/#base-url). | @@ -80,7 +77,7 @@ Postgres is the storage engine — the domain model is relational, and the schem | **Aurora Provisioned (PostgreSQL-compatible)**, smallest instance | ~$55–60/month single-AZ — Aurora's smallest instance class is larger than RDS's smallest | Pricier than Serverless v2's floor for a low-traffic start, with none of Serverless v2's scale-with-load benefit. Not competitive against either other option at Tier 0 volume. | | **Aurora Serverless v2 (PostgreSQL-compatible)**, 0.5 ACU minimum — **chosen** | ~$43/month floor, scales up with load | Costs more than bare RDS at idle. What it buys back: the **Data API**, which is why Tier 0 has no VPC at all (see [below](#why-no-vpc-and-specifically-no-nat-gateway)), and why [tenancy tiers](#tenancy-tiers--where-they-run) can route a request to any of several clusters — shared, or a specific customer's `isolated`/`dedicated_region` one — via one lookup-table value, with every tier using the identical connection mechanism. | -**The ~$30/month gap between RDS and Aurora Serverless v2 is a real, named cost** — not hand-waved away — paid specifically for not having to build and maintain two different database-connection code paths (one VPC-bound for a cheap shared tier, one Data-API-based for dedicated tenants) or a NAT Gateway. At Tier 0's actual scale (see [Growth trajectory](#growth-trajectory)) that $30/month is a smaller cost than the engineering time either alternative would take to build and keep correct. +**The ~$30/month gap between RDS and Aurora Serverless v2 is a real, named cost** — not hand-waved away — paid specifically for not having to build and maintain two different database-connection code paths (one VPC-bound for a cheap shared tier, one Data-API-based for dedicated tenants) or a NAT Gateway. At Tier 0's actual scale (see [Growth trajectory](../plan-remaining.md#growth-trajectory)) that $30/month is a smaller cost than the engineering time either alternative would take to build and keep correct. ### Why no VPC, and specifically no NAT Gateway @@ -89,13 +86,13 @@ A NAT Gateway (~$32/month plus per-GB data processing, easy to forget it's runni - The API Lambda talks to Aurora via the **Data API** (HTTPS, IAM-signed) instead of a direct Postgres connection — so it never needs to be placed inside the VPC in the first place. - The webhook-delivery worker needs genuine outbound internet access (POSTing to arbitrary downstream URLs) — it gets that for free by **not** being placed in a VPC either; a Lambda outside a VPC has normal internet access by default, at no extra cost. -Net effect: zero VPC, zero NAT, for the entire Tier 0 stack. This is revisited in [Scale-out](#scale-out) once throughput, not cost, becomes the binding constraint. +Net effect: zero VPC, zero NAT, for the entire Tier 0 stack. This is revisited in [Scale-out](../plan-remaining.md#scale-out) once throughput, not cost, becomes the binding constraint. ## AWS account & region Resolved: a **dedicated AWS account**, in **`us-east-1`**. -**Dedicated account**, under an AWS Organization rather than reusing an existing personal/other-project account, specifically so [Cost guardrails](#cost-guardrails) below — the Budgets, the Cost Anomaly Detection baseline, the billing alarm — are scoped to exactly this system's spend from day one. A shared account makes every one of those guardrails a cost-allocation-tag exercise instead of an account-level default; at the current team size (one person plus an AI assistant, per the product's own stated growth plan) that's avoidable overhead, not a safety net worth the complexity. AWS Organizations (free) is still the right wrapper even for a single member account — it's what makes adding a second account (the [Scale-out](#scale-out) trigger for a separate staging environment) a few-minutes operation later instead of a migration. +**Dedicated account**, under an AWS Organization rather than reusing an existing personal/other-project account, specifically so [Cost guardrails](#cost-guardrails) below — the Budgets, the Cost Anomaly Detection baseline, the billing alarm — are scoped to exactly this system's spend from day one. A shared account makes every one of those guardrails a cost-allocation-tag exercise instead of an account-level default; at the current team size (one person plus an AI assistant, per the product's own stated growth plan) that's avoidable overhead, not a safety net worth the complexity. AWS Organizations (free) is still the right wrapper even for a single member account — it's what makes adding a second account (the [Scale-out](../plan-remaining.md#scale-out) trigger for a separate staging environment) a few-minutes operation later instead of a migration. **`us-east-1`**, over other candidates, for three concrete reasons rather than "it's the default": @@ -103,6 +100,11 @@ Resolved: a **dedicated AWS account**, in **`us-east-1`**. - **Pricing.** `us-east-1` is at or near the lowest-cost tier for every service in the [Tier 0 component table](#first-deployment-tier-0) above (Lambda, Aurora Serverless v2, API Gateway, S3) — a handful of regions tie it, none meaningfully undercut it for this stack. - **No data-residency constraint pulling the other way.** Nothing in the [Compliance](https://compositecode.github.io/substratalapps.com/compliance/) posture or the current customer base requires EU or other non-US placement for the *shared* tier specifically — that requirement, when it exists, is what the `dedicated_region` [tenancy tier](#tenancy-tiers--where-they-run) is *for*, scoped to the one customer who needs it rather than forcing a region choice for everyone. +**As built (2026-10-20):** production is the member account `substratal-prod` in an AWS Organization created through the AWS console's Projects experience, with a separate management account (billing and policies) and an Identity delegated-admin account. Two consequences worth knowing: + +- That Organization's region-restriction SCP (`AdvancedModeRegionRestrictionSecurityControlPolicy`) allows only global services in `us-east-1` by default. Its `UsEast1Partitional` statement carries an exception for `substratal-prod` (`StringNotEquals aws:PrincipalAccount`) so the stack can run there. The Projects feature might restore the original; if production calls suddenly fail with an explicit SCP deny, check that exception first. +- Console and CLI access go through the Projects role (`AccountFullAccessRole`), and the CLI signs in with `aws login`. The account's identity source (AWS Builder ID) doesn't offer IAM Identity Center permission sets. + This is a Tier 0 default, not a permanent commitment — nothing above prevents a `dedicated_region` Tenant from landing anywhere else, and nothing prevents revisiting the *shared* region later if a majority of the customer base ends up needing otherwise. Revisit if that happens; don't pre-build for it. ## Cost guardrails @@ -130,38 +132,18 @@ Rough, region-dependent, **not a quote**: | MCP Lambda, Stripe webhook Lambda, low call volume each | Low single digits combined, same free-tier coverage as the API handlers' Lambda | | **Floor total** | **~$45–55/month**, before any real traffic | -## Growth trajectory - -The stated plan: dozens of users and a handful of Applications in the first 6–12 months, roughly 10x from there over the following 6–12 months, and another 10x (100x from today) beyond that. - -| Horizon | Rough scale | What this means for the architecture | -|---|---|---| -| 0–6/12 months | Dozens of users, a few Applications | **Tier 0 as specified, unchanged.** The floor cost table above is sized for exactly this. | -| 6–12 months out (~10x) | Low hundreds of users | Watch rate-limit defaults and the Aurora ACU ceiling (not just the floor) — this is when the *first* [Scale-out](#scale-out) triggers are plausible, not guaranteed. Check real metrics rather than upgrading preemptively. | -| 12+ months (~100x) | Low thousands of users | The horizon [Scale-out](#scale-out)'s read-replica, Provisioned Concurrency, and RDS Proxy rows are realistically aimed at. Also the point multi-tenancy query performance (the test/live Row-Level Security policy plus the `tenant_id` filters) is worth profiling specifically, not just trusting by design. | - ## Tenancy tiers & where they run +Only `shared` is built. Every Tenant lives in the Tier 0 cluster today; the `isolated` and `dedicated_region` rows below are the design, and their provisioning, request routing, and migration runbook are in [plan-remaining.md → Dedicated tenancy infrastructure](../plan-remaining.md#dedicated-tenancy-infrastructure). + [Tenancy](https://compositecode.github.io/substratalapps.com/domain-model/tenancy/) is a second, independent scale-out axis — triggered by one customer's requirement, not by aggregate volume, and can happen on day one for a single large customer well before the traffic-driven triggers above are anywhere close. | Tier | Infrastructure | Relationship to Tier 0 above | |---|---|---| -| `shared` | Exactly [Tier 0](#first-deployment-tier-0) as specified — the same Aurora cluster every other `shared`-tier Tenant uses, isolated by `tenant_id` in the data-access layer (see [NFR → Multi-tenancy](https://compositecode.github.io/substratalapps.com/non-functional-requirements/#multi-tenancy)). | Is Tier 0. No separate infrastructure exists for this tier. | +| `shared` | **Built.** Exactly [Tier 0](#first-deployment-tier-0) as specified — the same Aurora cluster every other `shared`-tier Tenant uses, isolated by `tenant_id` in the data-access layer (see [NFR → Multi-tenancy](https://compositecode.github.io/substratalapps.com/non-functional-requirements/#multi-tenancy)). | Is Tier 0. No separate infrastructure exists for this tier. | | `isolated` | A second (third, fourth, …) Aurora Serverless v2 cluster, its own Secrets Manager secret, same AWS account and region as Tier 0. The API Lambda's Data API calls are routed to the right cluster via a small `tenants` lookup table kept in the primary/shared cluster — no new Lambda functions, no code fork. | One extra Aurora floor (~$45–55/month) per `isolated` Tenant — priced per [Pricing](https://compositecode.github.io/substratalapps.com/pricing/#business-and-enterprise). | | `dedicated_region` | Like `isolated`, but the dedicated cluster — and, if latency to that region matters, a regional API Gateway + Lambda deployment in front of it — sits in the customer's chosen AWS region. | The `isolated` floor again, in a second region. | -### Migration mechanics - -A tier change (`shared → isolated`, or either `→ dedicated_region`) is support-run, during a brief scheduled maintenance window: - -1. Provision the destination cluster (new Aurora Serverless v2 cluster; for `dedicated_region`, in the target region) via the same [Build checklist](#build-checklist) steps used for Tier 0 itself. -2. Set the source Tenant's `status` to `migrating` — the API layer treats this as read-only for that Tenant's rows specifically, not a global outage. -3. Snapshot and restore (RDS snapshot copy, or `pg_dump`/`pg_restore` at this data volume) every row carrying that `tenant_id` into the destination cluster. -4. Update the `tenants` lookup row's cluster endpoint, flip `tier`/`region`/`status` back to `active`, decommission the old rows once the cutover is confirmed. -5. Record an Audit Event (`tenant.tier_changed`). - -Because Entitlement, AppProfile, and AppSettings rows all denormalize `tenant_id`, step 3 is a `WHERE tenant_id = ...` export per table — no cross-table join needed to find everything that has to move. - ## MCP server See the [API spec's MCP Server page](https://compositecode.github.io/substratalapps.com/mcp-server/) for what this component is and why it exists; this section is the build-out and cost side of it specifically. @@ -176,15 +158,6 @@ A dedicated Lambda (shown in the [Tier 0 diagram](#first-deployment-tier-0) abov This Lambda never talks to Aurora, Secrets Manager, or SQS directly — it only ever calls the existing API handlers over HTTPS, so its IAM execution role needs no data-plane permissions at all. It's the cheapest possible shape for this component: pure compute, pay-per-call, the same free-tier coverage as the main API handlers' Lambda. -### Scaling the MCP server - -| Trigger | Change | -|---|---| -| Sustained high-frequency tool-call traffic on hot tools | Provisioned Concurrency on the MCP Lambda specifically — same pattern and trigger as the [Scale-out](#scale-out) table's cold-start row. | -| A real need for server-initiated push — mid-call progress notifications, or resumable streams | Move `/mcp` from a plain Lambda-behind-API-Gateway integration to a **Lambda Function URL with streaming responses enabled** (`InvokeMode: RESPONSE_STREAM`) — API Gateway's HTTP API integration buffers the full Lambda response, so it can't carry a chunked `text/event-stream`; a Function URL can. | -| Session/stream-resumption state that no longer fits in a short-lived signed token | A small **DynamoDB** table, keyed by session ID, TTL'd like `idempotency_keys` — ephemeral, high-churn, single-key lookup: the textbook case Aurora is the wrong tool for and DynamoDB is the right one. | -| More than a handful of concurrent agent integrations with genuinely different trust levels | Per-integration API Keys scoped as narrowly as each integration actually needs — an operational practice, not an infrastructure change. | - ## Stripe Billing [Pricing](https://compositecode.github.io/substratalapps.com/pricing/) specifies the tiers and the [Stripe catalog](https://compositecode.github.io/substratalapps.com/pricing/#stripe-catalog). [API Reference → Billing](https://compositecode.github.io/substratalapps.com/api-reference/billing/) specifies the endpoints. This section is how the integration is built and run. The subscriber is a User (an individual subscription) or an Organization (for the members it covers), never a Tenant. @@ -259,14 +232,14 @@ Outbound webhooks use a **transactional outbox**. Every state-changing transacti [Push Notifications](https://compositecode.github.io/substratalapps.com/api-reference/notifications/) specifies the behavior. This is how it runs, on top of the webhook machinery above, with no new always-on component. -- **Outbox.** A send inserts the `notifications` row and one `notification_deliveries` row per eligible device in the same transaction ([Database Schema](https://compositecode.github.io/substratalapps.com/domain-model/database-schema/#push_credentials-devices-notifications-notification_deliveries)). After commit, the API nudges a `push` SQS queue. The dispatcher's existing sweep also picks up any `pending` delivery older than a few seconds, so a lost nudge only delays a push. -- **Worker.** The existing webhook worker Lambda takes the `push` queue as a second event source. It decrypts the Application's credential with the same symmetric KMS key used for webhook and TOTP secrets, and keeps a per-credential cache of provider auth across warm invocations: the APNs provider JWT (ES256, refreshed every 50 minutes, inside Apple's 20–60 minute window), the FCM OAuth 2.0 access token (refreshed before its one-hour expiry), and the VAPID JWT (ES256, at most 12 hours). APNs needs HTTP/2, which Go's standard client supports, and one connection per gateway (production, sandbox) is reused across deliveries in an invocation. +- **Outbox.** A send inserts the `notifications` row and one `notification_deliveries` row per eligible device in the same transaction ([Database Schema](https://compositecode.github.io/substratalapps.com/domain-model/database-schema/#push_credentials-devices-notifications-notification_deliveries)). After commit, the API nudges the webhook SQS queue. A nudge carries no payload, so one queue serves both outboxes. The existing 1-minute `webhook-dispatch` sweep runs push delivery too, so a lost nudge only delays a push. +- **Worker.** The existing webhook worker Lambda drains push deliveries first on every invocation, then webhooks, because a push outcome can emit webhook events (`device.deactivated`, `notification.push_failed`). Each delivery is claimed with a 2-minute lease, so no database transaction stays open across a provider call. The worker decrypts the Application's credential and the device token with the same symmetric KMS key used for webhook and TOTP secrets, and keeps a per-credential cache of provider auth across warm invocations: the APNs provider JWT (ES256, refreshed every 50 minutes, inside Apple's 20–60 minute window), the FCM OAuth 2.0 access token (refreshed before its one-hour expiry), and the VAPID JWT (ES256, at most 12 hours). APNs needs HTTP/2, which Go's standard client supports, and one connection per gateway (production, sandbox) is reused across deliveries in an invocation. - **No VPC, no NAT.** The worker already runs outside a VPC for outbound webhooks, so it reaches `api.push.apple.com`, `fcm.googleapis.com`, and browser push endpoints at no extra cost. -- **Retries** use SQS delays (+30 seconds, +2 minutes, +10 minutes) within the notification's `ttl_seconds`. Provider responses that mean the token is gone deactivate the device, and ones that mean the credential is bad mark the credential `failed` and email the Application's owner, both in the worker's own transaction. +- **Retries** are scheduled in the database (`next_attempt_at`: +30 seconds, +2 minutes, +10 minutes, within the notification's `ttl_seconds`) and picked up by the 1-minute sweep, so a retry runs up to a minute after it comes due. Provider responses that mean the token is gone deactivate the device, and ones that mean the credential is bad mark the credential `failed` and email the Application's owner, both in the worker's own transaction. - **Secrets.** Push credentials are per Application and live KMS-encrypted in Aurora, not in Secrets Manager, which would cost $0.40 a month per Application for no added safety. Device tokens are encrypted the same way. -- **Scheduled jobs** (the existing jobs Lambda): the stale-device sweep (daily), deleting inactive devices 30 days after deactivation (daily), and purging notifications after 90 days and deliveries after 30 (nightly). -- **Cost.** APNs, FCM, and Web Push charge nothing to send. At Tier 0 volume the queue, worker time, and KMS decrypts are a rounding error, and the decrypted-credential cache keeps KMS calls to one per credential per warm worker. -- **Locally**, the devserver delivers in-process, the same as webhooks. A local provider backend that records pushes instead of sending them lets tests and the Quickstart run without real Apple or Google credentials. +- **Scheduled jobs** (the existing jobs Lambda): `push-maintenance`, daily at 04:30 UTC, runs the 270-day stale-device sweep, deletes inactive devices 30 days after deactivation, and purges notifications after 90 days and delivery results after 30. +- **Cost.** APNs, FCM, and Web Push charge nothing to send. At Tier 0 volume the worker time is a rounding error, and so is KMS: two decrypts per delivery (credential and token), at $0.03 per 10,000. +- **`PUSH_BACKEND`** selects the sender: `live` (production's default) sends to the providers, and `log` (the local default) only logs each push. Locally, the devserver delivers in-process, the same as webhooks, so the whole flow works without real Apple or Google credentials. Tests use a recording fake. ## Audit log archival @@ -282,38 +255,6 @@ Outbound webhooks use a **transactional outbox**. Every state-changing transacti Cost is negligible at Tier 0 volume (Glacier Deep Archive is priced for exactly this shape of rarely-read, kept-forever data) — this is about bounding Aurora's hot-table size and honoring the redaction-on-age guarantee, not about saving money on S3 itself. -## Scale-out - -Nothing below is pre-built into Tier 0 — each is a deliberate upgrade, triggered by a specific, named condition, not a default growth path. - -| Trigger | Change | -|---|---| -| Sustained read load on `effective-permissions` / catalog reads | Add an Aurora read replica; consider API Gateway response caching for read-heavy, low-churn paths. | -| Lambda cold-start latency becomes user-visible on hot paths | Provisioned Concurrency on the `effective-permissions` and entitlement-toggle handlers specifically — not a blanket setting. | -| Sustained high-throughput DB access, where Data API's per-call overhead starts to matter | Move those paths from Data API to direct VPC connections through **RDS Proxy** — this is the point where a NAT Gateway (or a cheaper NAT instance) becomes worth its cost, because throughput now justifies it rather than convenience. | -| Geographically distributed callers, latency-sensitive | CloudFront in front of API Gateway. | -| Hub availability becomes a real incident risk for every downstream app, not a theoretical one | Multi-region active-passive — deliberately not a Tier 0/1 concern; premature multi-region is itself a cost and complexity risk. | -| Webhook delivery volume grows enough to need isolation from the main API's blast radius | Split the worker Lambda's concurrency/alerting from the API handlers' (it already runs as a separate function). | -| Push volume grows enough that APNs connection reuse, or isolation from webhook delivery, matters | Give push delivery its own worker function with reserved concurrency, so long-lived HTTP/2 connections to APNs survive between batches and a burst of pushes can't delay webhooks. | -| More than one engineer shipping concurrently | A second AWS account (via AWS Organizations) for staging, separate from production — Tier 0 runs on a single account with a `dev`/`prod` naming convention, which is enough until this trigger. | -| A specific customer requires dedicated or regional infrastructure | Run the [Tenancy tiers](#tenancy-tiers--where-they-run) migration for that one customer's Tenant — the one scale-out trigger in this table that's customer-specific rather than aggregate-volume-driven. | -| Agent/MCP tool-call traffic needs server-push, resumable streams, or dedicated concurrency | See [MCP server → Scaling the MCP server](#scaling-the-mcp-server) — its own trigger table. | -| Stripe billing volume or complexity outgrows a single webhook Lambda (e.g. real invoice-dunning logic, usage-based billing beyond simple per-seat metering) | Revisit as its own build, not folded into this table speculatively — nothing here is blocked on it. | - -## Build checklist - -The order this gets stood up in, once API implementation begins: - -1. AWS account (dedicated member account under Organizations, `us-east-1` — see [AWS account & region](#aws-account--region)) + billing contact + **Budgets and Cost Anomaly Detection from step 1**, before any other resource. -2. Route 53 hosted zone + ACM certificate for the chosen API domain. SES domain identity for `mail.substratalapps.com` (DKIM/SPF/DMARC) and an SES production-access request, which takes about a day, so start it here. -3. KMS keys: one asymmetric RSA-2048 signing key for JWTs (JWKS is published from its public half), and one symmetric key for encrypting TOTP and webhook secrets. Secrets Manager secrets: the Aurora master credential, and Stripe's secret key + webhook-signing secret (test-mode keys first — see [Stripe Billing](#stripe-billing)). Aurora Serverless v2 cluster with Data API enabled, minimum 0.5 / maximum capacity set deliberately. -4. IAM: one execution role per Lambda function (API handlers, MCP server, Stripe webhook handler, webhook worker, scheduled-jobs), each scoped to only the resources it actually needs — no shared mega-role. -5. API Gateway HTTP API + custom domain mapping; Lambda handlers deployed behind it, implementing the [API Reference](https://compositecode.github.io/substratalapps.com/api-reference/) / [openapi.yaml](https://compositecode.github.io/substratalapps.com/openapi.yaml) contract. -6. SQS queues (webhook fan-out/delivery, push delivery, email, Stripe events) and their worker Lambdas; EventBridge Scheduler rules for the entitlement-expiry sweep (every 5 minutes), the webhook outbox dispatcher, the erasure cascade (hourly), the Organization quantity sync (daily 00:15 UTC), the subscription-access reconcile (hourly), test-mode purge, expired-token and idempotency-key cleanup (nightly), the push stale-device sweep (daily) and notification retention purge (nightly), and [audit log archival](#audit-log-archival). JWKS is served by the API with a 1-hour `Cache-Control`; CloudFront in front of the whole domain waits for the [Scale-out](#scale-out) trigger, since caching only that path would mean fronting the entire API. -7. Stripe account: run the catalog sync script (test mode first), configure the Customer Portal (no plan switching), enable Stripe Tax, register the `/internal/stripe/webhook` endpoint for the events in [Webhook handling](#webhook-handling). -8. CloudWatch Logs with explicit retention on every log group; a small set of alarms (error rate, Lambda throttling, Aurora ACU near max) in addition to the billing guardrails from step 1. -9. CI/CD via OIDC federation (GitHub Actions → an AWS deploy role) — no long-lived IAM user access keys committed anywhere. - ## Local development The RDS Data API choice that keeps Tier 0 NAT-free has a real cost: it doesn't have a clean open-source local emulator, so a developer can't just point the production data-access code at "Data API, but local" the way they could with a plain Postgres driver. @@ -322,4 +263,4 @@ The fix is to not let that choice leak into the application code in the first pl Stripe's own test mode (plus the Stripe MCP server referenced in [Stripe Billing](#stripe-billing)) covers the equivalent problem on the billing side — no local Stripe emulator is needed, since Stripe's hosted test mode is free and already the standard way to develop against it. -The full local setup (Docker Compose services, the devserver, `make` targets) and the `dev` → `main` → production deployment flow are in [README.md → Local development](README.md#local-development) and [README.md → Branches, CI, and deployment](README.md#branches-ci-and-deployment). +The full local setup (Docker Compose services, the devserver, `make` targets) and the `dev` → `main` → production deployment flow are in [README.md → Local development](../README.md#local-development) and [README.md → Branches, CI, and deployment](../README.md#branches-ci-and-deployment). diff --git a/infra/terraform/bootstrap/main.tf b/infra/terraform/bootstrap/main.tf index 9f0a9ac..a548842 100644 --- a/infra/terraform/bootstrap/main.tf +++ b/infra/terraform/bootstrap/main.tf @@ -1,6 +1,6 @@ # Applied once, by hand, with admin credentials (README → One-time setup). # It's the only Terraform ever applied from a laptop. In order: -# 1. Cost guardrails, before any other billable resource (DEPLOYMENT.md → +# 1. Cost guardrails, before any other billable resource (infra/README.md → # Cost guardrails: "a hard budget alarm exists before the first # resource does"). # 2. The Terraform state bucket and the artifacts bucket. @@ -227,7 +227,7 @@ resource "aws_iam_openid_connect_provider" "github" { } locals { - oidc_sub = "repo:${var.github_repository}" + oidc_sub = var.github_oidc_subject } # Read-only: terraform plan on pull requests into main. diff --git a/infra/terraform/bootstrap/variables.tf b/infra/terraform/bootstrap/variables.tf index 919637c..875a842 100644 --- a/infra/terraform/bootstrap/variables.tf +++ b/infra/terraform/bootstrap/variables.tf @@ -3,10 +3,17 @@ variable "region" { default = "us-east-1" } -variable "github_repository" { - description = "owner/name of the repository allowed to assume the CI roles" +variable "github_oidc_subject" { + description = <<-EOT + The repository part of the GitHub OIDC `sub` claim the CI roles trust. + This repository uses GitHub's immutable subject claims, which embed the + owner and repository ids (owner@id/name@id), so a renamed or recreated + repository can't inherit the roles. Read the current value with + `gh api repos///actions/oidc/customization/sub` + (sub_claim_prefix). + EOT type = string - default = "CompositeCode/substratalapps.com" + default = "repo:CompositeCode@38675098/substratalapps.com@1403762290" } variable "billing_email" { diff --git a/infra/terraform/modules/lambda_function/main.tf b/infra/terraform/modules/lambda_function/main.tf index ce80495..83445ad 100644 --- a/infra/terraform/modules/lambda_function/main.tf +++ b/infra/terraform/modules/lambda_function/main.tf @@ -1,6 +1,6 @@ # One Lambda function the way every Substratal function is built: Go on # provided.al2023 / arm64, its own execution role scoped to exactly what it -# needs (DEPLOYMENT.md → Build checklist, step 4: no shared mega-role), +# needs (infra/README.md → Build checklist, step 4: no shared mega-role), # explicit log retention, and a "live" alias that API Gateway, SQS, and # the scheduler always invoke. Each deploy publishes a new version and # moves live to it; rollback.yml moves it back. @@ -70,7 +70,7 @@ resource "aws_iam_role_policy" "this" { } # Explicit retention: logs left at "never expire" are the classic growing -# CloudWatch bill (DEPLOYMENT.md → Tier 0 components). +# CloudWatch bill (infra/README.md → Tier 0 components). resource "aws_cloudwatch_log_group" "this" { name = "/aws/lambda/${var.name}" retention_in_days = var.log_retention_days diff --git a/infra/terraform/prod/main.tf b/infra/terraform/prod/main.tf index 9264280..d888e6c 100644 --- a/infra/terraform/prod/main.tf +++ b/infra/terraform/prod/main.tf @@ -1,4 +1,4 @@ -# Production: the Tier 0 stack from DEPLOYMENT.md → First deployment. +# Production: the Tier 0 stack from infra/README.md → First deployment. # No VPC for Lambda and no NAT Gateway: every function reaches Aurora over # the RDS Data API (IAM-signed HTTPS), and the webhook worker gets outbound # internet access for free by staying outside a VPC. @@ -13,7 +13,7 @@ locals { data "aws_caller_identity" "me" {} data "aws_route53_zone" "main" { name = var.hosted_zone } -# ── Keys (DEPLOYMENT.md → Build checklist, step 3) ───────────────────── +# ── Keys (infra/README.md → Build checklist, step 3) ───────────────────── # JWT signing: the private key never leaves KMS; JWKS publishes its public half. resource "aws_kms_key" "signing" { @@ -46,7 +46,7 @@ resource "aws_rds_cluster" "main" { cluster_identifier = "${local.prefix}-db" engine = "aurora-postgresql" engine_mode = "provisioned" - engine_version = "16.6" + engine_version = "16.15" database_name = local.db_name master_username = "substratal" manage_master_user_password = true # Secrets Manager–managed; the Data API authenticates with it @@ -428,6 +428,7 @@ locals { "cleanup" = "cron(0 3 * * ? *)" "audit-archive" = "cron(30 3 * * ? *)" "settings-projections" = "cron(0 4 * * ? *)" + "push-maintenance" = "cron(30 4 * * ? *)" } } @@ -461,7 +462,7 @@ resource "aws_scheduler_schedule" "job" { } } -# ── Alarms (DEPLOYMENT.md → Build checklist, step 8) ─────────────────── +# ── Alarms (infra/README.md → Build checklist, step 8) ─────────────────── resource "aws_sns_topic" "alarms" { name = "${local.prefix}-alarms" diff --git a/internal/billing/billing.go b/internal/billing/billing.go index a6e9022..d815186 100644 --- a/internal/billing/billing.go +++ b/internal/billing/billing.go @@ -1,5 +1,5 @@ // Package billing is the Stripe side of the Free/Standard subscription -// (DEPLOYMENT.md → Stripe Billing): Customers for subscribing Users and +// (infra/README.md → Stripe Billing): Customers for subscribing Users and // Organizations, Standard Checkout, the Customer Portal, the inbound // webhook, subscription sync, and the daily Organization quantity sync. // Third-party developers' own billing never touches it. diff --git a/internal/billing/webhook.go b/internal/billing/webhook.go index 2645ad4..f53d5d1 100644 --- a/internal/billing/webhook.go +++ b/internal/billing/webhook.go @@ -14,7 +14,7 @@ import ( "github.com/CompositeCode/substratalapps.com/internal/core" ) -// Webhook is POST /internal/stripe/webhook (DEPLOYMENT.md → Webhook +// Webhook is POST /internal/stripe/webhook (infra/README.md → Webhook // handling). It isn't Bearer-authenticated: it verifies Stripe-Signature // against the raw body. Every event id is recorded first (insert-or-skip), // so a duplicate returns 200 immediately; processing re-fetches the diff --git a/internal/core/applications.go b/internal/core/applications.go index bb578cb..f58c2da 100644 --- a/internal/core/applications.go +++ b/internal/core/applications.go @@ -26,6 +26,7 @@ type application struct { RedirectURIs []string `json:"redirect_uris"` SupportURL *string `json:"support_url"` EmailFromName *string `json:"email_from_name"` + ReceivesPlatform bool `json:"receives_platform_notifications"` SettingsSchema json.RawMessage `json:"settings_schema"` SettingsSchemaStats map[string]any `json:"settings_schema_stats"` Permissions []appPermission `json:"permissions"` @@ -64,7 +65,7 @@ func (a appRow) full(stats map[string]int) application { } return application{ID: a.ID, Slug: a.Slug, Name: a.Name, Description: a.Description, IconURL: a.IconURL, LaunchURL: a.LaunchURL, RedirectURIs: nonNil(a.RedirectURIs), SupportURL: a.SupportURL, - EmailFromName: a.EmailFromName, SettingsSchema: a.SettingsSchema, + EmailFromName: a.EmailFromName, ReceivesPlatform: a.ReceivesPlatform, SettingsSchema: a.SettingsSchema, SettingsSchemaStats: map[string]any{"stale_override_counts": stats}, Permissions: perms, AvailableAppRoles: nonNil(a.AvailableAppRoles), DefaultAppRole: a.DefaultAppRole, Visibility: a.Visibility, AccessTier: a.AccessTier, OwnerUserID: a.OwnerUserID, OwnerOrganizationID: a.OwnerOrganizationID, ReviewStatus: a.ReviewStatus, @@ -406,6 +407,7 @@ func (s *Server) ApplicationsCreate(w http.ResponseWriter, r *http.Request, para RedirectURIs []string `json:"redirect_uris"` SupportURL *string `json:"support_url"` EmailFromName *string `json:"email_from_name"` + ReceivesPlatform bool `json:"receives_platform_notifications"` SettingsSchema json.RawMessage `json:"settings_schema"` Permissions []appPermission `json:"permissions"` AvailableAppRoles []string `json:"available_app_roles"` @@ -467,14 +469,14 @@ func (s *Server) ApplicationsCreate(w http.ResponseWriter, r *http.Request, para if _, err := c.q.Exec(c.ctx, `insert into applications (id, slug, name, description, icon_url, launch_url, redirect_uris, support_url, email_from_name, settings_schema, permissions, available_app_roles, default_app_role, visibility, access_tier, owner_user_id, owner_organization_id, review_status, review_notes, - tenant_id, test_mode) + tenant_id, test_mode, receives_platform_notifications) values (:id, :slug, :name, :desc, :icon, :launch, :redir::text[], :support, :from, :schema::jsonb, - :perms::jsonb, :roles::text[], :defrole, :vis, :tier, :ou, :oo, :rs, :rn, :t, :tm)`, + :perms::jsonb, :roles::text[], :defrole, :vis, :tier, :ou, :oo, :rs, :rn, :t, :tm, :rpn)`, db.Args{"id": id, "slug": slug, "name": in.Name, "desc": in.Description, "icon": in.IconURL, "launch": in.LaunchURL, "redir": db.TextArray(in.RedirectURIs), "support": in.SupportURL, "from": in.EmailFromName, "schema": string(in.SettingsSchema), "perms": db.JSON(perms), "roles": db.TextArray(roles), "defrole": in.DefaultAppRole, "vis": visibility, "tier": accessTier, "ou": ownerUser, - "oo": ownerOrg, "rs": review, "rn": in.ReviewNotes, "t": t.ID, "tm": c.testMode()}); err != nil { + "oo": ownerOrg, "rs": review, "rn": in.ReviewNotes, "t": t.ID, "tm": c.testMode(), "rpn": in.ReceivesPlatform}); err != nil { if _, ok := db.UniqueViolation(err); ok { return httpx.Conflict("slug_taken", "An Application with this slug already exists.") } @@ -515,7 +517,8 @@ func (c *Call) seedAppRoles(appID string, names []string) error { // ── PATCH /v1/applications/{id} ──────────────────────────────────────── var ownerFields = []string{"name", "description", "icon_url", "launch_url", "redirect_uris", "support_url", - "email_from_name", "settings_schema", "permissions", "available_app_roles", "default_app_role"} + "email_from_name", "settings_schema", "permissions", "available_app_roles", "default_app_role", + "receives_platform_notifications"} var moderationFields = []string{"visibility", "access_tier", "review_status", "review_notes", "owner_user_id", "owner_organization_id"} func (s *Server) ApplicationsUpdate(w http.ResponseWriter, r *http.Request, id string, params gen.ApplicationsUpdateParams) { @@ -598,6 +601,13 @@ func (s *Server) ApplicationsUpdate(w http.ResponseWriter, r *http.Request, id s if v, present := p.String("access_tier", &f); present && v != nil { next.AccessTier = *v } + if p.Has("receives_platform_notifications") { + if p.IsNull("receives_platform_notifications") { + f.Add("receives_platform_notifications", "required").Message = "this field isn't nullable" + } else { + p.Get("receives_platform_notifications", &next.ReceivesPlatform, &f) + } + } ownerChange := p.Has("owner_user_id") || p.Has("owner_organization_id") if ownerChange { setStr("owner_user_id", &next.OwnerUserID) @@ -696,18 +706,18 @@ func (s *Server) ApplicationsUpdate(w http.ResponseWriter, r *http.Request, id s launch_url = :launch, redirect_uris = :redir::text[], support_url = :support, email_from_name = :from, settings_schema = :schema::jsonb, permissions = :perms::jsonb, available_app_roles = :roles::text[], default_app_role = :defrole, visibility = :vis, access_tier = :tier, owner_user_id = :ou, owner_organization_id = :oo, - review_status = :rs, review_notes = :rn, tenant_id = :t + review_status = :rs, review_notes = :rn, tenant_id = :t, receives_platform_notifications = :rpn where id = :id`, db.Args{"name": next.Name, "desc": next.Description, "icon": next.IconURL, "launch": next.LaunchURL, "redir": db.TextArray(next.RedirectURIs), "support": next.SupportURL, "from": next.EmailFromName, "schema": string(next.SettingsSchema), "perms": db.JSON(next.Permissions), "roles": db.TextArray(next.AvailableAppRoles), "defrole": next.DefaultAppRole, "vis": next.Visibility, "tier": next.AccessTier, "ou": next.OwnerUserID, "oo": next.OwnerOrganizationID, "rs": next.ReviewStatus, "rn": next.ReviewNotes, - "t": newTenant, "id": a.ID}); err != nil { + "t": newTenant, "id": a.ID, "rpn": next.ReceivesPlatform}); err != nil { return err } if newTenant != a.TenantID { - for _, table := range []string{"entitlements", "app_profiles", "app_settings"} { + for _, table := range []string{"entitlements", "app_profiles", "app_settings", "push_credentials", "devices", "notifications"} { if _, err := c.q.Exec(c.ctx, `update `+table+` set tenant_id = :t where application_id = :a`, db.Args{"t": newTenant, "a": a.ID}); err != nil { return err diff --git a/internal/core/auth_flows.go b/internal/core/auth_flows.go index 88eacc9..dc69215 100644 --- a/internal/core/auth_flows.go +++ b/internal/core/auth_flows.go @@ -408,6 +408,10 @@ func (s *Server) AuthVerifyEmail(w http.ResponseWriter, r *http.Request) { return err } after, _ := c.findUser("u.id = :id", db.Args{"id": u.ID}) + if err := c.platformAlert(u.ID, "security.email_changed", "Your Substratal email address was changed", + strp("Your account's email address is now "+after.Email+". If this wasn't you, contact support."), nil); err != nil { + return err + } if err := c.audit(Audit{Action: "user.email_changed", TargetType: "user", TargetID: u.ID, TargetUserID: &u.ID, Before: map[string]any{"email": u.Email}, After: map[string]any{"email": after.Email}, ActorType: "user", ActorID: u.ID}); err != nil { @@ -555,6 +559,10 @@ func (s *Server) AuthResetPassword(w http.ResponseWriter, r *http.Request) { if _, err := c.q.Exec(c.ctx, `update users set email_verified = true where id = :u`, db.Args{"u": u.ID}); err != nil { return err } + if err := c.platformAlert(u.ID, "security.password_changed", "Your Substratal password was changed", + strp("Your password was reset and every session was signed out. If this wasn't you, contact support."), nil); err != nil { + return err + } if err := c.revokeSessions(u.ID, "password_reset", ""); err != nil { return err } @@ -605,6 +613,10 @@ func (s *Server) AuthChangePassword(w http.ResponseWriter, r *http.Request, id g if _, err := c.setPassword(uid, hash); err != nil { return err } + if err := c.platformAlert(uid, "security.password_changed", "Your Substratal password was changed", + strp("Your password was changed and your other sessions were signed out. If this wasn't you, reset it now."), nil); err != nil { + return err + } if err := c.revokeSessions(uid, "password_changed", c.p.SessionID); err != nil { return err } @@ -839,6 +851,10 @@ func (s *Server) AuthDisableMfaTotp(w http.ResponseWriter, r *http.Request, id g if err := c.clearMFA(ident.ID); err != nil { return err } + if err := c.platformAlert(uid, "security.mfa_disabled", "Two-step verification is off", + strp("Two-step verification was turned off on your Substratal account. If this wasn't you, change your password."), nil); err != nil { + return err + } if err := c.audit(Audit{Action: "user.mfa_disabled", TargetType: "user", TargetID: uid, TargetUserID: &uid}); err != nil { return err } @@ -881,6 +897,10 @@ func (s *Server) AuthResetMfaTotp(w http.ResponseWriter, r *http.Request, id gen if err := c.clearMFA(ident.ID); err != nil { return err } + if err := c.platformAlert(uid, "security.mfa_disabled", "Two-step verification is off", + strp("Support reset two-step verification on your Substratal account and signed out every session."), nil); err != nil { + return err + } if err := c.revokeSessions(uid, "admin", ""); err != nil { return err } @@ -955,12 +975,11 @@ func (s *Server) AuthRevokeSession(w http.ResponseWriter, r *http.Request, id ge } reason = "admin" } - n, err := c.q.Exec(c.ctx, `update sessions set revoked_at = now(), revoked_reason = :r - where id = :s and user_id = :u and revoked_at is null`, db.Args{"r": reason, "s": sessionID, "u": uid}) + revoked, err := c.revokeSession("id = :s and user_id = :u", db.Args{"s": sessionID, "u": uid}, reason) if err != nil { return err } - if n == 0 { + if !revoked { return httpx.NotFoundResource("session") } return c.NoContent() diff --git a/internal/core/auth_session.go b/internal/core/auth_session.go index 3b8ff9e..02d0aee 100644 --- a/internal/core/auth_session.go +++ b/internal/core/auth_session.go @@ -69,6 +69,9 @@ func (c *Call) issueSession(u userRow, amr []string, identityID string) (authSes "idle": db.Time(c.now.Add(sessionIdleTTL)), "abs": db.Time(c.now.Add(sessionAbsTTL))}); err != nil { return authSession{}, err } + if err := c.alertNewSignIn(u.ID, sid, ua, c.ip); err != nil { + return authSession{}, err + } rtk := ids.Secret(ids.RefreshToken) if _, err := c.q.Exec(c.ctx, `insert into refresh_tokens (token_hash, session_id) values (:h, :s)`, db.Args{"h": ids.Hash(rtk), "s": sid}); err != nil { @@ -92,12 +95,44 @@ func (c *Call) issueSession(u userRow, amr []string, identityID string) (authSes SessionID: sid, UserID: u.ID, EmailVerified: u.EmailVerified}, nil } -// revokeSessions revokes a User's sessions (all, or all but `except`). +// revokeSessions revokes a User's sessions (all, or all but `except`), +// and deactivates the push devices they registered. Suspension and +// deletion deactivate every device the User holds, including ones an app +// key registered with no session (Push Notifications → Sessions and +// device lifecycle). func (c *Call) revokeSessions(userID, reason, except string) error { - _, err := c.q.Exec(c.ctx, `update sessions set revoked_at = now(), revoked_reason = :r - where user_id = :u and revoked_at is null and id <> :except`, + switch reason { + case "suspended": + if err := c.deactivateDevices("d.user_id = :u", db.Args{"u": userID}, "user_suspended"); err != nil { + return err + } + case "deleted": + if err := c.deactivateDevices("d.user_id = :u", db.Args{"u": userID}, "user_deleted"); err != nil { + return err + } + } + sids, err := db.All[string](c.ctx, c.q, `update sessions set revoked_at = now(), revoked_reason = :r + where user_id = :u and revoked_at is null and id <> :except returning to_jsonb(id)`, db.Args{"u": userID, "r": reason, "except": except}) - return err + if err != nil { + return err + } + return c.sessionsEnded(sids) +} + +// revokeSession revokes one session, if it's still live, and deactivates +// the push devices it registered. It reports whether anything changed. +func (c *Call) revokeSession(where string, args db.Args, reason string) (bool, error) { + a := db.Args{"r": reason} + for k, v := range args { + a[k] = v + } + sids, err := db.All[string](c.ctx, c.q, `update sessions set revoked_at = now(), revoked_reason = :r + where revoked_at is null and `+where+` returning to_jsonb(id)`, a) + if err != nil { + return false, err + } + return len(sids) > 0, c.sessionsEnded(sids) } func weakPassword(err error) error { @@ -503,8 +538,13 @@ func (c *Call) lookupRefresh(tok string) (refreshRow, userRow, error) { func (c *Call) rotateRefresh(rt refreshRow) (string, error) { if rt.RotatedAt != nil { c.CommitAnyway() - if _, err := c.q.Exec(c.ctx, `update sessions set revoked_at = now(), revoked_reason = 'refresh_token_reused' - where id = :s and revoked_at is null`, db.Args{"s": rt.SessionID}); err != nil { + // Warn the User's devices before the revocation signs them out. + if err := c.platformAlert(rt.UserID, "security.session_reuse_detected", "A session was signed out for your safety", + strp("Someone tried to reuse an old sign-in token, so that session was ended. If this wasn't you, change your password."), + map[string]string{"session_id": rt.SessionID}); err != nil { + return "", err + } + if _, err := c.revokeSession("id = :s", db.Args{"s": rt.SessionID}, "refresh_token_reused"); err != nil { return "", err } c.s.Log.Warn("security", "event", "refresh_token_reused", "user", rt.UserID, "session", rt.SessionID, @@ -586,8 +626,7 @@ func (s *Server) AuthLogout(w http.ResponseWriter, r *http.Request) { if in.AllSessions { err = c.revokeSessions(c.p.UserID, "logout", "") } else { - _, err = c.q.Exec(c.ctx, `update sessions set revoked_at = now(), revoked_reason = 'logout' - where id = :s and revoked_at is null`, db.Args{"s": c.p.SessionID}) + _, err = c.revokeSession("id = :s", db.Args{"s": c.p.SessionID}, "logout") } if err != nil { return err diff --git a/internal/core/common.go b/internal/core/common.go index 90a9f3b..62c86eb 100644 --- a/internal/core/common.go +++ b/internal/core/common.go @@ -228,6 +228,7 @@ type appRow struct { RedirectURIs []string `json:"redirect_uris"` SupportURL *string `json:"support_url"` EmailFromName *string `json:"email_from_name"` + ReceivesPlatform bool `json:"receives_platform_notifications"` SettingsSchema json.RawMessage `json:"settings_schema"` Permissions []appPermission `json:"permissions"` AvailableAppRoles []string `json:"available_app_roles"` diff --git a/internal/core/devices.go b/internal/core/devices.go new file mode 100644 index 0000000..371c217 --- /dev/null +++ b/internal/core/devices.go @@ -0,0 +1,728 @@ +package core + +import ( + "context" + "encoding/hex" + "encoding/json" + "net/http" + "net/url" + "strings" + "time" + + "github.com/CompositeCode/substratalapps.com/internal/api/gen" + "github.com/CompositeCode/substratalapps.com/internal/db" + "github.com/CompositeCode/substratalapps.com/internal/httpx" + "github.com/CompositeCode/substratalapps.com/internal/ids" + "github.com/CompositeCode/substratalapps.com/internal/push" +) + +// maxActiveDevices is the per-User, per-Application cap; a newer +// registration displaces the one registered longest ago. +const maxActiveDevices = 20 + +// device is the API view of a Device. The token is never part of it. +type device struct { + ID string `json:"id"` + UserID string `json:"user_id"` + ApplicationID string `json:"application_id"` + Provider string `json:"provider"` + Environment *string `json:"environment"` + TokenHint string `json:"token_hint"` + Name *string `json:"name"` + AppVersion *string `json:"app_version"` + SessionID *string `json:"session_id"` + Status string `json:"status"` + InactiveReason *string `json:"inactive_reason"` + LastRegisteredAt time.Time `json:"last_registered_at"` + LastDeliveryAt *time.Time `json:"last_delivery_at"` + TestMode bool `json:"test_mode"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` + version int + tokenCiphertext string +} + +// deviceJSON is a device row as JSON; it works in a select and in an +// update ... returning, wherever the table is aliased d. +const deviceJSON = `jsonb_build_object('id', d.id, 'user_id', d.user_id, 'application_id', d.application_id, + 'provider', d.provider, 'environment', d.environment, 'token_hint', d.token_hint, 'name', d.name, + 'app_version', d.app_version, 'session_id', d.session_id, 'status', d.status, 'inactive_reason', d.inactive_reason, + 'last_registered_at', d.last_registered_at, 'last_delivery_at', d.last_delivery_at, 'test_mode', d.test_mode, + 'created_at', d.created_at, 'updated_at', d.updated_at, 'version_', d.version, 'token_', d.token_ciphertext)` + +func decodeDevices(rows []json.RawMessage) ([]device, error) { + out := make([]device, 0, len(rows)) + for _, raw := range rows { + var x struct { + device + V int `json:"version_"` + T string `json:"token_"` + } + if err := json.Unmarshal(raw, &x); err != nil { + return nil, err + } + x.device.version, x.device.tokenCiphertext = x.V, x.T + out = append(out, x.device) + } + return out, nil +} + +func (c *Call) queryDevices(where string, args db.Args) ([]device, error) { + rows, err := c.q.Query(c.ctx, `select `+deviceJSON+` from devices d where `+where, args) + if err != nil { + return nil, err + } + return decodeDevices(rows) +} + +// deactivateDevices makes every active device matching where inactive for +// reason, and tells each Application through device.deactivated. +func (c *Call) deactivateDevices(where string, args db.Args, reason string) error { + a := db.Args{"reason": reason} + for k, v := range args { + a[k] = v + } + rows, err := c.q.Query(c.ctx, `update devices as d set status = 'inactive', inactive_reason = :reason, deactivated_at = now() + where d.status = 'active' and (`+where+`) returning `+deviceJSON, a) + if err != nil { + return err + } + devs, err := decodeDevices(rows) + if err != nil { + return err + } + for _, d := range devs { + app := d.ApplicationID + if err := c.emit("device.deactivated", &app, map[string]any{"device": d}); err != nil { + return err + } + } + return nil +} + +// sessionsEnded deactivates the devices registered under revoked sessions +// (API Reference → Push Notifications → Sessions and device lifecycle). +func (c *Call) sessionsEnded(sessionIDs []string) error { + if len(sessionIDs) == 0 { + return nil + } + return c.deactivateDevices("d.session_id = any(:sids::text[])", db.Args{"sids": db.TextArray(sessionIDs)}, "session_revoked") +} + +// deleteDevice removes a device and its pending deliveries, and tells the +// Application through device.deleted. +func (c *Call) deleteDevice(d device) error { + notes, err := db.All[string](c.ctx, c.q, `delete from notification_deliveries where device_id = :d and status = 'pending' + returning to_jsonb(notification_id)`, db.Args{"d": d.ID}) + if err != nil { + return err + } + if _, err := c.q.Exec(c.ctx, `delete from devices where id = :d`, db.Args{"d": d.ID}); err != nil { + return err + } + for _, n := range dedupe(notes) { + if err := c.settleNotification(n); err != nil { + return err + } + } + app := d.ApplicationID + return c.emit("device.deleted", &app, map[string]any{"device": d}) +} + +// deviceUser authorizes device access for the User in {id}: the User +// themself, the app's own key (for its own Application's devices, and only +// for Users with an access path to it), and support (users.list to read, +// users.manage to delete). It returns the User id and, for an app key, the +// Application the caller is confined to. +func (c *Call) deviceUser(id string, write, del bool) (string, *string, error) { + uid, err := c.selfID(id) + if err != nil { + return "", nil, err + } + if _, err := c.visibleUser(uid); err != nil { + return "", nil, err + } + if c.isSelf(uid) { + return uid, nil, nil + } + if app, ok := c.p.AppKey(); ok { + has, err := c.userHasAccessPath(uid, app) + if err != nil { + return "", nil, err + } + if !has { + return "", nil, httpx.NotFoundResource("user") + } + return uid, &app, nil + } + switch { + case del && c.can("users.manage"): + return uid, nil, nil + case !write && (c.can("users.list") || c.can("users.manage")): + return uid, nil, nil + case write || del: + return "", nil, httpx.Forbidden("users.manage") + } + return "", nil, httpx.Forbidden("users.list") +} + +func (c *Call) loadDevice(uid, deviceID string, app *string) (device, error) { + where, args := "d.id = :id and d.user_id = :u", db.Args{"id": deviceID, "u": uid} + if app != nil { + where += " and d.application_id = :app" + args["app"] = *app + } + ds, err := c.queryDevices(where, args) + if err != nil { + return device{}, err + } + if len(ds) == 0 { + return device{}, httpx.NotFoundResource("device") + } + return ds[0], nil +} + +// deviceToken is a validated provider token, ready to store. +type deviceToken struct { + hash, hint, plaintext string // plaintext: the token, or the subscription JSON +} + +func invalidDeviceToken(why string) error { + return httpx.E(422, "invalid_device_token", "The device token isn't valid for its provider: "+why+".") +} + +// validateToken checks token or subscription for provider. +func (c *Call) validateToken(provider string, token *string, sub *push.Subscription) (deviceToken, error) { + switch provider { + case push.APNs, push.FCM: + if sub != nil { + return deviceToken{}, invalidDeviceToken("subscription is only for web_push") + } + if token == nil || *token == "" { + var f httpx.Fields + f.Add("token", "required") + return deviceToken{}, f.Err() + } + t := strings.TrimSpace(*token) + if provider == push.APNs { + t = strings.ToLower(t) + if len(t) < 64 || len(t) > 200 || len(t)%2 != 0 { + return deviceToken{}, invalidDeviceToken("an APNs token is 64–200 hex characters") + } + if _, err := hex.DecodeString(t); err != nil { + return deviceToken{}, invalidDeviceToken("an APNs token is hex") + } + } else if len(t) > 4096 || strings.ContainsAny(t, " \t\r\n") { + return deviceToken{}, invalidDeviceToken("an FCM token is 1–4,096 characters with no whitespace") + } + return deviceToken{hash: sha256Hex([]byte(t)), hint: "…" + t[len(t)-4:], plaintext: t}, nil + case push.WebPush: + if token != nil { + return deviceToken{}, invalidDeviceToken("web_push takes subscription, not token") + } + if sub == nil || sub.Endpoint == "" { + var f httpx.Fields + f.Add("subscription", "required") + return deviceToken{}, f.Err() + } + u, err := url.Parse(sub.Endpoint) + if err != nil || u.Host == "" || u.User != nil || len(sub.Endpoint) > 2048 { + return deviceToken{}, invalidDeviceToken("subscription.endpoint isn't a URL") + } + if u.Scheme != "https" && !(AllowPrivateWebhookTargets && u.Scheme == "http") { + return deviceToken{}, invalidDeviceToken("subscription.endpoint must be https") + } + if !AllowPrivateWebhookTargets { + if err := CheckPublicHost(c.ctx, u.Hostname()); err != nil { + return deviceToken{}, invalidDeviceToken("subscription.endpoint must resolve to a public address") + } + } + if err := push.ValidateSubscriptionKeys(sub.Keys.P256dh, sub.Keys.Auth); err != nil { + return deviceToken{}, invalidDeviceToken(err.Error()) + } + raw, _ := json.Marshal(sub) + return deviceToken{hash: sha256Hex([]byte(sub.Endpoint)), hint: "…" + sub.Endpoint[len(sub.Endpoint)-4:], + plaintext: string(raw)}, nil + } + var f httpx.Fields + f.Enum("provider", push.Providers...) + return deviceToken{}, f.Err() +} + +// enforceDeviceCap deactivates the oldest active devices beyond the cap. +func (c *Call) enforceDeviceCap(uid, appID string) error { + return c.deactivateDevices(`d.id in (select id from devices where user_id = :u and application_id = :a and status = 'active' + order by last_registered_at desc, id desc offset :cap)`, + db.Args{"u": uid, "a": appID, "cap": int64(maxActiveDevices)}, "limit") +} + +func (s *Server) DevicesList(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, params gen.DevicesListParams) { + s.serve(w, r, opts{op: "devices.list"}, func(c *Call) error { + uid, app, err := c.deviceUser(id, false, false) + if err != nil { + return err + } + where, args := []string{"d.user_id = :u"}, db.Args{"u": uid} + if app != nil { + where = append(where, "d.application_id = :app") + args["app"] = *app + } + if params.ApplicationId != nil { + where = append(where, "d.application_id = :fapp") + args["fapp"] = *params.ApplicationId + } + if params.Provider != nil { + where = append(where, "d.provider = :prov") + args["prov"] = string(*params.Provider) + } + switch { + case params.Status != nil: + where = append(where, "d.status = :st") + args["st"] = string(*params.Status) + case params.IncludeInactive == nil || !*params.IncludeInactive: + where = append(where, "d.status = 'active'") + } + lim := httpx.Limit(params.Limit) + args["lim"] = int64(lim + 1) + filters := httpx.Filters(c.r) + pos, err := c.s.Cursors.Decode(deref(params.Cursor), filters) + if err != nil { + return err + } + if pos != nil { + where = append(where, "(d.created_at, d.id) < (:ps::timestamptz, :pi)") + args["ps"], args["pi"] = pos.Sort, pos.ID + } + ds, err := c.queryDevices(strings.Join(where, " and ")+" order by d.created_at desc, d.id desc limit :lim", args) + if err != nil { + return err + } + return c.OK(httpx.Paginate(c.s.Cursors, ds, lim, filters, func(x device) (string, string) { + return httpx.TimeKey(x.CreatedAt), x.ID + })) + }) +} + +func (s *Server) DevicesGet(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, deviceID string) { + s.serve(w, r, opts{op: "devices.get"}, func(c *Call) error { + uid, app, err := c.deviceUser(id, false, false) + if err != nil { + return err + } + d, err := c.loadDevice(uid, deviceID, app) + if err != nil { + return err + } + c.Versioned(d.version) + return c.OK(d) + }) +} + +func (s *Server) DevicesRegister(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe) { + s.serve(w, r, opts{op: "devices.register"}, func(c *Call) error { + uid, keyApp, err := c.deviceUser(id, true, false) + if err != nil { + return err + } + var in struct { + ApplicationID string `json:"application_id"` + Provider string `json:"provider"` + Token *string `json:"token"` + Subscription *push.Subscription `json:"subscription"` + Environment *string `json:"environment"` + Name *string `json:"name"` + AppVersion *string `json:"app_version"` + } + if err := c.Decode(&in); err != nil { + return err + } + var f httpx.Fields + if in.ApplicationID == "" { + f.Add("application_id", "required") + } + if in.Provider == "" { + f.Add("provider", "required") + } else if !validProvider(in.Provider) { + f.Enum("provider", push.Providers...) + } + if in.Provider == push.APNs { + if in.Environment == nil { + f.Add("environment", "required") + } else if *in.Environment != "production" && *in.Environment != "sandbox" { + f.Enum("environment", "production", "sandbox") + } + } else if in.Environment != nil { + f.Add("environment", "undeclared_key").Message = "environment is only for apns" + } + labels(&f, in.Name, in.AppVersion) + if err := f.Err(); err != nil { + return err + } + if keyApp != nil && *keyApp != in.ApplicationID { + return httpx.Forbidden("") + } + a, err := c.loadApp(in.ApplicationID) + if err == db.ErrNotFound { + f.Add("application_id", "unknown_value") + return f.Err() + } + if err != nil { + return err + } + t, err := c.loadTenant(a.TenantID) + if err != nil { + return err + } + if err := tenantWritable(t); err != nil { + return err + } + if err := c.requireAccess(uid, a.ID); err != nil { + return err + } + tok, err := c.validateToken(in.Provider, in.Token, in.Subscription) + if err != nil { + return err + } + var session any + if c.isSelf(uid) { + session = c.p.SessionID + } + existing, err := c.queryDevices("d.application_id = :a and d.provider = :p and d.token_hash = :h", + db.Args{"a": a.ID, "p": in.Provider, "h": tok.hash}) + if err != nil { + return err + } + enc, err := c.s.Encrypter.Encrypt(c.ctx, tok.plaintext) + if err != nil { + return err + } + if len(existing) == 1 && existing[0].UserID == uid { + // A refresh of the caller's own registration. + before := existing[0] + if _, err := c.q.Exec(c.ctx, `update devices set name = :n, app_version = :v, environment = :env, + session_id = :s, token_ciphertext = :tok, last_registered_at = now(), + status = 'active', inactive_reason = null, deactivated_at = null + where id = :id`, db.Args{"n": in.Name, "v": in.AppVersion, "env": in.Environment, "s": session, + "tok": enc, "id": before.ID}); err != nil { + return err + } + after, err := c.loadDevice(uid, before.ID, nil) + if err != nil { + return err + } + if before.Status == "inactive" { + app := a.ID + if err := c.emit("device.registered", &app, map[string]any{"device": after}); err != nil { + return err + } + if err := c.enforceDeviceCap(uid, a.ID); err != nil { + return err + } + } + c.Versioned(after.version) + return c.OK(after) + } + if len(existing) == 1 { + // Someone else signed in on this install: the token is theirs + // now, and the previous User's history doesn't come with it. + if err := c.deleteDevice(existing[0]); err != nil { + return err + } + } + devID := ids.New(ids.Device) + if _, err := c.q.Exec(c.ctx, `insert into devices (id, user_id, application_id, tenant_id, session_id, provider, + environment, token_hash, token_ciphertext, token_hint, name, app_version, test_mode) + values (:id, :u, :a, :t, :s, :p, :env, :h, :tok, :hint, :n, :v, :tm)`, + db.Args{"id": devID, "u": uid, "a": a.ID, "t": a.TenantID, "s": session, "p": in.Provider, + "env": in.Environment, "h": tok.hash, "tok": enc, "hint": tok.hint, "n": in.Name, "v": in.AppVersion, + "tm": c.testMode()}); err != nil { + return err + } + out, err := c.loadDevice(uid, devID, nil) + if err != nil { + return err + } + app := a.ID + if err := c.emit("device.registered", &app, map[string]any{"device": out}); err != nil { + return err + } + if err := c.enforceDeviceCap(uid, a.ID); err != nil { + return err + } + c.Versioned(out.version) + return c.Created(out) + }) +} + +func labels(f *httpx.Fields, name, appVersion *string) { + if name != nil && len([]rune(*name)) > 100 { + f.Max("name", "too_long", 100) + } + if appVersion != nil && len([]rune(*appVersion)) > 50 { + f.Max("app_version", "too_long", 50) + } +} + +// requireAccess is 403 entitlement_required unless allow() passes for the +// User and Application right now. +func (c *Call) requireAccess(uid, appID string) error { + res, err := c.resolveOne(uid, appID) + if err != nil { + return err + } + if !res.Allowed { + return httpx.E(403, "entitlement_required", "This user has no active entitlement to "+appID+"."). + With("application_id", appID).With("entitlement_status", res.Res.Status) + } + return nil +} + +func (s *Server) DevicesUpdate(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, deviceID string, params gen.DevicesUpdateParams) { + s.serve(w, r, opts{op: "devices.update"}, func(c *Call) error { + uid, app, err := c.deviceUser(id, true, false) + if err != nil { + return err + } + before, err := c.loadDevice(uid, deviceID, app) + if err != nil { + return err + } + if err := httpx.CheckIfMatch(c.r, before.version); err != nil { + return err + } + p, err := c.Patch() + if err != nil { + return err + } + if err := p.RejectReadOnly("id", "user_id", "application_id", "provider", "session_id", "status", "inactive_reason", + "token_hint", "last_registered_at", "last_delivery_at", "test_mode", "created_at", "updated_at"); err != nil { + return err + } + var in struct { + Token *string `json:"token"` + Subscription *push.Subscription `json:"subscription"` + Environment *string `json:"environment"` + Name *string `json:"name"` + AppVersion *string `json:"app_version"` + } + if err := c.Decode(&in); err != nil { + return err + } + var f httpx.Fields + next := before + if p.Has("name") { + next.Name = in.Name + } + if p.Has("app_version") { + next.AppVersion = in.AppVersion + } + labels(&f, next.Name, next.AppVersion) + if p.Has("environment") { + switch { + case before.Provider != push.APNs: + f.Add("environment", "undeclared_key").Message = "environment is only for apns" + case in.Environment == nil || (*in.Environment != "production" && *in.Environment != "sandbox"): + f.Enum("environment", "production", "sandbox") + default: + next.Environment = in.Environment + } + } + if err := f.Err(); err != nil { + return err + } + args := db.Args{"id": before.ID, "n": next.Name, "v": next.AppVersion, "env": next.Environment, + "tok": nil, "h": nil, "hint": nil} + if p.Has("token") || p.Has("subscription") { + tok, err := c.validateToken(before.Provider, in.Token, in.Subscription) + if err != nil { + return err + } + // One token, one Device: another registration of it goes. + others, err := c.queryDevices("d.application_id = :a and d.provider = :p and d.token_hash = :h and d.id <> :id", + db.Args{"a": before.ApplicationID, "p": before.Provider, "h": tok.hash, "id": before.ID}) + if err != nil { + return err + } + for _, o := range others { + if err := c.deleteDevice(o); err != nil { + return err + } + } + enc, err := c.s.Encrypter.Encrypt(c.ctx, tok.plaintext) + if err != nil { + return err + } + args["tok"], args["h"], args["hint"] = enc, tok.hash, tok.hint + } + if _, err := c.q.Exec(c.ctx, `update devices set name = :n, app_version = :v, environment = :env, + token_ciphertext = coalesce(:tok, token_ciphertext), token_hash = coalesce(:h, token_hash), + token_hint = coalesce(:hint, token_hint), + last_registered_at = case when :tok::text is null then last_registered_at else now() end, + status = case when :tok::text is null then status else 'active' end, + inactive_reason = case when :tok::text is null then inactive_reason end, + deactivated_at = case when :tok::text is null then deactivated_at end + where id = :id`, args); err != nil { + return err + } + after, err := c.loadDevice(uid, before.ID, nil) + if err != nil { + return err + } + if before.Status == "inactive" && after.Status == "active" { + a := after.ApplicationID + if err := c.emit("device.registered", &a, map[string]any{"device": after}); err != nil { + return err + } + if err := c.enforceDeviceCap(uid, after.ApplicationID); err != nil { + return err + } + } + c.Versioned(after.version) + return c.OK(after) + }) +} + +func (s *Server) DevicesDelete(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, deviceID string) { + s.serve(w, r, opts{op: "devices.delete"}, func(c *Call) error { + uid, app, err := c.deviceUser(id, false, true) + if err != nil { + return err + } + d, err := c.loadDevice(uid, deviceID, app) + if err != nil { + return err + } + if err := c.deleteDevice(d); err != nil { + return err + } + if app == nil && !c.isSelf(uid) { + devApp := d.ApplicationID + if err := c.audit(Audit{Action: "device.deleted", TargetType: "device", TargetID: d.ID, TargetUserID: &uid, + ApplicationID: &devApp, Before: d}); err != nil { + return err + } + } + return c.NoContent() + }) +} + +// testNotificationTitle is a device test's default title. +const testNotificationTitle = "Test notification from Substratal" + +func (s *Server) DevicesSendTest(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, deviceID string) { + s.serve(w, r, opts{op: "devices.sendTest"}, func(c *Call) error { + uid, app, err := c.deviceUser(id, true, false) + if err != nil { + return err + } + d, err := c.loadDevice(uid, deviceID, app) + if err != nil { + return err + } + var in struct { + Title *string `json:"title"` + Body *string `json:"body"` + } + if err := c.Decode(&in); err != nil { + return err + } + var f httpx.Fields + title := testNotificationTitle + if in.Title != nil { + title = *in.Title + if n := len([]rune(title)); n < 1 || n > 120 { + f.Max("title", "too_long", 120) + } + } + if in.Body != nil && len([]rune(*in.Body)) > 1000 { + f.Max("body", "too_long", 1000) + } + if err := f.Err(); err != nil { + return err + } + if err := c.hit("push-test:"+d.ID, perHour(10), false); err != nil { + return err + } + cred, err := c.loadCredential(d.ApplicationID, d.Provider) + if err != nil { + if isAPIError(err) { + return httpx.Conflict("push_not_configured", "The Application has no "+d.Provider+" credential.") + } + return err + } + pc, err := c.s.decryptCredential(c.ctx, cred) + if err != nil { + return err + } + target, err := c.s.decryptTarget(c.ctx, d) + if err != nil { + return err + } + appID := d.ApplicationID + msg := push.Message{Credential: pc, Target: target, Notification: push.Notification{ + ApplicationID: &appID, Source: "application", Category: "test", Priority: "normal", + Title: title, Body: in.Body, TTL: 0}} + ctx, cancel := context.WithTimeout(c.ctx, 10*time.Second) + res := c.s.pushSender().Send(ctx, msg) + cancel() + if err := c.applyOutcome(d, cred.ID, res); err != nil { + return err + } + status := "failed" + if res.Outcome == push.Sent { + status = "sent" + } + var errText *string + if res.Reason != "" && res.Outcome != push.Sent { + errText = &res.Reason + } + var provStatus *int + if res.Status != 0 { + provStatus = &res.Status + } + return c.OK(map[string]any{"device_id": d.ID, "status": status, "provider_status": provStatus, + "error": errText, "duration_ms": res.Duration.Milliseconds()}) + }) +} + +// decryptTarget rebuilds a device's provider target. +func (s *Server) decryptTarget(ctx context.Context, d device) (push.Target, error) { + plain, err := s.Encrypter.Decrypt(ctx, d.tokenCiphertext) + if err != nil { + return push.Target{}, err + } + t := push.Target{Provider: d.Provider, Environment: deref(d.Environment)} + if d.Provider == push.WebPush { + var sub push.Subscription + if err := json.Unmarshal([]byte(plain), &sub); err != nil { + return t, err + } + t.Subscription = &sub + } else { + t.Token = plain + } + return t, nil +} + +// applyOutcome records what a provider's answer means for the device and +// the credential: a gone token deactivates the device, a rejected +// credential is marked failed, and a success stamps both. +func (c *Call) applyOutcome(d device, credID string, res push.Result) error { + switch res.Outcome { + case push.Sent: + if _, err := c.q.Exec(c.ctx, `update devices set last_delivery_at = now() where id = :d`, db.Args{"d": d.ID}); err != nil { + return err + } + _, err := c.q.Exec(c.ctx, `update push_credentials set last_used_at = now() where id = :id`, db.Args{"id": credID}) + return err + case push.TokenGone: + return c.deactivateDevices("d.id = :id", db.Args{"id": d.ID}, "unregistered") + case push.CredentialRejected: + reason := res.Reason + if reason == "" { + reason = "rejected" + } + return c.credentialRejected(credID, reason) + } + return nil +} diff --git a/internal/core/export.go b/internal/core/export.go index 19c30de..cd0dbe8 100644 --- a/internal/core/export.go +++ b/internal/core/export.go @@ -97,6 +97,16 @@ func (s *Server) UsersExport(w http.ResponseWriter, r *http.Request, id gen.User return err } out["audit_events"] = events + devs, err := c.queryDevices("d.user_id = :u order by d.created_at", db.Args{"u": uid}) + if err != nil { + return err + } + out["devices"] = devs + notes, err := c.queryNotifications("n.user_id = :u order by n.created_at desc, n.id desc", db.Args{"u": uid}) + if err != nil { + return err + } + out["notifications"] = notes return c.OK(out) }) } diff --git a/internal/core/notifications.go b/internal/core/notifications.go index 4e83b31..ec5406c 100644 --- a/internal/core/notifications.go +++ b/internal/core/notifications.go @@ -1,114 +1,570 @@ package core import ( + "encoding/json" "net/http" + "net/url" + "regexp" + "strings" + "time" + "unicode/utf8" "github.com/CompositeCode/substratalapps.com/internal/api/gen" + "github.com/CompositeCode/substratalapps.com/internal/db" "github.com/CompositeCode/substratalapps.com/internal/httpx" + "github.com/CompositeCode/substratalapps.com/internal/ids" + "github.com/CompositeCode/substratalapps.com/internal/push" ) -// Push notifications (API Reference → Push Notifications) are specified -// but not built yet. Every operation runs the shared pipeline, so -// authentication, rate limits, and the destructive-operation gate already -// apply, and then returns 501 not_implemented, the same treatment as the -// SSO callback. Idempotency stays off until the real handlers land, so a -// stored 501 can never be replayed after they do. +// recipientHourlyLimit caps what one Application can send one User in a +// rolling hour (NFR → Rate limiting). Platform alerts don't count. +const recipientHourlyLimit = 60 -func notImplemented(c *Call) error { - return httpx.E(501, "not_implemented", "Push notifications aren't available yet.") +// notification is the API view of a Notification. +type notification struct { + ID string `json:"id"` + UserID string `json:"user_id"` + ApplicationID *string `json:"application_id"` + Source string `json:"source"` + Category string `json:"category"` + Priority string `json:"priority"` + Title string `json:"title"` + Body *string `json:"body"` + Data map[string]string `json:"data"` + ActionURL *string `json:"action_url"` + Badge *int `json:"badge"` + Sound *string `json:"sound"` + CollapseKey *string `json:"collapse_key"` + TTLSeconds int `json:"ttl_seconds"` + Push json.RawMessage `json:"push"` + State string `json:"state"` + ReadAt *time.Time `json:"read_at"` + ArchivedAt *time.Time `json:"archived_at"` + Sender json.RawMessage `json:"sender"` + TestMode bool `json:"test_mode"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` + Deliveries *[]notificationDelivery `json:"deliveries,omitempty"` + version int } -// PushCredentialsList is reserved until push notifications ship. -func (s *Server) PushCredentialsList(w http.ResponseWriter, r *http.Request, id string) { - s.serve(w, r, opts{op: "pushCredentials.list"}, notImplemented) +type notificationDelivery struct { + ID string `json:"id"` + DeviceID *string `json:"device_id"` + Provider string `json:"provider"` + Status string `json:"status"` + Attempts int `json:"attempts"` + Error *string `json:"error"` + SentAt *time.Time `json:"sent_at"` } -// PushCredentialsCreate is reserved until push notifications ship. -func (s *Server) PushCredentialsCreate(w http.ResponseWriter, r *http.Request, id string, params gen.PushCredentialsCreateParams) { - s.serve(w, r, opts{op: "pushCredentials.create", userOnly: true}, notImplemented) -} - -// PushCredentialsGet is reserved until push notifications ship. -func (s *Server) PushCredentialsGet(w http.ResponseWriter, r *http.Request, id string, provider gen.PushProvider) { - s.serve(w, r, opts{op: "pushCredentials.get"}, notImplemented) -} +const notificationSQL = `select jsonb_build_object('id', n.id, 'user_id', n.user_id, 'application_id', n.application_id, + 'source', n.source, 'category', n.category, 'priority', n.priority, 'title', n.title, 'body', n.body, 'data', n.data, + 'action_url', n.action_url, 'badge', n.badge, 'sound', n.sound, 'collapse_key', n.collapse_key, + 'ttl_seconds', n.ttl_seconds, 'push', ` + notificationPushJSON + `, + 'state', case when n.archived_at is not null then 'archived' when n.read_at is not null then 'read' else 'unread' end, + 'read_at', n.read_at, 'archived_at', n.archived_at, + 'sender', jsonb_build_object('type', n.sender_type, 'id', n.sender_id), + 'test_mode', n.test_mode, 'created_at', n.created_at, 'updated_at', n.updated_at, 'version_', n.version) + from notifications n` -// PushCredentialsUpdate is reserved until push notifications ship. -func (s *Server) PushCredentialsUpdate(w http.ResponseWriter, r *http.Request, id string, provider gen.PushProvider, params gen.PushCredentialsUpdateParams) { - s.serve(w, r, opts{op: "pushCredentials.update", userOnly: true}, notImplemented) +func (c *Call) queryNotifications(where string, args db.Args) ([]notification, error) { + rows, err := c.q.Query(c.ctx, notificationSQL+" where "+where, args) + if err != nil { + return nil, err + } + out := make([]notification, 0, len(rows)) + for _, raw := range rows { + var x struct { + notification + V int `json:"version_"` + } + if err := json.Unmarshal(raw, &x); err != nil { + return nil, err + } + x.notification.version = x.V + if x.notification.Data == nil { + x.notification.Data = map[string]string{} + } + out = append(out, x.notification) + } + return out, nil } -// PushCredentialsDelete is reserved until push notifications ship. -func (s *Server) PushCredentialsDelete(w http.ResponseWriter, r *http.Request, id string, provider gen.PushProvider) { - s.serve(w, r, opts{op: "pushCredentials.delete", userOnly: true}, func(c *Call) error { - if err := c.destructiveAlways(); err != nil { - return err +// inboxUser authorizes inbox access for the User in {id}: the User +// themself (everything), the app's own key (its own Application's +// notifications, never platform alerts, for Users with an access path to +// it), and support with users.list to read. +func (c *Call) inboxUser(id string, write bool) (string, *string, error) { + uid, err := c.selfID(id) + if err != nil { + return "", nil, err + } + if _, err := c.visibleUser(uid); err != nil { + return "", nil, err + } + if c.isSelf(uid) { + return uid, nil, nil + } + if app, ok := c.p.AppKey(); ok { + has, err := c.userHasAccessPath(uid, app) + if err != nil { + return "", nil, err } - return notImplemented(c) - }) + if !has { + return "", nil, httpx.NotFoundResource("user") + } + return uid, &app, nil + } + if !write && (c.can("users.list") || c.can("users.manage")) { + return uid, nil, nil + } + if write { + return "", nil, httpx.Forbidden("") + } + return "", nil, httpx.Forbidden("users.list") } -// DevicesList is reserved until push notifications ship. -func (s *Server) DevicesList(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, params gen.DevicesListParams) { - s.serve(w, r, opts{op: "devices.list"}, notImplemented) +func (c *Call) loadNotification(uid, notificationID string, app *string) (notification, error) { + where, args := "n.id = :id and n.user_id = :u", db.Args{"id": notificationID, "u": uid} + if app != nil { + where += " and n.application_id = :app" + args["app"] = *app + } + ns, err := c.queryNotifications(where, args) + if err != nil { + return notification{}, err + } + if len(ns) == 0 { + return notification{}, httpx.NotFoundResource("notification") + } + return ns[0], nil } -// DevicesRegister is reserved until push notifications ship. -func (s *Server) DevicesRegister(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe) { - s.serve(w, r, opts{op: "devices.register"}, notImplemented) -} +var ( + categoryRe = regexp.MustCompile(`^[a-z][a-z0-9_]*(\.[a-z0-9_]+)*$`) + collapseKeyRe = regexp.MustCompile(`^[A-Za-z0-9._-]+$`) + // A private-use URI scheme for native apps is reverse-DNS + // (com.example.app:/path), the same rule as redirect_uris. + privateSchemeRe = regexp.MustCompile(`^[a-z][a-z0-9+-]*(\.[a-z0-9+-]+)+$`) +) -// DevicesGet is reserved until push notifications ship. -func (s *Server) DevicesGet(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, deviceID string) { - s.serve(w, r, opts{op: "devices.get"}, notImplemented) +func reservedCategory(cat string) bool { + return strings.HasPrefix(cat, "security.") || strings.HasPrefix(cat, "account.") || strings.HasPrefix(cat, "billing.") } -// DevicesUpdate is reserved until push notifications ship. -func (s *Server) DevicesUpdate(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, deviceID string, params gen.DevicesUpdateParams) { - s.serve(w, r, opts{op: "devices.update"}, notImplemented) +func validActionURL(raw string) bool { + u, err := url.Parse(raw) + if err != nil { + return false + } + if u.Scheme == "https" { + return u.Host != "" + } + return privateSchemeRe.MatchString(u.Scheme) } -// DevicesDelete is reserved until push notifications ship. -func (s *Server) DevicesDelete(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, deviceID string) { - s.serve(w, r, opts{op: "devices.delete"}, func(c *Call) error { - if err := c.destructiveAlways(); err != nil { +func (s *Server) NotificationsSend(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, params gen.NotificationsSendParams) { + s.serve(w, r, opts{op: "notifications.send", idem: idemOptional}, func(c *Call) error { + uid, err := c.selfID(id) + if err != nil { + return err + } + raw, err := c.Patch() + if err != nil { + return err + } + var in struct { + Title *string `json:"title"` + Body *string `json:"body"` + Category *string `json:"category"` + Priority *string `json:"priority"` + Push *bool `json:"push"` + Data map[string]string `json:"data"` + ActionURL *string `json:"action_url"` + Badge *int `json:"badge"` + Sound *string `json:"sound"` + CollapseKey *string `json:"collapse_key"` + TTLSeconds *int `json:"ttl_seconds"` + ApplicationID *string `json:"application_id"` + } + if err := c.Decode(&in); err != nil { + return err + } + // Who's sending, for which Application. + var appID string + var f httpx.Fields + if keyApp, ok := c.p.AppKey(); ok { + if !c.appKeyCan("notifications.send", keyApp) { + return httpx.Forbidden("notifications.send") + } + if in.ApplicationID != nil && *in.ApplicationID != keyApp { + return httpx.Forbidden("") + } + appID = keyApp + } else if c.can("notifications.send") { + if in.ApplicationID == nil || *in.ApplicationID == "" { + f.Add("application_id", "required") + } else { + appID = *in.ApplicationID + } + } else { + return httpx.Forbidden("notifications.send") + } + if _, err := c.visibleUser(uid); err != nil { + return err + } + sp := notificationSpec{UserID: uid, Priority: "normal", TTL: 86400, Push: true, Data: in.Data, + Body: in.Body, ActionURL: in.ActionURL, Badge: in.Badge, CollapseKey: in.CollapseKey, + SenderType: c.p.Type, SenderID: c.p.ID} + switch { + case in.Title == nil || *in.Title == "": + f.Add("title", "required") + case utf8.RuneCountInString(*in.Title) > 120: + f.Max("title", "too_long", 120) + default: + sp.Title = *in.Title + } + if in.Body != nil && utf8.RuneCountInString(*in.Body) > 1000 { + f.Max("body", "too_long", 1000) + } + reserved := false + switch { + case in.Category == nil || *in.Category == "": + f.Add("category", "required") + case len(*in.Category) > 64: + f.Max("category", "too_long", 64) + case !categoryRe.MatchString(*in.Category): + f.Add("category", "invalid_format").Pattern = categoryRe.String() + case reservedCategory(*in.Category): + reserved = true + default: + sp.Category = *in.Category + } + if in.Priority != nil { + if *in.Priority != "normal" && *in.Priority != "high" { + f.Enum("priority", "normal", "high") + } else { + sp.Priority = *in.Priority + } + } + if in.Push != nil { + sp.Push = *in.Push + } + if len(in.Data) > 20 { + f.Max("data", "too_long", 20) + } + if _, ok := in.Data["substratal"]; ok { + f.Add("data.substratal", "unknown_value").Message = "the key substratal is reserved" + } + if b, _ := json.Marshal(in.Data); len(b) > 2048 { + f.Max("data", "too_long", 2048) + } + if in.ActionURL != nil && (len(*in.ActionURL) > 2048 || !validActionURL(*in.ActionURL)) { + f.Add("action_url", "invalid_format").Allowed = []string{"https", "reverse-DNS private-use scheme"} + } + if in.Badge != nil && (*in.Badge < 0 || *in.Badge > 99999) { + f.Add("badge", "out_of_range").Max = ip(99999) + } + if raw.Has("sound") { + if in.Sound != nil && *in.Sound != "default" { + f.Enum("sound", "default") + } + sp.Sound = in.Sound + } else if sp.Priority == "high" { + sp.Sound = strp("default") + } + if in.CollapseKey != nil && (len(*in.CollapseKey) > 64 || !collapseKeyRe.MatchString(*in.CollapseKey)) { + f.Add("collapse_key", "invalid_format").Pattern = collapseKeyRe.String() + } + if in.TTLSeconds != nil { + if *in.TTLSeconds < 0 || *in.TTLSeconds > 2419200 { + f.Add("ttl_seconds", "out_of_range").Max = ip(2419200) + } else { + sp.TTL = *in.TTLSeconds + } + } + if err := f.Err(); err != nil { + return err + } + if reserved { + return httpx.E(422, "reserved_category", "Categories under security., account., and billing. are reserved for platform alerts."). + With("category", *in.Category) + } + a, err := c.loadApp(appID) + if err == db.ErrNotFound { + f.Add("application_id", "unknown_value") + return f.Err() + } + if err != nil { return err } - return notImplemented(c) + sp.AppID = &a.ID + t, err := c.loadTenant(a.TenantID) + if err != nil { + return err + } + if err := tenantWritable(t); err != nil { + return err + } + if err := c.requireAccess(uid, a.ID); err != nil { + return err + } + if !c.s.DisableRateLimits { + var n int + if err := db.One(c.ctx, c.q, &n, `select to_jsonb(count(*)) from notifications + where user_id = :u and application_id = :a and created_at > now() - interval '1 hour'`, + db.Args{"u": uid, "a": a.ID}); err != nil { + return err + } + if n >= recipientHourlyLimit { + return httpx.E(429, "rate_limited", "This User has received the hourly limit of notifications from this Application."). + With("scope", "recipient").RetryAfter(60) + } + } + nid := ids.New(ids.Notification) + if size := push.Size(sp.pushNotification(nid)); size > push.MaxPayload { + return httpx.E(422, "notification_too_large", "The push payload would exceed 4 KB."). + With("bytes", size).With("max", push.MaxPayload) + } + if err := c.createNotification(nid, sp); err != nil { + return err + } + out, err := c.loadNotification(uid, nid, nil) + if err != nil { + return err + } + c.Versioned(out.version) + return c.Created(out) }) } -// DevicesSendTest is reserved until push notifications ship. -func (s *Server) DevicesSendTest(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, deviceID string) { - s.serve(w, r, opts{op: "devices.sendTest"}, notImplemented) -} - -// NotificationsList is reserved until push notifications ship. func (s *Server) NotificationsList(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, params gen.NotificationsListParams) { - s.serve(w, r, opts{op: "notifications.list"}, notImplemented) + s.serve(w, r, opts{op: "notifications.list"}, func(c *Call) error { + uid, app, err := c.inboxUser(id, false) + if err != nil { + return err + } + where, args := []string{"n.user_id = :u"}, db.Args{"u": uid} + if app != nil { + where = append(where, "n.application_id = :app") + args["app"] = *app + } + if params.State != nil { + switch *params.State { + case gen.Unread: + where = append(where, "n.read_at is null and n.archived_at is null") + case gen.Read: + where = append(where, "n.read_at is not null and n.archived_at is null") + case gen.Archived: + where = append(where, "n.archived_at is not null") + } + } else if params.IncludeInactive == nil || !*params.IncludeInactive { + where = append(where, "n.archived_at is null") + } + if params.ApplicationId != nil { + where = append(where, "n.application_id = :fapp") + args["fapp"] = *params.ApplicationId + } + if params.Source != nil { + where = append(where, "n.source = :src") + args["src"] = string(*params.Source) + } + if params.Category != nil { + where = append(where, "n.category = :cat") + args["cat"] = *params.Category + } + if params.Priority != nil { + where = append(where, "n.priority = :pri") + args["pri"] = string(*params.Priority) + } + if params.CreatedAfter != nil { + where = append(where, "n.created_at > :after::timestamptz") + args["after"] = db.Time(*params.CreatedAfter) + } + if params.CreatedBefore != nil { + where = append(where, "n.created_at < :before::timestamptz") + args["before"] = db.Time(*params.CreatedBefore) + } + lim := httpx.Limit(params.Limit) + args["lim"] = int64(lim + 1) + filters := httpx.Filters(c.r) + pos, err := c.s.Cursors.Decode(deref(params.Cursor), filters) + if err != nil { + return err + } + if pos != nil { + where = append(where, "(n.created_at, n.id) < (:ps::timestamptz, :pi)") + args["ps"], args["pi"] = pos.Sort, pos.ID + } + ns, err := c.queryNotifications(strings.Join(where, " and ")+" order by n.created_at desc, n.id desc limit :lim", args) + if err != nil { + return err + } + return c.OK(httpx.Paginate(c.s.Cursors, ns, lim, filters, func(x notification) (string, string) { + return httpx.TimeKey(x.CreatedAt), x.ID + })) + }) } -// NotificationsSend is reserved until push notifications ship. -func (s *Server) NotificationsSend(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, params gen.NotificationsSendParams) { - s.serve(w, r, opts{op: "notifications.send"}, notImplemented) +func (s *Server) NotificationsGet(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, notificationID string) { + s.serve(w, r, opts{op: "notifications.get"}, func(c *Call) error { + uid, app, err := c.inboxUser(id, false) + if err != nil { + return err + } + n, err := c.loadNotification(uid, notificationID, app) + if err != nil { + return err + } + ds, err := db.All[notificationDelivery](c.ctx, c.q, `select jsonb_build_object('id', id, 'device_id', device_id, + 'provider', provider, 'status', status, 'attempts', attempts, 'error', error, 'sent_at', sent_at) + from notification_deliveries where notification_id = :n order by created_at, id`, db.Args{"n": n.ID}) + if err != nil { + return err + } + n.Deliveries = &ds + c.Versioned(n.version) + return c.OK(n) + }) } -// NotificationsUnreadCount is reserved until push notifications ship. -func (s *Server) NotificationsUnreadCount(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, params gen.NotificationsUnreadCountParams) { - s.serve(w, r, opts{op: "notifications.unreadCount"}, notImplemented) +func (s *Server) NotificationsUpdate(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, notificationID string, params gen.NotificationsUpdateParams) { + s.serve(w, r, opts{op: "notifications.update"}, func(c *Call) error { + uid, app, err := c.inboxUser(id, true) + if err != nil { + return err + } + before, err := c.loadNotification(uid, notificationID, app) + if err != nil { + return err + } + if err := httpx.CheckIfMatch(c.r, before.version); err != nil { + return err + } + p, err := c.Patch() + if err != nil { + return err + } + for k := range p { + if k != "state" { + return httpx.ReadOnly(k) + } + } + var state string + var f httpx.Fields + if !p.Get("state", &state, &f) { + if err := f.Err(); err != nil { + return err + } + f.Add("state", "required") + return f.Err() + } + var set string + switch state { + case "read": + set = "read_at = coalesce(read_at, now()), archived_at = null" + case "unread": + set = "read_at = null, archived_at = null" + case "archived": + set = "archived_at = coalesce(archived_at, now()), read_at = coalesce(read_at, now())" + default: + f.Enum("state", "unread", "read", "archived") + return f.Err() + } + if _, err := c.q.Exec(c.ctx, `update notifications set `+set+` where id = :id`, db.Args{"id": before.ID}); err != nil { + return err + } + after, err := c.loadNotification(uid, before.ID, nil) + if err != nil { + return err + } + c.Versioned(after.version) + return c.OK(after) + }) } -// NotificationsMarkRead is reserved until push notifications ship. func (s *Server) NotificationsMarkRead(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe) { - s.serve(w, r, opts{op: "notifications.markRead"}, notImplemented) -} - -// NotificationsGet is reserved until push notifications ship. -func (s *Server) NotificationsGet(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, notificationID string) { - s.serve(w, r, opts{op: "notifications.get"}, notImplemented) + s.serve(w, r, opts{op: "notifications.markRead"}, func(c *Call) error { + uid, app, err := c.inboxUser(id, true) + if err != nil { + return err + } + var in struct { + ApplicationID *string `json:"application_id"` + Source *string `json:"source"` + Category *string `json:"category"` + Before *time.Time `json:"before"` + } + if err := c.Decode(&in); err != nil { + return err + } + where, args := []string{"user_id = :u", "read_at is null", "archived_at is null"}, db.Args{"u": uid} + if app != nil { + where = append(where, "application_id = :app") + args["app"] = *app + } + if in.ApplicationID != nil { + where = append(where, "application_id = :fapp") + args["fapp"] = *in.ApplicationID + } + if in.Source != nil { + if *in.Source != "application" && *in.Source != "platform" { + var f httpx.Fields + f.Enum("source", "application", "platform") + return f.Err() + } + where = append(where, "source = :src") + args["src"] = *in.Source + } + if in.Category != nil { + where = append(where, "category = :cat") + args["cat"] = *in.Category + } + if in.Before != nil { + where = append(where, "created_at <= :before::timestamptz") + args["before"] = db.Time(*in.Before) + } + n, err := c.q.Exec(c.ctx, `update notifications set read_at = now() where `+strings.Join(where, " and "), args) + if err != nil { + return err + } + return c.OK(map[string]any{"updated_count": n}) + }) } -// NotificationsUpdate is reserved until push notifications ship. -func (s *Server) NotificationsUpdate(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, notificationID string, params gen.NotificationsUpdateParams) { - s.serve(w, r, opts{op: "notifications.update"}, notImplemented) +func (s *Server) NotificationsUnreadCount(w http.ResponseWriter, r *http.Request, id gen.UserIdOrMe, params gen.NotificationsUnreadCountParams) { + s.serve(w, r, opts{op: "notifications.unreadCount"}, func(c *Call) error { + uid, app, err := c.inboxUser(id, true) + if err != nil { + return err + } + where, args := []string{"user_id = :u", "read_at is null", "archived_at is null"}, db.Args{"u": uid} + if app != nil { + where = append(where, "application_id = :app") + args["app"] = *app + } + if params.ApplicationId != nil { + where = append(where, "application_id = :fapp") + args["fapp"] = *params.ApplicationId + } + if params.Source != nil { + where = append(where, "source = :src") + args["src"] = string(*params.Source) + } + type group struct { + ApplicationID *string `json:"application_id"` + UnreadCount int `json:"unread_count"` + } + groups, err := db.All[group](c.ctx, c.q, `select jsonb_build_object('application_id', application_id, 'unread_count', count(*)) + from notifications where `+strings.Join(where, " and ")+` group by application_id order by application_id nulls last`, args) + if err != nil { + return err + } + total := 0 + for _, g := range groups { + total += g.UnreadCount + } + return c.OK(map[string]any{"unread_count": total, "applications": groups}) + }) } diff --git a/internal/core/notifications_test.go b/internal/core/notifications_test.go index e47527e..2dced0f 100644 --- a/internal/core/notifications_test.go +++ b/internal/core/notifications_test.go @@ -1,32 +1,518 @@ package core_test import ( + "context" + "crypto/ecdh" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/rsa" + "crypto/x509" + "encoding/base64" + "encoding/json" + "encoding/pem" + "strings" "testing" + "github.com/CompositeCode/substratalapps.com/internal/db" + "github.com/CompositeCode/substratalapps.com/internal/push" "github.com/CompositeCode/substratalapps.com/internal/testenv" ) -// Push notifications are specified but not built: every route exists, runs -// the shared pipeline (so authentication and the user-token rule apply -// first), and answers 501 not_implemented in the standard error envelope. -func TestPushNotificationsReserved(t *testing.T) { +func p8Key(t *testing.T) string { + t.Helper() + k, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + der, _ := x509.MarshalPKCS8PrivateKey(k) + return string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: der})) +} + +func serviceAccount(t *testing.T) map[string]any { + t.Helper() + k, _ := rsa.GenerateKey(rand.Reader, 2048) + der, _ := x509.MarshalPKCS8PrivateKey(k) + return map[string]any{"type": "service_account", "project_id": "timetrack-test", "private_key_id": "k1", + "private_key": string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: der})), + "client_email": "push@timetrack-test.iam.gserviceaccount.com"} +} + +func apnsToken() string { return strings.Repeat("ab", 32) } + +// pushApp is an Application owned by a signed-up User, with an APNs +// credential, an app key holding notifications.send, and a second User +// with access to it. +type pushApp struct { + ID string + Owner testenv.User + Key string + Member testenv.User + DeviceID string +} + +func newPushApp(t *testing.T, e *testenv.Env) pushApp { + t.Helper() + owner := e.Signup() + app := e.App(owner.ID, nil) + e.Post("/v1/applications/"+app+"/push-credentials", owner.Token, map[string]any{"provider": "apns", + "team_id": "ABCDE12345", "key_id": "XYZ987WVU6", "topic": "com.example.app", "private_key": p8Key(t)}).Expect(201) + key := e.Post("/v1/api-keys", owner.Token, map[string]any{"name": "backend", "scope": app, + "permissions": []string{"notifications.send"}}).Expect(201).Str("secret") + member := e.Signup() + e.Grant(e.Admin, member.ID, app) + dev := e.Post("/v1/users/me/devices", member.Token, map[string]any{"application_id": app, "provider": "apns", + "token": apntok(), "environment": "production", "name": "Jordan's iPhone"}).Expect(201).Str("id") + return pushApp{ID: app, Owner: owner, Key: key, Member: member, DeviceID: dev} +} + +func apntok() string { return apnsToken() } + +func TestPushCredentials(t *testing.T) { + e := testenv.New(t) + owner := e.Signup() + app := e.App(owner.ID, nil) + base := "/v1/applications/" + app + "/push-credentials" + + r := e.Post(base, owner.Token, map[string]any{"provider": "apns", "team_id": "ABCDE12345", "key_id": "XYZ987WVU6", + "topic": "com.example.app", "private_key": p8Key(t)}).Expect(201) + if r.Str("status") != "active" || r.Str("topic") != "com.example.app" || strings.Contains(string(r.Body), "PRIVATE KEY") { + t.Fatalf("apns credential = %s", r.Body) + } + e.Post(base, owner.Token, map[string]any{"provider": "apns", "team_id": "ABCDE12345", "key_id": "XYZ987WVU6", + "topic": "com.example.app", "private_key": p8Key(t)}).ExpectErr(409, "push_credential_exists") + e.Post(base, owner.Token, map[string]any{"provider": "fcm", "service_account": map[string]any{"type": "service_account"}}). + ExpectErr(422, "invalid_push_credential") + fcm := e.Post(base, owner.Token, map[string]any{"provider": "fcm", "service_account": serviceAccount(t)}).Expect(201) + if fcm.Str("project_id") != "timetrack-test" || strings.Contains(string(fcm.Body), "private_key") { + t.Fatalf("fcm credential = %s", fcm.Body) + } + wp := e.Post(base, owner.Token, map[string]any{"provider": "web_push", "subject": "mailto:ops@example.com"}).Expect(201) + if pub, _ := base64.RawURLEncoding.DecodeString(wp.Str("vapid_public_key")); len(pub) != 65 { + t.Fatalf("vapid key = %q", wp.Str("vapid_public_key")) + } + e.Post(base, owner.Token, map[string]any{"provider": "apns", "team_id": "short"}).ExpectErr(422, "validation_failed") + + list := e.Get(base, owner.Token).Expect(200) + if n := len(list.JSON()["data"].([]any)); n != 3 { + t.Fatalf("list = %d credentials", n) + } + // The app's own key can read, never write. + key := e.Post("/v1/api-keys", owner.Token, map[string]any{"name": "k", "scope": app, "permissions": []string{}}).Expect(201).Str("secret") + e.Get(base+"/web_push", key).Expect(200) + e.Patch(base+"/web_push", key, map[string]any{"subject": "mailto:x@example.com"}).ExpectErr(400, "user_token_required") + // Another User can't see or change any of it. + other := e.Signup() + e.Get(base, other.Token).ExpectErr(404, "application_not_found") + e.Post(base, other.Token, map[string]any{"provider": "fcm"}).ExpectErr(403, "forbidden") + + // PATCH: rotate the key; only fields of that provider; status rules. + e.Patch(base+"/apns", owner.Token, map[string]any{"service_account": map[string]any{}}).ExpectErr(422, "validation_failed") + e.Patch(base+"/apns", owner.Token, map[string]any{"provider": "fcm"}).ExpectErr(422, "read_only_field") + e.Patch(base+"/apns", owner.Token, map[string]any{"status": "failed"}).ExpectErr(409, "invalid_status_transition") + rot := e.Patch(base+"/apns", owner.Token, map[string]any{"key_id": "NEWKEY1234", "private_key": p8Key(t)}).Expect(200) + if rot.Str("key_id") != "NEWKEY1234" { + t.Fatalf("rotated = %s", rot.Body) + } + e.Get(base+"/nope", owner.Token).ExpectErr(404, "push_credential_not_found") + e.Delete(base+"/fcm", owner.Token).Expect(204) + e.Get(base+"/fcm", owner.Token).ExpectErr(404, "push_credential_not_found") + + audit := e.Get("/v1/audit-events?target_type=push_credential", e.Admin).Expect(200) + actions := map[string]bool{} + for _, ev := range audit.JSON()["data"].([]any) { + m := ev.(map[string]any) + actions[m["action"].(string)] = true + if strings.Contains(mustJSON(m), "PRIVATE KEY") { + t.Fatal("key material in an Audit Event") + } + } + for _, a := range []string{"push_credential.created", "push_credential.updated", "push_credential.deleted"} { + if !actions[a] { + t.Errorf("missing audit action %s (got %v)", a, actions) + } + } +} + +func mustJSON(v any) string { + b, _ := json.Marshal(v) + return string(b) +} + +func TestDevices(t *testing.T) { + e := testenv.New(t) + pa := newPushApp(t, e) + m := pa.Member + devs := "/v1/users/me/devices" + + // Re-registering the same token refreshes the same Device. + again := e.Post(devs, m.Token, map[string]any{"application_id": pa.ID, "provider": "apns", "token": apntok(), + "environment": "sandbox", "app_version": "2.5.0"}).Expect(200) + if again.Str("id") != pa.DeviceID || again.Str("environment") != "sandbox" || again.Str("token_hint") != "…abab" { + t.Fatalf("refresh = %s", again.Body) + } + if strings.Contains(string(again.Body), apntok()) { + t.Fatal("the token itself was returned") + } + e.Post(devs, m.Token, map[string]any{"application_id": pa.ID, "provider": "apns", "token": "zz", "environment": "production"}). + ExpectErr(422, "invalid_device_token") + e.Post(devs, m.Token, map[string]any{"application_id": pa.ID, "provider": "apns", "token": apntok()}). + ExpectErr(422, "validation_failed") + // No access, no device. + stranger := e.Signup() + e.Post(devs, stranger.Token, map[string]any{"application_id": pa.ID, "provider": "fcm", "token": "fcm-token-1"}). + ExpectErr(403, "entitlement_required") + + // A browser subscription. + ua, _ := ecdh.P256().GenerateKey(rand.Reader) + auth := make([]byte, 16) + _, _ = rand.Read(auth) + sub := map[string]any{"endpoint": "https://push.example.com/send/abc123", + "keys": map[string]any{"p256dh": base64.RawURLEncoding.EncodeToString(ua.PublicKey().Bytes()), + "auth": base64.RawURLEncoding.EncodeToString(auth)}} + web := e.Post(devs, m.Token, map[string]any{"application_id": pa.ID, "provider": "web_push", "subscription": sub}).Expect(201) + sub["keys"].(map[string]any)["auth"] = "short" + e.Post(devs, m.Token, map[string]any{"application_id": pa.ID, "provider": "web_push", "subscription": sub}). + ExpectErr(422, "invalid_device_token") + + list := e.Get(devs, m.Token).Expect(200) + if n := len(list.JSON()["data"].([]any)); n != 2 { + t.Fatalf("devices = %d", n) + } + // The app's own key manages its users' devices; other users' devices are invisible. + e.Get("/v1/users/"+m.ID+"/devices/"+pa.DeviceID, pa.Key).Expect(200) + e.Get("/v1/users/"+stranger.ID+"/devices", pa.Key).ExpectErr(404, "user_not_found") + e.Get("/v1/users/"+m.ID+"/devices/"+pa.DeviceID, stranger.Token).ExpectErr(403, "forbidden") + + // PATCH a rotated token. + newTok := strings.Repeat("cd", 32) + p := e.Patch(devs+"/"+pa.DeviceID, m.Token, map[string]any{"token": newTok, "name": "New phone"}).Expect(200) + if p.Str("token_hint") != "…cdcd" || p.Str("name") != "New phone" { + t.Fatalf("patch = %s", p.Body) + } + e.Patch(devs+"/"+pa.DeviceID, m.Token, map[string]any{"provider": "fcm"}).ExpectErr(422, "read_only_field") + + // Logging out deactivates the devices that session registered. + e.Post("/v1/auth/logout", m.Token, map[string]any{}).Expect(204) + login := e.Login(m.Email, m.Password) + tok := login.Token() + got := e.Get(devs+"/"+pa.DeviceID, tok).Expect(200) + if got.Str("status") != "inactive" || got.Str("inactive_reason") != "session_revoked" { + t.Fatalf("after logout = %s", got.Body) + } + if n := len(e.Get(devs, tok).Expect(200).JSON()["data"].([]any)); n != 0 { + t.Fatalf("active devices after logout = %d", n) + } + if n := len(e.Get(devs+"?include_inactive=true", tok).Expect(200).JSON()["data"].([]any)); n != 2 { + t.Fatalf("all devices = %d", n) + } + // Signing back in and re-registering reactivates it. + re := e.Post(devs, tok, map[string]any{"application_id": pa.ID, "provider": "apns", "token": newTok, "environment": "production"}).Expect(200) + if re.Str("status") != "active" || re.Str("session_id") != login.Str("session_id") { + t.Fatalf("re-register = %s", re.Body) + } + + // Someone else signs in on the same install: the token moves to them. + other := e.Signup() + e.Grant(e.Admin, other.ID, pa.ID) + moved := e.Post(devs, other.Token, map[string]any{"application_id": pa.ID, "provider": "apns", "token": newTok, "environment": "production"}).Expect(201) + if moved.Str("id") == pa.DeviceID { + t.Fatal("the previous User's Device was handed over") + } + e.Get(devs+"/"+pa.DeviceID, tok).ExpectErr(404, "device_not_found") + if len(appEvents(e, "device.deleted", pa.ID)) == 0 || len(appEvents(e, "device.registered", pa.ID)) < 3 { + t.Error("missing device webhook events") + } + + // Device test: synchronous, and 409 without a credential for the provider. + tr := e.Post("/v1/users/me/devices/"+moved.Str("id")+"/test", other.Token, map[string]any{}).Expect(200) + if tr.Str("status") != "sent" { + t.Fatalf("test = %s", tr.Body) + } + if msgs := e.Push.Sent(); msgs[len(msgs)-1].Notification.Title != "Test notification from Substratal" { + t.Fatalf("test push = %+v", msgs[len(msgs)-1]) + } + e.Post("/v1/users/me/devices/"+web.Str("id")+"/test", tok, map[string]any{}).ExpectErr(409, "push_not_configured") + + // Suspension deactivates everything; DELETE removes a device. + e.Post("/v1/users/"+other.ID+"/suspend", e.Admin, map[string]any{}).Expect(200) + sd := e.Get("/v1/users/"+other.ID+"/devices?include_inactive=true", e.Admin).Expect(200).JSON()["data"].([]any) + if sd[0].(map[string]any)["inactive_reason"] != "user_suspended" { + t.Fatalf("after suspend = %v", sd) + } + e.Delete(devs+"/"+web.Str("id"), tok).Expect(204) + e.Delete("/v1/users/"+other.ID+"/devices/"+moved.Str("id"), e.Admin).Expect(204) + if len(e.Get("/v1/audit-events?action=device.deleted", e.Admin).Expect(200).JSON()["data"].([]any)) != 1 { + t.Error("an admin device delete should be audited (and only that one)") + } +} + +func TestSendAndInbox(t *testing.T) { e := testenv.New(t) - u := e.Signup() - app := e.App("usr_01JAG0SYSTEM00000000000000", nil) - - e.Get("/v1/users/me/notifications", "").ExpectErr(401, "unauthenticated") - e.Get("/v1/users/me/notifications", u.Token).ExpectErr(501, "not_implemented") - e.Get("/v1/users/me/notifications/unread-count", u.Token).ExpectErr(501, "not_implemented") - e.Post("/v1/users/me/devices", u.Token, map[string]any{ - "application_id": app, "provider": "fcm", "token": "fcm-registration-token", - }).ExpectErr(501, "not_implemented") - e.Delete("/v1/users/me/devices/dvc_01JAH2B3C4D5E6F7G8H9J0K1M2", u.Token).ExpectErr(501, "not_implemented") - - // Push credentials are managed by people only: an API Key is refused - // before the route's 501. - key := e.Post("/v1/api-keys", e.Admin, map[string]any{"name": "app", "scope": app, "permissions": []string{}}). - Expect(201).Str("secret") - e.Post("/v1/applications/"+app+"/push-credentials", key, map[string]any{"provider": "web_push", "subject": "mailto:ops@example.com"}). - ExpectErr(400, "user_token_required") - e.Get("/v1/applications/"+app+"/push-credentials", key).ExpectErr(501, "not_implemented") + pa := newPushApp(t, e) + m := pa.Member + send := "/v1/users/" + m.ID + "/notifications" + body := map[string]any{"category": "timesheet.reminder", "title": "Timesheet due tomorrow", + "body": "Due Friday at 5 pm.", "priority": "high", "data": map[string]any{"timesheet_id": "ts_8841"}, + "action_url": "com.example.app:/timesheets/ts_8841", "badge": 1, "collapse_key": "timesheet-due"} + + n := e.PostIdem(send, pa.Key, testenv.Unique(), body).Expect(201) + push1 := n.JSON()["push"].(map[string]any) + if push1["status"] != "queued" || push1["devices_targeted"] != float64(1) || n.Str("state") != "unread" || + n.Str("sound") != "default" || n.JSON()["sender"].(map[string]any)["type"] != "api_key" { + t.Fatalf("send = %s", n.Body) + } + e.DeliverPush() + msgs := e.Push.Sent() + if len(msgs) != 1 || msgs[0].Target.Token != apntok() || msgs[0].Notification.Title != "Timesheet due tomorrow" || + msgs[0].Credential.Topic != "com.example.app" || msgs[0].Notification.Data["timesheet_id"] != "ts_8841" { + t.Fatalf("pushes = %+v", msgs) + } + got := e.Get("/v1/users/me/notifications/"+n.Str("id"), m.Token).Expect(200).JSON() + if p := got["push"].(map[string]any); p["status"] != "sent" || p["devices_sent"] != float64(1) { + t.Fatalf("after delivery = %v", got) + } + if ds := got["deliveries"].([]any); len(ds) != 1 || ds[0].(map[string]any)["status"] != "sent" { + t.Fatalf("deliveries = %v", got["deliveries"]) + } + + // Errors. + noPerm := e.Post("/v1/api-keys", pa.Owner.Token, map[string]any{"name": "k2", "scope": pa.ID, "permissions": []string{}}).Expect(201).Str("secret") + e.Post(send, noPerm, body).ExpectErr(403, "forbidden") + stranger := e.Signup() + e.Post("/v1/users/"+stranger.ID+"/notifications", pa.Key, body).ExpectErr(403, "entitlement_required") + e.Post(send, pa.Key, map[string]any{"category": "security.alert", "title": "x"}).ExpectErr(422, "reserved_category") + e.Post(send, pa.Key, map[string]any{"title": "x"}).ExpectErr(422, "validation_failed") + e.Post(send, pa.Key, map[string]any{"category": "c", "title": "x", "data": map[string]any{"substratal": "x"}}).ExpectErr(422, "validation_failed") + e.Post(send, pa.Key, map[string]any{"category": "c", "title": "x", "action_url": "http://insecure.example.com"}).ExpectErr(422, "validation_failed") + // Every field within its own limit, but 4-byte characters push the + // payload past the providers' shared 4 KB. + e.Post(send, pa.Key, map[string]any{"category": "c", "title": strings.Repeat("😀", 120), "body": strings.Repeat("😀", 1000)}). + ExpectErr(422, "notification_too_large") + // A platform caller names the Application. + e.Post(send, e.Admin, body).ExpectErr(422, "validation_failed") + adm := e.Post(send, e.Admin, map[string]any{"category": "ops.notice", "title": "Maintenance tonight", "application_id": pa.ID, "push": false}).Expect(201) + if adm.JSON()["push"].(map[string]any)["skip_reason"] != "push_not_requested" || adm.JSON()["sender"].(map[string]any)["type"] != "user" { + t.Fatalf("inbox-only = %s", adm.Body) + } + + // Preferences: push off for this app → inbox only. + e.Patch("/v1/users/me/apps/"+pa.ID+"/settings", m.Token, map[string]any{"overrides": map[string]any{"notifications": map[string]any{"push": false}}}).Expect(200) + off := e.Post(send, pa.Key, map[string]any{"category": "c.d", "title": "muted"}).Expect(201) + if off.JSON()["push"].(map[string]any)["skip_reason"] != "push_disabled" { + t.Fatalf("muted = %s", off.Body) + } + if len(appEvents(e, "notification.push_failed", pa.ID)) != 0 { + t.Error("push_disabled is a choice, not a failure") + } + e.Patch("/v1/users/me/apps/"+pa.ID+"/settings", m.Token, map[string]any{"overrides": map[string]any{"notifications": map[string]any{"push": nil}}}).Expect(200) + + // No devices: skipped, and the app hears about it. + other := e.Signup() + e.Grant(e.Admin, other.ID, pa.ID) + nd := e.Post("/v1/users/"+other.ID+"/notifications", pa.Key, map[string]any{"category": "c", "title": "hi"}).Expect(201) + if nd.JSON()["push"].(map[string]any)["skip_reason"] != "no_devices" || len(appEvents(e, "notification.push_failed", pa.ID)) != 1 { + t.Fatalf("no devices = %s", nd.Body) + } + + // Inbox. + inbox := "/v1/users/me/notifications" + if l := e.Get(inbox, m.Token).Expect(200).JSON()["data"].([]any); len(l) != 3 { + t.Fatalf("inbox = %d", len(l)) + } + cnt := e.Get(inbox+"/unread-count", m.Token).Expect(200).JSON() + if cnt["unread_count"] != float64(3) { + t.Fatalf("unread = %v", cnt) + } + e.Patch(inbox+"/"+n.Str("id"), m.Token, map[string]any{"state": "archived"}).Expect(200) + e.Patch(inbox+"/"+n.Str("id"), m.Token, map[string]any{"title": "x"}).ExpectErr(422, "read_only_field") + if l := e.Get(inbox, m.Token).Expect(200).JSON()["data"].([]any); len(l) != 2 { + t.Fatalf("archived still listed: %d", len(l)) + } + if l := e.Get(inbox+"?state=archived", m.Token).Expect(200).JSON()["data"].([]any); len(l) != 1 { + t.Fatalf("archived = %d", len(l)) + } + mr := e.Post(inbox+"/mark-read", m.Token, map[string]any{"application_id": pa.ID}).Expect(200) + if mr.JSON()["updated_count"] != float64(2) { + t.Fatalf("mark-read = %s", mr.Body) + } + if e.Get(inbox+"/unread-count", m.Token).Expect(200).JSON()["unread_count"] != float64(0) { + t.Fatal("still unread") + } + // The app's key sees only its own notifications; support reads, never writes. + e.Get("/v1/users/"+m.ID+"/notifications?state=read", pa.Key).Expect(200) + e.Get("/v1/users/"+m.ID+"/notifications", e.Admin).Expect(200) + e.Post("/v1/users/"+m.ID+"/notifications/mark-read", e.Admin, map[string]any{}).ExpectErr(403, "forbidden") + + // Export carries devices (never the token) and notifications. + ex := e.Get("/v1/users/me/export", m.Token).Expect(200) + if strings.Contains(string(ex.Body), apntok()) || len(ex.JSON()["devices"].([]any)) != 1 || len(ex.JSON()["notifications"].([]any)) != 3 { + t.Fatalf("export = %s", ex.Body) + } +} + +func TestPushOutcomes(t *testing.T) { + e := testenv.New(t) + pa := newPushApp(t, e) + send := func() string { + return e.Post("/v1/users/"+pa.Member.ID+"/notifications", pa.Key, map[string]any{"category": "c", "title": "hi"}).Expect(201).Str("id") + } + note := func(id string) map[string]any { + return e.Get("/v1/users/me/notifications/"+id, pa.Member.Token).Expect(200).JSON() + } + + // A transient failure stays pending, with the attempt counted. + e.Push.Script(apntok(), push.Result{Outcome: push.Retry, Status: 503, Reason: "ServiceUnavailable"}) + id := send() + e.DeliverPush() + d := note(id)["deliveries"].([]any)[0].(map[string]any) + if d["status"] != "pending" || d["attempts"] != float64(1) || note(id)["push"].(map[string]any)["status"] != "queued" { + t.Fatalf("retry = %v", d) + } + + // A gone token deactivates the device and fails the push. + e.Push.Script(apntok(), push.Result{Outcome: push.TokenGone, Status: 410, Reason: "Unregistered"}) + expireLease(t, e) + e.DeliverPush() + if p := note(id)["push"].(map[string]any); p["status"] != "failed" || p["devices_failed"] != float64(1) { + t.Fatalf("token gone = %v", p) + } + dev := e.Get("/v1/users/me/devices/"+pa.DeviceID, pa.Member.Token).Expect(200) + if dev.Str("inactive_reason") != "unregistered" || len(appEvents(e, "device.deactivated", pa.ID)) != 1 || + len(appEvents(e, "notification.push_failed", pa.ID)) != 1 { + t.Fatalf("device after 410 = %s", dev.Body) + } + + // A rejected credential is marked failed, reported, and emailed. + e.Post("/v1/users/me/devices", pa.Member.Token, map[string]any{"application_id": pa.ID, "provider": "apns", + "token": apntok(), "environment": "production"}).Expect(200) + e.Push.Script(apntok(), push.Result{Outcome: push.CredentialRejected, Status: 403, Reason: "InvalidProviderToken"}) + send() + e.DeliverPush() + cred := e.Get("/v1/applications/"+pa.ID+"/push-credentials/apns", pa.Owner.Token).Expect(200) + if cred.Str("status") != "failed" || cred.Str("failure_reason") != "InvalidProviderToken" || len(appEvents(e, "push_credential.failed", pa.ID)) != 1 { + t.Fatalf("credential = %s", cred.Body) + } + if _, ok := e.Mail.Last(pa.Owner.Email, "push_credential_failed"); !ok { + t.Error("the owner wasn't emailed") + } + // Further sends skip until it's fixed. + n := e.Post("/v1/users/"+pa.Member.ID+"/notifications", pa.Key, map[string]any{"category": "c", "title": "again"}).Expect(201) + if n.JSON()["push"].(map[string]any)["skip_reason"] != "credential_failed" { + t.Fatalf("after failure = %s", n.Body) + } + e.Patch("/v1/applications/"+pa.ID+"/push-credentials/apns", pa.Owner.Token, map[string]any{"status": "active"}).Expect(200) +} + +// expireLease makes leased-but-retrying deliveries due now. +func expireLease(t *testing.T, e *testenv.Env) { + t.Helper() + err := e.DB.Tx(context.Background(), db.Settings{AllModes: true}, func(q db.Querier) error { + _, err := q.Exec(context.Background(), `update notification_deliveries set next_attempt_at = now() where status = 'pending'`, nil) + return err + }) + if err != nil { + t.Fatal(err) + } +} + +func sqlExec(t *testing.T, e *testenv.Env, q string, args db.Args) { + t.Helper() + if err := e.DB.Tx(context.Background(), db.Settings{AllModes: true}, func(qq db.Querier) error { + _, err := qq.Exec(context.Background(), q, args) + return err + }); err != nil { + t.Fatal(err) + } +} + +func TestPlatformAlerts(t *testing.T) { + e := testenv.New(t) + pa := newPushApp(t, e) + m := pa.Member + alerts := func(token string) map[string]map[string]any { + out := map[string]map[string]any{} + for _, x := range e.Get("/v1/users/me/notifications?source=platform", token).Expect(200).JSON()["data"].([]any) { + n := x.(map[string]any) + out[n["category"].(string)] = n + } + return out + } + + // Without the opt-in, alerts land in the inbox but push nowhere. + e.Do(testenv.Req{Method: "POST", Path: "/v1/auth/login", Body: map[string]any{"email": m.Email, "password": m.Password}, + Header: map[string]string{"User-Agent": "TimeTrack/2.4 (iOS 19.0)"}}).Expect(200) + got := alerts(m.Token) + si, ok := got["security.new_sign_in"] + if !ok || si["application_id"] != nil || si["priority"] != "high" || si["push"].(map[string]any)["skip_reason"] != "no_devices" { + t.Fatalf("new sign-in = %v", got) + } + // An app key never sees platform alerts. + if l := e.Get("/v1/users/"+m.ID+"/notifications?source=platform", pa.Key).Expect(200).JSON()["data"].([]any); len(l) != 0 { + t.Fatal("an app key saw a platform alert") + } + + // With the opt-in, a password reset's alert reaches the device even + // though the reset signs every session out. + e.Patch("/v1/applications/"+pa.ID, pa.Owner.Token, map[string]any{"receives_platform_notifications": true}).Expect(200) + e.Push.Reset() + e.Post("/v1/auth/password/forgot", "", map[string]any{"email": m.Email}).Expect(202) + e.Post("/v1/auth/password/reset", "", map[string]any{"token": e.Token(m.Email, "password_reset"), + "new_password": "an entirely new passphrase"}).Expect(204) + e.DeliverPush() + msgs := e.Push.Sent() + if len(msgs) != 1 || msgs[0].Notification.Category != "security.password_changed" || msgs[0].Notification.Source != "platform" { + t.Fatalf("password alert pushes = %+v", msgs) + } + tok := e.Login(m.Email, "an entirely new passphrase").Token() + if d := e.Get("/v1/users/me/devices/"+pa.DeviceID, tok).Expect(200); d.Str("inactive_reason") != "session_revoked" { + t.Fatalf("device after reset = %s", d.Body) + } + + // account.access_granted: developer grants only, never in the first minute. + newbie := e.Signup() + e.Grant(e.Admin, newbie.ID, e.App("usr_01JAG0SYSTEM00000000000000", nil)) + if _, ok := alerts(newbie.Token)["account.access_granted"]; ok { + t.Fatal("a brand-new account got an access alert") + } + sqlExec(t, e, `update users set created_at = now() - interval '2 minutes' where id = :u`, db.Args{"u": newbie.ID}) + e.Grant(e.Admin, newbie.ID, e.App("usr_01JAG0SYSTEM00000000000000", nil)) + ag, ok := alerts(newbie.Token)["account.access_granted"] + if !ok || !strings.HasPrefix(ag["title"].(string), "You now have access to") { + t.Fatalf("access alert = %v", alerts(newbie.Token)) + } + + // An Organization invitation for an existing User. + org := e.Post("/v1/organizations", pa.Owner.Token, map[string]any{"name": "Acme"}).Expect(201).Str("id") + e.Post("/v1/organizations/"+org+"/members", pa.Owner.Token, map[string]any{"email": newbie.Email}).Expect(201) + if inv, ok := alerts(newbie.Token)["account.organization_invitation"]; !ok || inv["data"].(map[string]any)["organization_id"] != org { + t.Fatalf("invitation alert = %v", alerts(newbie.Token)) + } +} + +func TestPushMaintenance(t *testing.T) { + e := testenv.New(t) + pa := newPushApp(t, e) + sqlExec(t, e, `update devices set last_registered_at = now() - interval '300 days' where id = :d`, db.Args{"d": pa.DeviceID}) + if err := e.Core.PushMaintenance(context.Background()); err != nil { + t.Fatal(err) + } + d := e.Get("/v1/users/me/devices/"+pa.DeviceID, pa.Member.Token).Expect(200) + if d.Str("inactive_reason") != "stale" { + t.Fatalf("stale = %s", d.Body) + } + sqlExec(t, e, `update devices set deactivated_at = now() - interval '31 days' where id = :d`, db.Args{"d": pa.DeviceID}) + if err := e.Core.PushMaintenance(context.Background()); err != nil { + t.Fatal(err) + } + e.Get("/v1/users/me/devices/"+pa.DeviceID, pa.Member.Token).ExpectErr(404, "device_not_found") +} + +// appEvents is the outbox events of a type for one Application (tests in +// a package share a database, so global counts would be meaningless). +func appEvents(e *testenv.Env, typ, appID string) []map[string]any { + var out []map[string]any + for _, ev := range e.Events(typ) { + if ev["application_id"] == appID { + out = append(out, ev) + } + } + return out } diff --git a/internal/core/organizations.go b/internal/core/organizations.go index 4a8eb8b..afb004c 100644 --- a/internal/core/organizations.go +++ b/internal/core/organizations.go @@ -501,6 +501,19 @@ func (s *Server) OrganizationsAddMember(w http.ResponseWriter, r *http.Request, TargetUserID: &uid, OrganizationID: &o.ID, After: map[string]any{"user_id": uid, "role": role}}); err != nil { return err } + if !direct { + // An existing active User hears about it in their inbox too; a + // new or invited one has no inbox yet, only the email. The + // caller can't tell which, so nothing about the account leaks. + data := map[string]string{"organization_id": o.ID} + if c.p.IsUser() { + data["invited_by"] = c.p.UserID + } + if err := c.platformAlert(uid, "account.organization_invitation", "You've been invited to join "+o.Name, + strp("Open the invitation to accept it."), data); err != nil { + return err + } + } if direct { pairs, err := c.orgAppPairs(uid, o.ID) if err != nil { diff --git a/internal/core/push_credentials.go b/internal/core/push_credentials.go new file mode 100644 index 0000000..10a5df6 --- /dev/null +++ b/internal/core/push_credentials.go @@ -0,0 +1,555 @@ +package core + +import ( + "context" + "encoding/json" + "net/http" + "slices" + "strings" + "time" + + "github.com/CompositeCode/substratalapps.com/internal/api/gen" + "github.com/CompositeCode/substratalapps.com/internal/db" + "github.com/CompositeCode/substratalapps.com/internal/email" + "github.com/CompositeCode/substratalapps.com/internal/httpx" + "github.com/CompositeCode/substratalapps.com/internal/ids" + "github.com/CompositeCode/substratalapps.com/internal/push" +) + +// pushCredential is the API view of an Application's credential for one +// provider (API Reference → Push Notifications → Push credentials). Key +// material is never part of it. +type pushCredential struct { + ID string `json:"id"` + ApplicationID string `json:"application_id"` + Provider string `json:"provider"` + Status string `json:"status"` + TeamID string `json:"team_id,omitempty"` + KeyID string `json:"key_id,omitempty"` + Topic string `json:"topic,omitempty"` + ProjectID string `json:"project_id,omitempty"` + ClientEmail string `json:"client_email,omitempty"` + Subject string `json:"subject,omitempty"` + VAPIDPublicKey string `json:"vapid_public_key,omitempty"` + FailureReason *string `json:"failure_reason"` + FailedAt *time.Time `json:"failed_at"` + LastUsedAt *time.Time `json:"last_used_at"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` + version int + secret string +} + +// credConfig is the non-secret part, stored as config jsonb. +type credConfig struct { + TeamID string `json:"team_id,omitempty"` + KeyID string `json:"key_id,omitempty"` + Topic string `json:"topic,omitempty"` + ProjectID string `json:"project_id,omitempty"` + ClientEmail string `json:"client_email,omitempty"` + Subject string `json:"subject,omitempty"` + VAPIDPublicKey string `json:"vapid_public_key,omitempty"` +} + +func (c *Call) queryCredentials(where string, args db.Args) ([]pushCredential, error) { + type row struct { + ID string `json:"id"` + ApplicationID string `json:"application_id"` + Provider string `json:"provider"` + Status string `json:"status"` + Config credConfig `json:"config"` + Secret string `json:"secret_ciphertext"` + FailureReason *string `json:"failure_reason"` + FailedAt *time.Time `json:"failed_at"` + LastUsedAt *time.Time `json:"last_used_at"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` + Version int `json:"version"` + } + rows, err := db.All[row](c.ctx, c.q, `select to_jsonb(p) from push_credentials p where `+where, args) + if err != nil { + return nil, err + } + out := make([]pushCredential, len(rows)) + for i, r := range rows { + out[i] = pushCredential{ID: r.ID, ApplicationID: r.ApplicationID, Provider: r.Provider, Status: r.Status, + TeamID: r.Config.TeamID, KeyID: r.Config.KeyID, Topic: r.Config.Topic, ProjectID: r.Config.ProjectID, + ClientEmail: r.Config.ClientEmail, Subject: r.Config.Subject, VAPIDPublicKey: r.Config.VAPIDPublicKey, + FailureReason: r.FailureReason, FailedAt: r.FailedAt, LastUsedAt: r.LastUsedAt, + CreatedAt: r.CreatedAt, UpdatedAt: r.UpdatedAt, version: r.Version, secret: r.Secret} + } + return out, nil +} + +func (p pushCredential) config() credConfig { + return credConfig{TeamID: p.TeamID, KeyID: p.KeyID, Topic: p.Topic, ProjectID: p.ProjectID, + ClientEmail: p.ClientEmail, Subject: p.Subject, VAPIDPublicKey: p.VAPIDPublicKey} +} + +// decrypt rebuilds the provider credential the push package signs with. +func (s *Server) decryptCredential(ctx context.Context, p pushCredential) (push.Credential, error) { + secret, err := s.Encrypter.Decrypt(ctx, p.secret) + if err != nil { + return push.Credential{}, err + } + cred := push.Credential{Provider: p.Provider, TeamID: p.TeamID, KeyID: p.KeyID, Topic: p.Topic, + Subject: p.Subject, VAPIDPublicKey: p.VAPIDPublicKey} + switch p.Provider { + case push.APNs: + cred.PrivateKey = secret + case push.FCM: + sa, err := push.ParseServiceAccount([]byte(secret)) + if err != nil { + return cred, err + } + cred.ServiceAccount = sa + case push.WebPush: + cred.VAPIDPrivate = secret + } + return cred, nil +} + +// credentialApp loads an Application for push-credential access: its +// owner, its own app keys (read only), and applications.manage. Anyone +// else gets 404, so the configuration's existence isn't leaked. +func (c *Call) credentialApp(appID string, write bool) (appRow, error) { + a, err := c.loadApp(appID) + if err == db.ErrNotFound { + return a, httpx.NotFoundResource("application") + } + if err != nil { + return a, err + } + if c.can("applications.manage") { + return a, nil + } + if key, ok := c.p.AppKey(); ok { + if key != appID { + return a, httpx.NotFoundResource("application") + } + if write { + return a, httpx.ErrUserTokenRequired() + } + return a, nil + } + owner, err := c.isAppOwner(a) + if err != nil { + return a, err + } + if !owner { + if write { + return a, httpx.Forbidden("applications.manage") + } + return a, httpx.NotFoundResource("application") + } + return a, nil +} + +func validProvider(p string) bool { return slices.Contains(push.Providers, p) } + +func invalidCredential(why string) error { + return httpx.E(422, "invalid_push_credential", "The push credential isn't valid: "+why+".") +} + +// credentialInput is the provider-specific body of a POST or PATCH. +type credentialInput struct { + TeamID *string `json:"team_id"` + KeyID *string `json:"key_id"` + Topic *string `json:"topic"` + PrivateKey *string `json:"private_key"` + ServiceAccount json.RawMessage `json:"service_account"` + Subject *string `json:"subject"` +} + +var providerFields = map[string][]string{ + push.APNs: {"team_id", "key_id", "topic", "private_key"}, + push.FCM: {"service_account"}, + push.WebPush: {"subject"}, +} + +// applyCredential validates in against provider and folds it into cfg, +// returning the new secret when key material changed ("" otherwise). +// create requires every field the provider needs. +func applyCredential(provider string, raw httpx.Patch, in credentialInput, cfg *credConfig, create bool) (string, error) { + var f httpx.Fields + for k := range raw { + if k == "provider" || k == "status" { + continue + } + if !slices.Contains(providerFields[provider], k) { + f.Add(k, "undeclared_key").Message = "not a field of a " + provider + " credential" + } + } + if err := f.Err(); err != nil { + return "", err + } + need := func(field string, v *string) bool { + if v == nil || strings.TrimSpace(*v) == "" { + if create || v != nil { + f.Add(field, "required") + } + return false + } + return true + } + secret := "" + switch provider { + case push.APNs: + if need("team_id", in.TeamID) { + if len(*in.TeamID) != 10 { + f.Add("team_id", "invalid_format").Message = "10 characters" + } + cfg.TeamID = *in.TeamID + } + if need("key_id", in.KeyID) { + if len(*in.KeyID) != 10 { + f.Add("key_id", "invalid_format").Message = "10 characters" + } + cfg.KeyID = *in.KeyID + } + if need("topic", in.Topic) { + if len(*in.Topic) > 255 { + f.Max("topic", "too_long", 255) + } + cfg.Topic = *in.Topic + } + if need("private_key", in.PrivateKey) { + if err := f.Err(); err != nil { + return "", err + } + if _, err := push.ParseAPNsKey(*in.PrivateKey); err != nil { + return "", invalidCredential("private_key: " + err.Error()) + } + secret = *in.PrivateKey + } + case push.FCM: + if len(in.ServiceAccount) == 0 || string(in.ServiceAccount) == "null" { + if create || raw.Has("service_account") { + f.Add("service_account", "required") + } + break + } + sa, err := push.ParseServiceAccount(in.ServiceAccount) + if err != nil { + return "", invalidCredential("service_account: " + err.Error()) + } + cfg.ProjectID, cfg.ClientEmail = sa.ProjectID, sa.ClientEmail + secret = string(in.ServiceAccount) + case push.WebPush: + if need("subject", in.Subject) { + s := *in.Subject + if len(s) > 255 || !(strings.HasPrefix(s, "mailto:") || strings.HasPrefix(s, "https://")) { + f.Add("subject", "invalid_format").Allowed = []string{"mailto:", "https://"} + } + cfg.Subject = s + } + if create { + pub, priv, err := push.GenerateVAPID() + if err != nil { + return "", err + } + cfg.VAPIDPublicKey, secret = pub, priv + } + } + return secret, f.Err() +} + +func (s *Server) PushCredentialsList(w http.ResponseWriter, r *http.Request, id string) { + s.serve(w, r, opts{op: "pushCredentials.list"}, func(c *Call) error { + a, err := c.credentialApp(id, false) + if err != nil { + return err + } + creds, err := c.queryCredentials("p.application_id = :a order by p.provider", db.Args{"a": a.ID}) + if err != nil { + return err + } + return c.OK(map[string]any{"data": creds}) + }) +} + +func (s *Server) PushCredentialsGet(w http.ResponseWriter, r *http.Request, id string, provider gen.PushProvider) { + s.serve(w, r, opts{op: "pushCredentials.get"}, func(c *Call) error { + a, err := c.credentialApp(id, false) + if err != nil { + return err + } + p, err := c.loadCredential(a.ID, string(provider)) + if err != nil { + return err + } + c.Versioned(p.version) + return c.OK(p) + }) +} + +func (c *Call) loadCredential(appID, provider string) (pushCredential, error) { + if !validProvider(provider) { + return pushCredential{}, httpx.NotFoundResource("push_credential") + } + creds, err := c.queryCredentials("p.application_id = :a and p.provider = :p", db.Args{"a": appID, "p": provider}) + if err != nil { + return pushCredential{}, err + } + if len(creds) == 0 { + return pushCredential{}, httpx.NotFoundResource("push_credential") + } + return creds[0], nil +} + +func (s *Server) PushCredentialsCreate(w http.ResponseWriter, r *http.Request, id string, params gen.PushCredentialsCreateParams) { + s.serve(w, r, opts{op: "pushCredentials.create", userOnly: true, idem: idemOptional}, func(c *Call) error { + a, err := c.credentialApp(id, true) + if err != nil { + return err + } + raw, err := c.Patch() + if err != nil { + return err + } + var in struct { + Provider string `json:"provider"` + credentialInput + } + if err := c.Decode(&in); err != nil { + return err + } + if !validProvider(in.Provider) { + var f httpx.Fields + if in.Provider == "" { + f.Add("provider", "required") + } else { + f.Enum("provider", push.Providers...) + } + return f.Err() + } + t, err := c.loadTenant(a.TenantID) + if err != nil { + return err + } + if err := tenantWritable(t); err != nil { + return err + } + var cfg credConfig + secret, err := applyCredential(in.Provider, raw, in.credentialInput, &cfg, true) + if err != nil { + return err + } + if existing, err := c.queryCredentials("p.application_id = :a and p.provider = :p", + db.Args{"a": a.ID, "p": in.Provider}); err != nil { + return err + } else if len(existing) > 0 { + return httpx.Conflict("push_credential_exists", "This Application already has a "+in.Provider+" credential; PATCH it instead.") + } + enc, err := c.s.Encrypter.Encrypt(c.ctx, secret) + if err != nil { + return err + } + credID := ids.New(ids.PushCredential) + if _, err := c.q.Exec(c.ctx, `insert into push_credentials (id, application_id, tenant_id, provider, config, secret_ciphertext) + values (:id, :a, :t, :p, :cfg::jsonb, :sec)`, + db.Args{"id": credID, "a": a.ID, "t": a.TenantID, "p": in.Provider, "cfg": db.JSON(cfg), "sec": enc}); err != nil { + if _, ok := db.UniqueViolation(err); ok { + return httpx.Conflict("push_credential_exists", "This Application already has a "+in.Provider+" credential; PATCH it instead.") + } + return err + } + out, err := c.loadCredential(a.ID, in.Provider) + if err != nil { + return err + } + if err := c.audit(Audit{Action: "push_credential.created", TargetType: "push_credential", TargetID: out.ID, + ApplicationID: &a.ID, After: out}); err != nil { + return err + } + c.Versioned(out.version) + return c.Created(out) + }) +} + +func (s *Server) PushCredentialsUpdate(w http.ResponseWriter, r *http.Request, id string, provider gen.PushProvider, params gen.PushCredentialsUpdateParams) { + s.serve(w, r, opts{op: "pushCredentials.update", userOnly: true}, func(c *Call) error { + a, err := c.credentialApp(id, true) + if err != nil { + return err + } + before, err := c.loadCredential(a.ID, string(provider)) + if err != nil { + return err + } + if err := httpx.CheckIfMatch(c.r, before.version); err != nil { + return err + } + raw, err := c.Patch() + if err != nil { + return err + } + if err := raw.RejectReadOnly("id", "application_id", "provider", "vapid_public_key", "project_id", "client_email", + "failure_reason", "failed_at", "last_used_at", "created_at", "updated_at"); err != nil { + return err + } + var in struct { + Status *string `json:"status"` + credentialInput + } + if err := c.Decode(&in); err != nil { + return err + } + if in.Status != nil && *in.Status != "active" { + return httpx.Conflict("invalid_status_transition", "A credential's status can only go from failed to active.") + } + t, err := c.loadTenant(a.TenantID) + if err != nil { + return err + } + if err := tenantWritable(t); err != nil { + return err + } + cfg := before.config() + secret, err := applyCredential(before.Provider, raw, in.credentialInput, &cfg, false) + if err != nil { + return err + } + // Any change to key material, or an explicit status: active, makes + // a failed credential active again. + reactivate := secret != "" || in.Status != nil + args := db.Args{"id": before.ID, "cfg": db.JSON(cfg), "react": reactivate, "sec": nil} + if secret != "" { + if args["sec"], err = c.s.Encrypter.Encrypt(c.ctx, secret); err != nil { + return err + } + } + if _, err := c.q.Exec(c.ctx, `update push_credentials set config = :cfg::jsonb, + secret_ciphertext = coalesce(:sec, secret_ciphertext), + status = case when :react then 'active' else status end, + failure_reason = case when :react then null else failure_reason end, + failed_at = case when :react then null else failed_at end + where id = :id`, args); err != nil { + return err + } + after, err := c.loadCredential(a.ID, before.Provider) + if err != nil { + return err + } + if err := c.audit(Audit{Action: "push_credential.updated", TargetType: "push_credential", TargetID: after.ID, + ApplicationID: &a.ID, Before: before, After: after}); err != nil { + return err + } + c.Versioned(after.version) + return c.OK(after) + }) +} + +func (s *Server) PushCredentialsDelete(w http.ResponseWriter, r *http.Request, id string, provider gen.PushProvider) { + s.serve(w, r, opts{op: "pushCredentials.delete", userOnly: true}, func(c *Call) error { + a, err := c.credentialApp(id, true) + if err != nil { + return err + } + before, err := c.loadCredential(a.ID, string(provider)) + if err != nil { + return err + } + if _, err := c.q.Exec(c.ctx, `delete from push_credentials where id = :id`, db.Args{"id": before.ID}); err != nil { + return err + } + if err := c.audit(Audit{Action: "push_credential.deleted", TargetType: "push_credential", TargetID: before.ID, + ApplicationID: &a.ID, Before: before}); err != nil { + return err + } + // One credential serves both modes, but a request sees only its + // own; the effects on devices and pending deliveries run for both + // right after commit. + appID, prov := a.ID, before.Provider + c.After(func(ctx context.Context) { + if err := c.s.credentialRemoved(ctx, appID, prov); err != nil { + c.s.Log.Error("push credential removal effects", "application", appID, "provider", prov, "err", err) + } + }) + return c.NoContent() + }) +} + +// credentialRemoved skips pending deliveries that depended on a deleted +// credential and, for web_push, deactivates the Application's browser +// subscriptions, which were bound to the deleted VAPID key. +func (s *Server) credentialRemoved(ctx context.Context, appID, provider string) error { + return eachMode(func(tm bool) error { + return s.system(ctx, tm, db.Settings{}, func(c *Call) error { + notes, err := db.All[string](ctx, c.q, `update notification_deliveries d set status = 'skipped', error = 'credential_removed', + next_attempt_at = null + from notifications n where n.id = d.notification_id and d.status = 'pending' and d.provider = :p + and (n.application_id = :a or d.device_id in (select id from devices where application_id = :a)) + returning to_jsonb(d.notification_id)`, db.Args{"a": appID, "p": provider}) + if err != nil { + return err + } + for _, n := range dedupe(notes) { + if err := c.settleNotification(n); err != nil { + return err + } + } + if provider == push.WebPush { + return c.deactivateDevices("d.application_id = :a and d.provider = 'web_push'", db.Args{"a": appID}, "credential_removed") + } + return nil + }) + }) +} + +// credentialRejected marks a credential failed after the provider refused +// it, records why, and tells the Application's owner, once per failure. +func (c *Call) credentialRejected(credID, reason string) error { + var cred struct { + ApplicationID string `json:"application_id"` + Provider string `json:"provider"` + } + err := db.One(c.ctx, c.q, &cred, `update push_credentials set status = 'failed', failure_reason = :r, failed_at = now() + where id = :id and status = 'active' returning jsonb_build_object('application_id', application_id, 'provider', provider)`, + db.Args{"id": credID, "r": reason}) + if err == db.ErrNotFound { + return nil // already failed, or deleted meanwhile + } + if err != nil { + return err + } + app := cred.ApplicationID + if err := c.audit(Audit{Action: "push_credential.updated", TargetType: "push_credential", TargetID: credID, + ApplicationID: &app, Before: map[string]any{"status": "active"}, + After: map[string]any{"status": "failed", "failure_reason": reason}, ActorType: "system", ActorID: "system"}); err != nil { + return err + } + if err := c.emit("push_credential.failed", &app, map[string]any{"provider": cred.Provider, "failure_reason": reason}); err != nil { + return err + } + // Owners are live Users, invisible to a test-mode transaction, so the + // email goes out from a live-mode read after commit. + provider := cred.Provider + c.After(func(ctx context.Context) { c.s.emailCredentialOwners(ctx, app, provider, reason) }) + return nil +} + +func (s *Server) emailCredentialOwners(ctx context.Context, app, provider, reason string) { + var owners []string + err := s.system(ctx, false, db.Settings{}, func(c *Call) error { + var err error + owners, err = db.All[string](ctx, c.q, `select to_jsonb(u.email) from applications a join users u on u.id = a.owner_user_id + where a.id = :a + union select to_jsonb(u.email) from applications a + join organization_memberships m on m.organization_id = a.owner_organization_id and m.role = 'org_admin' and m.status = 'active' + join users u on u.id = m.user_id where a.id = :a`, db.Args{"a": app}) + return err + }) + if err != nil { + s.Log.Error("push credential owners", "application", app, "err", err) + return + } + for _, to := range owners { + if err := s.Email.Send(ctx, email.Message{To: to, Kind: "push_credential_failed", + Subject: "A push credential for " + app + " was rejected", + Text: "The " + provider + " push credential for " + app + " was rejected by the provider (" + reason + "). " + + "Pushes to that provider are paused until you replace it: PATCH /v1/applications/" + app + "/push-credentials/" + provider + "."}); err != nil { + s.Log.Error("email push credential owner", "application", app, "err", err) + } + } +} diff --git a/internal/core/push_delivery.go b/internal/core/push_delivery.go new file mode 100644 index 0000000..3f8068d --- /dev/null +++ b/internal/core/push_delivery.go @@ -0,0 +1,645 @@ +package core + +import ( + "context" + "encoding/json" + "fmt" + "strings" + "time" + + "github.com/CompositeCode/substratalapps.com/internal/db" + "github.com/CompositeCode/substratalapps.com/internal/ids" + "github.com/CompositeCode/substratalapps.com/internal/push" +) + +// pushSender is the configured provider client, or a log-only one. +func (s *Server) pushSender() push.Sender { + if s.Push == nil { + return push.Log{Logger: s.Log} + } + return s.Push +} + +// ── Creating a notification and its deliveries ───────────────────────── + +// notificationSpec is one notification to create, from an Application's +// send or from a platform alert. +type notificationSpec struct { + UserID string + AppID *string // nil for a platform alert + Category string + Priority string + Title string + Body *string + Data map[string]string + ActionURL *string + Badge *int + Sound *string + CollapseKey *string + TTL int + Push bool + SenderType string + SenderID string +} + +func (sp notificationSpec) source() string { + if sp.AppID == nil { + return "platform" + } + return "application" +} + +// pushNotification is the provider-facing view of a spec. +func (sp notificationSpec) pushNotification(id string) push.Notification { + return push.Notification{ID: id, ApplicationID: sp.AppID, Source: sp.source(), Category: sp.Category, + Priority: sp.Priority, Title: sp.Title, Body: sp.Body, Data: sp.Data, ActionURL: sp.ActionURL, + Badge: sp.Badge, Sound: sp.Sound, CollapseKey: sp.CollapseKey, TTL: time.Duration(sp.TTL) * time.Second} +} + +type pushTarget struct { + DeviceID string `json:"id"` + AppID string `json:"application_id"` + Provider string `json:"provider"` + CredOK *bool `json:"cred_ok"` // nil: no credential for the provider + PushOn bool `json:"push_on"` +} + +// pushEnabledSQL resolves notifications.push for (d.user_id, d.application_id): +// the per-app override, then global Settings, and a missing key is on. +const pushEnabledSQL = `coalesce( + (select case jsonb_typeof(o.overrides->'notifications'->'push') when 'boolean' then (o.overrides->'notifications'->>'push')::boolean end + from app_settings o where o.user_id = d.user_id and o.application_id = d.application_id), + (select case jsonb_typeof(s.notifications->'push') when 'boolean' then (s.notifications->>'push')::boolean end + from settings s where s.user_id = d.user_id), + true)` + +// planDeliveries picks the devices a notification is pushed to, or why it +// isn't pushed at all (API Reference → Push Notifications → skip_reason). +func (c *Call) planDeliveries(sp notificationSpec) ([]pushTarget, string, error) { + if !sp.Push { + return nil, "push_not_requested", nil + } + if sp.AppID != nil { + var on bool + if err := db.One(c.ctx, c.q, &on, `select to_jsonb(`+pushEnabledSQL+`) from (select :u::text as user_id, :a::text as application_id) d`, + db.Args{"u": sp.UserID, "a": *sp.AppID}); err != nil { + return nil, "", err + } + if !on { + return nil, "push_disabled", nil + } + } + appFilter := "d.application_id = :a" + args := db.Args{"u": sp.UserID} + if sp.AppID != nil { + args["a"] = *sp.AppID + } else { + // A device belongs to one Application and only its credentials + // reach it, so platform alerts go through apps that opt in. + appFilter = "d.application_id in (select id from applications where receives_platform_notifications)" + } + devs, err := db.All[pushTarget](c.ctx, c.q, `select jsonb_build_object('id', d.id, 'application_id', d.application_id, + 'provider', d.provider, 'push_on', `+pushEnabledSQL+`, + 'cred_ok', (select p.status = 'active' from push_credentials p where p.application_id = d.application_id and p.provider = d.provider)) + from devices d where d.user_id = :u and d.status = 'active' and `+appFilter+` + order by d.last_registered_at desc, d.id`, args) + if err != nil { + return nil, "", err + } + if len(devs) == 0 { + return nil, "no_devices", nil + } + var on []pushTarget + for _, d := range devs { + if d.PushOn { + on = append(on, d) + } + } + if len(on) == 0 { + return nil, "push_disabled", nil + } + var ok []pushTarget + anyCred := false + for _, d := range on { + if d.CredOK != nil { + anyCred = true + if *d.CredOK { + ok = append(ok, d) + } + } + } + switch { + case len(ok) > 0: + return ok, "", nil + case anyCred: + return nil, "credential_failed", nil + } + return nil, "no_credentials", nil +} + +// createNotification writes a notification and one pending delivery per +// target device in the request's transaction, the push outbox, and nudges +// the worker once it commits. +func (c *Call) createNotification(id string, sp notificationSpec) error { + targets, skip, err := c.planDeliveries(sp) + if err != nil { + return err + } + status := "queued" + var skipArg any + if skip != "" { + status, skipArg = "skipped", skip + } + var tenant any + if sp.AppID != nil { + var t string + if err := db.One(c.ctx, c.q, &t, `select to_jsonb(tenant_id) from applications where id = :a`, db.Args{"a": *sp.AppID}); err != nil { + return err + } + tenant = t + } + data := sp.Data + if data == nil { + data = map[string]string{} + } + var badge any + if sp.Badge != nil { + badge = int64(*sp.Badge) + } + if _, err := c.q.Exec(c.ctx, `insert into notifications (id, user_id, application_id, tenant_id, source, category, priority, + title, body, data, action_url, badge, sound, collapse_key, ttl_seconds, push_requested, push_status, skip_reason, + push_devices_targeted, sender_type, sender_id, test_mode) + values (:id, :u, :a, :t, :src, :cat, :pri, :title, :body, :data::jsonb, :url, :badge, :sound, :ck, :ttl, :push, + :st, :skip, :n, :stype, :sid, :tm)`, + db.Args{"id": id, "u": sp.UserID, "a": sp.AppID, "t": tenant, "src": sp.source(), "cat": sp.Category, + "pri": sp.Priority, "title": sp.Title, "body": sp.Body, "data": db.JSON(data), "url": sp.ActionURL, + "badge": badge, "sound": sp.Sound, "ck": sp.CollapseKey, "ttl": int64(sp.TTL), "push": sp.Push, + "st": status, "skip": skipArg, "n": int64(len(targets)), "stype": sp.SenderType, "sid": sp.SenderID, + "tm": c.testMode()}); err != nil { + return err + } + for _, t := range targets { + if _, err := c.q.Exec(c.ctx, `insert into notification_deliveries (id, notification_id, device_id, provider, next_attempt_at, test_mode) + values (:id, :n, :d, :p, now(), :tm)`, + db.Args{"id": ids.New(ids.NotificationDelivery), "n": id, "d": t.DeviceID, "p": t.Provider, "tm": c.testMode()}); err != nil { + return err + } + } + if len(targets) > 0 { + c.After(func(ctx context.Context) { c.s.Notifier.EventCommitted(ctx, id) }) + } else if skip != "" && sp.AppID != nil && pushFailureSkip(skip) { + return c.emitPushFailed(id) + } + return nil +} + +// pushFailureSkip reports whether a skip is a failure an Application +// should hear about, rather than a choice (push_disabled, push_not_requested). +func pushFailureSkip(reason string) bool { + return reason == "no_devices" || reason == "no_credentials" || reason == "credential_failed" +} + +func (c *Call) emitPushFailed(notificationID string) error { + var n struct { + UserID string `json:"user_id"` + AppID *string `json:"application_id"` + Category string `json:"category"` + Push json.RawMessage `json:"push"` + } + if err := db.One(c.ctx, c.q, &n, `select jsonb_build_object('user_id', n.user_id, 'application_id', n.application_id, + 'category', n.category, 'push', `+notificationPushJSON+`) from notifications n where n.id = :id`, + db.Args{"id": notificationID}); err != nil { + return err + } + if n.AppID == nil { + return nil // platform alerts are between the platform and the User + } + return c.emit("notification.push_failed", n.AppID, map[string]any{"notification_id": notificationID, + "user_id": n.UserID, "category": n.Category, "push": n.Push}) +} + +// ── Platform alerts (API Reference → Push Notifications → Platform alerts) ── + +// platformAlertPriority is the catalog: each category and its priority. +var platformAlertPriority = map[string]string{ + "security.new_sign_in": "high", + "security.password_changed": "high", + "security.email_changed": "high", + "security.mfa_disabled": "high", + "security.session_reuse_detected": "high", + "account.organization_invitation": "normal", + "account.access_granted": "normal", +} + +// platformAlert raises one platform alert for an active User, in the +// current transaction. An action that also revokes sessions raises it +// first, so its deliveries target the devices that were active. +func (c *Call) platformAlert(userID, category, title string, body *string, data map[string]string) error { + pri, ok := platformAlertPriority[category] + if !ok { + return fmt.Errorf("unknown platform alert %q", category) + } + var status string + if err := db.One(c.ctx, c.q, &status, `select to_jsonb(status) from users where id = :u`, db.Args{"u": userID}); err != nil { + if err == db.ErrNotFound { + return nil + } + return err + } + if status != "active" { + return nil + } + var sound *string + if pri == "high" { + sound = strp("default") + } + return c.createNotification(ids.New(ids.Notification), notificationSpec{UserID: userID, Category: category, + Priority: pri, Title: title, Body: body, Data: data, Sound: sound, TTL: 86400, Push: true, + SenderType: "system", SenderID: "system"}) +} + +// alertNewSignIn raises security.new_sign_in when a login's user agent +// hasn't been seen in the User's sessions for 90 days. A User's very first +// session (signup) raises nothing. +func (c *Call) alertNewSignIn(userID, sessionID, userAgent, ip string) error { + var seen struct { + Prior bool `json:"prior"` + Known bool `json:"known"` + } + if err := db.One(c.ctx, c.q, &seen, `select jsonb_build_object( + 'prior', exists (select 1 from sessions where user_id = :u and id <> :s), + 'known', exists (select 1 from sessions where user_id = :u and id <> :s + and user_agent is not distinct from :ua and created_at > now() - interval '90 days'))`, + db.Args{"u": userID, "s": sessionID, "ua": nullIfEmpty(userAgent)}); err != nil { + return err + } + if !seen.Prior || seen.Known { + return nil + } + where := "an unrecognized device" + if userAgent != "" { + where = userAgent + } + data := map[string]string{"session_id": sessionID, "user_agent": userAgent} + if ip != "" { + data["ip_address"] = ip + } + return c.platformAlert(userID, "security.new_sign_in", "New sign-in to your Substratal account", + strp("Signed in from "+where+". If this wasn't you, change your password."), data) +} + +func nullIfEmpty(s string) any { + if s == "" { + return nil + } + return s +} + +// developerSources are the Entitlement sources a developer or admin +// grants. Platform-managed free and subscription rows never raise +// account.access_granted: signing up, subscribing to Standard, or a new +// free Application would otherwise alert once per app. +var developerSources = map[string]bool{"purchase": true, "trial": true, "admin_grant": true, "org_seat": true} + +// alertAccessGranted raises account.access_granted when access a developer +// or admin granted becomes the User's resolved access, except in the first +// 60 seconds after signup. +func (c *Call) alertAccessGranted(p pair, r resolved) error { + if r.Res.ActiveVia == nil || !developerSources[r.Res.ActiveVia.Source] { + return nil + } + var info struct { + New bool `json:"new"` + Name string `json:"name"` + } + if err := db.One(c.ctx, c.q, &info, `select jsonb_build_object( + 'new', (select created_at > now() - interval '60 seconds' from users where id = :u), + 'name', (select name from applications where id = :a))`, db.Args{"u": p.User, "a": p.App}); err != nil { + return err + } + if info.New { + return nil + } + return c.platformAlert(p.User, "account.access_granted", "You now have access to "+info.Name, nil, + map[string]string{"application_id": p.App}) +} + +// ── Settling a notification's push summary ───────────────────────────── + +// notificationPushJSON is the push summary object of a notification n. +const notificationPushJSON = `jsonb_build_object('requested', n.push_requested, 'status', n.push_status, + 'devices_targeted', n.push_devices_targeted, 'devices_sent', n.push_devices_sent, + 'devices_failed', n.push_devices_failed, 'skip_reason', n.skip_reason)` + +// settleNotification recomputes a notification's push summary from its +// deliveries and, when the push ends without reaching any device, tells +// the Application through notification.push_failed. +func (c *Call) settleNotification(notificationID string) error { + var st struct { + Status string `json:"push_status"` + Pending int `json:"pending"` + Sent int `json:"sent"` + Failed int `json:"failed"` + Skipped []string `json:"skipped"` + } + err := db.One(c.ctx, c.q, &st, `select jsonb_build_object('push_status', n.push_status, + 'pending', count(d.id) filter (where d.status = 'pending'), + 'sent', count(d.id) filter (where d.status = 'sent'), + 'failed', count(d.id) filter (where d.status = 'failed'), + 'skipped', coalesce(array_agg(d.error) filter (where d.status = 'skipped'), '{}')) + from notifications n left join notification_deliveries d on d.notification_id = n.id + where n.id = :n group by n.id`, db.Args{"n": notificationID}) + if err == db.ErrNotFound { + return nil + } + if err != nil { + return err + } + if st.Status != "queued" { + return nil // already settled (or never queued) + } + status, skip := "queued", "" + switch { + case st.Pending > 0: + case st.Sent > 0: + status = "sent" + case st.Failed > 0: + status = "failed" + default: + status, skip = "skipped", skipFromDeliveries(st.Skipped) + } + var skipArg any + if skip != "" { + skipArg = skip + } + if _, err := c.q.Exec(c.ctx, `update notifications set push_status = :st, skip_reason = :skip, + push_devices_sent = :sent, push_devices_failed = :failed where id = :n`, + db.Args{"st": status, "skip": skipArg, "sent": int64(st.Sent), "failed": int64(st.Failed), "n": notificationID}); err != nil { + return err + } + if status == "failed" || (status == "skipped" && pushFailureSkip(skip)) { + return c.emitPushFailed(notificationID) + } + return nil +} + +// skipFromDeliveries names why a push whose every delivery was skipped +// never went out. +func skipFromDeliveries(errs []string) string { + has := func(e string) bool { + for _, x := range errs { + if x == e { + return true + } + } + return false + } + switch { + case has("expired"): + return "expired" + case has("credential_failed"): + return "credential_failed" + case has("credential_removed"): + return "no_credentials" + } + return "no_devices" +} + +// ── The delivery worker ──────────────────────────────────────────────── + +// PushRetrySchedule is the delay before attempts 2 through 4 (API +// Reference → Push Notifications → Delivery). +var PushRetrySchedule = []time.Duration{30 * time.Second, 2 * time.Minute, 10 * time.Minute} + +const ( + pushMaxAttempts = 4 + pushLease = 2 * time.Minute +) + +// RunPush delivers every due push until none are left. The API nudges it +// after a send commits; the scheduled sweep runs it too, so the outbox is +// the only source of truth. +func (s *Server) RunPush(ctx context.Context) error { + for i := 0; i < 50; i++ { + n, err := s.DeliverPush(ctx, 50) + if err != nil || n == 0 { + return err + } + } + return nil +} + +// DeliverPush claims up to limit due deliveries with a short lease, so no +// transaction stays open across a provider call, and sends each. +func (s *Server) DeliverPush(ctx context.Context, limit int) (int, error) { + type claim struct { + ID string `json:"id"` + TestMode bool `json:"test_mode"` + } + var batch []claim + err := s.DB.Tx(ctx, db.Settings{AllModes: true}, func(q db.Querier) error { + var err error + batch, err = db.All[claim](ctx, q, `update notification_deliveries set next_attempt_at = now() + :lease::interval + where id in (select id from notification_deliveries where status = 'pending' and next_attempt_at <= now() + order by next_attempt_at limit :lim for update skip locked) + returning jsonb_build_object('id', id, 'test_mode', test_mode)`, + db.Args{"lease": fmt.Sprintf("%d seconds", int(pushLease.Seconds())), "lim": int64(limit)}) + return err + }) + if err != nil { + return 0, err + } + for _, cl := range batch { + if err := s.deliverOne(ctx, cl.ID, cl.TestMode); err != nil { + s.Log.Error("push delivery", "delivery", cl.ID, "err", err) + } + } + return len(batch), nil +} + +// notificationRow is the stored notification, as the worker reads it. +type notificationRow struct { + ID string `json:"id"` + ApplicationID *string `json:"application_id"` + Source string `json:"source"` + Category string `json:"category"` + Priority string `json:"priority"` + Title string `json:"title"` + Body *string `json:"body"` + Data map[string]string `json:"data"` + ActionURL *string `json:"action_url"` + Badge *int `json:"badge"` + Sound *string `json:"sound"` + CollapseKey *string `json:"collapse_key"` + TTL int `json:"ttl_seconds"` + CreatedAt time.Time `json:"created_at"` +} + +func (n notificationRow) push() push.Notification { + return push.Notification{ID: n.ID, ApplicationID: n.ApplicationID, Source: n.Source, Category: n.Category, + Priority: n.Priority, Title: n.Title, Body: n.Body, Data: n.Data, ActionURL: n.ActionURL, Badge: n.Badge, + Sound: n.Sound, CollapseKey: n.CollapseKey, TTL: time.Duration(n.TTL) * time.Second} +} + +// deliverOne sends one delivery in three steps: decide in a transaction, +// send outside any transaction, and record the outcome in another. +func (s *Server) deliverOne(ctx context.Context, id string, testMode bool) error { + var x struct { + Attempts int `json:"attempts"` + Provider string `json:"provider"` + DeviceID *string `json:"device_id"` + Note notificationRow `json:"notification"` + } + var msg push.Message + var dev device + var credID string + ready := false + err := s.system(ctx, testMode, db.Settings{}, func(c *Call) error { + if err := db.One(ctx, c.q, &x, `select jsonb_build_object('attempts', d.attempts, 'provider', d.provider, + 'device_id', d.device_id, 'notification', to_jsonb(n)) + from notification_deliveries d join notifications n on n.id = d.notification_id + where d.id = :id and d.status = 'pending' for update of d`, db.Args{"id": id}); err != nil { + if err == db.ErrNotFound { + return nil // settled meanwhile + } + return err + } + finish := func(status, why string) error { + if _, err := c.q.Exec(ctx, `update notification_deliveries set status = :st, error = :e, next_attempt_at = null + where id = :id`, db.Args{"st": status, "e": why, "id": id}); err != nil { + return err + } + return c.settleNotification(x.Note.ID) + } + // ttl_seconds is how long a provider may keep trying an offline + // device; 0 means now or never. + deadline := x.Note.CreatedAt.Add(time.Duration(x.Note.TTL) * time.Second) + if (x.Note.TTL == 0 && x.Attempts > 0) || (x.Note.TTL > 0 && c.now.After(deadline)) { + if x.Attempts == 0 { + return finish("skipped", "expired") + } + return finish("failed", "expired") + } + if x.DeviceID == nil { + return finish("skipped", "device_deleted") + } + devs, err := c.queryDevices("d.id = :id", db.Args{"id": *x.DeviceID}) + if err != nil { + return err + } + if len(devs) == 0 { + return finish("skipped", "device_deleted") + } + dev = devs[0] + // A security alert raised by an action that also signs the User out + // still reaches the devices that were active when it was raised. + if dev.Status != "active" && !strings.HasPrefix(x.Note.Category, "security.") { + return finish("skipped", "device_inactive") + } + cred, err := c.loadCredential(dev.ApplicationID, x.Provider) + if err != nil { + if isAPIError(err) { + return finish("skipped", "credential_removed") + } + return err + } + if cred.Status != "active" { + return finish("skipped", "credential_failed") + } + pc, err := s.decryptCredential(ctx, cred) + if err != nil { + return err + } + target, err := s.decryptTarget(ctx, dev) + if err != nil { + return err + } + msg, credID, ready = push.Message{Credential: pc, Target: target, Notification: x.Note.push()}, cred.ID, true + return nil + }) + if err != nil || !ready { + return err + } + res := s.pushSender().Send(ctx, msg) + return s.system(ctx, testMode, db.Settings{}, func(c *Call) error { + attempts := x.Attempts + 1 + var provStatus any + if res.Status != 0 { + provStatus = int64(res.Status) + } + var why any + if res.Reason != "" { + why = res.Reason + } + record := func(status string, next *time.Time) error { + var nextArg any + if next != nil { + nextArg = db.Time(*next) + } + _, err := c.q.Exec(ctx, `update notification_deliveries set status = :st, attempts = :a, provider_status = :ps, + error = :e, next_attempt_at = :next::timestamptz, + sent_at = case when :st = 'sent' then now() else sent_at end + where id = :id`, db.Args{"st": status, "a": int64(attempts), "ps": provStatus, "e": why, "next": nextArg, "id": id}) + return err + } + if res.Outcome == push.Retry { + next := c.now.Add(PushRetrySchedule[min(attempts, len(PushRetrySchedule))-1]) + if ra := c.now.Add(res.RetryAfter); ra.After(next) { + next = ra + } + deadline := x.Note.CreatedAt.Add(time.Duration(x.Note.TTL) * time.Second) + if attempts < pushMaxAttempts && x.Note.TTL > 0 && !next.After(deadline) { + return record("pending", &next) + } + if err := record("failed", nil); err != nil { + return err + } + return c.settleNotification(x.Note.ID) + } + status := "failed" + if res.Outcome == push.Sent { + status, why = "sent", nil + } + if err := record(status, nil); err != nil { + return err + } + if err := c.applyOutcome(dev, credID, res); err != nil { + return err + } + return c.settleNotification(x.Note.ID) + }) +} + +// ── Maintenance (daily) ──────────────────────────────────────────────── + +// PushMaintenance runs the push lifecycle rules: devices not re-registered +// for 270 days go inactive (stale), inactive devices are deleted 30 days +// after deactivation, and notifications and delivery results age out of +// the inbox after 90 and 30 days. +func (s *Server) PushMaintenance(ctx context.Context) error { + return eachMode(func(tm bool) error { + return s.system(ctx, tm, db.Settings{}, func(c *Call) error { + if err := c.deactivateDevices("d.last_registered_at < now() - interval '270 days'", nil, "stale"); err != nil { + return err + } + old, err := c.queryDevices("d.status = 'inactive' and d.deactivated_at < now() - interval '30 days' limit 1000", nil) + if err != nil { + return err + } + for _, d := range old { + if err := c.deleteDevice(d); err != nil { + return err + } + } + for _, q := range []string{ + `delete from notification_deliveries where created_at < now() - interval '30 days'`, + `delete from notifications where created_at < now() - interval '90 days'`, + } { + if _, err := c.q.Exec(ctx, q, nil); err != nil { + return err + } + } + return nil + }) + }) +} diff --git a/internal/core/quickstart_test.go b/internal/core/quickstart_test.go index e25ead6..27dd5a0 100644 --- a/internal/core/quickstart_test.go +++ b/internal/core/quickstart_test.go @@ -14,7 +14,7 @@ func TestMain(m *testing.M) { os.Exit(code) } -// TestQuickstart runs docs/quickstart.md end to end: PLAN.md's Phase 1 +// TestQuickstart runs docs/quickstart.md end to end: the Phase 1 // acceptance test. func TestQuickstart(t *testing.T) { e := testenv.New(t) diff --git a/internal/core/ratelimit.go b/internal/core/ratelimit.go index c3be7d4..c5f9e87 100644 --- a/internal/core/ratelimit.go +++ b/internal/core/ratelimit.go @@ -28,6 +28,7 @@ var principalLimits = map[string]limit{ "organizations.grantEntitlement": perMinute(20), "permissions.getEffective": perMinute(300), "auth.createAppToken": perMinute(300), + "notifications.send": perMinute(600), } // publicLimits are per client IP on unauthenticated endpoints. diff --git a/internal/core/resolve.go b/internal/core/resolve.go index 6845ae2..57f577d 100644 --- a/internal/core/resolve.go +++ b/internal/core/resolve.go @@ -162,6 +162,9 @@ func (c *Call) flip(w *accessWatch, grantedReason, revokedReason string) error { if a { err = c.emit("access.granted", &app, map[string]any{ "user_id": p.User, "application_id": p.App, "reason": grantedReason, "entitlement_status": "active"}) + if err == nil { + err = c.alertAccessGranted(p, after[p]) + } } else { err = c.emit("access.revoked", &app, map[string]any{ "user_id": p.User, "application_id": p.App, "reason": revokedReason, diff --git a/internal/core/roles.go b/internal/core/roles.go index 10b86ff..2f2dcaa 100644 --- a/internal/core/roles.go +++ b/internal/core/roles.go @@ -37,6 +37,7 @@ var PlatformPermissions = []PlatformPermission{ {"webhooks.manage", "platform", "Manage any webhook subscription, and create platform-scoped ones."}, {"api_keys.manage", "platform", "Create, rotate, and revoke any API Key."}, {"tenants.manage", "platform", "View any Tenant and request a tier change on a customer's behalf."}, + {"notifications.send", "platform", "Send notifications to any User on behalf of any Application. On an app-scoped key, confined to that Application's own users."}, } func isPlatformPermission(k string) bool { @@ -48,9 +49,9 @@ func isPlatformPermission(k string) bool { return false } -// appConfinable are the four platform permissions an app-scoped key may +// appConfinable are the five platform permissions an app-scoped key may // carry in confined form (API Keys → App-confined permissions). -var appConfinable = []string{"entitlements.manage", "roles.manage", "users.list", "audit.view"} +var appConfinable = []string{"entitlements.manage", "roles.manage", "users.list", "audit.view", "notifications.send"} // ── GET /v1/permissions ──────────────────────────────────────────────── diff --git a/internal/core/server.go b/internal/core/server.go index b9bb5e0..0014500 100644 --- a/internal/core/server.go +++ b/internal/core/server.go @@ -22,6 +22,7 @@ import ( "github.com/CompositeCode/substratalapps.com/internal/email" "github.com/CompositeCode/substratalapps.com/internal/httpx" "github.com/CompositeCode/substratalapps.com/internal/ids" + "github.com/CompositeCode/substratalapps.com/internal/push" ) // Notifier is told that a webhook outbox event committed, so delivery can @@ -63,7 +64,8 @@ type Deps struct { Encrypter auth.Encrypter Email email.Sender Notifier Notifier - Billing Billing // nil disables checkout/portal (503 billing_unavailable) + Billing Billing // nil disables checkout/portal (503 billing_unavailable) + Push push.Sender // nil logs pushes instead of sending them (PUSH_BACKEND=log) Cursors httpx.Cursors Log *slog.Logger diff --git a/internal/core/subscriptions.go b/internal/core/subscriptions.go index 8ccedc8..b12ba2c 100644 --- a/internal/core/subscriptions.go +++ b/internal/core/subscriptions.go @@ -874,7 +874,7 @@ func (s *Server) BillingCreateOrganizationPortalSession(w http.ResponseWriter, r }) } -// ── Stripe sync (DEPLOYMENT.md → Webhook handling) ───────────────────── +// ── Stripe sync (infra/README.md → Webhook handling) ───────────────────── // StripeSubscription is the subscription state the sync applies, read // fresh from Stripe by the billing package. diff --git a/internal/core/system.go b/internal/core/system.go index ef1fcef..b700700 100644 --- a/internal/core/system.go +++ b/internal/core/system.go @@ -172,6 +172,9 @@ func (s *Server) erase(ctx context.Context, tm bool, uid string) error { `delete from refresh_tokens where session_id in (select id from sessions where user_id = :u)`, `delete from sessions where user_id = :u`, `delete from auth_tokens where user_id = :u`, + // Push: notifications (their deliveries cascade) and devices. + `delete from notifications where user_id = :u`, + `delete from devices where user_id = :u`, } { if _, err := c.q.Exec(ctx, q, db.Args{"u": uid}); err != nil { return err @@ -361,6 +364,9 @@ func (s *Server) PurgeTestData(ctx context.Context) error { old := `test_mode and created_at < now() - interval '30 days'` users := `(select id from users where ` + old + `)` for _, q := range []string{ + `delete from notification_deliveries where ` + old, + `delete from notifications where ` + old, + `delete from devices where ` + old, `delete from webhook_deliveries where ` + old, `delete from webhook_events where ` + old, `delete from webhook_subscriptions where ` + old, diff --git a/internal/core/users.go b/internal/core/users.go index 7285c7a..6ef4120 100644 --- a/internal/core/users.go +++ b/internal/core/users.go @@ -270,6 +270,10 @@ func (s *Server) UsersUpdate(w http.ResponseWriter, r *http.Request, id gen.User if err := c.sendVerification(u.ID, newEmail, nil, nil); err != nil { return err } + if err := c.platformAlert(u.ID, "security.email_changed", "Your Substratal email address was changed", + strp("Your account's email address is now "+newEmail+". If this wasn't you, contact support."), nil); err != nil { + return err + } if err := c.audit(Audit{Action: "user.email_changed", TargetType: "user", TargetID: u.ID, TargetUserID: &u.ID, Before: map[string]any{"email": u.Email}, After: map[string]any{"email": newEmail}}); err != nil { return err diff --git a/internal/core/webhooks.go b/internal/core/webhooks.go index 8cc1baa..618cc9a 100644 --- a/internal/core/webhooks.go +++ b/internal/core/webhooks.go @@ -22,7 +22,9 @@ var WebhookEventTypes = []string{ "entitlement.granted", "entitlement.disabled", "entitlement.revoked", "entitlement.expired", "entitlement.updated", "entitlement.member_scope_changed", "entitlement.deleted", "role.assigned", "role.removed", "user.suspended", "user.reactivated", "user.deleted", - "organization.member_added", "organization.member_removed", "application.review_status_changed", "webhook.test", + "organization.member_added", "organization.member_removed", "application.review_status_changed", + "device.registered", "device.deactivated", "device.deleted", "notification.push_failed", "push_credential.failed", + "webhook.test", } // PublishedWebhookVersions are the payload versions a subscription may pin. diff --git a/internal/db/dataapi/dataapi.go b/internal/db/dataapi/dataapi.go index 8728e1d..c2df7e9 100644 --- a/internal/db/dataapi/dataapi.go +++ b/internal/db/dataapi/dataapi.go @@ -1,5 +1,5 @@ // Package dataapi implements db.DB over the RDS Data API: IAM-signed HTTPS -// to Aurora Serverless v2, with no VPC attachment (see DEPLOYMENT.md → Why +// to Aurora Serverless v2, with no VPC attachment (see infra/README.md → Why // no VPC). It's what every Lambda runs in AWS. package dataapi diff --git a/internal/ids/ids.go b/internal/ids/ids.go index 3c56fe1..0720d8f 100644 --- a/internal/ids/ids.go +++ b/internal/ids/ids.go @@ -17,19 +17,23 @@ import ( // Resource id prefixes. const ( - User = "usr_" - Organization = "org_" - Tenant = "tnt_" - TierChange = "tcr_" - Entitlement = "ent_" - AuditEvent = "evt_" - Webhook = "whk_" - APIKey = "key_" - UserIdentity = "uid_" - SSOConnection = "ssc_" - Session = "ses_" - WebhookEvent = "wev_" - Delivery = "dlv_" + User = "usr_" + Organization = "org_" + Tenant = "tnt_" + TierChange = "tcr_" + Entitlement = "ent_" + AuditEvent = "evt_" + Webhook = "whk_" + APIKey = "key_" + UserIdentity = "uid_" + SSOConnection = "ssc_" + Session = "ses_" + WebhookEvent = "wev_" + Delivery = "dlv_" + Device = "dvc_" + PushCredential = "pcr_" + Notification = "ntf_" + NotificationDelivery = "ndl_" ) // Credential prefixes. Never logged, never echoed after issuance. diff --git a/internal/jobs/jobs.go b/internal/jobs/jobs.go index 9337d05..f1ac082 100644 --- a/internal/jobs/jobs.go +++ b/internal/jobs/jobs.go @@ -1,4 +1,4 @@ -// Package jobs is every scheduled job (DEPLOYMENT.md → Build checklist, +// Package jobs is every scheduled job (infra/README.md → Build checklist, // step 6), one entry point each. EventBridge Scheduler invokes the jobs // Lambda with {"job": ""}; locally, `make job name=` runs one. package jobs @@ -47,10 +47,17 @@ func (r *Runner) Jobs() map[string]func(context.Context) error { r.Base.Log.Info("entitlement sweep", "expired", exp, "started", started) return err }, - // every minute: the safety net under SQS nudges + // every minute: the safety net under SQS nudges, for webhooks and + // push deliveries alike (push events can produce webhooks, so + // pushes go first) "webhook-dispatch": func(ctx context.Context) error { + if err := r.Core.RunPush(ctx); err != nil { + return err + } return r.Base.Deliverer().Run(ctx) }, + // daily: stale devices, inactive-device deletion, inbox retention + "push-maintenance": func(ctx context.Context) error { return r.Core.PushMaintenance(ctx) }, // hourly "erasure-cascade": func(ctx context.Context) error { n, err := r.Core.RunErasures(ctx) diff --git a/internal/platform/config/config.go b/internal/platform/config/config.go index 8cba737..e84770f 100644 --- a/internal/platform/config/config.go +++ b/internal/platform/config/config.go @@ -46,6 +46,10 @@ type Config struct { // Empty = the devserver's in-process worker polls instead. WebhookQueueURL string + // Push: "live" sends to APNs, FCM, and Web Push; "log" only logs + // (the local default, so no Apple or Google credentials are needed). + PushBackend string + // Stripe. StripeSecretKey string StripeWebhookSecret string @@ -92,6 +96,7 @@ func Load() (Config, error) { SMTPAddr: get("SMTP_ADDR", "localhost:51025"), EmailFrom: get("EMAIL_FROM", "Substratal "), WebhookQueueURL: os.Getenv("WEBHOOK_QUEUE_URL"), + PushBackend: get("PUSH_BACKEND", "log"), StripeSecretKey: os.Getenv("STRIPE_SECRET_KEY"), StripeWebhookSecret: os.Getenv("STRIPE_WEBHOOK_SECRET"), StripePrices: StripePrices{ @@ -107,6 +112,9 @@ func Load() (Config, error) { if c.EmailBackend == "smtp" && os.Getenv("EMAIL_BACKEND") == "" { c.EmailBackend = "ses" } + if os.Getenv("PUSH_BACKEND") == "" { + c.PushBackend = "live" + } } return c, nil } diff --git a/internal/platform/wire/deps.go b/internal/platform/wire/deps.go index 5a1ec12..60e286b 100644 --- a/internal/platform/wire/deps.go +++ b/internal/platform/wire/deps.go @@ -20,6 +20,7 @@ import ( "github.com/CompositeCode/substratalapps.com/internal/email" "github.com/CompositeCode/substratalapps.com/internal/httpx" "github.com/CompositeCode/substratalapps.com/internal/platform/config" + "github.com/CompositeCode/substratalapps.com/internal/push" "github.com/CompositeCode/substratalapps.com/internal/webhooks" ) @@ -111,6 +112,23 @@ func Billing(cfg config.Config, log *slog.Logger) core.Billing { return billing.New(cfg.StripeSecretKey, cfg.StripePrices, log) } +// Push builds the push sender for PUSH_BACKEND. +func Push(cfg config.Config, log *slog.Logger) (push.Sender, error) { + switch cfg.PushBackend { + case "live": + l := push.NewLive() + if cfg.Env != "production" && os.Getenv("WEBHOOKS_ALLOW_PRIVATE") == "true" { + l.AllowHTTP = true // a local Web Push fake on http://localhost + } else { + l.CheckHost = core.CheckPublicHost + } + return l, nil + case "log": + return push.Log{Logger: log}, nil + } + return nil, fmt.Errorf("wire: unknown PUSH_BACKEND %q", cfg.PushBackend) +} + // Base is everything every entry point shares. type Base struct { Config config.Config @@ -119,6 +137,7 @@ type Base struct { Signer auth.Signer Encrypter auth.Encrypter Email email.Sender + Push push.Sender } // NewBase builds the shared dependencies. @@ -144,7 +163,11 @@ func NewBase(ctx context.Context) (*Base, error) { if err != nil { return nil, err } - return &Base{Config: cfg, Log: log, DB: d, Signer: signer, Encrypter: enc, Email: mail}, nil + sender, err := Push(cfg, log) + if err != nil { + return nil, err + } + return &Base{Config: cfg, Log: log, DB: d, Signer: signer, Encrypter: enc, Email: mail, Push: sender}, nil } // Deliverer is the webhook delivery engine. @@ -174,7 +197,7 @@ func (b *Base) Server(n core.Notifier) *core.Server { } core.AllowPrivateWebhookTargets = b.Config.Env != "production" && os.Getenv("WEBHOOKS_ALLOW_PRIVATE") == "true" return core.New(core.Deps{ - DB: b.DB, Signer: b.Signer, Encrypter: b.Encrypter, Email: b.Email, Notifier: n, + DB: b.DB, Signer: b.Signer, Encrypter: b.Encrypter, Email: b.Email, Notifier: n, Push: b.Push, Billing: Billing(b.Config, b.Log), Cursors: httpx.Cursors{Key: key}, Log: b.Log, Issuer: b.Config.Issuer, PublicBaseURL: b.Config.PublicBaseURL, AuthorizeURL: b.Config.AuthorizeURL, DashboardURL: b.Config.DashboardURL, CORSOrigins: b.Config.CORSOrigins, diff --git a/internal/push/credentials.go b/internal/push/credentials.go new file mode 100644 index 0000000..5980331 --- /dev/null +++ b/internal/push/credentials.go @@ -0,0 +1,209 @@ +package push + +import ( + "crypto" + "crypto/ecdh" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/rsa" + "crypto/sha256" + "crypto/x509" + "encoding/base64" + "encoding/json" + "encoding/pem" + "errors" + "fmt" + "math/big" + "sort" + "strings" +) + +// ServiceAccount is the subset of a Google service-account JSON key the +// FCM HTTP v1 OAuth flow needs. +type ServiceAccount struct { + Type string `json:"type"` + ProjectID string `json:"project_id"` + PrivateKeyID string `json:"private_key_id"` + PrivateKey string `json:"private_key"` + ClientEmail string `json:"client_email"` + TokenURI string `json:"token_uri"` +} + +// googleTokenURI is where service-account assertions are exchanged. +const googleTokenURI = "https://oauth2.googleapis.com/token" + +// ParseAPNsKey parses a .p8 key: PEM PKCS#8 holding a P-256 ECDSA key. +func ParseAPNsKey(pemText string) (*ecdsa.PrivateKey, error) { + block, _ := pem.Decode([]byte(pemText)) + if block == nil { + return nil, errors.New("not a PEM block") + } + k, err := x509.ParsePKCS8PrivateKey(block.Bytes) + if err != nil { + return nil, fmt.Errorf("not a PKCS#8 private key: %w", err) + } + ec, ok := k.(*ecdsa.PrivateKey) + if !ok || ec.Curve != elliptic.P256() { + return nil, errors.New("not a P-256 ECDSA key") + } + return ec, nil +} + +// ParseServiceAccount parses and checks a service-account JSON key. +func ParseServiceAccount(raw []byte) (*ServiceAccount, error) { + var sa ServiceAccount + if err := json.Unmarshal(raw, &sa); err != nil { + return nil, errors.New("not a JSON object") + } + var missing []string + for name, v := range map[string]string{"type": sa.Type, "project_id": sa.ProjectID, + "private_key": sa.PrivateKey, "client_email": sa.ClientEmail} { + if strings.TrimSpace(v) == "" { + missing = append(missing, name) + } + } + if len(missing) > 0 { + sort.Strings(missing) + return nil, fmt.Errorf("missing %s", strings.Join(missing, ", ")) + } + if sa.Type != "service_account" { + return nil, errors.New(`type must be "service_account"`) + } + if _, err := parseRSAKey(sa.PrivateKey); err != nil { + return nil, fmt.Errorf("private_key: %w", err) + } + if sa.TokenURI == "" { + sa.TokenURI = googleTokenURI + } + return &sa, nil +} + +func parseRSAKey(pemText string) (*rsa.PrivateKey, error) { + block, _ := pem.Decode([]byte(pemText)) + if block == nil { + return nil, errors.New("not a PEM block") + } + if k, err := x509.ParsePKCS8PrivateKey(block.Bytes); err == nil { + if rk, ok := k.(*rsa.PrivateKey); ok { + return rk, nil + } + return nil, errors.New("not an RSA key") + } + return x509.ParsePKCS1PrivateKey(block.Bytes) +} + +// GenerateVAPID returns a fresh VAPID key pair: the public key as base64url +// of the uncompressed P-256 point (what PushManager.subscribe takes as +// applicationServerKey), and the private scalar as base64url. +func GenerateVAPID() (public, private string, err error) { + k, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + return "", "", err + } + pub, err := k.PublicKey.Bytes() + if err != nil { + return "", "", err + } + priv, err := k.Bytes() + if err != nil { + return "", "", err + } + return b64(pub), b64(priv), nil +} + +func parseVAPID(private string) (*ecdsa.PrivateKey, error) { + raw, err := unb64(private) + if err != nil { + return nil, err + } + return ecdsa.ParseRawPrivateKey(elliptic.P256(), raw) +} + +// ── JWTs ─────────────────────────────────────────────────────────────── + +func b64(b []byte) string { return base64.RawURLEncoding.EncodeToString(b) } + +// unb64 accepts base64url with or without padding (browsers and libraries +// differ), and standard base64 as a courtesy. +func unb64(s string) ([]byte, error) { + s = strings.TrimRight(strings.TrimSpace(s), "=") + if b, err := base64.RawURLEncoding.DecodeString(s); err == nil { + return b, nil + } + return base64.RawStdEncoding.DecodeString(s) +} + +func jwtSigningInput(header, claims any) (string, error) { + h, err := json.Marshal(header) + if err != nil { + return "", err + } + c, err := json.Marshal(claims) + if err != nil { + return "", err + } + return b64(h) + "." + b64(c), nil +} + +// signES256 returns a JWT signed with ES256: the signature is r‖s, each +// left-padded to 32 bytes (RFC 7518 §3.4), not ASN.1 DER. +func signES256(k *ecdsa.PrivateKey, header, claims any) (string, error) { + in, err := jwtSigningInput(header, claims) + if err != nil { + return "", err + } + sum := sha256.Sum256([]byte(in)) + r, s, err := ecdsa.Sign(rand.Reader, k, sum[:]) + if err != nil { + return "", err + } + sig := make([]byte, 64) + r.FillBytes(sig[:32]) + s.FillBytes(sig[32:]) + return in + "." + b64(sig), nil +} + +// verifyES256 checks an ES256 JWT's signature (tests and fakes). +func verifyES256(pub *ecdsa.PublicKey, token string) bool { + i := strings.LastIndexByte(token, '.') + if i < 0 { + return false + } + sig, err := unb64(token[i+1:]) + if err != nil || len(sig) != 64 { + return false + } + sum := sha256.Sum256([]byte(token[:i])) + return ecdsa.Verify(pub, sum[:], new(big.Int).SetBytes(sig[:32]), new(big.Int).SetBytes(sig[32:])) +} + +func signRS256(k *rsa.PrivateKey, header, claims any) (string, error) { + in, err := jwtSigningInput(header, claims) + if err != nil { + return "", err + } + sum := sha256.Sum256([]byte(in)) + sig, err := rsa.SignPKCS1v15(rand.Reader, k, crypto.SHA256, sum[:]) + if err != nil { + return "", err + } + return in + "." + b64(sig), nil +} + +// ValidateSubscriptionKeys checks a browser subscription's keys: p256dh +// must be an uncompressed P-256 point and auth 16 bytes, both base64url. +func ValidateSubscriptionKeys(p256dh, auth string) error { + pub, err := unb64(p256dh) + if err != nil { + return errors.New("p256dh isn't base64url") + } + if _, err := ecdh.P256().NewPublicKey(pub); err != nil { + return errors.New("p256dh isn't an uncompressed P-256 public key") + } + a, err := unb64(auth) + if err != nil || len(a) != 16 { + return errors.New("auth must be 16 bytes of base64url") + } + return nil +} diff --git a/internal/push/ece.go b/internal/push/ece.go new file mode 100644 index 0000000..0685f63 --- /dev/null +++ b/internal/push/ece.go @@ -0,0 +1,139 @@ +package push + +import ( + "crypto/aes" + "crypto/cipher" + "crypto/ecdh" + "crypto/hkdf" + "crypto/rand" + "crypto/sha256" + "encoding/binary" + "errors" + "io" +) + +// recordSize is the aes128gcm record size advertised in the header. Push +// payloads are a single record, so it only needs to exceed the payload. +const recordSize = 4096 + +// encryptAES128GCM encrypts plaintext for a browser subscription per +// RFC 8291 (Message Encryption for Web Push) over RFC 8188's aes128gcm +// content coding, as a single record: +// +// salt(16) ‖ rs(4) ‖ idlen(1) ‖ keyid = as_public(65) ‖ AEAD(plaintext ‖ 0x02) +// +// asKey and salt are normally fresh per message; they're parameters so the +// RFC's own test vector can pin them. +func encryptAES128GCM(plaintext []byte, uaPublic, authSecret []byte, asKey *ecdh.PrivateKey, salt []byte) ([]byte, error) { + curve := ecdh.P256() + ua, err := curve.NewPublicKey(uaPublic) + if err != nil { + return nil, errors.New("p256dh isn't a valid P-256 public key") + } + if len(authSecret) != 16 { + return nil, errors.New("auth must be 16 bytes") + } + if asKey == nil { + if asKey, err = curve.GenerateKey(rand.Reader); err != nil { + return nil, err + } + } + if salt == nil { + salt = make([]byte, 16) + if _, err := io.ReadFull(rand.Reader, salt); err != nil { + return nil, err + } + } + shared, err := asKey.ECDH(ua) + if err != nil { + return nil, err + } + asPublic := asKey.PublicKey().Bytes() + + // IKM = HKDF(auth_secret, ecdh_secret, "WebPush: info" ‖ 0x00 ‖ ua_public ‖ as_public, 32) + prkKey, err := hkdf.Extract(sha256.New, shared, authSecret) + if err != nil { + return nil, err + } + keyInfo := append(append([]byte("WebPush: info\x00"), uaPublic...), asPublic...) + ikm, err := hkdf.Expand(sha256.New, prkKey, string(keyInfo), 32) + if err != nil { + return nil, err + } + prk, err := hkdf.Extract(sha256.New, ikm, salt) + if err != nil { + return nil, err + } + cek, err := hkdf.Expand(sha256.New, prk, "Content-Encoding: aes128gcm\x00", 16) + if err != nil { + return nil, err + } + nonce, err := hkdf.Expand(sha256.New, prk, "Content-Encoding: nonce\x00", 12) + if err != nil { + return nil, err + } + block, err := aes.NewCipher(cek) + if err != nil { + return nil, err + } + gcm, err := cipher.NewGCM(block) + if err != nil { + return nil, err + } + // A single (and so last) record: the plaintext, then the 0x02 delimiter. + sealed := gcm.Seal(nil, nonce, append(append([]byte{}, plaintext...), 0x02), nil) + + out := make([]byte, 0, 16+4+1+len(asPublic)+len(sealed)) + out = append(out, salt...) + out = binary.BigEndian.AppendUint32(out, recordSize) + out = append(out, byte(len(asPublic))) + out = append(out, asPublic...) + return append(out, sealed...), nil +} + +// decryptAES128GCM reverses encryptAES128GCM with the user agent's private +// key. Browsers do this; it's here for tests and the local fake. +func decryptAES128GCM(body []byte, uaKey *ecdh.PrivateKey, authSecret []byte) ([]byte, error) { + if len(body) < 21 { + return nil, errors.New("short body") + } + salt, idlen := body[:16], int(body[20]) + if len(body) < 21+idlen { + return nil, errors.New("short body") + } + asPub, err := ecdh.P256().NewPublicKey(body[21 : 21+idlen]) + if err != nil { + return nil, err + } + shared, err := uaKey.ECDH(asPub) + if err != nil { + return nil, err + } + prkKey, _ := hkdf.Extract(sha256.New, shared, authSecret) + keyInfo := append(append([]byte("WebPush: info\x00"), uaKey.PublicKey().Bytes()...), asPub.Bytes()...) + ikm, _ := hkdf.Expand(sha256.New, prkKey, string(keyInfo), 32) + prk, _ := hkdf.Extract(sha256.New, ikm, salt) + cek, _ := hkdf.Expand(sha256.New, prk, "Content-Encoding: aes128gcm\x00", 16) + nonce, _ := hkdf.Expand(sha256.New, prk, "Content-Encoding: nonce\x00", 12) + block, err := aes.NewCipher(cek) + if err != nil { + return nil, err + } + gcm, err := cipher.NewGCM(block) + if err != nil { + return nil, err + } + plain, err := gcm.Open(nil, nonce, body[21+idlen:], nil) + if err != nil { + return nil, err + } + // Strip padding back to the 0x02 last-record delimiter. + i := len(plain) - 1 + for i >= 0 && plain[i] == 0 { + i-- + } + if i < 0 || plain[i] != 2 { + return nil, errors.New("missing record delimiter") + } + return plain[:i], nil +} diff --git a/internal/push/fake.go b/internal/push/fake.go new file mode 100644 index 0000000..eddc985 --- /dev/null +++ b/internal/push/fake.go @@ -0,0 +1,80 @@ +package push + +import ( + "context" + "crypto/sha256" + "log/slog" + "sync" +) + +func sha256Sum(b []byte) []byte { + s := sha256.Sum256(b) + return s[:] +} + +// Recorder is a Sender that records every message instead of sending it, +// for tests and for local development without Apple or Google +// credentials. Every push succeeds unless an outcome is scripted for its +// token (or, for Web Push, its endpoint). +type Recorder struct { + mu sync.Mutex + sent []Message + outcomes map[string]Result +} + +// Send records m and returns its scripted result, or Sent. +func (r *Recorder) Send(_ context.Context, m Message) Result { + r.mu.Lock() + defer r.mu.Unlock() + r.sent = append(r.sent, m) + if res, ok := r.outcomes[TargetKey(m.Target)]; ok { + return res + } + return Result{Outcome: Sent, Status: 200} +} + +// Script makes every push to key (a token or a Web Push endpoint) return res. +func (r *Recorder) Script(key string, res Result) { + r.mu.Lock() + defer r.mu.Unlock() + if r.outcomes == nil { + r.outcomes = map[string]Result{} + } + r.outcomes[key] = res +} + +// Sent returns every recorded message, oldest first. +func (r *Recorder) Sent() []Message { + r.mu.Lock() + defer r.mu.Unlock() + return append([]Message(nil), r.sent...) +} + +// Reset forgets recorded messages and scripted outcomes. +func (r *Recorder) Reset() { + r.mu.Lock() + defer r.mu.Unlock() + r.sent, r.outcomes = nil, nil +} + +// TargetKey is what identifies a device to a provider: its token, or a +// browser subscription's endpoint. +func TargetKey(t Target) string { + if t.Subscription != nil { + return t.Subscription.Endpoint + } + return t.Token +} + +// Log is a Sender that only logs (PUSH_BACKEND=log): every push succeeds, +// and nothing leaves the machine. The token is never logged. +type Log struct{ Logger *slog.Logger } + +// Send logs m and reports success. +func (l Log) Send(_ context.Context, m Message) Result { + if l.Logger != nil { + l.Logger.Info("push (log backend)", "provider", m.Target.Provider, "notification", m.Notification.ID, + "category", m.Notification.Category, "title", m.Notification.Title) + } + return Result{Outcome: Sent, Status: 200} +} diff --git a/internal/push/live.go b/internal/push/live.go new file mode 100644 index 0000000..a179006 --- /dev/null +++ b/internal/push/live.go @@ -0,0 +1,456 @@ +package push + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/url" + "strconv" + "strings" + "sync" + "time" +) + +// Provider endpoints. +const ( + APNsProduction = "https://api.push.apple.com" + APNsSandbox = "https://api.sandbox.push.apple.com" + FCMEndpoint = "https://fcm.googleapis.com" + fcmScope = "https://www.googleapis.com/auth/firebase.messaging" +) + +// sendTimeout is the per-request timeout (API Reference → Push +// Notifications → Delivery: a 5-second timeout counts as a retryable failure). +const sendTimeout = 5 * time.Second + +// Live sends to the real providers. The zero value is not usable; build it +// with NewLive. Provider auth (the APNs provider JWT, the FCM OAuth access +// token, and the VAPID JWT) is cached per credential, so a warm worker +// signs or fetches each at most once per validity window. +type Live struct { + // Client must speak HTTP/2 to reach APNs; Go's default transport does + // over TLS. + Client *http.Client + + APNsProductionURL string + APNsSandboxURL string + FCMURL string + // TokenURL, when set, overrides every service account's token_uri. + TokenURL string + + // CheckHost vets a Web Push endpoint's host at send time (the same + // public-address rule as webhook URLs). Nil skips the check. + CheckHost func(ctx context.Context, host string) error + // AllowHTTP accepts plain-http Web Push endpoints (local fakes only). + AllowHTTP bool + + Now func() time.Time + + mu sync.Mutex + tokens map[string]cachedToken +} + +type cachedToken struct { + value string + expires time.Time +} + +// NewLive returns a Live sender for the real provider endpoints. +func NewLive() *Live { + return &Live{ + Client: &http.Client{Timeout: sendTimeout, CheckRedirect: noRedirects}, + APNsProductionURL: APNsProduction, APNsSandboxURL: APNsSandbox, FCMURL: FCMEndpoint, + Now: time.Now, + } +} + +func noRedirects(*http.Request, []*http.Request) error { return http.ErrUseLastResponse } + +func (l *Live) now() time.Time { + if l.Now == nil { + return time.Now() + } + return l.Now() +} + +func (l *Live) cached(key string, mint func() (string, time.Time, error)) (string, error) { + l.mu.Lock() + defer l.mu.Unlock() + if l.tokens == nil { + l.tokens = map[string]cachedToken{} + } + if t, ok := l.tokens[key]; ok && l.now().Before(t.expires) { + return t.value, nil + } + v, exp, err := mint() + if err != nil { + return "", err + } + l.tokens[key] = cachedToken{v, exp} + return v, nil +} + +func (l *Live) forget(key string) { + l.mu.Lock() + defer l.mu.Unlock() + delete(l.tokens, key) +} + +// Send sends one message and classifies the answer. +func (l *Live) Send(ctx context.Context, m Message) Result { + start := time.Now() + var r Result + switch m.Target.Provider { + case APNs: + r = l.sendAPNs(ctx, m) + case FCM: + r = l.sendFCM(ctx, m) + case WebPush: + r = l.sendWebPush(ctx, m) + default: + r = Result{Outcome: Failed, Reason: "unknown provider"} + } + r.Duration = time.Since(start) + return r +} + +// do runs one request, returning the status and up to 4 KB of body. A +// transport failure is a retryable timeout or connection error. +func (l *Live) do(ctx context.Context, req *http.Request) (int, []byte, http.Header, *Result) { + rctx, cancel := context.WithTimeout(ctx, sendTimeout) + defer cancel() + resp, err := l.Client.Do(req.WithContext(rctx)) + if err != nil { + reason := "connection_error" + if errors.Is(err, context.DeadlineExceeded) || strings.Contains(err.Error(), "Timeout") { + reason = "timeout" + } + return 0, nil, nil, &Result{Outcome: Retry, Reason: reason} + } + defer func() { _ = resp.Body.Close() }() + b, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10)) + return resp.StatusCode, b, resp.Header, nil +} + +func retryAfter(h http.Header) time.Duration { + if h == nil { + return 0 + } + v := h.Get("Retry-After") + if v == "" { + return 0 + } + if n, err := strconv.Atoi(v); err == nil && n > 0 { + return time.Duration(n) * time.Second + } + if t, err := http.ParseTime(v); err == nil { + return time.Until(t) + } + return 0 +} + +// ── APNs ─────────────────────────────────────────────────────────────── + +// apnsJWTLifetime keeps the provider token inside Apple's 20–60 minute +// refresh window: refreshed after 50 minutes, never more often than 20. +const apnsJWTLifetime = 50 * time.Minute + +func (l *Live) apnsToken(c Credential) (string, error) { + return l.cached("apns:"+c.TeamID+":"+c.KeyID, func() (string, time.Time, error) { + k, err := ParseAPNsKey(c.PrivateKey) + if err != nil { + return "", time.Time{}, err + } + now := l.now() + tok, err := signES256(k, map[string]string{"alg": "ES256", "kid": c.KeyID}, + map[string]any{"iss": c.TeamID, "iat": now.Unix()}) + return tok, now.Add(apnsJWTLifetime), err + }) +} + +func (l *Live) sendAPNs(ctx context.Context, m Message) Result { + c, n := m.Credential, m.Notification + tok, err := l.apnsToken(c) + if err != nil { + return Result{Outcome: CredentialRejected, Reason: "InvalidProviderKey"} + } + base := l.APNsProductionURL + if m.Target.Environment == "sandbox" { + base = l.APNsSandboxURL + } + req, _ := http.NewRequest(http.MethodPost, base+"/3/device/"+url.PathEscape(m.Target.Token), bytes.NewReader(APNsPayload(n))) + req.Header.Set("Authorization", "bearer "+tok) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("apns-topic", c.Topic) + req.Header.Set("apns-push-type", "alert") + if n.Priority == "high" { + req.Header.Set("apns-priority", "10") + } else { + req.Header.Set("apns-priority", "5") + } + exp := "0" + if n.TTL > 0 { + exp = strconv.FormatInt(l.now().Add(n.TTL).Unix(), 10) + } + req.Header.Set("apns-expiration", exp) + if n.CollapseKey != nil { + req.Header.Set("apns-collapse-id", *n.CollapseKey) + } + status, body, h, fail := l.do(ctx, req) + if fail != nil { + return *fail + } + var e struct { + Reason string `json:"reason"` + } + _ = json.Unmarshal(body, &e) + r := Result{Status: status, Reason: e.Reason, RetryAfter: retryAfter(h)} + switch { + case status == http.StatusOK: + r.Outcome = Sent + case status == http.StatusGone, e.Reason == "BadDeviceToken", e.Reason == "DeviceTokenNotForTopic", e.Reason == "Unregistered": + r.Outcome = TokenGone + case e.Reason == "ExpiredProviderToken": + // Ours to fix: mint a fresh token and try again. + l.forget("apns:" + c.TeamID + ":" + c.KeyID) + r.Outcome = Retry + case status == http.StatusForbidden, e.Reason == "TopicDisallowed", e.Reason == "BadTopic": + r.Outcome = CredentialRejected + case status == http.StatusTooManyRequests || status >= 500: + r.Outcome = Retry + default: + r.Outcome = Failed + } + return r +} + +// ── FCM HTTP v1 ──────────────────────────────────────────────────────── + +var errCredential = errors.New("credential rejected") + +func (l *Live) fcmToken(ctx context.Context, sa *ServiceAccount) (string, error) { + return l.cached("fcm:"+sa.ClientEmail, func() (string, time.Time, error) { + key, err := parseRSAKey(sa.PrivateKey) + if err != nil { + return "", time.Time{}, errCredential + } + tokenURL := sa.TokenURI + if l.TokenURL != "" { + tokenURL = l.TokenURL + } + if tokenURL == "" { + tokenURL = googleTokenURI + } + now := l.now() + assertion, err := signRS256(key, map[string]string{"alg": "RS256", "typ": "JWT", "kid": sa.PrivateKeyID}, + map[string]any{"iss": sa.ClientEmail, "scope": fcmScope, "aud": tokenURL, + "iat": now.Unix(), "exp": now.Add(time.Hour).Unix()}) + if err != nil { + return "", time.Time{}, err + } + form := url.Values{"grant_type": {"urn:ietf:params:oauth:grant-type:jwt-bearer"}, "assertion": {assertion}} + req, _ := http.NewRequest(http.MethodPost, tokenURL, strings.NewReader(form.Encode())) + req.Header.Set("Content-Type", "application/x-www-form-urlencoded") + status, body, _, fail := l.do(ctx, req) + if fail != nil { + return "", time.Time{}, fmt.Errorf("token endpoint: %s", fail.Reason) + } + if status == http.StatusBadRequest || status == http.StatusUnauthorized || status == http.StatusForbidden { + return "", time.Time{}, errCredential + } + var tr struct { + AccessToken string `json:"access_token"` + ExpiresIn int `json:"expires_in"` + } + if status != http.StatusOK || json.Unmarshal(body, &tr) != nil || tr.AccessToken == "" { + return "", time.Time{}, fmt.Errorf("token endpoint: HTTP %d", status) + } + life := time.Duration(tr.ExpiresIn) * time.Second + if life <= 0 { + life = time.Hour + } + // Refresh five minutes before Google says it expires. + return tr.AccessToken, now.Add(life - 5*time.Minute), nil + }) +} + +// fcmMessage is the HTTP v1 message body. +func fcmMessage(token string, n Notification) []byte { + notif := map[string]any{"title": n.Title} + if n.Body != nil { + notif["body"] = *n.Body + } + android := map[string]any{"priority": "NORMAL", "ttl": strconv.FormatInt(int64(n.TTL/time.Second), 10) + "s"} + if n.Priority == "high" { + android["priority"] = "HIGH" + } + if n.CollapseKey != nil { + android["collapse_key"] = *n.CollapseKey + } + an := map[string]any{} + if n.Sound != nil { + an["sound"] = *n.Sound + } + if n.Badge != nil { + an["notification_count"] = *n.Badge + } + if len(an) > 0 { + android["notification"] = an + } + b, _ := json.Marshal(map[string]any{"message": map[string]any{ + "token": token, "notification": notif, "data": fcmData(n), "android": android}}) + return b +} + +func (l *Live) sendFCM(ctx context.Context, m Message) Result { + sa := m.Credential.ServiceAccount + if sa == nil { + return Result{Outcome: CredentialRejected, Reason: "missing_service_account"} + } + tok, err := l.fcmToken(ctx, sa) + if errors.Is(err, errCredential) { + return Result{Outcome: CredentialRejected, Reason: "invalid_grant"} + } + if err != nil { + return Result{Outcome: Retry, Reason: "token_unavailable"} + } + req, _ := http.NewRequest(http.MethodPost, l.FCMURL+"/v1/projects/"+url.PathEscape(sa.ProjectID)+"/messages:send", + bytes.NewReader(fcmMessage(m.Target.Token, m.Notification))) + req.Header.Set("Authorization", "Bearer "+tok) + req.Header.Set("Content-Type", "application/json") + status, body, h, fail := l.do(ctx, req) + if fail != nil { + return *fail + } + var e struct { + Error struct { + Status string `json:"status"` + Message string `json:"message"` + Details []struct { + ErrorCode string `json:"errorCode"` + } `json:"details"` + } `json:"error"` + } + _ = json.Unmarshal(body, &e) + code := e.Error.Status + for _, d := range e.Error.Details { + if d.ErrorCode != "" { + code = d.ErrorCode + } + } + r := Result{Status: status, Reason: code, RetryAfter: retryAfter(h)} + switch { + case status == http.StatusOK: + r.Outcome, r.Reason = Sent, "" + case code == "UNREGISTERED", code == "SENDER_ID_MISMATCH", status == http.StatusNotFound: + r.Outcome = TokenGone + case code == "INVALID_ARGUMENT" && strings.Contains(strings.ToLower(e.Error.Message), "registration token"): + r.Outcome = TokenGone + case status == http.StatusUnauthorized || code == "THIRD_PARTY_AUTH_ERROR" || code == "PERMISSION_DENIED" || status == http.StatusForbidden: + l.forget("fcm:" + sa.ClientEmail) + r.Outcome = CredentialRejected + case status == http.StatusTooManyRequests || status >= 500: + r.Outcome = Retry + default: + r.Outcome = Failed + } + return r +} + +// ── Web Push ─────────────────────────────────────────────────────────── + +// vapidLifetime is how long a VAPID JWT is valid (RFC 8292 allows up to +// 24 hours); it's reused until an hour before it expires. +const vapidLifetime = 12 * time.Hour + +func (l *Live) vapidToken(c Credential, audience string) (string, error) { + return l.cached("vapid:"+c.VAPIDPublicKey+":"+audience, func() (string, time.Time, error) { + k, err := parseVAPID(c.VAPIDPrivate) + if err != nil { + return "", time.Time{}, errCredential + } + exp := l.now().Add(vapidLifetime) + tok, err := signES256(k, map[string]string{"typ": "JWT", "alg": "ES256"}, + map[string]any{"aud": audience, "exp": exp.Unix(), "sub": c.Subject}) + return tok, exp.Add(-time.Hour), err + }) +} + +// webPushTopic fits a collapse key into the Topic header's alphabet +// (base64url) and 32-character limit, hashing anything that doesn't. +func webPushTopic(key string) string { + ok := len(key) <= 32 + for _, r := range key { + if !(r >= 'A' && r <= 'Z' || r >= 'a' && r <= 'z' || r >= '0' && r <= '9' || r == '-' || r == '_') { + ok = false + } + } + if ok { + return key + } + return b64(sha256Sum([]byte(key)))[:32] +} + +func (l *Live) sendWebPush(ctx context.Context, m Message) Result { + s := m.Target.Subscription + if s == nil { + return Result{Outcome: TokenGone, Reason: "missing_subscription"} + } + u, err := url.Parse(s.Endpoint) + if err != nil || u.Host == "" || (u.Scheme != "https" && !(l.AllowHTTP && u.Scheme == "http")) { + return Result{Outcome: TokenGone, Reason: "invalid_endpoint"} + } + if l.CheckHost != nil { + if err := l.CheckHost(ctx, u.Hostname()); err != nil { + return Result{Outcome: TokenGone, Reason: "endpoint_not_public"} + } + } + uaPublic, err1 := unb64(s.Keys.P256dh) + auth, err2 := unb64(s.Keys.Auth) + if err1 != nil || err2 != nil { + return Result{Outcome: TokenGone, Reason: "invalid_keys"} + } + body, err := encryptAES128GCM(WebPushPayload(m.Notification), uaPublic, auth, nil, nil) + if err != nil { + return Result{Outcome: TokenGone, Reason: "invalid_keys"} + } + tok, err := l.vapidToken(m.Credential, u.Scheme+"://"+u.Host) + if err != nil { + return Result{Outcome: CredentialRejected, Reason: "invalid_vapid_key"} + } + req, _ := http.NewRequest(http.MethodPost, s.Endpoint, bytes.NewReader(body)) + req.Header.Set("Authorization", "vapid t="+tok+", k="+m.Credential.VAPIDPublicKey) + req.Header.Set("Content-Encoding", "aes128gcm") + req.Header.Set("Content-Type", "application/octet-stream") + req.Header.Set("TTL", strconv.FormatInt(int64(m.Notification.TTL/time.Second), 10)) + if m.Notification.Priority == "high" { + req.Header.Set("Urgency", "high") + } else { + req.Header.Set("Urgency", "normal") + } + if m.Notification.CollapseKey != nil { + req.Header.Set("Topic", webPushTopic(*m.Notification.CollapseKey)) + } + status, _, h, fail := l.do(ctx, req) + if fail != nil { + return *fail + } + r := Result{Status: status, RetryAfter: retryAfter(h)} + switch { + case status >= 200 && status < 300: + r.Outcome = Sent + case status == http.StatusNotFound || status == http.StatusGone: + r.Outcome, r.Reason = TokenGone, "subscription_expired" + case status == http.StatusUnauthorized || status == http.StatusForbidden: + r.Outcome, r.Reason = CredentialRejected, "vapid_rejected" + case status == http.StatusTooManyRequests || status >= 500: + r.Outcome, r.Reason = Retry, http.StatusText(status) + default: + r.Outcome, r.Reason = Failed, http.StatusText(status) + } + return r +} diff --git a/internal/push/push.go b/internal/push/push.go new file mode 100644 index 0000000..4180857 --- /dev/null +++ b/internal/push/push.go @@ -0,0 +1,220 @@ +// Package push sends push notifications through the three providers the +// API supports (API Reference → Push Notifications): Apple Push +// Notification service (APNs), Firebase Cloud Messaging (FCM HTTP v1), and +// Web Push (RFC 8030, with VAPID per RFC 8292 and payload encryption per +// RFC 8291). +// +// It knows nothing about the database. The core package decides who gets a +// notification and records the outcome; this package turns one +// notification for one device into one provider request, and classifies +// the provider's answer so the caller knows whether to retry, deactivate +// the device, or mark the credential failed. +package push + +import ( + "context" + "encoding/json" + "time" +) + +// Providers. +const ( + APNs = "apns" + FCM = "fcm" + WebPush = "web_push" +) + +// Providers lists every supported provider. +var Providers = []string{APNs, FCM, WebPush} + +// Credential is an Application's decrypted credential for one provider. +type Credential struct { + Provider string + + // APNs (token-based authentication). + TeamID string + KeyID string + Topic string + PrivateKey string // the .p8 file: PEM PKCS#8, P-256 + + // FCM. + ServiceAccount *ServiceAccount + + // Web Push (VAPID). + Subject string + VAPIDPublicKey string // base64url, uncompressed P-256 point + VAPIDPrivate string // base64url, 32-byte scalar +} + +// Target is one device: the provider token, or a browser subscription. +type Target struct { + Provider string + Token string // apns, fcm + Environment string // apns: production | sandbox + Subscription *Subscription // web_push +} + +// Subscription is a browser PushSubscription. +type Subscription struct { + Endpoint string `json:"endpoint"` + Keys struct { + P256dh string `json:"p256dh"` + Auth string `json:"auth"` + } `json:"keys"` +} + +// Notification is what a device shows, plus the platform's own block. +type Notification struct { + ID string // ntf_…; empty for a device test + ApplicationID *string + Source string // application | platform + Category string + Priority string // normal | high + Title string + Body *string + Data map[string]string + ActionURL *string + Badge *int + Sound *string + CollapseKey *string + TTL time.Duration +} + +// Message is one push to one device. +type Message struct { + Credential Credential + Target Target + Notification Notification +} + +// Outcome classifies a provider's answer. +type Outcome int + +const ( + // Sent: the provider accepted the push. + Sent Outcome = iota + // Retry: a transient failure (429, 5xx, timeout); try again later. + Retry + // TokenGone: the provider says the device token is invalid or + // unregistered; deactivate the device and never retry. + TokenGone + // CredentialRejected: the provider rejected the Application's + // credential itself; mark it failed. + CredentialRejected + // Failed: a permanent failure for this push only. + Failed +) + +func (o Outcome) String() string { + return [...]string{"sent", "retry", "token_gone", "credential_rejected", "failed"}[o] +} + +// Result is the classified answer to one Send. +type Result struct { + Outcome Outcome + Status int // the provider's HTTP status; 0 when no response arrived + Reason string // the provider's reason code, or "timeout" + RetryAfter time.Duration + Duration time.Duration +} + +// Sender sends one message. +type Sender interface { + Send(ctx context.Context, m Message) Result +} + +// MaxPayload is the largest push payload all three providers accept. +const MaxPayload = 4096 + +// webPushOverhead is the aes128gcm header (86 bytes for a 65-byte key id), +// the padding delimiter, and the AEAD tag, which come out of the 4096-byte +// record Web Push services accept. +const webPushOverhead = 86 + 1 + 16 + +// substratalBlock is the platform's own block in every payload. +func substratalBlock(n Notification) map[string]any { + b := map[string]any{"application_id": n.ApplicationID, "source": n.Source, "category": n.Category} + if n.ID != "" { + b["notification_id"] = n.ID + } else { + b["notification_id"] = nil + } + if n.ActionURL != nil { + b["action_url"] = *n.ActionURL + } + return b +} + +// APNsPayload is the JSON body sent to APNs. +func APNsPayload(n Notification) []byte { + alert := map[string]any{"title": n.Title} + if n.Body != nil { + alert["body"] = *n.Body + } + aps := map[string]any{"alert": alert, "category": n.Category} + if n.Priority == "high" { + aps["interruption-level"] = "time-sensitive" + } else { + aps["interruption-level"] = "active" + } + if n.Badge != nil { + aps["badge"] = *n.Badge + } + if n.Sound != nil { + aps["sound"] = *n.Sound + } + body := map[string]any{} + for k, v := range n.Data { + body[k] = v + } + body["aps"] = aps + body["substratal"] = substratalBlock(n) + b, _ := json.Marshal(body) + return b +} + +// WebPushPayload is the plaintext JSON a service worker receives. +func WebPushPayload(n Notification) []byte { + body := map[string]any{"title": n.Title, "category": n.Category, "priority": n.Priority, + "data": n.Data, "substratal": substratalBlock(n)} + if n.Data == nil { + body["data"] = map[string]string{} + } + if n.Body != nil { + body["body"] = *n.Body + } + if n.Badge != nil { + body["badge"] = *n.Badge + } + if n.CollapseKey != nil { + body["tag"] = *n.CollapseKey + } + b, _ := json.Marshal(body) + return b +} + +// fcmData is the string-only data map FCM carries: the notification's own +// data plus the substratal block as a JSON string. +func fcmData(n Notification) map[string]string { + out := map[string]string{} + for k, v := range n.Data { + out[k] = v + } + sb, _ := json.Marshal(substratalBlock(n)) + out["substratal"] = string(sb) + out["category"] = n.Category + return out +} + +// Size is the largest payload n produces across the providers, in the +// 4096-byte budget they share: the APNs body, and the Web Push plaintext +// plus its encryption overhead. FCM's data message is no larger than +// either. It's what 422 notification_too_large reports. +func Size(n Notification) int { + a := len(APNsPayload(n)) + w := len(WebPushPayload(n)) + webPushOverhead + if w > a { + return w + } + return a +} diff --git a/internal/push/push_test.go b/internal/push/push_test.go new file mode 100644 index 0000000..0ff2f5b --- /dev/null +++ b/internal/push/push_test.go @@ -0,0 +1,372 @@ +package push + +import ( + "context" + "crypto/ecdh" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/rsa" + "crypto/x509" + "encoding/json" + "encoding/pem" + "io" + "net/http" + "net/http/httptest" + "net/url" + "strings" + "sync" + "testing" + "time" +) + +func mustB64(t *testing.T, s string) []byte { + t.Helper() + b, err := unb64(s) + if err != nil { + t.Fatal(err) + } + return b +} + +// RFC 8291 Appendix A: the worked example, byte for byte. +func TestEncryptRFC8291Vector(t *testing.T) { + asPriv, err := ecdh.P256().NewPrivateKey(mustB64(t, "yfWPiYE-n46HLnH0KqZOF1fJJU3MYrct3AELtAQ-oRw")) + if err != nil { + t.Fatal(err) + } + got, err := encryptAES128GCM([]byte("When I grow up, I want to be a watermelon"), + mustB64(t, "BCVxsr7N_eNgVRqvHtD0zTZsEc6-VV-JvLexhqUzORcxaOzi6-AYWXvTBHm4bjyPjs7Vd8pZGH6SRpkNtoIAiw4"), + mustB64(t, "BTBZMqHH6r4Tts7J_aSIgg"), asPriv, mustB64(t, "DGv6ra1nlYgDCS1FRnbzlw")) + if err != nil { + t.Fatal(err) + } + want := "DGv6ra1nlYgDCS1FRnbzlwAAEABBBP4z9KsN6nGRTbVYI_c7VJSPQTBtkgcy27mlmlMoZIIgDll6e3vCYLocInmYWAmS6TlzAC8wEqKK6PBru3jl7A_yl95bQpu6cVPTpK4Mqgkf1CXztLVBSt2Ks3oZwbuwXPXLWyouBWLVWGNWQexSgSxsj_Qulcy4a-fN" + if b64(got) != want { + t.Fatalf("ciphertext mismatch:\n got %s\nwant %s", b64(got), want) + } + uaPriv, _ := ecdh.P256().NewPrivateKey(mustB64(t, "q1dXpw3UpT5VOmu_cf_v6ih07Aems3njxI-JWgLcM94")) + plain, err := decryptAES128GCM(got, uaPriv, mustB64(t, "BTBZMqHH6r4Tts7J_aSIgg")) + if err != nil || string(plain) != "When I grow up, I want to be a watermelon" { + t.Fatalf("decrypt = %q, %v", plain, err) + } +} + +func TestEncryptRoundTrip(t *testing.T) { + ua, _ := ecdh.P256().GenerateKey(rand.Reader) + auth := make([]byte, 16) + _, _ = rand.Read(auth) + msg := []byte(`{"title":"hello"}`) + body, err := encryptAES128GCM(msg, ua.PublicKey().Bytes(), auth, nil, nil) + if err != nil { + t.Fatal(err) + } + if len(body) != len(msg)+webPushOverhead { + t.Fatalf("overhead = %d, want %d", len(body)-len(msg), webPushOverhead) + } + plain, err := decryptAES128GCM(body, ua, auth) + if err != nil || string(plain) != string(msg) { + t.Fatalf("round trip = %q, %v", plain, err) + } +} + +func p8(t *testing.T) (string, *ecdsa.PrivateKey) { + t.Helper() + k, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + der, _ := x509.MarshalPKCS8PrivateKey(k) + return string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: der})), k +} + +func TestParseCredentials(t *testing.T) { + pemText, _ := p8(t) + if _, err := ParseAPNsKey(pemText); err != nil { + t.Fatal(err) + } + if _, err := ParseAPNsKey("not a key"); err == nil { + t.Fatal("garbage parsed as an APNs key") + } + rk, _ := rsa.GenerateKey(rand.Reader, 2048) + der, _ := x509.MarshalPKCS8PrivateKey(rk) + rsaPEM := string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: der})) + if _, err := ParseAPNsKey(rsaPEM); err == nil { + t.Fatal("an RSA key parsed as an APNs key") + } + good, _ := json.Marshal(map[string]string{"type": "service_account", "project_id": "p", "private_key": rsaPEM, "client_email": "a@p.iam.gserviceaccount.com"}) + sa, err := ParseServiceAccount(good) + if err != nil || sa.TokenURI != googleTokenURI { + t.Fatalf("service account = %+v, %v", sa, err) + } + if _, err := ParseServiceAccount([]byte(`{"type":"service_account"}`)); err == nil || + !strings.Contains(err.Error(), "client_email, private_key, project_id") { + t.Fatalf("missing fields err = %v", err) + } + pub, priv, err := GenerateVAPID() + if err != nil || len(mustB64(t, pub)) != 65 || len(mustB64(t, priv)) != 32 { + t.Fatalf("vapid = %q %q %v", pub, priv, err) + } + if _, err := parseVAPID(priv); err != nil { + t.Fatal(err) + } +} + +func note() Notification { + app := "app_timetrack" + body, key, badge, sound := "Due Friday.", "timesheet-due", 1, "default" + return Notification{ID: "ntf_1", ApplicationID: &app, Source: "application", Category: "timesheet.reminder", + Priority: "high", Title: "Timesheet due", Body: &body, Data: map[string]string{"timesheet_id": "ts_1"}, + Badge: &badge, Sound: &sound, CollapseKey: &key, TTL: time.Hour} +} + +// An APNs fake over HTTP/2 + TLS, checking what Apple checks. +func TestAPNs(t *testing.T) { + pemText, key := p8(t) + var mu sync.Mutex + var got *http.Request + var gotBody []byte + srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + mu.Lock() + got, gotBody = r, b + mu.Unlock() + switch { + case strings.HasSuffix(r.URL.Path, "/gone"): + w.WriteHeader(http.StatusGone) + _, _ = w.Write([]byte(`{"reason":"Unregistered"}`)) + case strings.HasSuffix(r.URL.Path, "/badkey"): + w.WriteHeader(http.StatusForbidden) + _, _ = w.Write([]byte(`{"reason":"InvalidProviderToken"}`)) + case strings.HasSuffix(r.URL.Path, "/busy"): + w.Header().Set("Retry-After", "7") + w.WriteHeader(http.StatusTooManyRequests) + _, _ = w.Write([]byte(`{"reason":"TooManyRequests"}`)) + default: + w.WriteHeader(http.StatusOK) + } + })) + srv.EnableHTTP2 = true + srv.StartTLS() + defer srv.Close() + l := NewLive() + l.Client = srv.Client() + l.APNsProductionURL, l.APNsSandboxURL = srv.URL, srv.URL+"/sandbox" + cred := Credential{Provider: APNs, TeamID: "ABCDE12345", KeyID: "XYZ987WVU6", Topic: "com.example.app", PrivateKey: pemText} + send := func(token, env string) Result { + return l.Send(context.Background(), Message{Credential: cred, Target: Target{Provider: APNs, Token: token, Environment: env}, Notification: note()}) + } + + if r := send("abc123", "production"); r.Outcome != Sent { + t.Fatalf("send = %+v", r) + } + mu.Lock() + if got.ProtoMajor != 2 { + t.Errorf("proto = %s, APNs requires HTTP/2", got.Proto) + } + if got.URL.Path != "/3/device/abc123" || got.Header.Get("apns-topic") != "com.example.app" || + got.Header.Get("apns-priority") != "10" || got.Header.Get("apns-push-type") != "alert" || + got.Header.Get("apns-collapse-id") != "timesheet-due" || got.Header.Get("apns-expiration") == "0" { + t.Errorf("headers = %v %v", got.URL.Path, got.Header) + } + tok := strings.TrimPrefix(got.Header.Get("Authorization"), "bearer ") + if !verifyES256(&key.PublicKey, tok) { + t.Error("provider JWT doesn't verify against the .p8 key") + } + var hdr map[string]string + _ = json.Unmarshal(mustB64(t, strings.Split(tok, ".")[0]), &hdr) + if hdr["kid"] != "XYZ987WVU6" || hdr["alg"] != "ES256" { + t.Errorf("jwt header = %v", hdr) + } + var payload map[string]any + _ = json.Unmarshal(gotBody, &payload) + aps := payload["aps"].(map[string]any) + if aps["alert"].(map[string]any)["title"] != "Timesheet due" || aps["interruption-level"] != "time-sensitive" || + payload["timesheet_id"] != "ts_1" || payload["substratal"].(map[string]any)["notification_id"] != "ntf_1" { + t.Errorf("payload = %s", gotBody) + } + mu.Unlock() + + // The provider token is cached, not re-signed per push. + send("abc123", "production") + mu.Lock() + if strings.TrimPrefix(got.Header.Get("Authorization"), "bearer ") != tok { + t.Error("provider JWT wasn't reused") + } + mu.Unlock() + + send("abc123", "sandbox") + mu.Lock() + if !strings.HasPrefix(got.URL.Path, "/sandbox/3/device/") { + t.Errorf("sandbox device went to %s", got.URL.Path) + } + mu.Unlock() + if r := send("gone", "production"); r.Outcome != TokenGone || r.Reason != "Unregistered" { + t.Errorf("410 = %+v", r) + } + if r := send("badkey", "production"); r.Outcome != CredentialRejected { + t.Errorf("403 = %+v", r) + } + if r := send("busy", "production"); r.Outcome != Retry || r.RetryAfter != 7*time.Second { + t.Errorf("429 = %+v", r) + } +} + +func TestFCM(t *testing.T) { + rk, _ := rsa.GenerateKey(rand.Reader, 2048) + der, _ := x509.MarshalPKCS8PrivateKey(rk) + saJSON, _ := json.Marshal(map[string]string{"type": "service_account", "project_id": "proj-1", "private_key_id": "kid1", + "private_key": string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: der})), + "client_email": "sender@proj-1.iam.gserviceaccount.com"}) + sa, err := ParseServiceAccount(saJSON) + if err != nil { + t.Fatal(err) + } + tokenCalls := 0 + var msg map[string]any + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/token": + tokenCalls++ + _ = r.ParseForm() + if r.Form.Get("grant_type") != "urn:ietf:params:oauth:grant-type:jwt-bearer" || strings.Count(r.Form.Get("assertion"), ".") != 2 { + w.WriteHeader(400) + _, _ = w.Write([]byte(`{"error":"invalid_grant"}`)) + return + } + _, _ = w.Write([]byte(`{"access_token":"ya29.test","expires_in":3600}`)) + case "/v1/projects/proj-1/messages:send": + if r.Header.Get("Authorization") != "Bearer ya29.test" { + w.WriteHeader(401) + return + } + var body map[string]any + _ = json.NewDecoder(r.Body).Decode(&body) + msg = body["message"].(map[string]any) + if msg["token"] == "dead" { + w.WriteHeader(404) + _, _ = w.Write([]byte(`{"error":{"code":404,"status":"NOT_FOUND","details":[{"@type":"type.googleapis.com/google.firebase.fcm.v1.FcmError","errorCode":"UNREGISTERED"}]}}`)) + return + } + _, _ = w.Write([]byte(`{"name":"projects/proj-1/messages/1"}`)) + } + })) + defer srv.Close() + l := NewLive() + l.FCMURL, l.TokenURL = srv.URL, srv.URL+"/token" + cred := Credential{Provider: FCM, ServiceAccount: sa} + for i := 0; i < 2; i++ { + if r := l.Send(context.Background(), Message{Credential: cred, Target: Target{Provider: FCM, Token: "tok-1"}, Notification: note()}); r.Outcome != Sent { + t.Fatalf("send = %+v", r) + } + } + if tokenCalls != 1 { + t.Errorf("token endpoint called %d times; the access token should be cached", tokenCalls) + } + data := msg["data"].(map[string]any) + android := msg["android"].(map[string]any) + if data["timesheet_id"] != "ts_1" || !strings.Contains(data["substratal"].(string), "ntf_1") || + android["priority"] != "HIGH" || android["ttl"] != "3600s" || android["collapse_key"] != "timesheet-due" { + t.Errorf("message = %v", msg) + } + if r := l.Send(context.Background(), Message{Credential: cred, Target: Target{Provider: FCM, Token: "dead"}, Notification: note()}); r.Outcome != TokenGone || r.Reason != "UNREGISTERED" { + t.Errorf("unregistered = %+v", r) + } + // A rejected assertion is the credential's fault. + l2 := NewLive() + l2.FCMURL, l2.TokenURL = srv.URL, srv.URL+"/token?x=1" + srv.Config.Handler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(400) + _, _ = w.Write([]byte(`{"error":"invalid_grant"}`)) + }) + if r := l2.Send(context.Background(), Message{Credential: cred, Target: Target{Provider: FCM, Token: "tok-1"}, Notification: note()}); r.Outcome != CredentialRejected { + t.Errorf("invalid_grant = %+v", r) + } +} + +// A browser push service fake that decrypts what it receives, so the test +// proves a real user agent could read it. +func TestWebPush(t *testing.T) { + ua, _ := ecdh.P256().GenerateKey(rand.Reader) + auth := make([]byte, 16) + _, _ = rand.Read(auth) + pub, priv, _ := GenerateVAPID() + vapidPub, _ := unb64(pub) + vk, err := ecdsa.ParseUncompressedPublicKey(elliptic.P256(), vapidPub) + if err != nil { + t.Fatal(err) + } + var plain []byte + var hdr http.Header + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if strings.HasSuffix(r.URL.Path, "/expired") { + w.WriteHeader(http.StatusGone) + return + } + hdr = r.Header + body, _ := io.ReadAll(r.Body) + var err error + plain, err = decryptAES128GCM(body, ua, auth) + if err != nil { + w.WriteHeader(400) + return + } + w.WriteHeader(http.StatusCreated) + })) + defer srv.Close() + l := NewLive() + l.AllowHTTP = true + sub := &Subscription{Endpoint: srv.URL + "/push/abc"} + sub.Keys.P256dh, sub.Keys.Auth = b64(ua.PublicKey().Bytes()), b64(auth) + cred := Credential{Provider: WebPush, Subject: "mailto:ops@example.com", VAPIDPublicKey: pub, VAPIDPrivate: priv} + r := l.Send(context.Background(), Message{Credential: cred, Target: Target{Provider: WebPush, Subscription: sub}, Notification: note()}) + if r.Outcome != Sent || r.Status != 201 { + t.Fatalf("send = %+v", r) + } + var payload map[string]any + if err := json.Unmarshal(plain, &payload); err != nil || payload["title"] != "Timesheet due" { + t.Fatalf("decrypted payload = %s", plain) + } + if hdr.Get("Content-Encoding") != "aes128gcm" || hdr.Get("Urgency") != "high" || hdr.Get("TTL") != "3600" || + hdr.Get("Topic") != "timesheet-due" { + t.Errorf("headers = %v", hdr) + } + authz := hdr.Get("Authorization") + if !strings.HasPrefix(authz, "vapid t=") || !strings.HasSuffix(authz, ", k="+pub) { + t.Fatalf("authorization = %q", authz) + } + jwt := strings.TrimSuffix(strings.TrimPrefix(authz, "vapid t="), ", k="+pub) + if !verifyES256(vk, jwt) { + t.Error("VAPID JWT doesn't verify against the public key") + } + var claims map[string]any + _ = json.Unmarshal(mustB64(t, strings.Split(jwt, ".")[1]), &claims) + su, _ := url.Parse(srv.URL) + if claims["aud"] != "http://"+su.Host || claims["sub"] != "mailto:ops@example.com" { + t.Errorf("claims = %v", claims) + } + sub.Endpoint = srv.URL + "/push/expired" + if r := l.Send(context.Background(), Message{Credential: cred, Target: Target{Provider: WebPush, Subscription: sub}, Notification: note()}); r.Outcome != TokenGone { + t.Errorf("410 = %+v", r) + } + // Plain http is refused unless explicitly allowed. + l.AllowHTTP = false + if r := l.Send(context.Background(), Message{Credential: cred, Target: Target{Provider: WebPush, Subscription: sub}, Notification: note()}); r.Outcome != TokenGone { + t.Errorf("http endpoint = %+v", r) + } +} + +func TestTopicAndSize(t *testing.T) { + if webPushTopic("short_key-1") != "short_key-1" { + t.Error("a safe key should pass through") + } + if got := webPushTopic("timesheet.due.2026-10-12"); len(got) != 32 || strings.Contains(got, ".") { + t.Errorf("hashed topic = %q", got) + } + n := note() + if s := Size(n); s <= len(APNsPayload(n)) && s <= len(WebPushPayload(n)) { + t.Errorf("Size = %d", s) + } + big := strings.Repeat("x", 4000) + n.Body = &big + if Size(n) <= MaxPayload { + t.Error("a 4000-character body must exceed the budget") + } +} diff --git a/internal/testenv/testenv.go b/internal/testenv/testenv.go index 9309ff1..6d326b1 100644 --- a/internal/testenv/testenv.go +++ b/internal/testenv/testenv.go @@ -42,6 +42,7 @@ import ( "github.com/CompositeCode/substratalapps.com/internal/db/pg" "github.com/CompositeCode/substratalapps.com/internal/email" "github.com/CompositeCode/substratalapps.com/internal/httpx" + "github.com/CompositeCode/substratalapps.com/internal/push" "github.com/CompositeCode/substratalapps.com/internal/webhooks" "github.com/CompositeCode/substratalapps.com/migrations" ) @@ -57,6 +58,7 @@ type Env struct { DB db.DB Mail *email.Capture Deliverer *webhooks.Deliverer + Push *push.Recorder // every push the API sends, instead of APNs/FCM/Web Push Signer *auth.LocalSigner Admin string // superadmin access token AdminID string @@ -220,12 +222,13 @@ func NewWith(t testing.TB, opts Options) *Env { core.AllowPrivateWebhookTargets = true deliverer := webhooks.New(d, enc, mail, log) deliverer.AllowPrivate = true - s := core.New(core.Deps{DB: d, Signer: signer(), Encrypter: enc, Email: mail, + recorder := &push.Recorder{} + s := core.New(core.Deps{DB: d, Signer: signer(), Encrypter: enc, Email: mail, Push: recorder, Cursors: httpx.Cursors{Key: []byte("test-cursor-key")}, Log: log, Issuer: "https://api.substratalapps.com", PublicBaseURL: "http://localhost", AuthorizeURL: "https://auth.substratalapps.com/authorize", DashboardURL: "https://dashboard.test", DisableRateLimits: !opts.RateLimits, Billing: opts.Billing}) srv := httptest.NewServer(s.Handler()) - e := &Env{T: t, Server: srv, Core: s, DB: d, Mail: mail, Deliverer: deliverer, Signer: signer(), Validate: true, + e := &Env{T: t, Server: srv, Core: s, DB: d, Mail: mail, Deliverer: deliverer, Push: recorder, Signer: signer(), Validate: true, validator: specValidator(t)} t.Cleanup(func() { srv.Close(); d.Close() }) adminEmail := "admin+" + Unique() + "@example.com" @@ -484,6 +487,14 @@ func (e *Env) Deliver() { } } +// DeliverPush runs the push delivery worker until nothing is due. +func (e *Env) DeliverPush() { + e.T.Helper() + if err := e.Core.RunPush(context.Background()); err != nil { + e.T.Fatalf("deliver push: %v", err) + } +} + // Events lists outbox events of a type, newest first. func (e *Env) Events(eventType string) []map[string]any { e.T.Helper() diff --git a/internal/webhooks/notifier.go b/internal/webhooks/notifier.go index e00feb8..d518e68 100644 --- a/internal/webhooks/notifier.go +++ b/internal/webhooks/notifier.go @@ -34,6 +34,12 @@ type LocalNotifier struct { // NewLocal starts the in-process loop and returns its notifier. func NewLocal(ctx context.Context, d *Deliverer, every time.Duration) *LocalNotifier { + return NewLocalRunner(ctx, d.Log, every, d.Run) +} + +// NewLocalRunner is NewLocal for any pass, such as push delivery followed +// by webhook delivery. +func NewLocalRunner(ctx context.Context, log *slog.Logger, every time.Duration, run func(context.Context) error) *LocalNotifier { n := &LocalNotifier{kick: make(chan struct{}, 1)} go func() { t := time.NewTicker(every) @@ -45,8 +51,8 @@ func NewLocal(ctx context.Context, d *Deliverer, every time.Duration) *LocalNoti case <-n.kick: case <-t.C: } - if err := d.Run(ctx); err != nil && ctx.Err() == nil { - d.Log.Error("webhook worker pass failed", "err", err) + if err := run(ctx); err != nil && ctx.Err() == nil { + log.Error("worker pass failed", "err", err) } } }() diff --git a/migrations/0003_push.sql b/migrations/0003_push.sql new file mode 100644 index 0000000..2f95755 --- /dev/null +++ b/migrations/0003_push.sql @@ -0,0 +1,171 @@ +-- 0003_push: push notifications and the notification inbox (API Reference → +-- Push Notifications; Database Schema → push_credentials, devices, +-- notifications, notification_deliveries). Expand-only. + +-- ── Push credentials: Application configuration, shared by both modes ── + +create table push_credentials ( + id text primary key, + application_id text not null references applications(id), + tenant_id text not null references tenants(id), + provider text not null check (provider in ('apns','fcm','web_push')), + config jsonb not null, + secret_ciphertext text not null, + status text not null default 'active' check (status in ('active','failed')), + failure_reason text, + failed_at timestamptz, + last_used_at timestamptz, + created_at timestamptz not null default now(), + updated_at timestamptz not null default now(), + version integer not null default 1, + unique (application_id, provider), + check ((status = 'failed') = (failed_at is not null)) +); + +-- ── Devices ─────────────────────────────────────────────────────────── + +create table devices ( + id text primary key, + user_id text not null references users(id), + application_id text not null references applications(id), + tenant_id text not null references tenants(id), + session_id text references sessions(id) on delete set null, + provider text not null check (provider in ('apns','fcm','web_push')), + environment text check (environment in ('production','sandbox')), + token_hash text not null, + token_ciphertext text not null, + token_hint text not null, + name text check (char_length(name) <= 100), + app_version text check (char_length(app_version) <= 50), + status text not null default 'active' check (status in ('active','inactive')), + inactive_reason text check (inactive_reason in + ('unregistered','session_revoked','user_suspended','user_deleted','stale','limit','credential_removed')), + deactivated_at timestamptz, + last_registered_at timestamptz not null default now(), + last_delivery_at timestamptz, + test_mode boolean not null default false, + created_at timestamptz not null default now(), + updated_at timestamptz not null default now(), + version integer not null default 1, + check ((provider = 'apns') = (environment is not null)), + check ((status = 'inactive') = (inactive_reason is not null)), + check ((status = 'inactive') = (deactivated_at is not null)) +); +create unique index devices_token_key on devices (application_id, provider, token_hash, test_mode); +create index devices_user_app_active on devices (user_id, application_id) where status = 'active'; +create index devices_session on devices (session_id) where session_id is not null; +create index devices_stale on devices (last_registered_at) where status = 'active'; +create index devices_inactive on devices (deactivated_at) where status = 'inactive'; + +-- ── Notifications: the unit of sending, and the inbox item ──────────── + +create table notifications ( + id text primary key, + user_id text not null references users(id), + application_id text references applications(id), + tenant_id text references tenants(id), + source text not null check (source in ('application','platform')), + category text not null check (category ~ '^[a-z][a-z0-9_]*(\.[a-z0-9_]+)*$' and char_length(category) <= 64), + priority text not null default 'normal' check (priority in ('normal','high')), + title text not null check (char_length(title) between 1 and 120), + body text check (char_length(body) <= 1000), + data jsonb not null default '{}', + action_url text check (char_length(action_url) <= 2048), + badge integer check (badge between 0 and 99999), + sound text check (sound = 'default'), + collapse_key text check (char_length(collapse_key) <= 64), + ttl_seconds integer not null default 86400 check (ttl_seconds between 0 and 2419200), + push_requested boolean not null default true, + push_status text not null check (push_status in ('queued','sent','failed','skipped')), + skip_reason text check (skip_reason in + ('push_not_requested','push_disabled','no_devices','no_credentials','credential_failed','expired')), + push_devices_targeted integer not null default 0, + push_devices_sent integer not null default 0, + push_devices_failed integer not null default 0, + sender_type text not null check (sender_type in ('api_key','user','system')), + sender_id text not null, + read_at timestamptz, + archived_at timestamptz, + test_mode boolean not null default false, + created_at timestamptz not null default now(), + updated_at timestamptz not null default now(), + version integer not null default 1, + check ((source = 'platform') = (application_id is null)), + check ((application_id is null) = (tenant_id is null)), + check ((push_status = 'skipped') = (skip_reason is not null)), + check (source = 'platform' or category !~ '^(security|account|billing)\.') +); +create index notifications_inbox on notifications (user_id, created_at desc, id) where archived_at is null; +create index notifications_user_app on notifications (user_id, application_id, created_at desc); +create index notifications_unread on notifications (user_id) where read_at is null and archived_at is null; +create index notifications_created on notifications (created_at); + +-- ── Deliveries: the push outbox, one per (notification, device) ─────── + +create table notification_deliveries ( + id text primary key, + notification_id text not null references notifications(id) on delete cascade, + device_id text references devices(id) on delete set null, + provider text not null check (provider in ('apns','fcm','web_push')), + status text not null default 'pending' check (status in ('pending','sent','failed','skipped')), + attempts int not null default 0 check (attempts between 0 and 4), + provider_status int, + error text, + next_attempt_at timestamptz, + sent_at timestamptz, + test_mode boolean not null default false, + created_at timestamptz not null default now() +); +create index notification_deliveries_due on notification_deliveries (next_attempt_at) where status = 'pending'; +create index notification_deliveries_notification on notification_deliveries (notification_id); +create index notification_deliveries_device on notification_deliveries (device_id) where device_id is not null; + +-- ── Triggers and test/live isolation ────────────────────────────────── + +do $$ +declare t text; +begin + foreach t in array array['push_credentials','devices','notifications'] loop + execute format('create trigger %I before update on %I for each row execute function substratal_util.touch_updated_at()', + t || '_touch_updated_at', t); + execute format('create trigger %I before update on %I for each row execute function substratal_util.bump_version()', + t || '_bump_version', t); + end loop; + -- push_credentials is Application configuration, like applications: + -- one credential serves both modes, so it has no mode policy. + foreach t in array array['devices','notifications','notification_deliveries'] loop + execute format('alter table %I enable row level security', t); + execute format('alter table %I force row level security', t); + execute format('create policy %I on %I using (substratal_util.mode_visible(test_mode)) with check (substratal_util.mode_visible(test_mode))', + t || '_mode', t); + end loop; +end $$; + +-- ── Applications opt in to platform alert pushes ────────────────────── + +alter table applications add column receives_platform_notifications boolean not null default false; + +-- ── notifications.send: superadmin holds every platform permission ──── + +update roles set permissions = array_append(permissions, 'notifications.send') + where id = 'role_platform_superadmin' and not ('notifications.send' = any(permissions)); + +-- ── Audit ───────────────────────────────────────────────────────────── + +alter table audit_events drop constraint audit_events_action_check; +alter table audit_events add constraint audit_events_action_check check (action in ( + 'user.created','user.updated','user.email_changed','user.suspended','user.reactivated', + 'user.deleted','user.erasure_requested','user.erasure_cancelled','user.erased', + 'user.password_changed','user.password_reset','user.mfa_enabled','user.mfa_disabled','user.mfa_reset', + 'entitlement.granted','entitlement.disabled','entitlement.revoked','entitlement.expired', + 'entitlement.updated','entitlement.member_scope_changed','entitlement.deleted', + 'role.created','role.updated','role.deleted','role.assigned','role.removed', + 'profile.updated','settings.updated', + 'application.created','application.updated','application.review_status_changed', + 'organization.created','organization.updated','organization.member_invited','organization.member_added', + 'organization.member_removed','organization.member_role_changed', + 'subscription.created','subscription.status_changed','subscription.members_changed', + 'tenant.tier_change_requested','tenant.tier_change_request_updated','tenant.tier_changed', + 'api_key.created','api_key.updated','api_key.rotated','api_key.revoked','api_key.restrict_destructive_disabled', + 'webhook.created','webhook.updated','webhook.deleted','webhook.secret_rotated', + 'push_credential.created','push_credential.updated','push_credential.deleted','device.deleted')) not valid; diff --git a/plan-remaining.md b/plan-remaining.md new file mode 100644 index 0000000..2477dd4 --- /dev/null +++ b/plan-remaining.md @@ -0,0 +1,132 @@ +# Plan: what's left to build + +Everything specified for the Platform API that isn't built yet, in one place. What *is* built is described by the [API specification](https://compositecode.github.io/substratalapps.com/) (the system of record), [README.md](README.md) (code, CI/CD, local development), and [infra/README.md](infra/README.md) (deployment architecture as built). One-time setup that has to be done by hand, rather than built, is in [remaining_manual_steps.md](remaining_manual_steps.md). + +This replaces `PLAN.md`, `DEPLOYMENT.md`, and `ORIGINAL-SPEC-DRAFT.md`, which were retired on 2026-10-20 once everything else in them was built and documented where it belongs. They remain in git history. + +1. [Status](#status) +2. [Verify the build in production](#verify-the-build-in-production) +3. [Test-mode User expiry](#test-mode-user-expiry) +4. [First-party web pages](#first-party-web-pages) +5. [Phase 3: marketplace and scale features](#phase-3-marketplace-and-scale-features) +6. [Dedicated tenancy infrastructure](#dedicated-tenancy-infrastructure) +7. [Growth trajectory](#growth-trajectory) +8. [Scale-out](#scale-out) +9. [Scaling the MCP server](#scaling-the-mcp-server) +10. [Rules for every change](#rules-for-every-change) + +--- + +## Status + +As of 2026-10-20, every operation in `openapi.yaml` is implemented, including subscriber billing, push notifications, and the hosted PKCE endpoints. SSO returns `501 not_implemented` until the broker integration, as specified. Production infrastructure exists in AWS (`substratal-prod`, `us-east-1`); the first deploy is finishing (see [remaining_manual_steps.md](remaining_manual_steps.md)). No product decisions are open: each is written into the spec page it governs and recorded in the [Changelog](https://compositecode.github.io/substratalapps.com/changelog/). + +## Verify the build in production + +The phase exit criteria have only been checked against the local devserver. Once production is up, check them there: + +- **Phase 0:** `GET /v1/health` returns `200` through the real Lambda and API Gateway path. (Done: it returned `200` on 2026-10-20.) +- **Phase 1:** the [Quickstart](https://compositecode.github.io/substratalapps.com/quickstart/) runs end to end against production: create a user, grant an app, check access, turn it off. Run `scripts/quickstart.sh` with `SUBSTRATAL_API=https://api.substratalapps.com/v1` and a test-mode key. +- **Phase 2:** a third-party-shaped app signs a user in through the embedded flow, receives `access.revoked` and kills the session, calls introspection, has its `settings_schema` validated and typed on write, and receives a push sent through its app key. A test-mode User and a test-mode Organization subscribe to Standard through Checkout (needs a Stripe test-mode key in a non-production environment, or temporarily in production), gain the standard Applications, and lapse back to Free. + +## Test-mode User expiry + +Specified 2026-10-20 ([Auth → Test-mode Users → Lifetime](https://compositecode.github.io/substratalapps.com/api-reference/auth/#lifetime), [Changelog](https://compositecode.github.io/substratalapps.com/changelog/)); **not yet built**. Each step can land and be tested on its own, in this order: + +1. **Migration `0005_test_user_expiry`** (`0004` is taken by `0004_protected_superadmin`). Add `users.expires_at` and `users.expired_processed_at`. Backfill existing test-mode Users with `created_at + interval '30 days'`, which is when the purge would have removed them anyway, so nothing changes for them. Then add `check ((expires_at is not null) = test_mode)` and `check (expires_at <= created_at + interval '30 days')`, and the partial index `(expires_at) where test_mode and expired_processed_at is null`. Add `test_user_expired` to the `sessions.revoked_reason` and `devices.inactive_reason` checks. +2. **Creation** (`internal/core/users_create.go`, `createUser`). Test-mode Users get `expires_at` from the request, or now + 24 hours by default. Validate the range (≥ 15 minutes from now, ≤ `created_at` + 30 days) as `422 validation_failed` with `expires_at_out_of_range`, and reject it on a live create with `live_user`. This covers both `POST /v1/users` and test-mode signup. +3. **Enforcement at the edge** (`internal/core/principal.go`, the per-request User lookup). Load `expires_at` alongside `status`, and reject an expired test-mode User's tokens the same way a suspended User's are. Login, `token/refresh`, `app-tokens`, `oauth/token`, and the `mfa/verify` step return `403 test_user_expired`. This is what makes expiry exact to the second. The sweep below only tidies up. +4. **Token lifetimes** (`internal/core/auth_session.go`). Cap the access token's `exp`, the app token's `exp`, and the session's `absolute_expires_at` at the User's `expires_at`, so Applications that verify app tokens locally against JWKS stop accepting them on time without calling the API. +5. **`PATCH /v1/users/{id}` `expires_at`** (`users.manage` only; self → `403 status_change_forbidden`). Same range check. Moving it later on an expired User clears `expired_processed_at` and fires `access.granted` (reason `test_user_extended`) for every Application they still have active access to. Writes `user.updated`. +6. **Expiry sweep**, in the existing `entitlement-sweep` job (every 5 minutes, no new schedule). Select `test_mode and expires_at <= now() and expired_processed_at is null`. For each, in one transaction: revoke sessions (`test_user_expired`), deactivate devices, fire `access.revoked` (reason `test_user_expired`) per Application with active access, enqueue the `user.expired` webhook, write the `user.expired` Audit Event (actor `system`), and set `expired_processed_at`. Reuse the suspension path's revoke-and-notify code, with the reason passed in. +7. **Subscriptions.** The test-mode covered-member count excludes expired Users (`internal/core/subscriptions.go`). Live counts are unaffected, because they never include test-mode Users. +8. **OpenAPI.** Add `expires_at` to `User`, to the `POST /v1/users` and signup request bodies, and to `PATCH /v1/users/{id}`. Add `test_user_expired` to the error, revoke-reason, and device-reason enums, and `user.expired` to the webhook and audit enums. Bump `info.version`, then run `make gen`. +9. **Tests.** Integration tests for each of these: the 24-hour default, the range errors, a token rejected after `expires_at` (with a test clock), `exp` capping, the sweep's events firing exactly once, extension restoring access, a live create rejecting `expires_at`, and the backfill leaving existing test Users working. Update the Quickstart acceptance test if it relies on a test User living longer than 24 hours. +10. **Docs.** Remove the "Not yet built" line from the 2026-10-20 Changelog entry and add a short "built" entry, as was done for push notifications. + +## First-party web pages + +The API is complete, but the user-facing pages its emails and redirects point to don't exist yet. Both are clients of the API, not part of it: + +- **Dashboard** (`dashboard.substratalapps.com`, `DASHBOARD_URL`): the email-verification, password-reset, invitation-acceptance, and email-change links land here. Also where a User subscribes to Standard and manages their account, devices, and inbox, and where org admins manage members and the Organization's subscription. It's the first-party browser origin in the API's CORS allow-list. **At minimum, the verify-email and reset-password pages are needed before email-based signup works for real users.** +- **Hosted sign-in** (`auth.substratalapps.com/authorize`, `AUTHORIZE_URL`): the page behind the hosted authorization-code + PKCE flow ([Auth → Getting an app token](https://compositecode.github.io/substratalapps.com/api-reference/auth/#getting-an-app-token)). The endpoints are built; only the page is missing. Needed before any third-party Application, and by any first-party app that prefers it to embedded login. + +## Phase 3: marketplace and scale features + +Matches [Roadmap → Phase 3](https://compositecode.github.io/substratalapps.com/roadmap/#phase-3) — explicitly **not** built speculatively ahead of the triggers named in [Scale-out](#scale-out) and [Applications → The review lifecycle](https://compositecode.github.io/substratalapps.com/domain-model/applications/#the-review-lifecycle): + +- **SSO broker integration** (`sso_connections` becoming real, an actual WorkOS-or-equivalent wiring behind `POST /v1/auth/sso/{provider}/callback`) — the schema and the route exist from Phase 1 per [Auth → Native auth and per-Organization SSO](https://compositecode.github.io/substratalapps.com/api-reference/auth/#native-auth-and-per-organization-sso), deliberately built in advance precisely so this phase is additive wiring, not a migration. Triggered by a real Organization actually asking for it, not a phase-number default. +- Self-service Application registration + review queue. The *process* is already fully specified — reviewer model (no dedicated assignment, any `applications.manage` holder works the FIFO queue), a 5-business-day policy SLA, the `rejected` status with required `review_notes` and edit-to-resubmit, non-cascading suspension — see [Applications → The review lifecycle](https://compositecode.github.io/substratalapps.com/domain-model/applications/#the-review-lifecycle) and [API Reference → Applications → Reviewing a submission](https://compositecode.github.io/substratalapps.com/api-reference/applications/#reviewing-a-submission). What's actually deferred to this phase is just the self-service *submission* endpoint itself (`POST /v1/applications` opening up beyond platform-admin-only) — the review mechanics it feeds into are build-ready today. +- SCIM-style provisioning. +- Richer audit/compliance views (filtering, export) — driven by actual [Compliance](https://compositecode.github.io/substratalapps.com/compliance/) needs (a SOC 2 audit, a customer's data-export request), not built ahead of either. +- **The hosted sign-in page** above, if no first-party app has needed it sooner. + +## Dedicated tenancy infrastructure + +Only the `shared` tier is built: the schema, `tenants` table, tier-change requests, and `migrating` status all exist, but no `isolated` or `dedicated_region` cluster can be provisioned yet, and requests aren't routed per Tenant. Build this when the first customer actually needs it, alongside the business and enterprise pricing that charges for it ([Pricing → Business and Enterprise](https://compositecode.github.io/substratalapps.com/pricing/#business-and-enterprise)). The tier design is in [infra/README.md → Tenancy tiers](infra/README.md#tenancy-tiers--where-they-run). + +To build: + +- Terraform for an additional Aurora Serverless v2 cluster and secret per dedicated Tenant (and, for `dedicated_region`, optionally a regional API Gateway and Lambda deployment). +- Per-Tenant request routing: a lookup from `tenant_id` to cluster ARN and secret, read on each request by the data-access layer. +- Per-request allowed-set RLS policies (`app.allowed_tenant_ids`, with an explicit platform bypass) as a second line of defense inside a dedicated cluster ([NFR → Multi-tenancy](https://compositecode.github.io/substratalapps.com/non-functional-requirements/#multi-tenancy)). +- Tooling for the runbook below. + +### Migration runbook + +A tier change (`shared → isolated`, or either `→ dedicated_region`) is support-run, during a brief scheduled maintenance window: + +1. Provision the destination cluster (new Aurora Serverless v2 cluster; for `dedicated_region`, in the target region) via the same [Build checklist](README.md#one-time-setup-phase-0) steps used for Tier 0 itself. +2. Set the source Tenant's `status` to `migrating` — the API layer treats this as read-only for that Tenant's rows specifically, not a global outage. +3. Snapshot and restore (RDS snapshot copy, or `pg_dump`/`pg_restore` at this data volume) every row carrying that `tenant_id` into the destination cluster. +4. Update the `tenants` lookup row's cluster endpoint, flip `tier`/`region`/`status` back to `active`, decommission the old rows once the cutover is confirmed. +5. Record an Audit Event (`tenant.tier_changed`). + +Because Entitlement, AppProfile, and AppSettings rows all denormalize `tenant_id`, step 3 is a `WHERE tenant_id = ...` export per table — no cross-table join needed to find everything that has to move. + +## Growth trajectory + +The stated plan: dozens of users and a handful of Applications in the first 6–12 months, roughly 10x from there over the following 6–12 months, and another 10x (100x from today) beyond that. + +| Horizon | Rough scale | What this means for the architecture | +|---|---|---| +| 0–6/12 months | Dozens of users, a few Applications | **Tier 0 as specified, unchanged.** The [floor cost table](infra/README.md#illustrative-tier-0-floor-cost) is sized for exactly this. | +| 6–12 months out (~10x) | Low hundreds of users | Watch rate-limit defaults and the Aurora ACU ceiling (not just the floor) — this is when the *first* [Scale-out](#scale-out) triggers are plausible, not guaranteed. Check real metrics rather than upgrading preemptively. | +| 12+ months (~100x) | Low thousands of users | The horizon [Scale-out](#scale-out)'s read-replica, Provisioned Concurrency, and RDS Proxy rows are realistically aimed at. Also the point multi-tenancy query performance (the test/live Row-Level Security policy plus the `tenant_id` filters) is worth profiling specifically, not just trusting by design. | + +## Scale-out + +Nothing below is pre-built into Tier 0 — each is a deliberate upgrade, triggered by a specific, named condition, not a default growth path. + +| Trigger | Change | +|---|---| +| Sustained read load on `effective-permissions` / catalog reads | Add an Aurora read replica; consider API Gateway response caching for read-heavy, low-churn paths. | +| Lambda cold-start latency becomes user-visible on hot paths | Provisioned Concurrency on the `effective-permissions` and entitlement-toggle handlers specifically — not a blanket setting. | +| Sustained high-throughput DB access, where Data API's per-call overhead starts to matter | Move those paths from Data API to direct VPC connections through **RDS Proxy** — this is the point where a NAT Gateway (or a cheaper NAT instance) becomes worth its cost, because throughput now justifies it rather than convenience. | +| Geographically distributed callers, latency-sensitive | CloudFront in front of API Gateway. | +| Hub availability becomes a real incident risk for every downstream app, not a theoretical one | Multi-region active-passive — deliberately not a Tier 0/1 concern; premature multi-region is itself a cost and complexity risk. | +| Webhook delivery volume grows enough to need isolation from the main API's blast radius | Split the worker Lambda's concurrency/alerting from the API handlers' (it already runs as a separate function). | +| Push volume grows enough that APNs connection reuse, or isolation from webhook delivery, matters | Give push delivery its own worker function with reserved concurrency, so long-lived HTTP/2 connections to APNs survive between batches and a burst of pushes can't delay webhooks. | +| More than one engineer shipping concurrently | A second AWS account (via AWS Organizations) for staging, separate from production — Tier 0 runs on a single account with a `dev`/`prod` naming convention, which is enough until this trigger. | +| A specific customer requires dedicated or regional infrastructure | Run the [Tenancy tiers](infra/README.md#tenancy-tiers--where-they-run) migration for that one customer's Tenant — the one scale-out trigger in this table that's customer-specific rather than aggregate-volume-driven. | +| Agent/MCP tool-call traffic needs server-push, resumable streams, or dedicated concurrency | See [MCP server → Scaling the MCP server](#scaling-the-mcp-server) — its own trigger table. | +| Stripe billing volume or complexity outgrows a single webhook Lambda (e.g. real invoice-dunning logic, usage-based billing beyond simple per-seat metering) | Revisit as its own build, not folded into this table speculatively — nothing here is blocked on it. | + +## Scaling the MCP server + +| Trigger | Change | +|---|---| +| Sustained high-frequency tool-call traffic on hot tools | Provisioned Concurrency on the MCP Lambda specifically — same pattern and trigger as the [Scale-out](#scale-out) table's cold-start row. | +| A real need for server-initiated push — mid-call progress notifications, or resumable streams | Move `/mcp` from a plain Lambda-behind-API-Gateway integration to a **Lambda Function URL with streaming responses enabled** (`InvokeMode: RESPONSE_STREAM`) — API Gateway's HTTP API integration buffers the full Lambda response, so it can't carry a chunked `text/event-stream`; a Function URL can. | +| Session/stream-resumption state that no longer fits in a short-lived signed token | A small **DynamoDB** table, keyed by session ID, TTL'd like `idempotency_keys` — ephemeral, high-churn, single-key lookup: the textbook case Aurora is the wrong tool for and DynamoDB is the right one. | +| More than a handful of concurrent agent integrations with genuinely different trust levels | Per-integration API Keys scoped as narrowly as each integration actually needs — an operational practice, not an infrastructure change. | + +## Rules for every change + +These applied from the first line of code and still apply to everything above: + +- **Access Control's permission checks on every endpoint**, from the first commit of it. Retrofitting authorization onto endpoints built "open" is how it gets missed somewhere. +- **Conventions' error shape, pagination, idempotency, and `me` self-addressing** come from the shared helpers. New endpoints get them for free and don't reimplement them. +- **Contract tests against `openapi.yaml`** for every endpoint, the day it's built. +- **The spec pages and the [Changelog](https://compositecode.github.io/substratalapps.com/changelog/) stay current.** An implementation detail that contradicts or settles something in the spec gets written into the page it governs, with a Changelog entry, immediately. +- **Migrations are expand-only per release**, and numbered in the order they'll reach production ([README → Migrations](README.md#migrations-expand-then-contract)). diff --git a/remaining_manual_steps.md b/remaining_manual_steps.md new file mode 100644 index 0000000..697eea3 --- /dev/null +++ b/remaining_manual_steps.md @@ -0,0 +1,116 @@ +# Remaining manual steps + +One-time setup that has to be done by hand, in a console or from a laptop, to take the Platform API live. Code that still has to be built is in [plan-remaining.md](plan-remaining.md); how the deployment works is in [infra/README.md](infra/README.md). + +Already done (2026-10-20): the AWS account (`substratal-prod`, `196936048600`) and its management account; CLI access with `aws login`; DNS moved from Vercel to Route 53; the Organization SCP exception for `substratal-prod`; the bootstrap Terraform (budgets, anomaly detection, state and artifact buckets, GitHub OIDC roles); GitHub variables, the `production` environment limited to `main`, and the `main` ruleset; and a first partial deploy (Aurora, all five Lambdas, API Gateway, the certificate, SES domain). `https://api.substratalapps.com/v1/health` returns `200`. + +Work through these in order. The first four make the API usable; the rest make it ready for real customers. + +1. [Raise the Lambda concurrency quota](#1-raise-the-lambda-concurrency-quota) +2. [Re-run the production deploy](#2-re-run-the-production-deploy) +3. [Set the superadmin's password](#3-set-the-superadmins-password) +4. [Confirm the alert emails](#4-confirm-the-alert-emails) +5. [Get SES out of the sandbox](#5-get-ses-out-of-the-sandbox) +6. [Set up Stripe](#6-set-up-stripe) +7. [Merge push notifications and add push credentials](#7-merge-push-notifications-and-add-push-credentials) +8. [Register Substratal's Applications](#8-register-substratals-applications) +9. [Fix the pull-request plan job](#9-fix-the-pull-request-plan-job) +10. [Legal and compliance](#10-legal-and-compliance) +11. [Optional: transfer the domain registration](#11-optional-transfer-the-domain-registration) + +--- + +## 1. Raise the Lambda concurrency quota + +New AWS accounts can run only **10** concurrent Lambda executions, and AWS keeps 10 unreserved, so the webhook worker's reserved concurrency of 5 fails the deploy (`decreases account's UnreservedConcurrentExecution below its minimum value of [10]`). It would also cap the whole API at 10 simultaneous requests. + +1. Sign in to the **substratal-prod** console session, region **US East (N. Virginia)**. +2. Open https://us-east-1.console.aws.amazon.com/servicequotas/home/services/lambda/quotas/L-B99A9384 (**Service Quotas → AWS services → AWS Lambda → Concurrent executions**). +3. **Request increase at account level**, enter **1000**, and submit. +4. Watch it under **Service Quotas → Quota request history**. It's usually approved within hours, sometimes a couple of days, and AWS emails the account contact. + +From the CLI instead: `aws service-quotas request-service-quota-increase --service-code lambda --quota-code L-B99A9384 --desired-value 1000 --profile substratal-prod --region us-east-1`. + +## 2. Re-run the production deploy + +After the quota shows **1000** (`aws lambda get-account-settings --profile substratal-prod --query AccountLimit.ConcurrentExecutions`): + +- **Easiest:** GitHub → **Actions → Deploy → Run workflow**, branch `main`, **Run workflow**. +- Or from a terminal: `gh workflow run deploy.yml -R CompositeCode/substratalapps.com --ref main`. + +The run builds the Lambdas, applies Terraform (it picks up where the first run stopped), runs the migrations on the new cluster, makes sure the protected superadmin exists (`SUPERADMIN_EMAIL`), and smoke-tests `/v1/health` and JWKS. It takes about 5–10 minutes. Every later merge into `main` deploys the same way, automatically. + +## 3. Set the superadmin's password + +The deploy creates `adronhall@proton.me` as the **protected superadmin**: active, verified, holding `superadmin`, and impossible to suspend, delete, erase, or demote ([Users → The protected superadmin](https://compositecode.github.io/substratalapps.com/api-reference/users/#the-protected-superadmin)). It has no password until you set one. From a clone of the repository, signed in with `aws login --profile substratal-prod`: + +```bash +export AWS_PROFILE=substratal-prod AWS_REGION=us-east-1 +export DATABASE_BACKEND=dataapi \ + DB_CLUSTER_ARN=arn:aws:rds:us-east-1:196936048600:cluster:substratal-prod-db \ + DB_SECRET_ARN=$(aws rds describe-db-clusters --db-cluster-identifier substratal-prod-db \ + --query 'DBClusters[0].MasterUserSecret.SecretArn' --output text) +go run ./cmd/admin bootstrap -email adronhall@proton.me # prompts for the password (12+ characters) +``` + +Then sign in: `curl -s https://api.substratalapps.com/v1/auth/login -H 'content-type: application/json' -d '{"email":"adronhall@proton.me","password":"…"}'`. Turn on MFA right away ([Auth → MFA](https://compositecode.github.io/substratalapps.com/api-reference/auth/#mfa-totp)). + +## 4. Confirm the alert emails + +- **Budget and anomaly alerts:** SNS sent a subscription confirmation ("AWS Notification - Subscription Confirmation") for the `substratal-billing-alerts` topic to `adronhall@proton.me`. Click **Confirm subscription**; until then the $75 warning, the $150 hard stop, and anomaly alerts reach no one. +- **Operational alarms** (5xx rate, Lambda errors and throttles, Aurora near capacity, webhook DLQ, security log): the `alarm_email` Terraform variable is empty, so these alarms have no recipient. Decide the address; setting it is a one-line change to `infra/terraform/prod/variables.tf` and a confirmation click after the next deploy. + +## 5. Get SES out of the sandbox + +SES is still in the **sandbox**: it can only send to verified addresses, so signup verification, password resets, and invitations don't reach real users. + +1. **substratal-prod** console → **Amazon SES → Account dashboard → Request production access**. +2. Mail type **Transactional**; website `https://compositecode.github.io/substratalapps.com/`; describe the use: account verification, password resets, invitations, and security notices for users of Substratal's Applications, sent from `mail.substratalapps.com` (DKIM/SPF/DMARC configured), with bounces and complaints suppressed. +3. AWS usually answers within a day. + +Until then, verify any address you need to test with under **SES → Identities → Create identity → Email address**. + +## 6. Set up Stripe + +The billing code (Free and Standard at $6 per user per month) is built and deployed; Stripe itself isn't set up. See [Pricing](https://compositecode.github.io/substratalapps.com/pricing/) and [infra/README.md → Stripe Billing](infra/README.md#stripe-billing). + +1. **Activate the Stripe account:** business details, bank account, and the public statement descriptor. Live mode needs a privacy policy and terms of service URL (step 10). +2. **Enable Stripe Tax** and add your tax registrations (billing is tax-exclusive). +3. **Create the catalog**, test mode first: `STRIPE_SECRET_KEY=sk_test_… make stripe-catalog` creates the `substratal_standard` Product and the `standard_monthly_usd` Price ($6.00 per user, monthly). Repeat with the live key (`CONFIRM_LIVE=yes`) when going live. +4. **Configure the Customer Portal** (Settings → Billing → Customer portal): allow updating payment methods, viewing invoices, and canceling at period end. **Turn plan switching off.** +5. **Register the webhook endpoint** `https://api.substratalapps.com/internal/stripe/webhook` with these events: `checkout.session.completed`, `customer.subscription.created`, `customer.subscription.updated`, `customer.subscription.deleted`, `invoice.paid`, `invoice.payment_failed`. Copy its signing secret. +6. **Put the keys into the app secret** `substratal-prod/app` (Secrets Manager → `substratal-prod/app` → Retrieve secret value → Edit): set `stripe_secret_key` (a restricted key with Customers, Subscriptions, Checkout Sessions, Customer Portal, and Prices read access) and `stripe_webhook_secret`. Terraform never overwrites these. The Lambdas pick them up on their next cold start; re-running the deploy forces it. +7. **Decide test or live.** Production holds one key. With a test key, the whole flow can be tried end to end with Stripe's test cards; with the live key, test-mode Users can't check out (`409 test_mode_billing_unavailable`, by design). Start with test, then switch to live. + +## 7. Merge push notifications and add push credentials + +Push notifications are built in PR #12 (`dev`), not yet on `main`; until it merges, the 17 push operations return `501` in production. Merging it runs migration `0003_push.sql`, which is independent of `0004_protected_superadmin.sql` that will already be applied. + +After that, for each Application that sends pushes, its owner uploads credentials with `POST /v1/applications/{id}/push-credentials` (and rotates them with `PATCH …/push-credentials/{provider}`): an APNs key (`.p8`, key id, team id, bundle id), an FCM service account, or Web Push VAPID keys ([Push Notifications → Push credentials](https://compositecode.github.io/substratalapps.com/api-reference/notifications/#push-credentials)). + +## 8. Register Substratal's Applications + +For each of Substratal's apps, with the superadmin's token ([Applications → Owner onboarding](https://compositecode.github.io/substratalapps.com/api-reference/applications/#owner-onboarding-end-to-end)): + +1. `POST /v1/applications` with its slug, name, `launch_url`, `redirect_uris`, `permissions`, `available_app_roles`, and **`access_tier`** (`free` or `standard`, which decides who gets it under [Pricing](https://compositecode.github.io/substratalapps.com/pricing/)). +2. Give each seeded Role its permissions (`PATCH /v1/roles/role__`). +3. Create the app backend's key (`POST /v1/api-keys`, scope the app, `entitlements.manage` and `users.list`), plus a test-mode key. +4. Subscribe the backend to `access.revoked` and `role.removed` (`POST /v1/webhooks`): the required revocation integration. + +## 9. Fix the pull-request plan job + +The `plan` job on pull requests into `main` fails: the read-only plan role can't read the app secret's value (`secretsmanager:GetSecretValue`), which `terraform plan` needs to refresh `aws_secretsmanager_secret_version.app`. It isn't a required check, so it doesn't block merges, but its plan comment is the best preview of what a release changes in AWS. Choose one: + +- Allow the plan role `secretsmanager:GetSecretValue` on `substratal-prod/app` only (it then can read the Stripe keys, as anyone who can open a PR into `main` could), or +- Stop Terraform reading the value: keep the secret's initial value out of Terraform state and plan with the secret version untracked. This is a small Terraform change. + +## 10. Legal and compliance + +- **Privacy policy and terms of service**, published at stable URLs. Stripe live mode and SES production access both expect them. +- **Data processing agreements** with the sub-processors listed on [Compliance](https://compositecode.github.io/substratalapps.com/compliance/): AWS (AWS Artifact → Agreements), Stripe, and the push providers (Apple, Google, and the browser push services). + +## 11. Optional: transfer the domain registration + +DNS is on Route 53; the registration is still at Vercel (reseller for Tucows), renewing 2027-08-07. To consolidate: in Vercel, **Domains → substratalapps.com → Transfer out** to get the authorization code, then `aws route53domains transfer-domain --domain-name substratalapps.com --duration-in-years 1 --auth-code … --region us-east-1` with the registrant contact details (about $15, adds a year). Keep the Route 53 nameservers during the transfer. + +Also keep an eye on the Organization SCP exception for `substratal-prod` ([infra/README.md → AWS account & region](infra/README.md#aws-account--region)): if production calls suddenly fail with an explicit SCP deny in `us-east-1`, that exception was removed. diff --git a/scripts/quickstart.sh b/scripts/quickstart.sh index 320a6ae..e8ee227 100755 --- a/scripts/quickstart.sh +++ b/scripts/quickstart.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash # The docs/quickstart.md walkthrough as a script, against a running API # (default: the local devserver). It's the Phase 1 acceptance test from -# PLAN.md, so CI and `make quickstart` run it on every change. +# plan-remaining.md, so CI and `make quickstart` run it on every change. # # SUBSTRATAL_API=http://localhost:8080/v1 TOKEN= scripts/quickstart.sh #