Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions cmd/devserver/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,20 +56,22 @@ func main() {
Stripe: stripe, Core: server, Log: base.Log})
}

// The 5-minute entitlement sweep, in-process.
// The 5-minute entitlement sweep and the audit-export worker, in-process.
runner := jobs.New(base)
go func() {
t := time.NewTicker(5 * time.Minute)
every := func(d time.Duration, job string) {
t := time.NewTicker(d)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
_ = runner.Run(ctx, "entitlement-sweep")
_ = runner.Run(ctx, job)
}
}
}()
}
go every(5*time.Minute, "entitlement-sweep")
go every(15*time.Second, "audit-exports")

srv := &http.Server{Addr: base.Config.ListenAddr, Handler: mux, ReadHeaderTimeout: 10 * time.Second}
go func() {
Expand Down
39 changes: 38 additions & 1 deletion docs/api-reference/audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,10 @@ See [Domain Model → Audit Event](../../domain-model/orders-and-audit/#audit-ev
|---|---|---|---|
| `GET` | `/audit-events` | `audit.view` (platform or app-confined); or self via `target_user_id=me`; or a Tenant owner via `tenant_id` | Query the audit log. |
| `GET` | `/audit-events/{id}` | same as above, for that event | Fetch one event. |
| `POST` | `/audit-exports` | same as `GET /audit-events`, for the filters sent | Export matching events to a file. |
| `GET` | `/audit-exports/{id}` | the export's requester, or `audit.view` | Check an export, and get its download link. |

There's deliberately no write endpoint. Audit Events are produced only as a side effect of other writes, never created directly, and never edited or deleted through the API.
There's deliberately no write endpoint for Audit Events. They're produced only as a side effect of other writes, never created directly, and never edited or deleted through the API. An export is a read: it copies events into a file and changes none of them.

## `GET /audit-events`

Expand Down Expand Up @@ -79,6 +81,40 @@ GET /audit-events?target_user_id=usr_01JAG3Z9X8QS3F6K2M4N5P6R7S&since=2026-09-01

Returns one event, same shape. `404 audit_event_not_found` if it doesn't exist, isn't visible to the caller, or has been archived.

## Exporting

For compliance evidence (a SOC 2 audit, a customer's request for their records) or offline analysis, export the log to a file instead of paging through it.

```json
// POST /audit-exports (Idempotency-Key required)
{ "format": "csv", "tenant_id": "tnt_01JAG1SYSTEM00000000000000", "since": "2026-01-01T00:00:00Z" }
```
```json
// Response — 202
{
"id": "aex_01JAH2K4Q9R0S1T2U3V4W5X6Y7",
"status": "pending",
"format": "csv",
"filters": { "tenant_id": "tnt_01JAG1SYSTEM00000000000000", "since": "2026-01-01T00:00:00Z" },
"row_count": null,
"download_url": null,
"download_url_expires_at": null,
"error": null,
"created_at": "2026-10-21T18:00:00Z",
"completed_at": null,
"expires_at": "2026-10-28T18:00:00Z",
"test_mode": false
}
```

- **Filters and scope** are exactly `GET /audit-events`'s, as body fields: `action` is an array (up to 100), `target_user_id` may be `me`, and the caller is confined to the same scope as on the list. An app-confined key exports only its own Application's events, a User without `audit.view` only their own trail (`target_user_id: "me"`) or a Tenant they own. Hot window only, like the list.
- **`format`** is `ndjson` (default), one Audit Event object per line, or `csv`, with the columns `id, timestamp, action, actor_type, actor_id, target_type, target_id, target_user_id, application_id, organization_id, tenant_id, request_id, before, after` (`before`/`after` as JSON text). Rows are oldest first.
- **Asynchronous.** The request returns `202` at once, and a job writes the file, usually within a minute. Poll `GET /audit-exports/{id}` until `status` is `ready` (or `failed`, with `error` saying why: `too_many_rows` over 1,000,000 events, so narrow the filters).
- **Downloading.** Once `ready`, `download_url` is a signed link valid for 15 minutes (`download_url_expires_at`). Fetch the export again for a fresh one. The file and the record are deleted 7 days after the request (`expires_at`).
- **Who can see it:** whoever requested it, and `audit.view`. Anyone else gets `404 audit_export_not_found`.
- **It's audited.** When the file is ready, `audit.exported` is written with the requester as actor, the filters, the format, and the row count, in the export's own scope, so it appears in the same trail it exported from.
- Limited to 20 per caller per day.

## Retention

Audit Events are never deleted or edited through the API, including after the record they describe is hard-deleted. For example, a user's right-to-erasure request removes their Profile/Settings, but the audit trail of *that deletion itself* persists, with snapshot values redacted. See [Non-Functional Requirements → Data retention](../../non-functional-requirements/#data-retention).
Expand All @@ -88,4 +124,5 @@ Audit Events are never deleted or edited through the API, including after the re
| Code | Status | When |
|---|---|---|
| `audit_event_not_found` | 404 | See above. |
| `audit_export_not_found` | 404 | The export doesn't exist, has expired, or wasn't requested by the caller (and they lack `audit.view`). |
| `forbidden` | 403 | No `audit.view`, and the query isn't scoped to `me` or an owned Tenant. |
10 changes: 10 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,16 @@ What's changed in this specification over time. Every decision is written into t

---

## 2026-10-21 — Audit Event exports

The audit log can be exported to a file. See [API Reference → Audit → Exporting](../api-reference/audit/#exporting). `openapi.yaml` is now at `1.0.0-draft.8`, with 123 operations.

- **`POST /audit-exports`** takes the same filters as `GET /audit-events`, as body fields, plus `format` (`ndjson` or `csv`). It confines the caller to the same scope as the list, requires `Idempotency-Key`, and returns `202`. A job writes the file, usually within a minute. Limited to 20 per caller per day.
- **`GET /audit-exports/{id}`** shows the status, and once it's `ready`, a download link signed for 15 minutes. Only the requester and `audit.view` can see an export. Files and records are deleted after 7 days.
- New audit action `audit.exported` (target type `audit_export`), written when the file is ready, with the requester as actor.
- **Retention** needed no new code. It's already enforced: the one-year hot window, and the nightly archival of older events, shape-only, to indefinite cold storage ([Audit log lifecycle](../non-functional-requirements/#audit-log-lifecycle)).
- Migration `0011_audit_exports`. Infrastructure: a private exports bucket with a 7-day lifecycle rule, and an `audit-exports` schedule every minute.

## 2026-10-21 — SCIM provisioning

An Organization's identity provider can invite and remove its members with SCIM 2.0. See [API Reference → SCIM](../api-reference/scim/). `openapi.yaml` is now at `1.0.0-draft.5`, with 121 operations.
Expand Down
18 changes: 18 additions & 0 deletions docs/domain-model/database-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,24 @@ No `updated_at`, no soft-delete column — this table is append-only by design (

**Enforcement, not just storage:** the `restrict_destructive` check happens in the same request-handling layer as the permission check — both read from the authenticated key's row, both must pass. The classification of which operations count as destructive lives in exactly one place in the implementation (a lookup by HTTP method + the specific Entitlement/Role/Organization-member/Tenant-tier mutations named in [API Keys → Agent keys & `restrict_destructive`](../../api-reference/api-keys/#agent-keys--restrict_destructive)), shared by this check and by the MCP server's `destructiveHint` tool-annotation generation — not reimplemented twice and left to drift.

## audit_exports

| Column | Type | Constraint |
|---|---|---|
| `id` | `text` | PK, `aex_…` |
| `requested_by_type`, `requested_by_id` | `text` | not null — `user` or `api_key`, and its id |
| `scope` | `jsonb` | not null — the scope the requester was confined to (`application_id`, `target_user_id`, or `tenant_id`), or `{}` for `audit.view` |
| `filters` | `jsonb` | not null — as requested, `me` resolved |
| `format` | `text` | `ndjson` or `csv` |
| `status` | `text` | `pending`, `running`, `ready`, or `failed` |
| `row_count`, `object_key` | `integer`, `text` | set exactly when `ready` |
| `error` | `text` | set exactly when `failed` |
| `test_mode` | `boolean` | mode policy, like every other test-mode table |
| `created_at`, `started_at`, `completed_at` | `timestamptz` | |
| `expires_at` | `timestamptz` | not null — `created_at` + 7 days; the nightly cleanup deletes the row, and an S3 lifecycle rule the file |

**Index:** `(created_at) where status in ('pending','running')` for the export worker, which claims one at a time with `for update skip locked` and picks up a `running` export again if its worker hasn't finished within 20 minutes.

## scim_tokens

| Column | Type | Constraint |
Expand Down
1 change: 1 addition & 0 deletions docs/domain-model/orders-and-audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ Every value `action` can take. It's mirrored exactly by the `AuditAction` enum i
| `push_credential.created` / `push_credential.updated` / `push_credential.deleted` | An Application's [push credential](../../api-reference/notifications/#push-credentials) is configured, changed, or removed. The platform marking one `failed` (or a change returning it to `active`) is `push_credential.updated` with actor `system`. Key material never appears in `before`/`after` | `push_credential` |
| `device.deleted` | `users.manage` deletes another User's [Device](../../api-reference/notifications/#delete-usersiddevicesdeviceid) — **admin-only** | `device` |
| `organization.scim_token_created` / `organization.scim_token_revoked` | An Organization's [SCIM](../../api-reference/scim/) token is created (or rotated), or revoked | `organization` |
| `audit.exported` | An [Audit Event export](../../api-reference/audit/#exporting) is written; actor is whoever requested it, with the filters, format, and row count in `after` | `audit_export` |

This list is the authoritative source for `action` values. If an endpoint's page describes a write that isn't represented here, that's a spec bug; file it the same way as any other inconsistency.

Expand Down
1 change: 1 addition & 0 deletions docs/non-functional-requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ Per-API-key token bucket, returning `429` with a `Retry-After` header on exhaust
| `POST /auth/token/refresh` | 30 requests/minute per session |
| `GET /users/{id}/export` | 5 per user per day |
| `POST /organizations` | 10 per user per day |
| `POST /audit-exports` | 20 per API Key or User per day |
| `POST /users/{id}/notifications` | 600 requests/minute per API Key, and 60 notifications per recipient User per Application per rolling hour (`429 rate_limited`, `details.scope: "recipient"`). Platform alerts don't count toward the per-recipient limit. |
| `POST /users/{id}/devices/{deviceId}/test` | 10 per device per hour |

Expand Down
83 changes: 81 additions & 2 deletions docs/openapi.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
openapi: 3.1.0
info:
title: Substratal Apps — Platform API
version: "1.0.0-draft.5"
version: "1.0.0-draft.8"
summary: >-
User accounts, auth, roles, per-app entitlements (the on/off access
switch), profiles, settings, organizations, tenancy, subscription billing,
Expand Down Expand Up @@ -2013,6 +2013,51 @@ paths:
schema: { $ref: "#/components/schemas/AuditEvent" }
"404": { $ref: "#/components/responses/NotFound" }

/audit-exports:
post:
operationId: audit.createExport
tags: [Audit]
summary: Export Audit Events to a file (asynchronous)
description: >-
Exports every Audit Event matching the same filters as GET /audit-events, in the caller's
scope there, as NDJSON or CSV. Hot window only. Returns 202 at once; a job writes the file,
usually within a minute. Poll GET /audit-exports/{id} until status is ready, then download
from download_url. Exports are kept 7 days. Writes audit.exported when the file is ready.
x-substratal-phase: phase3
parameters:
- $ref: "#/components/parameters/IdempotencyKey"
requestBody:
required: true
content:
application/json:
schema: { $ref: "#/components/schemas/AuditExportRequest" }
responses:
"202":
description: Export queued
content:
application/json:
schema: { $ref: "#/components/schemas/AuditExport" }
"403": { $ref: "#/components/responses/Forbidden" }
"422": { $ref: "#/components/responses/Unprocessable" }
"429": { $ref: "#/components/responses/TooManyRequests" }

/audit-exports/{id}:
parameters:
- { name: id, in: path, required: true, schema: { type: string } }
get:
operationId: audit.getExport
tags: [Audit]
summary: Fetch an export's status, and its download link once ready
description: "Visible to whoever requested it, and to audit.view. download_url is freshly signed on every call and valid for 15 minutes."
x-substratal-phase: phase3
responses:
"200":
description: The export
content:
application/json:
schema: { $ref: "#/components/schemas/AuditExport" }
"404": { $ref: "#/components/responses/NotFound" }

# ───────────────────────────── Webhooks ─────────────────────────────

/webhooks:
Expand Down Expand Up @@ -3473,6 +3518,7 @@ components:
- user.mfa_enabled
- user.mfa_disabled
- user.mfa_reset
- audit.exported
- entitlement.granted
- entitlement.disabled
- entitlement.revoked
Expand Down Expand Up @@ -3520,7 +3566,40 @@ components:

AuditTargetType:
type: string
enum: [user, entitlement, role, role_assignment, application, organization, subscription, tenant, tier_change_request, api_key, webhook, push_credential, device]
enum: [user, entitlement, role, role_assignment, application, organization, subscription, tenant, tier_change_request, api_key, webhook, push_credential, device, audit_export]

AuditExportRequest:
type: object
description: "The same filters as GET /audit-events, as body fields, plus format."
properties:
format: { type: string, enum: [ndjson, csv], default: ndjson }
target_user_id: { type: string, description: "A User id, or me." }
target_type: { $ref: "#/components/schemas/AuditTargetType" }
target_id: { type: string }
actor_id: { type: string }
actor_type: { type: string, enum: [user, api_key, system] }
action: { type: array, items: { $ref: "#/components/schemas/AuditAction" }, maxItems: 100 }
application_id: { type: string }
organization_id: { type: string }
tenant_id: { type: string }
since: { type: string, format: date-time }
until: { type: string, format: date-time }

AuditExport:
type: object
properties:
id: { type: string }
status: { type: string, enum: [pending, running, ready, failed] }
format: { type: string, enum: [ndjson, csv] }
filters: { type: object, description: "The filters as requested, with target_user_id me resolved to the caller's id." }
row_count: { type: [integer, "null"], description: "Set once ready." }
download_url: { type: [string, "null"], format: uri, description: "Set once ready. Signed, valid for 15 minutes; fetch the export again for a fresh one." }
download_url_expires_at: { type: [string, "null"], format: date-time }
error: { type: [string, "null"], description: "Set when failed, e.g. too_many_rows." }
created_at: { type: string, format: date-time }
completed_at: { type: [string, "null"], format: date-time }
expires_at: { type: string, format: date-time, description: "When the file and this record are deleted: 7 days after the request." }
test_mode: { type: boolean }

ScimToken:
type: object
Expand Down
2 changes: 1 addition & 1 deletion docs/openapi/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ The complete Platform API contract as an [OpenAPI 3.1](https://spec.openapis.org
| API version | `Substratal-Version: 1` request header, required (see [Conventions → Versioning](../api-reference/conventions/#versioning)) |
| Authentication | `Authorization: Bearer <token>`: a user access token or an API Key (`bearerAuth`) |
| Content type | `application/json; charset=utf-8` |
| Operations | 121, every one with a stable `operationId` |
| Operations | 123, every one with a stable `operationId` |
| Schemas | 109 reusable components under `components.schemas` |

The prose pages under [API Reference](../api-reference/) are authoritative. `openapi.yaml` is kept in sync with them by hand, and the implementation's contract tests validate every response against it, so a drift between the two fails CI rather than reaching a client. See [Conventions → Machine-readable](../api-reference/conventions/#machine-readable).
Expand Down
2 changes: 1 addition & 1 deletion docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Enough to replace "someone manually emails a link and flips a flag in a spreadsh

## Phase 3

- Richer audit/compliance views — filtering, export, retention policy enforcement; see [Compliance & Data Protection](../compliance/) for what's driving this (SOC 2 readiness, GDPR data-subject requests).
- Richer audit/compliance views — filtering, export, retention policy enforcement; see [Compliance & Data Protection](../compliance/) for what's driving this (SOC 2 readiness, GDPR data-subject requests). **Built 2026-10-21:** filtering on [`GET /audit-events`](../api-reference/audit/), [exports](../api-reference/audit/#exporting), and retention through the [Audit log lifecycle](../non-functional-requirements/#audit-log-lifecycle).
- SCIM-style provisioning for larger customers who want their own IdP to push user lifecycle events into the hub automatically. **Built 2026-10-21:** [SCIM 2.0 Users](../api-reference/scim/), as invitations and membership removal.
- The **hosted sign-in page** and the authorization-code + PKCE flow ([Auth → Getting an app token](../api-reference/auth/#getting-an-app-token)). The API side is specified now; this is the UI. It's required before any third-party Application goes live.
- Self-service Application registration and review queue for third-party developers — the marketplace phase. See [App developer/publisher model](../domain-model/applications/#the-review-lifecycle).
Expand Down
4 changes: 4 additions & 0 deletions infra/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,10 @@ The delivery worker reserves 5 concurrent executions, so a burst of deliveries c

Cost is negligible at Tier 0 volume (Glacier Deep Archive is priced for exactly this shape of rarely-read, kept-forever data) — this is about bounding Aurora's hot-table size and honoring the redaction-on-age guarantee, not about saving money on S3 itself.

### Audit exports

[`POST /audit-exports`](https://compositecode.github.io/substratalapps.com/api-reference/audit/#exporting) files go to a second private bucket, `substratal-prod-audit-exports-<account>` (SSE-S3, public access blocked), with a lifecycle rule that deletes every object after 7 days, matching the export record's `expires_at`. The `audit-exports` job runs every minute on the jobs Lambda, pages through the matching events into a temporary file, and uploads it (`AUDIT_EXPORT_BUCKET`, `s3:PutObject`). The API Lambda signs the 15-minute download links, so it holds `s3:GetObject` on that bucket only. Locally, exports are written under `.dev/audit-exports` and the devserver runs the worker every 15 seconds.

## Local development

The RDS Data API choice that keeps Tier 0 NAT-free has a real cost: it doesn't have a clean open-source local emulator, so a developer can't just point the production data-access code at "Data API, but local" the way they could with a plain Postgres driver.
Expand Down
Loading
Loading