diff --git a/.changeset/prisma-migrate-history.md b/.changeset/prisma-migrate-history.md new file mode 100644 index 00000000..a7f5aee5 --- /dev/null +++ b/.changeset/prisma-migrate-history.md @@ -0,0 +1,5 @@ +--- +'lens': minor +--- + +Adopt Prisma migrate: commit a baseline migration (`prisma/migrations/0_init`), make CI fail on drift between `schema.prisma` and the migrations, switch app startup and deploys to `prisma migrate deploy`, and remove the `db push` path. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a3ecf75e..28a53f98 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -21,3 +21,58 @@ jobs: - run: npx tsc --noEmit - run: npm run build - run: npm test --if-present + + db-migrations: + # #233 — schema changes must be reviewable migrations, not `db push`. + # This job is the authority on drift: it replays every committed migration + # against a real Postgres and fails if the result disagrees with + # prisma/schema.prisma. It also proves a new migration can be generated, + # applied and rolled back on a clean database. + name: Migration drift & deploy + runs-on: ubuntu-latest + needs: build + services: + postgres: + image: postgres:15-alpine + env: + POSTGRES_USER: lens + POSTGRES_PASSWORD: lens + POSTGRES_DB: lens + ports: + - 5432:5432 + options: >- + --health-cmd "pg_isready -U lens -d lens" + --health-interval 5s + --health-timeout 5s + --health-retries 10 + env: + DATABASE_URL: postgresql://lens:lens@localhost:5432/lens + DIRECT_DATABASE_URL: postgresql://lens:lens@localhost:5432/lens + SHADOW_DATABASE_URL: postgresql://lens:lens@localhost:5432/lens_shadow + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: '20' + cache: npm + - run: npm ci || npm install + - run: node node_modules/prisma/build/index.js generate + + - name: Create empty shadow database + run: | + node -e ' + const { Client } = require("pg"); + (async () => { + const c = new Client({ connectionString: process.env.DATABASE_URL }); + await c.connect(); + await c.query("DROP DATABASE IF EXISTS lens_shadow"); + await c.query("CREATE DATABASE lens_shadow"); + await c.end(); + })(); + ' + + - name: Fail on drift between schema.prisma and prisma/migrations + run: bash scripts/db-drift-check.sh + + - name: Prove generate / apply / rollback on a clean database + run: bash scripts/db-migration-cycle.sh diff --git a/.github/workflows/load.yml b/.github/workflows/load.yml index fb262cf6..8c14e629 100644 --- a/.github/workflows/load.yml +++ b/.github/workflows/load.yml @@ -81,8 +81,8 @@ jobs: - name: Build run: npm run build - - name: Apply database schema - run: node node_modules/prisma/build/index.js db push --accept-data-loss + - name: Apply committed migrations + run: node node_modules/prisma/build/index.js migrate deploy - name: Start server run: | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ac301046..d098a19a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -46,7 +46,7 @@ git clone https://github.com/Miracle656/Lens.git cd Lens npm install cp .env.example .env # fill in DATABASE_URL, REDIS_URL, WATCHED_PAIRS -npx prisma db push +npm run db:migrate # apply committed migrations (prisma migrate dev) — do NOT `db push` npm run dev ``` diff --git a/README.md b/README.md index 13334c3b..c7abbe3d 100644 --- a/README.md +++ b/README.md @@ -380,8 +380,8 @@ npm install # 3. Copy env cp .env.example .env -# 4. Push database schema -npm run db:push +# 4. Apply database migrations +npm run db:deploy # 5. Seed the database with fixture data npm run seed diff --git a/docs/backup-restore.md b/docs/backup-restore.md index c697a83b..9f68733c 100644 --- a/docs/backup-restore.md +++ b/docs/backup-restore.md @@ -85,7 +85,7 @@ before loading. Delete the decrypted `.dump.gz` when you are done. 4. Point the matching Lens deployment at the new instance: - `DATABASE_URL` — pooled URL is fine for the app - - `DIRECT_DATABASE_URL` — direct URL (Prisma migrations / `db push`) + - `DIRECT_DATABASE_URL` — direct URL (Prisma `migrate deploy` / CLI) 5. Check `/status`, then one `/prices/history` (or GraphQL equivalent) for a watched pair. 6. Rollback is swapping those two env vars back to the previous instance. diff --git a/docs/database-migrations.md b/docs/database-migrations.md new file mode 100644 index 00000000..6a3c378a --- /dev/null +++ b/docs/database-migrations.md @@ -0,0 +1,130 @@ +# Database migrations + +Issue **#233**. Before this, `prisma/` contained only `schema.prisma` — there was +no `migrations/` directory, and every schema change was applied with +`prisma db push`. Lens now uses **`prisma migrate`**: schema changes are +committed as reviewable SQL, and deployments apply them with `migrate deploy`. + +## Why not `db push` + +`prisma db push` diffs the *live* database against `schema.prisma` and applies +whatever it decides, on the spot. That gives you none of the things a service +that holds money-moving data needs: + +- **Nothing is reviewable.** A schema change is whatever the diff produced at + that instant, not a file a reviewer read. +- **No reproducible deploy.** Two environments pushed at different times can + land on different shapes; nothing in the repo describes the live shape. +- **No rollback.** There is no record of what changed, so there is nothing to + undo. With `--accept-data-loss` a push can silently drop a column. +- **It has already bitten us.** A `db push` in the sibling service half-applied + and left the database on a shape no commit described. That is exactly the + failure `prisma migrate` exists to prevent. + +So: **do not run `prisma db push` against any environment.** It is no longer a +`package.json` script. The `db push` call at app startup (`src/index.ts`) now +runs `migrate deploy` instead. + +## What lives in the repo now + +``` +prisma/ + schema.prisma # the source of truth for the data model + migrations/ + migration_lock.toml # provider = "postgresql" (do not hand-edit) + 0_init/ + migration.sql # baseline: recreates the production schema exactly +``` + +`0_init/migration.sql` is the **baseline** — the SQL that reproduces the current +production schema from an empty database. It is generated, not hand-written: + +```bash +prisma migrate diff \ + --from-empty \ + --to-schema-datamodel prisma/schema.prisma \ + --script > prisma/migrations/0_init/migration.sql +``` + +## Daily workflow (local development) + +Edit `schema.prisma`, then generate a migration. Do **not** `db push`. + +```bash +npm run db:migrate # prisma migrate dev — creates and applies a new migration +``` + +`migrate dev` writes a new timestamped folder under `prisma/migrations/`. +**Commit that folder.** It is now the reviewable record of the change. + +```bash +npm run db:status # which migrations are applied / pending +npm run db:reset # drop schema and re-apply all migrations (NEVER on prod) +``` + +## Deploying + +Deploys apply committed migrations only — they never diff the live database: + +```bash +npm run db:deploy # prisma migrate deploy +``` + +App startup (`src/index.ts`) runs `migrate deploy` for you, so a container that +starts against a database with pending migrations applies them and only them, +deterministically, or fails loudly if the database is ahead of the code. + +### Adopting the baseline on an EXISTING production database + +Production already matches `schema.prisma`, so the baseline's `CREATE TABLE` +statements must **not** be re-run there — `migrate deploy` would collide with +the tables that are already there. Instead, mark the baseline as already applied +without executing it, which records it in Prisma's `_prisma_migrations` table: + +```bash +npx prisma migrate resolve --applied 0_init +``` + +After that, `migrate deploy` sees `0_init` as applied and only runs future +migrations. This one-time step is the standard Prisma path for adopting +`migrate` on a database that already exists; it is why the baseline is committed +separately rather than generated on deploy. + +## The CI drift guard + +The `db-migrations` job in `.github/workflows/ci.yml` brings up a Postgres +service and runs: + +- `scripts/db-drift-check.sh` — deploys every committed migration, then + `migrate diff --from-migrations prisma/migrations --to-schema-datamodel + prisma/schema.prisma --exit-code` against a shadow database. It exits non-zero + — failing CI — whenever `schema.prisma` and `prisma/migrations` disagree. This + is the check that would have caught a half-applied `db push`. +- `scripts/db-migration-cycle.sh` — proves a new migration can be generated, + applied and rolled back on a clean database, and that the drift guard actually + fires on an un-migrated schema change. + +The DB-free unit test `tests/prisma-migrations.test.ts` (part of `npm test`) +additionally locks the **baseline** to `schema.prisma`: it fails while the repo +has no migration history, and fails if you edit the schema without regenerating +the baseline, so a missing migration is caught even without a database. + +Run the database-backed checks locally against the compose Postgres: + +```bash +docker compose up -d postgres +# create an empty shadow database alongside `lens` +DATABASE_URL=postgresql://lens:lens@localhost:5432/lens \ +DIRECT_DATABASE_URL=postgresql://lens:lens@localhost:5432/lens \ +SHADOW_DATABASE_URL=postgresql://lens:lens@localhost:5432/lens_shadow \ + npm run db:drift-check +``` + +## Rolling back a bad migration + +Prisma does not generate down-migrations. In a **non-production** environment, +`npm run db:reset` drops the schema and re-applies every committed migration, +which is the clean, reproducible way to back out of a half-applied change. In +production, roll forward: commit a new migration that reverses the change and +`migrate deploy` it. Either way the history in `prisma/migrations/` is the +source of truth — the thing `db push` never gave us. diff --git a/package.json b/package.json index a56deda9..1a921c94 100644 --- a/package.json +++ b/package.json @@ -20,8 +20,13 @@ "mcp:server": "tsx examples/mcp-server/run.ts", "mcp:agent": "tsx examples/mcp-server/agent.ts", "key:issue": "tsx scripts/issue-api-key.ts", - "db:push": "prisma db push", "db:generate": "prisma generate", + "db:migrate": "prisma migrate dev", + "db:deploy": "prisma migrate deploy", + "db:reset": "prisma migrate reset", + "db:status": "prisma migrate status", + "db:drift-check": "bash scripts/db-drift-check.sh", + "db:migration-cycle": "bash scripts/db-migration-cycle.sh", "db:migrate:partitioning": "tsx scripts/migrate-price-points-partitioning.ts", "seed": "tsx scripts/seed.ts" }, diff --git a/prisma/migrations/0_init/migration.sql b/prisma/migrations/0_init/migration.sql new file mode 100644 index 00000000..d860d07b --- /dev/null +++ b/prisma/migrations/0_init/migration.sql @@ -0,0 +1,202 @@ +-- CreateTable +CREATE TABLE "price_points" ( + "id" TEXT NOT NULL, + "network" TEXT NOT NULL DEFAULT 'testnet', + "asset_a" TEXT NOT NULL, + "asset_b" TEXT NOT NULL, + "pair_key" TEXT NOT NULL, + "source" TEXT NOT NULL, + "pool_id" TEXT, + "price" DECIMAL(36,18) NOT NULL, + "base_volume" DECIMAL(36,7) NOT NULL, + "counter_volume" DECIMAL(36,7) NOT NULL, + "ledger" INTEGER NOT NULL, + "timestamp" TIMESTAMP(3) NOT NULL, + "event_id" TEXT, + + CONSTRAINT "price_points_pkey" PRIMARY KEY ("id","timestamp") +); + +-- CreateTable +CREATE TABLE "pool_snapshots" ( + "id" TEXT NOT NULL, + "network" TEXT NOT NULL DEFAULT 'testnet', + "pool_id" TEXT NOT NULL, + "asset_a" TEXT NOT NULL, + "asset_b" TEXT NOT NULL, + "reserve_a" DECIMAL(36,7) NOT NULL, + "reserve_b" DECIMAL(36,7) NOT NULL, + "spot_price" DECIMAL(36,18) NOT NULL, + "total_shares" DECIMAL(36,7), + "fee_bp" INTEGER NOT NULL DEFAULT 30, + "ledger" INTEGER NOT NULL, + "timestamp" TIMESTAMP(3) NOT NULL, + + CONSTRAINT "pool_snapshots_pkey" PRIMARY KEY ("id","timestamp") +); + +-- CreateTable +CREATE TABLE "price_aggregates" ( + "pair_key" TEXT NOT NULL, + "network" TEXT NOT NULL DEFAULT 'testnet', + "window" TEXT NOT NULL, + "bucket" TIMESTAMP(3) NOT NULL, + "vwap" DECIMAL(36,18) NOT NULL, + "sdex_vwap" DECIMAL(36,18), + "amm_vwap" DECIMAL(36,18), + "volume" DECIMAL(36,7) NOT NULL DEFAULT 0, + "sdex_volume" DECIMAL(36,7) DEFAULT 0, + "amm_volume" DECIMAL(36,7) DEFAULT 0, + "trade_count" INTEGER DEFAULT 0, + "open_price" DECIMAL(36,18), + "close_price" DECIMAL(36,18), + "high_price" DECIMAL(36,18), + "low_price" DECIMAL(36,18), + + CONSTRAINT "price_aggregates_pkey" PRIMARY KEY ("network","pair_key","window","bucket") +); + +-- CreateTable +CREATE TABLE "price_snapshots" ( + "pair" TEXT NOT NULL, + "network" TEXT NOT NULL DEFAULT 'testnet', + "ts" TIMESTAMP(3) NOT NULL, + "price" DECIMAL(36,18) NOT NULL, + "volume" DECIMAL(36,7) NOT NULL DEFAULT 0, + + CONSTRAINT "price_snapshots_pkey" PRIMARY KEY ("network","pair","ts") +); + +-- CreateTable +CREATE TABLE "indexer_state" ( + "id" TEXT NOT NULL, + "network" TEXT NOT NULL DEFAULT 'testnet', + "last_cursor" TEXT, + "last_ledger" INTEGER, + "last_processed_at" TIMESTAMP(3), + "updated_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP, + + CONSTRAINT "indexer_state_pkey" PRIMARY KEY ("network","id") +); + +-- CreateTable +CREATE TABLE "pair_configs" ( + "pair_key" TEXT NOT NULL, + "network" TEXT NOT NULL DEFAULT 'testnet', + "asset_a_code" TEXT NOT NULL, + "asset_a_issuer" TEXT, + "asset_b_code" TEXT NOT NULL, + "asset_b_issuer" TEXT, + "added_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP, + + CONSTRAINT "pair_configs_pkey" PRIMARY KEY ("network","pair_key") +); + +-- CreateTable +CREATE TABLE "bazaar_resources" ( + "id" TEXT NOT NULL, + "type" TEXT NOT NULL, + "network" TEXT NOT NULL, + "url" TEXT NOT NULL, + "description" TEXT, + "mime_type" TEXT, + "service_name" TEXT, + "tags" TEXT[] DEFAULT ARRAY[]::TEXT[], + "icon_url" TEXT, + "mcp_tool_name" TEXT, + "http_method" TEXT, + "accepts" JSONB NOT NULL, + "pay_to" TEXT NOT NULL, + "bazaar_info" JSONB NOT NULL, + "bazaar_schema" JSONB NOT NULL, + "route_template" TEXT, + "extension_keys" TEXT[] DEFAULT ARRAY['bazaar']::TEXT[], + "created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP, + "updated_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP, + + CONSTRAINT "bazaar_resources_pkey" PRIMARY KEY ("id") +); + +-- CreateTable +CREATE TABLE "settlement_attempts" ( + "id" TEXT NOT NULL, + "network" TEXT NOT NULL DEFAULT 'testnet', + "tx_hash" TEXT NOT NULL, + "state" TEXT NOT NULL, + "payer" TEXT, + "pay_to" TEXT, + "asset" TEXT, + "amount" TEXT, + "error_reason" TEXT, + "error_message" TEXT, + "response" JSONB, + "created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP, + "updated_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP, + + CONSTRAINT "settlement_attempts_pkey" PRIMARY KEY ("id") +); + +-- CreateTable +CREATE TABLE "webhooks" ( + "id" TEXT NOT NULL, + "network" TEXT NOT NULL DEFAULT 'testnet', + "url" TEXT NOT NULL, + "asset_a" TEXT NOT NULL, + "asset_b" TEXT NOT NULL, + "threshold" DOUBLE PRECISION NOT NULL, + "direction" TEXT NOT NULL, + "secret" TEXT NOT NULL, + "created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP, + + CONSTRAINT "webhooks_pkey" PRIMARY KEY ("id") +); + +-- CreateTable +CREATE TABLE "api_keys" ( + "id" TEXT NOT NULL, + "hash" TEXT NOT NULL, + "label" TEXT NOT NULL, + "rate_per_min" INTEGER NOT NULL DEFAULT 60, + "rate_per_day" INTEGER NOT NULL DEFAULT 10000, + "revoked_at" TIMESTAMP(3), + "created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP, + "monthly_quota_cents" INTEGER DEFAULT 10000, + "daily_quota_cents" INTEGER DEFAULT 500, + "overage_policy" TEXT NOT NULL DEFAULT 'block', + + CONSTRAINT "api_keys_pkey" PRIMARY KEY ("id") +); + +-- CreateIndex +CREATE INDEX "price_points_network_pair_key_timestamp_idx" ON "price_points"("network", "pair_key", "timestamp" DESC); + +-- CreateIndex +CREATE INDEX "price_points_network_pair_key_source_timestamp_idx" ON "price_points"("network", "pair_key", "source", "timestamp" DESC); + +-- CreateIndex +CREATE INDEX "pool_snapshots_network_pool_id_timestamp_idx" ON "pool_snapshots"("network", "pool_id", "timestamp" DESC); + +-- CreateIndex +CREATE INDEX "price_snapshots_network_pair_ts_idx" ON "price_snapshots"("network", "pair", "ts"); + +-- CreateIndex +CREATE INDEX "bazaar_resources_network_type_pay_to_created_at_id_idx" ON "bazaar_resources"("network", "type", "pay_to", "created_at" DESC, "id"); + +-- CreateIndex +CREATE INDEX "bazaar_resources_extension_keys_idx" ON "bazaar_resources" USING GIN ("extension_keys"); + +-- CreateIndex +CREATE UNIQUE INDEX "bazaar_http_identity" ON "bazaar_resources"("network", "url", "http_method"); + +-- CreateIndex +CREATE UNIQUE INDEX "bazaar_mcp_identity" ON "bazaar_resources"("network", "url", "mcp_tool_name"); + +-- CreateIndex +CREATE UNIQUE INDEX "settlement_attempt_identity" ON "settlement_attempts"("network", "tx_hash"); + +-- CreateIndex +CREATE INDEX "webhooks_network_asset_a_asset_b_idx" ON "webhooks"("network", "asset_a", "asset_b"); + +-- CreateIndex +CREATE UNIQUE INDEX "api_keys_hash_key" ON "api_keys"("hash"); + diff --git a/prisma/migrations/migration_lock.toml b/prisma/migrations/migration_lock.toml new file mode 100644 index 00000000..044d57cd --- /dev/null +++ b/prisma/migrations/migration_lock.toml @@ -0,0 +1,3 @@ +# Please do not edit this file manually +# It should be added in your version-control system (e.g., Git) +provider = "postgresql" diff --git a/scripts/db-drift-check.sh b/scripts/db-drift-check.sh new file mode 100644 index 00000000..18d579da --- /dev/null +++ b/scripts/db-drift-check.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# +# #233 — General schema/migration drift guard. +# +# The DB-free vitest test (tests/prisma-migrations.test.ts) proves the baseline +# reproduces the schema while there is a single migration. This script is the +# general check: it replays *all* committed migrations against a real database +# and diffs the result against schema.prisma, so it stays correct no matter how +# many migrations the repo accumulates. Exits non-zero on drift, which is what +# makes CI fail when schema.prisma and prisma/migrations disagree. +# +# Needs a live Postgres (CI service container, or `docker compose up postgres`). +# DATABASE_URL app connection (used to deploy the migrations) +# DIRECT_DATABASE_URL direct/pooler-bypassing URL +# SHADOW_DATABASE_URL an EMPTY database Prisma uses to replay migrations +# +# Usage: bash scripts/db-drift-check.sh +set -euo pipefail + +PRISMA="node node_modules/prisma/build/index.js" + +for v in DATABASE_URL DIRECT_DATABASE_URL SHADOW_DATABASE_URL; do + if [ -z "${!v:-}" ]; then + echo "ERROR: $v is required (point it at a scratch Postgres, or run 'docker compose up -d postgres')." >&2 + exit 2 + fi +done + +echo "==> Deploying committed migrations to DATABASE_URL" +$PRISMA migrate deploy + +echo "==> Replaying migrations and diffing against schema.prisma (no drift expected)" +# --from-migrations replays prisma/migrations into the shadow DB; +# --exit-code makes the command fail (exit 1) if the replayed shape differs from +# the schema datamodel. --script is intentionally absent: exit code is the signal. +$PRISMA migrate diff \ + --from-migrations prisma/migrations \ + --to-schema-datamodel prisma/schema.prisma \ + --shadow-database-url "$SHADOW_DATABASE_URL" \ + --exit-code + +echo "OK: prisma/migrations and schema.prisma agree." diff --git a/scripts/db-migration-cycle.sh b/scripts/db-migration-cycle.sh new file mode 100644 index 00000000..9c043d99 --- /dev/null +++ b/scripts/db-migration-cycle.sh @@ -0,0 +1,118 @@ +#!/usr/bin/env bash +# +# #233 — Prove a migration can be generated, applied and rolled back on a clean +# database, and that the drift guard actually fires. This is the acceptance +# check for "A new migration can be generated, applied and rolled back in a +# clean database" and "CI fails on drift between schema and migrations." +# +# Runs entirely against throwaway databases (CI service container, or +# `docker compose up -d postgres` + a second empty DB). It never leaves the repo +# dirty: the scratch migration it generates is removed before it exits. +# +# DATABASE_URL empty-ish app DB (reset to a clean state here) +# DIRECT_DATABASE_URL direct URL +# SHADOW_DATABASE_URL empty database for Prisma to replay migrations +# +# Usage: bash scripts/db-migration-cycle.sh +set -euo pipefail + +PRISMA="node node_modules/prisma/build/index.js" +SCRATCH_DIR="prisma/migrations/29990101000000_drift_canary" +SCRATCH_SCHEMA="$(mktemp -t drift-canary-XXXXXX).prisma" +trap 'rm -rf "$SCRATCH_DIR" "$SCRATCH_SCHEMA"' EXIT + +for v in DATABASE_URL DIRECT_DATABASE_URL SHADOW_DATABASE_URL; do + if [ -z "${!v:-}" ]; then + echo "ERROR: $v is required." >&2 + exit 2 + fi +done + +# Assert a table exists (1) or does not (0) in DATABASE_URL, via the app's pg dep. +table_exists() { + node -e ' + const { Client } = require("pg"); + (async () => { + const c = new Client({ connectionString: process.env.DATABASE_URL }); + await c.connect(); + const { rows } = await c.query( + "SELECT 1 FROM information_schema.tables WHERE table_name = $1", + [process.argv[1]], + ); + await c.end(); + process.exit(rows.length ? 0 : 1); + })().catch(() => process.exit(1)); + ' "$1" +} + +require_true() { # $? == 0 already checked by caller; used for readable failures + if [ "$1" -ne 0 ]; then echo "FAIL: $2" >&2; exit 1; fi +} + +echo "==> 1. Start from a clean database: drop everything, apply the committed baseline" +$PRISMA migrate reset --force --skip-seed + +echo "==> 2. Generate a NEW migration from a schema change (add a canary table)" +cp prisma/schema.prisma "$SCRATCH_SCHEMA" +cat >> "$SCRATCH_SCHEMA" <<'PRISMA_EOF' + +model DriftCanary { + id String @id + @@map("drift_canary") +} +PRISMA_EOF + +mkdir -p "$SCRATCH_DIR" +$PRISMA migrate diff \ + --from-migrations prisma/migrations \ + --to-schema-datamodel "$SCRATCH_SCHEMA" \ + --shadow-database-url "$SHADOW_DATABASE_URL" \ + --script > "$SCRATCH_DIR/migration.sql" + +if grep -q 'CREATE TABLE "drift_canary"' "$SCRATCH_DIR/migration.sql"; then + echo " generated migration creates drift_canary — good" +else + echo "FAIL: generated migration did not create the canary table" >&2 + cat "$SCRATCH_DIR/migration.sql" >&2 + exit 1 +fi + +echo "==> 3. Apply it to the clean database" +$PRISMA migrate deploy +table_exists "drift_canary"; require_true $? "canary table should exist after apply" +echo " drift_canary present after migrate deploy — good" + +echo "==> 4. Roll it back on the clean database (drop scratch, reset re-applies only the baseline)" +rm -rf "$SCRATCH_DIR" +$PRISMA migrate reset --force --skip-seed +if table_exists "drift_canary"; then + echo "FAIL: drift_canary survived the rollback — migrate reset did not drop it" >&2 + exit 1 +fi +echo " drift_canary gone after reset — generate/apply/rollback cycle verified" + +echo "==> 5. Negative proof: an un-migrated schema change must be flagged as drift" +cp prisma/schema.prisma "$SCRATCH_SCHEMA" +cat >> "$SCRATCH_SCHEMA" <<'PRISMA_EOF' + +model DriftCanary { + id String @id + @@map("drift_canary") +} +PRISMA_EOF +# --exit-code returns 1 when the replayed migrations differ from the schema. +set +e +$PRISMA migrate diff \ + --from-migrations prisma/migrations \ + --to-schema-datamodel "$SCRATCH_SCHEMA" \ + --shadow-database-url "$SHADOW_DATABASE_URL" \ + --exit-code +drift_exit=$? +set -e +if [ "$drift_exit" -eq 0 ]; then + echo "FAIL: drift guard did NOT fire on an un-migrated schema change" >&2 + exit 1 +fi +echo " drift guard fired (exit $drift_exit) on the un-migrated schema — CI would fail — good" + +echo "OK: migration generate/apply/rollback works on a clean DB and drift is detected." diff --git a/src/index.ts b/src/index.ts index 55ed4c2b..3d899878 100644 --- a/src/index.ts +++ b/src/index.ts @@ -29,8 +29,12 @@ const log = createLogger('startup') async function main() { // ── Ensure DB schema is up-to-date ──────────────────────────────────────── - log.info('running database migrations') - execSync('node node_modules/prisma/build/index.js db push --accept-data-loss', { stdio: 'inherit' }) + // #233: apply committed, reviewable migrations — never `db push`. `migrate + // deploy` only runs migrations already in prisma/migrations and fails loudly + // if the DB is ahead of them, instead of silently diffing the live schema and + // applying whatever it decides. See docs/database-migrations.md. + log.info('applying committed database migrations') + execSync('node node_modules/prisma/build/index.js migrate deploy', { stdio: 'inherit' }) log.info('database ready') // ── Connect dependencies ────────────────────────────────────────────────── diff --git a/tests/prisma-migrations.test.ts b/tests/prisma-migrations.test.ts new file mode 100644 index 00000000..fe70900e --- /dev/null +++ b/tests/prisma-migrations.test.ts @@ -0,0 +1,131 @@ +import { describe, it, expect } from 'vitest' +import { execFileSync } from 'node:child_process' +import { existsSync, readdirSync, readFileSync } from 'node:fs' +import { fileURLToPath } from 'node:url' +import { join } from 'node:path' + +/** + * #233 — There is no migration history, only `db push`. + * + * The defect this guards against is invisible to every other test in the repo: + * `prisma/` shipped only `schema.prisma`, and every schema change was applied + * with `prisma db push`, which diffs the live DB against the schema and applies + * whatever it decides. Nothing in the codebase could detect a schema that had + * drifted away from any committed, reviewable migration. + * + * These tests are the "fails first" proof. Before adoption `prisma/migrations` + * does not exist, so `has a committed migration history` fails; the moment the + * baseline lands it passes. The drift test then locks `schema.prisma` and the + * baseline together: edit the schema without regenerating a migration and the + * two diverge, exactly the failure mode #233 exists to prevent. + * + * They are deliberately DB-free: they only run + * `prisma migrate diff --from-empty --to-schema-datamodel`, which reads the + * schema file and never opens a connection, so they run inside plain `npm test` + * and the CI build job with no Postgres present. The general, multi-migration + * drift guard (replaying every migration against a real database) lives in the + * `db-migration-drift` CI job and scripts/db-drift-check.sh, both of which need + * a live database. + */ + +const repoRoot = fileURLToPath(new URL('../', import.meta.url)) +const migrationsDir = join(repoRoot, 'prisma', 'migrations') +const schemaFile = join(repoRoot, 'prisma', 'schema.prisma') + +/** Prisma's engines need the datasource env vars defined, but + * `--from-empty --to-schema-datamodel` never connects, so point them at a + * closed port: if the command ever *did* try to reach the DB it fails loudly + * instead of silently touching someone's real database. */ +const OFFLINE_ENV = { + DATABASE_URL: 'postgresql://off:off@127.0.0.1:1/off', + DIRECT_DATABASE_URL: 'postgresql://off:off@127.0.0.1:1/off', + PRISMA_HIDE_UPDATE_MESSAGE: '1', + CHECKPOINT_DISABLE: '1', +} + +function prismaMigrateDiffToSchemaScript(): string { + return execFileSync( + 'node', + [ + 'node_modules/prisma/build/index.js', + 'migrate', + 'diff', + '--from-empty', + '--to-schema-datamodel', + schemaFile, + '--script', + ], + { cwd: repoRoot, env: { ...process.env, ...OFFLINE_ENV }, encoding: 'utf8' }, + ) +} + +/** Migration folders, in the order Prisma would replay them (sorted). */ +function committedMigrations(): string[] { + if (!existsSync(migrationsDir)) return [] + return readdirSync(migrationsDir, { withFileTypes: true }) + .filter((e) => e.isDirectory() && existsSync(join(migrationsDir, e.name, 'migration.sql'))) + .map((e) => e.name) + .sort() +} + +function readMigrationSql(name: string): string { + return readFileSync(join(migrationsDir, name, 'migration.sql'), 'utf8') +} + +/** Reduce a migration script to its meaningful DDL: drop `--` comment banners + * and blank lines, trim trailing whitespace so formatting differences that + * carry no schema meaning do not masquerade as drift. */ +function normalizeSql(sql: string): string { + return sql + .split('\n') + .map((line) => line.replace(/\s+$/, '')) + .filter((line) => line.trim() !== '' && !line.trim().startsWith('--')) + .join('\n') + .trim() +} + +describe('#233 Prisma migration history replaces db push', () => { + it('has a committed migration history (fails while the repo only db-pushes)', () => { + expect(existsSync(migrationsDir), 'prisma/migrations/ must exist — schema changes must be reviewable').toBe(true) + + const migrations = committedMigrations() + expect(migrations.length, 'prisma/migrations/ must contain at least one migration folder').toBeGreaterThan(0) + }) + + it('pins the migration lock to the Postgres provider', () => { + const lock = readFileSync(join(migrationsDir, 'migration_lock.toml'), 'utf8') + expect(lock).toMatch(/provider\s*=\s*"postgresql"/) + }) + + it('has an init/baseline migration that reproduces schema.prisma exactly', () => { + const migrations = committedMigrations() + expect(migrations.length).toBeGreaterThan(0) + + // The baseline is the migration that recreates the whole schema from empty. + // At adoption there is exactly one; as the repo grows, `migrate deploy` + + // `migrate diff --exit-code` against a real DB (scripts/db-drift-check.sh, + // run by CI) becomes the authoritative general guard, while this test keeps + // proving the initial baseline reproduces the schema byte-for-byte. + const initName = migrations.find((m) => /init|baseline/i.test(m)) ?? migrations[0] + const baselineSql = readMigrationSql(initName) + + const expected = normalizeSql(prismaMigrateDiffToSchemaScript()) + const actual = normalizeSql(baselineSql) + + expect( + actual, + `schema.prisma and prisma/migrations/${initName}/migration.sql have drifted: regenerate the baseline (or add a new migration) so they agree`, + ).toBe(expected) + }, 120_000) + + it('carries the money table and its network-leading indexes into the baseline', () => { + // A query without a network predicate reads both chains and the planner + // cannot use the index properly, so losing these from the migration is both + // wrong and slow — guard that the committed baseline keeps them. + const allSql = committedMigrations().map(readMigrationSql).join('\n') + + expect(allSql).toMatch(/CREATE TABLE "price_points"/) + expect(allSql).toMatch(/"price_points"\("network", "pair_key", "timestamp" DESC\)/) + expect(allSql).toMatch(/"price_points"\("network", "pair_key", "source", "timestamp" DESC\)/) + }) +})