[oratis]

Where this came from

Writing a telemetry plan for TakoAPI (docs/analytics-plan-2026-08-21.md, last commit
here) meant reading the gateway and billing paths closely. That turned up a set of
defects that had nothing to do with telemetry. Each one was then independently verified
against origin/main and, where it concerned production, against read-only gcloud
output. This PR fixes the ones a PR can fix.

Two of the findings were refuted during verification and are deliberately NOT
"fixed" here — see Not changed, on purpose below. One of them was refuted because
the plan had been drafted on a working tree 18 commits behind origin/main; that
correction is part of the doc commit.

Branch is cut from origin/main = b868319c.

❗ CI on this PR is red for a pre-existing reason — not this PR

The i18n workflow fails at its npm ci step, before check:i18n ever runs:
package.json and package-lock.json are out of sync (@emnapi/*, @swc/helpers).
The same failure is already on main: the push run for b868319c (PR #51,
2026-08-08) failed identically — run 31250086352. Every run of this workflow since
then has failed.
This PR changes neither package.json nor package-lock.json.
npm run check:i18n itself passes — verified locally: All 14 locales match en.json
(618 keys). Regenerating the lockfile is a separate, one-line-intent PR; it is
deliberately not bundled here because a ~2500-line lockfile rewrite generated on
macOS/npm 11 cannot be validated against this workflow's linux/node-20 environment
from here, and getting it wrong would also break the Docker build. See operator
action 9.

---

⚠️ Read this first: changes to billing semantics

Three commits change how and when money moves. Nothing here changes a price — but
they change when a charge is written, whether it can silently not happen, and who gets
rate-limited.

#	Change	Before	After
1	Billed calls block on the debit	void meterInvocation(...), response returned without waiting	await debitInvocation(...) before the response. Costs one Cloud SQL round-trip on billed calls only
2	A partially delivered SSE stream is now free	Charged in full as soon as upstream headers looked OK	Charged only if the stream runs to completion over a <400 response
3	One API key = one rate-limit bucket	gw:<keyId>:<ip> — 120/60s per IP, and the IP is caller-supplied	gw:<keyId> — 120/60s per key, in total. A caller legitimately spread across many IPs will start seeing 429s; ApiKey.rateLimit is the per-key escape hatch (now honoured, see below)
4	PayPal webhook can fail	Always answered 200, discarding the outcome	Returns 500 when an approved order could not be credited, so PayPal retries. captureAndCredit is idempotent, so a redelivery finding the money already credited answers already + 200
5	?key=<secret> no longer authenticates cron	Authorization header or query string	Header only

🔴 #1 and #2 share one operator prerequisite that this PR cannot satisfy: Cloud Run
must be given --no-cpu-throttling. The service currently has no such annotation, so
CPU is only allocated during request handling. #1 improves matters regardless (the
debit now completes while the request is still open). #2 does not — stream
settlement inherently runs after the response headers are flushed, so moving the charge
to the end of the stream makes it more dependent on CPU staying allocated, not less.
Without that flag, #2 relocates the charge to a less reliable moment. Please apply the
flag before or together with merging this.

---

What's fixed

Each finding is one commit.

Money

• F1 — fix(billing): await the credit debit before responding
  The DEBIT ledger entry and balance decrement lived inside a promise that was
  deliberately not awaited, in the same transaction as the Invocation row. On a
  throttled or reclaimed instance both vanish together: a 200 served, nothing charged,
  nothing recorded. billing.ts also swallowed every failure in an empty catch {}.
  Split into debitInvocation (money, awaited) and meterInvocation (telemetry,
  fire-and-forget); empty catch replaced with console.error.

• F2 — fix(billing): charge SSE streams when the stream ends
  The stream route priced the call off upstream.status before reading a byte, then
  four lines later could return 502 for a missing body — leaving a paid, status-200
  row behind a failed request. Mid-stream breaks, client disconnects and the 120s abort
  were likewise invisible. Now: startInvocation opens the row unpriced before
  relaying, settleInvocation closes that same row at termination (so a stream is
  still one Invocation and one callsCount), pull() gained the try/catch it never
  had, and three errorCode values are introduced — NO_STREAM_BODY,
  STREAM_ABORTED, CLIENT_DISCONNECTED (column already exists, no migration).

• fix(billing): stop losing the PayPal top-up fee (PAYPAL_ENV=live — real money)
  The idempotency probe looks for a TOPUP row, but gross and fee were two separate
  transactions. A crash in between left the fee unwritten, and the retry then saw the
  TOPUP row, said "already", and never charged it. Merged into one transaction
  (applyTopUp). Separately, three return "failed" paths sat after the PayPal
  capture — non-COMPLETED capture, unattributable payment, non-positive amount — and
  wrote nothing, logged nothing, alerted nothing. All now log with full context.

Security

• fix(security): stop bucketing gateway rate limits by a spoofable client IP
extractClientIp reads the first element of X-Forwarded-For, which Cloud Run
  appends to rather than replaces — so it is caller-controlled, and src/proxy.ts
  cannot normalise it (its matcher excludes api|v1|mcp). Varying the header per
  request minted a fresh bucket each time. This matters for money, not just load:
  checkCreditPreflight reads the balance with no lock before the debit, and
  billing.ts documents the limiter as the thing bounding that concurrency.
  RateLimitConfig gains perIp (default true, unchanged for the 11 anonymous
  endpoints); the three /v1 routes pass false.

• fix(security): accept CRON_SECRET only in the Authorization header
?key=<redacted> was accepted, and Cloud Run logs full URLs including query strings
  with 30-day retention — so one hand-rolled curl would have copied the secret into a
  log readable by anyone with logging.viewer, a far wider audience than the Secret
  Manager binding. Nothing used the query form (all six Scheduler jobs send the
  header). Comparison also moved to crypto.timingSafeEqual.

• F5 — fix(privacy): hash RequestLog client IPs and add a retention sweep
RequestLog.ip stored raw addresses and nothing ever deleted from the table — the
  only sweep in the codebase targets RateLimitBucket. Now a salted SHA-256 with the
  UTC date mixed in (so correlation is bounded to one day), plus a new
  /api/cron/retention route deleting rows older than 90 days in batches.
  ⚠️ Migration 010 is destructive on purpose: it DROPs ip rather than renaming
  it — renaming would preserve the plaintext this change exists to remove, under a
  name claiming it was hashed. Nothing reads the column. Snapshot the rows first if
  they are wanted for an investigation.

• feat(gateway): honour ApiKey.rateLimit
scopes, rateLimit and monthlyQuota had existed since the gateway shipped with
  nothing reading any of them. rateLimit is now honoured (null → 120, the previous
  literal, so existing keys are unaffected). scopes and monthlyQuota are marked
  NOT ENFORCED YET in the schema rather than left looking like working guarantees.

Correctness / observability

• F4 — feat(gateway): emit attributable telemetry for rejected calls
  Correcting the finding as filed: 401/402/429 are not unrecorded — Cloud Run's
  platform log has them. What is missing is attribution: no userId, no apiKeyId, no
  agent, no reason, and 402/429 are indistinguishable there. So this adds a structured
  stdout line (reason, route template, apiKeyId/userId/agentSlug, plus
  requiredUsd/balanceUsd on 402) rather than DB rows — wrapping these routes in
  withRequestLog would have added a write per rejection to an unbounded table.
  No response, status or body changes.

• F7 — feat(health): record agent health transitions
Agent.healthStatus is overwritten hourly and there is no history model anywhere, so
  "since when has this agent been down" had no answer. New AgentHealthCheck written
  only on a status flip, keeping the table proportional to flips rather than
  (agents × hours). HealthSummary gains transitions; the cron route now logs the
  summary, which was previously returned to Cloud Scheduler and discarded.

• F6 — docs(content): stop presenting scraped ClawSkills counts as live install data
Skill.downloads is scraped from clawskills.sh and refreshed nightly; nothing here
  ever increments it. The trending page nonetheless called it, in all 15 locales, "the
  most-downloaded … right now — a live leaderboard of what coding agents are
  installing" — not live, not now, and not installs. Copy corrected in all 15 locales
  (values only; check:i18n still reports 618 keys matching) and the column
  documented in the schema.

Docs

• docs(analytics): add the telemetry plan, with its refuted claims corrected
  Also redacts the local filesystem path and the other product's name that the draft
  carried, since this repo is public.

---

Not changed, on purpose

• F3 (cloudbuild.yaml drift) — REFUTED, no code change. PR Salvage: cloudbuild safety + tsx devDependency (May 10 local commits) #51 already fixed that
  file on origin/main; the placeholder strings existed only in a stale local
  checkout. And even the old file would not have "overwritten production secrets" —
  gcloud raises ConfigurationError client-side when a literal collides with an
  existing secretKeyRef, so the deploy step would have failed without writing
  anything. Corrected in the plan doc.
• F8 (prisma/sql preview feature, DELETE … LIMIT) — REFUTED, no code change.
  Neither trap is present. Both were real constraints for the new retention sweep
  though, which is why it uses ctid IN (SELECT … LIMIT n).

Deliberately deferred (reasons, not oversights)

• User.apiKey stored in plaintext. Real, and it is a full admin path via
  requireAdmin. Not fixed here because it needs a credential migration: five lookup
  sites, a schema change, a backfill on the live DB, and re-issuing every existing key.
  Hashing only new keys would silently break every current one. Needs its own PR plus
  an operator step.
• Renaming Skill.downloads → externalDownloads. The cleaner fix, but a column
  rename across ~15 call sites with a manual gcloud run deploy and no CI build means
  the old revision would query a dropped column during rollout. Not worth bundling with
  billing changes.
• ApiKey.scopes / monthlyQuota enforcement. Needs product decisions (scope
  taxonomy, month boundary, behaviour at the cap).
• RequestLog.userAgent is still stored in full.
• PAYMENT.CAPTURE.COMPLETED webhook subscriber. A PENDING capture that settles
  later is now logged but still not credited automatically; adding this needs a new
  subscription configured on the PayPal side too.

---

🔧 Operator actions required (this PR cannot do these)

1. --no-cpu-throttling on Cloud Run — blocking for F2, strongly wanted for F1.
   Without it, moving stream billing to stream-end makes the charge less reliable.
   Changes the billing model (always-on CPU costs more), so it needs a human decision.
   cloudbuild.yaml uses --update-* and will not clobber it.
2. Apply migrations 010 and 011 (prisma migrate deploy). 010 drops
   RequestLog.ip and its data — snapshot first if it is wanted.
3. Create the TAKO_IP_SALT secret and bind it to the service. Until it exists,
   ipHash is written as null (deliberately — an unsalted hash of a 32-bit address
   space is trivially reversible).
4. Create a Cloud Scheduler job for /api/cron/retention with
   Authorization: Bearer <redacted>, same shape as takoapi-health. Until then the
   route is inert and RequestLog keeps growing.
5. Purge the existing plaintext IP backlog — item 2 removes the column, which
   handles it; if a snapshot is taken first, delete the snapshot on a schedule.
6. Rotate CRON_SECRET if it was ever passed as ?key= in a shell, CI job or
   browser. Check Cloud Run logs for /api/cron/* URLs carrying a query string.
7. Verify no automation used ?key= before merging — it stops working.
8. Plan the User.apiKey credential migration (see above); it is the weakest
   credential in the system and it grants admin.
9. Regenerate package-lock.json — CI has been broken on main since 2026-08-08.
   npm ci fails on @emnapi/* and @swc/helpers desync, on GitHub Actions
   (linux/node-20) and locally alike, so the i18n workflow has not actually run
   check:i18n on any commit since b868319c. Needs npm install on a machine whose
   platform matches CI, then a check that both npm ci and the Docker build still
   work. Pre-dates this PR; deliberately kept out of it.

---

Verification

No test framework exists in this repo, so:

Check	Result
npx tsc --noEmit (whole project)	✅ exit 0
npm run build (next build)	✅ exit 0, compiled successfully; /api/cron/retention registers as dynamic
npx eslint on all 14 changed .ts files	✅ exit 0
npm run check:i18n (the only CI-enforced check)	✅ All 14 locales match en.json (618 keys)
npx prisma validate	✅ valid

npx eslint src/ reports 18 pre-existing errors, all react-hooks/set-state-in-effect
in admin/dashboard pages not touched by this PR (verified against the changed-file
list). They exist on origin/main unchanged.

Secrets: git diff origin/main was read through and grepped for
password|secret|token|private_key|BEGIN … PRIVATE|api[_-]?key|passwd|credential.
Every hit is a variable name, an env-var name, or prose. No literal credential values
are added. Secret values are referred to as <redacted> throughout.

🤖 Generated with Claude Code