A hung dependency restarts the container instead of hanging the site

[ralyodio]

Twice now Redis has gone away and tipoffwatch has stayed down until a person noticed:
2026-09-13, and again on 2026-09-22 while I was clearing the volume. Both times the
shape was identical:

• /healthz → 200 in 50ms
• / → nothing at all, until curl gave up
• Railway → service Online, every liveness signal green
• restarting Redis did not fix it; only redeploying the web service did

Why it cannot recover on its own

One line in the shared client. BullMQ requires maxRetriesPerRequest: null, and that turns
a disconnect into a hang: a command issued while the socket is down is queued until it
returns rather than rejected.

Verified rather than assumed — a bare ping() against a dead port, with the app's own
client options:

bare ping() against a dead redis -> HUNG      (never settled)
watchdog gave up: true
reason names redis: true

The page render does catch Redis errors and fall through to Postgres, but there is no error
to catch, so the request simply stops. /healthz touches neither dependency, which is why
nothing ever went red.

What this adds

genrewatch already solved this for Postgres in its PR #23, after the same failure on its
pool on 2026-09-07. The rule it established is the important part: the probe must go
through the client the requests actually use. A separate connection is the one thing
guaranteed to look healthy during this failure.

So that module is ported here, given a subject, and both dependencies get a watchdog:

	probe	gives up after
the database pool	select 1 on the shared sql handle	3 consecutive failures
redis	PING on the shared ioredis connection	4 consecutive failures

Redis gets one more failure than the pool on purpose. A healthy Redis here is routinely
unreachable for a while, because it restarts by reading an RDB off the volume before
accepting anything — 26 seconds now, 124 before the event streams were trimmed in #78. Boot
already throws if Redis is absent, so an impatient watchdog would turn one Redis deploy into
a deploy loop on this service.

Both watchdogs run for either role: the web side reads the page cache and writes passkey
challenges through the same client the workers queue on, and a wedged worker stops every
score updating without anything going red. They are stopped first on SIGTERM, so a clean
drain is never mistaken for a wedge.

Every knob has a working default (DB_WATCHDOG_*, REDIS_WATCHDOG_*), so no service needs
a new variable.

On exiting

It reads as drastic for a web server and it is the cheapest correct move: the failure is
process-local state no request can repair, a restart demonstrably clears it (both outages
ended with one), and Railway replaces the container in about a minute. Hanging forever is
not the safer option — it is the outage.

Testing

test/db-watchdog.test.js — 10 pass, 0 fail. Seven cases ported from genrewatch
(gives up only on consecutive failures, a success clears the count, a rejection counts the
same as a hang, it does not keep firing after giving up, it refuses to start without a
probe, it does not hold the process open). Three new ones for this repo's half: the reason
names redis, a PING queued forever is caught, and a Redis that is merely slow to load its
RDB does not trigger a restart.

Also: 77 pass / 0 fail across the watchdog plus the config, nichedb and odds files;
bun build apps/web/src/main.js bundles clean (1160 modules), which is what checks the new
connection import actually resolves; biome check clean on all three files.

🤖 Generated with Claude Code