The live tick was storing every synced row in Redis, sixty times an hour

[ralyodio]

Redis-g3Yk filled its 50GB volume and stopped being able to save at all, which took
tipoffwatch.com down: the container crash-looped every 2¼ minutes, loading for 124s,
accepting connections, starting a background save and getting killed.

Why it filled

The dataset was 8.65GB on disk and 25.3GB in memory across 209 keys. Two of those
keys were 23.5GB of it:

key	size	per entry
bull:live-scores:events	15.1 GB	~1.5 MB
bull:live-plays:events	8.4 GB	~844 KB

What a BullMQ processor returns is written twice, into the job hash and into the queue's
completed event. Since the mirror landed (#68, #69 on 2026-09-10) both live pollers
returned mirror: out — the whole nichedb payload, every row they had just synced —
alongside the counts they already reported. Nothing reads mirror, and nothing reads
returnvalue anywhere in the repo. At one tick a minute against BullMQ's default ceiling
of 10,000 events, that is ten thousand copies of a sync.

The disk then failed in the usual spiral. --save 60 1 writes a full second copy of the
dataset to temp-<pid>.rdb before renaming it over dump.rdb; that 8.65GB write no
longer fit, so every save died partway and left its partial file behind. Eleven of them,
30GB, the oldest dated the day after the mirror shipped — each one eating the space the
next save needed.

What this changes

• syncLiveScores / syncPlays return counts, not the payload. The counts were
  already in the object, so no caller changes.
• Every queue and worker caps its event stream at 1,000 entries. BullMQ's 10,000 is a
  sensible count and a dangerous size: it costs 10,000× whatever the fattest job returns.
  These streams are observability only and nothing here reads them back, so the smaller
  ceiling turns the same mistake into a slow query rather than an outage.

Already done on the Railway service

Not in this diff, for the record:

• purged the 11 orphaned temp-*.rdb files — /data went 36G/79% → 8.1G/18%
• trimmed the live streams — memory 25.32G → 5.26G, dump.rdb 8.65GB → 1.80GB,
  RDB load 124.5s → 25.9s
• start command now clears stale temp-*.rdb at boot and saves on 900 100 instead of
  60 1, so a snapshot is not rewriting the whole dataset every minute
• first DB saved on disk since Sep 11; verified live with x-cache: miss then hit

Testing

The seven test files that touch the changed surface (mirror, syncLiveScores,
syncPlays, startWorkers, @tipoff/queue) were run against origin/main's version of
the three files and then against this branch, same command, back to back:

origin/main code:  143 pass, 0 fail   (7 files)
this branch:       143 pass, 0 fail   (7 files)

The full bun test suite is not deterministic in a local shell and should not be read as
a gate here: it reports 3 or 4 failures depending on the run, and the test count varies
(1567 vs 1554). robots-auth-paths ×2 and playlist-editing ×1 fail identically on a
clean main; live-feed.test.js needs DATABASE_URL, makes real ESPN requests through
the bandwidth-capped proxy, and times out at module load under CPU contention. None of
them import @tipoff/queue or call the changed functions. The repo has no
.github/workflows, so Socket Security is the only CI gate.

biome check reports the same 2 pre-existing warnings as main and no new ones.

Worth a separate look

The app never reconnects after a Redis restart. Each time Redis bounced, /healthz kept
answering 200 while / hung indefinitely, and only a redeploy of the web service brought
it back — the same signature as the 2026-09-13 outage. The db-watchdog from genrewatch
PR #23 is still unported, and it is what turns any Redis restart into a silent outage.

🤖 Generated with Claude Code