Repository navigation
Drain The Telemetry Hub Sentry Backlog Within A Bounded Budget - #1910
Merged
Merged
Conversation
…d budget The hub's Sentry pass read at most readLimit events (200) per ten-second tick. When a collector drains a durable outbox into the receiver — several thousand rows at once, the exact shape UM-1649 fixed on the collector side — the journal backlog grew faster than the pass drained it. Delivery itself was never at risk (the journal is durable and offsets advance only after success), but catch-up lagged behind enqueue by minutes and compaction stayed wedged behind the slow destination. A pass now keeps reading and delivering batches until a short read says the journal is caught up, bounded by a 5-second wall-clock budget and 25 read batches per pass, so one pass drains ~5,000 events instead of 200 while never becoming an unbounded run. The offset advances per fully delivered batch, so progress is durable even when a later batch fails. Failure semantics are unchanged: whatever a failed batch delivered before failing stays delivered and is marked, the pass stops there, and the next tick resumes. The dry-run path still works with no endpoint configured. Seven new vitest cases in __tests__/sentry-drain.test.mjs pin the drain, both budgets, mid-drain failure resume, the empty journal and sent-set skipping. Full hub suite (79 tests) and eslint pass. Refs UM-1661.
jaywedgeworth22
enabled auto-merge (squash)
November 7, 2026 23:41
Code Review Completed! 🔥The code review was successfully completed based on your current configurations. Kody Guide: Usage and ConfigurationInteracting with Kody
Current Kody ConfigurationReview OptionsThe following review options are enabled or disabled:
|
![kody-ai[bot]](https://avatars.githubusercontent.com/in/413034?s=60&v=4){kind=small}
Kody flagged that the drain budgets were only rechecked after a whole journal batch, so one retry-heavy batch could run far past the bound. Each chunk now rechecks the wall clock and a budget stop returns cleanly between chunks with ok:true and budgetStopped:true: the offset stays put, the remainder is re-read next pass, and the sent set keeps delivered events from going out twice. The wall-clock test now pins the stop, the unchanged offset, and the resume. Hub suite still green.
jaywedgeworth22
added a commit
that referenced
this pull request
Nov 8, 2026
Receipts for the UM-1661 runtime acceptance: the hub is on merged main (PR #1910 bounded drain) with its original journal volume preserved through an external mount after a re-parse briefly flipped it; the hub source map carries zcode and kimi-code; UM producer scoping carries both new producers; the detached Mac runtime is installed after PR #1909 fixed the updater fresh-install failure; the ZCode collector is receipt-verified on Mac (3,059 first pass, idempotency re-proof) and server (4,415 backfill); Kimi is installed but its Mac build writes no usage rows yet; the server collector wrapper node symlink was repaired (BotFleet had been failing every tick). Refs UM-1661.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs UM-1661.
Summary
This PR changes the Telemetry Hub Sentry sender
runOnce()from onereadLimitbatch per pass to a bounded catch-up drain. Each pass now keeps reading and delivering batches until a short read indicates catch-up, or the wall-clock budget or batch-count budget stops it, then resumes on the next tick. Delivery-failure semantics are unchanged: already-delivered events stay marked delivered, the pass stops at failure, and offset advances per fully delivered batch so a backlog does not wedge compaction. This addresses the backlog-growth shape where a collector drains a durable outbox into the receiver (UM-1649 pattern) within UM-1661 scope.Changes
DEFAULT_DRAIN_BUDGET_MS(5,000 ms) andDEFAULT_DRAIN_MAX_BATCHES(25) with newdrainBudgetMsanddrainMaxBatchesoptions oncreateSentrySender. DefaultreadLimitremains 200 events per journal read.runOnce()into a loop that scrubs, splits spans/logs, delivers, marks sent, aggregatesdelivered/quarantinedtotals, updates the journal offset per batch, and exits on short read, deadline, or batch-count exhaustion. Dry-run, empty-journal, missing-configuration, and Sentry refusal/held handling are preserved per batch.scripts/telemetry-hub/__tests__/sentry-drain.test.mjscovering multi-batch drain in one pass, request-count budget stop/resume, wall-clock budget stop, partial delivery preservation on mid-drain Sentry failure with later resume, empty journal response, sent-set deduplication, and default budget values. Run results are not shown in the changes, so passing behavior is unverified from the diff alone.Risks / Review
Automated review reported 1 medium finding at
scripts/telemetry-hub/sentry-sender.mjs:529that drain budgets are checked only after a batch, allowing one pass to exceed the time bound. Production drain pace under sustained backlog, wall-clock expiry mid-batch, and Sentry error paths remain unverified from the diff alone.