[vpetersson-bot]

What's wrong today

The project's Sentry quota is consumed faster than it replenishes, so errors stop being accepted partway through each billing period. The cost is not the missing events — it is that every issue's lastSeen freezes on the day the allowance ran out, so "it has not recurred since" stops being evidence of anything. A regression can land and leave no record.

What spends the quota is depth, not breadth. A device stuck in a retry loop repeats one error thousands of times while the rest of the fleet contributes a handful each. In a fully recorded day, one board produced 1,761 of an issue's 1,762 events (ANTHIAS-3H) and another produced 1,063 of 1,093 (ANTHIAS-2D) — two boards crowding out every other issue.

Nothing bounds this today. before_send filters transient noise (redis blips, DNS failures offline, yt-dlp download errors), but a genuine error repeated ten thousand times passes every one of those checks.

What this does

Counts occurrences per issue, per process, over a one-hour window, and keeps the 1st, 2nd, 4th, 8th, 16th, … — dropping what falls between two powers of two. A loop costs O(log n) events instead of n: 1,761 becomes 11.

The magnitude is not lost with the events. Each surviving event carries its true running count in a throttle.occurrence tag, so triage can still tell the 4th occurrence from the 4th distinct event.

Two properties the design turns on:

• Nothing can be hidden by this. Every device still reports every distinct issue it hits, at least once per window. Only repetition of something already reported is dropped.
• The gate runs last in before_send. Noise filtered above it is not a report and must not spend a key's budget — otherwise a loop of filtered redis blips would throttle the one real error raised alongside it.

Issues are keyed by exception type plus the innermost Anthias frame (falling back to the innermost frame of any kind), or for logging-integration events by logger plus the unformatted message template. The template is what keeps "Timed out waiting for UP message from <ForkPoolWorker-26>" and the same line for worker 27 a single key rather than one per worker (ANTHIAS-1Q). An event matching neither shape is left unthrottled rather than given a coarse key that could merge unrelated issues — this filter must never be the reason a real bug goes unseen.

Bookkeeping is bounded at 512 live keys per process, evicting expired windows first and then the oldest; the overflow path can only let more events through, never fewer. The window clock is time.monotonic, so a board whose wall clock jumps years when NTP first lands doesn't expire or freeze every window at once.

How I know it's right

15 new tests in tests/test_sentry.py covering: the powers-of-two sequence; the occurrence tag (and that the first event is untagged, and that an existing tag mapping survives); that distinct issues don't share a budget; that noise dropped earlier spends no budget; that one log template is one key across its arguments while different templates are not; that an unkeyable event is never throttled; window expiry resetting the count; and both eviction paths.

Full unit suite: 2402 passed. The one failure on this branch, test_viewer.py::TestWaitForWaylandSocket::test_probe_reports_not_ready_for_a_non_socket_path, reproduces identically on untouched master under pytest -n auto and passes when run alone — a pre-existing parallel-run flake, unrelated to this change. ruff check, ruff format --check and mypy are clean.

A new module-level autouse fixture clears the throttle state around every test in test_sentry.py; the counters are module scope, so without it a key counted by one test would still be counted in the next.

Follow-up, not in this PR

Thinning the loops buys headroom but does not by itself bring the fleet under its allowance. The quota level is a separate decision, and the per-device loops that produce these bursts (ANTHIAS-3H, ANTHIAS-D) are still worth fixing at source.

🤖 Generated with Claude Code

[Copilot]

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud