[pufit]

GitHub Repos source: report the last day after a gap, not nothing

Why

Since #593, the polling GitHub Repos source re-reads a listing as a new baseline when it was last read more than a day ago, and reports nothing for it. A daily schedule (0 9 * * *) comes back a day and a moment after the last check (the scheduler's delay, and listed_at is stamped to the second), so practically every check is such a listing: the watch stays silent every morning, with only an info log. A weekday schedule loses everything from Friday evening to Monday. Hosted mode is not affected in practice, because the gateway polls every minute.

What

"Nothing older than a day is backfilled" now means that the last day is reported, not nothing:

• A listing that was read before and is more than a day old is read as of a day and an hour ago. Its cursor becomes max(cursor, now − (24 h + 1 h)): items created or published within the last day, with an hour of grace for the scheduler's delay, are reported (within the one page, as for any listing), and older ones are not. So an item created seconds after the previous check is not lost when the next daily check comes a moment late.
• If its own cursor falls within that window, the listing is read as it is, including the keys of the items it saw in its cursor's second. Otherwise the window has no keys, since nothing listed back then was created after it. The window never starts before the listing's own cursor, so nothing is reported twice; the inclusive same-second rule and the record-ID dedup are unchanged.
• The pull requests the listing follows stay followed, so updates: true works with a daily schedule: a followed pull request that changed and was updated within the last day is reported after the gap, as it is without one.
• A listing never read before (a new repository or type) still takes a baseline.
• The earlier version's timestamp cursor, when more than a day old, now reports the last day too.

Last check	This check	Reported
23 h ago	now	everything new since the last check (unchanged)
24 h + a moment ago (daily schedule)	now	everything new since the last check (within the last day and its hour of grace)
7 days ago	now	only what is new in the last 25 h
never (new repository or type)	now	nothing: the baseline (unchanged)

Switching the watch off and on in poll mode resumes where it left off within the day. Beyond a day, it reports the last day's items. In hosted mode, the gateway forgets the watch when it is switched off, so switching it on starts fresh.

Companion gateway change: https://github.com/ClickHouse/nerve-private/pull/573.

Changes

• nerve/sources/github_repos.py: STALE_GRACE and _window(). The fetch loop reads a stale listing against it and keeps the fingerprints. Also the STALE_AFTER comment and the module docstring.
• docs/sources.md (the "No backfill beyond a day" bullet, the Upgrades bullet, the hosted bullet), docs/config.md (the repos row), config.example.yaml.
• tests/test_github_repos_source.py.

Verification

• New tests:
  • A check a day and a second after the last reports the items of the last day, but not an older item that only changed.
  • A check a day and five seconds after the last reports a pull request created 2 s after the previous check.
  • A week's gap reports only the last day and its hour of grace (an item 24.5 h old in, one 26 h old out).
  • A followed pull request pushed within the last day is reported after a 2-day gap.
  • A listing whose read time is unknown is read against the last day.
  • A stale listing whose cursor falls within the day keeps its keys.
  • The earlier cursor more than a day old reports the last day.
  • The 23 h case is unchanged.
• Mutation checks, each of which makes tests fail:
  • "stale → report nothing" (7 failures).
  • No hour of grace: the item created 2 s after the previous check is lost (3 failures).
  • The window starting before the listing's own cursor, which reports an item twice (1).
  • A stale listing read from its old cursor, which backfills beyond the day (3).
  • The window dropping the keys of a cursor within the day (1).
  • A stale listing forgetting its followed pull requests (1).
• tests/test_github_repos_source.py and tests/test_hosted_sources.py: 183 passed. CI runs the full suite. No new ruff finding.

Generated by Nerve

[nerveai]

Nerve PR review · ✅ Ready to merge

This fixes the regression I flagged on #593. Since that merge, a poll-mode listing last read more than 24 h ago took a new baseline, so a daily schedule (always a moment past 24 h) never reported anything. Now a stale listing is read against max(own cursor, now − 25 h) and keeps its followed PRs. I traced _window and _scan: when the window replaces the cursor, nothing listed at the last check can have created_at after the window start, so nothing is reported twice, and a daily run (gap < 25 h) behaves exactly like an unbroken one. Hosted mode uses HostedGitHubSource (registry.py:245-249), so this change is poll-only. The cursor format is unchanged.

Completeness: 8 new or rewritten tests cover the daily, week-long, legacy-cursor, unknown-listed_at and followed-PR cases. docs/sources.md, docs/config.md and config.example.yaml are updated. No config or migration change. The hosted bullet (docs/sources.md:414-415) assumes nerve-private#573 is deployed.

Worth considering:

• A weekday schedule (0 9 * * 1-5) still drops Friday 09:00 → Sunday 08:00 on Monday, with no warning (github_repos.py:341). The one-page cap already bounds a burst, so a longer horizon would cover the weekend. If the day stays, a warning that names the skipped span when the gap exceeds 25 h would make the loss visible: today :690 logs at info level for every stale listing.

_{Checked: 10 source/hosted/sync test files in an offline sandbox at 00281af: 489 passed. Read the github_repos.py fetch/scan/window code, the runner's cursor persistence and the registry wiring. CI backend job still running · automated review by nerveai — mention @nerveai to ask a follow-up.}