Skip to content

fix: stop the duplicate scan from fetching every listed post - #90

Merged
Toldry merged 1 commit into
masterfrom
claude/scan-no-lazy-fetch
Oct 16, 2026
Merged

Toldry merged 1 commit into
masterfrom
claude/scan-no-lazy-fetch

Conversation

@Toldry

@Toldry Toldry commented Oct 16, 2026

Copy link
Copy Markdown
Owner

Requested by Querent · project thread

Before: on v2.14.3, docker compose exec bot python -m racb --process-one crashed during the image duplicate scan with prawcore.exceptions.TooManyRequests: received 429 HTTP response. #89 made the scan read crosspost_parent_list and similar fields. Most listed posts don't have them, and when code reads a field a listed post lacks, PRAW downloads the whole post again. That was one request per post, up to about 1000 in a full scan.

After: the scan reads media fields only from what the listing already returned (duplicate_detector.post_attribute), so classifying listed posts makes no requests. A post known only by its id, like a comment's submission, is fetched once. Separately, a rate limit or network error while Phase 3 handles an entry no longer crashes --process-one or aborts the scheduled job with a traceback. It logs a warning, keeps the row for the next run, and stops the batch.

How: post_attribute reads vars(post) for PRAW objects. TRANSIENT_NETWORK_ERRORS moved from main.py to racb.core.reddit so Phase 3 can use it. phase3.handle_comment_or_defer wraps handle_comment. The new unit test builds real PRAW Submissions as a listing creates them, with _fetch patched to fail. It fails on master and passes here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PXeu2v7gGePkS9NhaCgwDy


Generated by Claude Code

Reading a field a listed post lacks (crosspost_parent_list) made PRAW
fetch the post, one request per post, and the scan hit HTTP 429. Read
fields from what the listing returned, and keep a Phase 3 entry for the
next run instead of crashing when Reddit rate limits or fails.

Co-Authored-By: Toldry <toldry@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXeu2v7gGePkS9NhaCgwDy
@Toldry Toldry self-assigned this Oct 16, 2026
@Toldry
Toldry merged commit b4d9e81 into master Oct 16, 2026
4 checks passed
Toldry added a commit that referenced this pull request Oct 16, 2026
Co-Authored-By: Toldry <toldry@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bi2vprJUUCtGDcGHxadEba
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants