feat!: run jobs and Phase 1 off the stream thread, per-bot switches

[Toldry]

Requested by Querent · project thread

Before: the bot ran on one thread. While Phase 2, Phase 3, the inbox pass, the cleanup or a Phase 1 duplicate search (up to 200 preview downloads) ran, nobody read the r/all stream. PRAW's stream only catches up on the newest 100 comments, so comments posted meanwhile were lost. LISTEN_ONLY=True only stopped crossposting; the three auxiliary bots still replied and the cleanup job still deleted posts.

After: the main thread only reads the stream. Every scheduled job runs on its own thread, Phase 1 runs on a pool of worker threads (PHASE1_WORKERS, default 4), and docker stop shuts all of them down cleanly. Each bot account has its own on/off switch, which replaces LISTEN_ONLY.

Breaking (hence feat!):

• LISTEN_ONLY is removed and ignored. An .env with LISTEN_ONLY=True must switch to AUTO_CROSSPOST_BOT_ENABLED=False, or the bot starts crossposting.
• The --only-phase2 flag is removed. It was added for an old Heroku Procfile entry, and nothing uses it now.

How:

• Threads
  • racb/workers.py adds PeriodicJob, one thread per job. A job's next run starts one interval after its previous run ends. Phase 2 and Phase 3 share a lock, so as before they never overlap.
  • It also adds Phase1Workers. The stream thread runs only the regex and queues the comment ID of each match. The queue holds 1000 IDs; matches that don't fit are dropped and counted in the heartbeat. A worker then fetches the comment again through its own PRAW instance, which costs one extra request per match.
  • reddit.get_reddit_instance() returns a separate PRAW instance per thread and account, because PRAW isn't thread safe. db.get_connection() gives each thread its own connection. The schema script runs on one connection at a time, because concurrent CREATE OR REPLACE FUNCTION can fail.
• Shutdown
  • racb/core/shutdown.py holds a stop signal that SIGTERM and SIGINT set.
  • Back-offs wait with shutdown.stop_requested_within(), and Phase 2 and Phase 3 stop between entries.
  • start_bot() waits up to 50 s for the threads, and compose gets stop_grace_period: 60s.
  • A crash on any thread (with DEBUG, or from a bug) stops everything and is re-raised, so Docker restarts the process as before.
• Per-bot switches
  • The switches are AUTO_CROSSPOST_BOT_ENABLED, SUB_DOESNT_EXIST_BOT_ENABLED, SAME_SUBREDDIT_BOT_ENABLED and SAME_POST_BOT_ENABLED, all True by default.
  • A disabled bot never comments, posts or deletes, and its inbox stays unread. Turning AutoCrosspostBot off also skips Phase 3 and the cleanup job.
  • With all four off, only the stream, filtering, DB inserts, Phase 2 and the known-subreddits buffer run.
• --process-one authenticates only AutoCrosspostBot, the one account it uses. It exits with code 2 if that bot is disabled.
• Other changes
  • StreamMetrics and the known_subreddits buffer get locks, and log lines name their thread.
  • The schedule dependency is removed.
  • Master (fix: stop the duplicate scan from fetching every listed post #90) is merged in. Phase 3 keeps fix: stop the duplicate scan from fetching every listed post #90's per-entry deferral and adds a shutdown check between entries.
  • AGENTS.md, .env.example and the CHANGELOG are updated, and all 1725 unit tests pass.

Rate limits: each thread's prawcore limiter paces itself on Reddit's shared per-account quota headers. Together they can overshoot by a few requests near the end of a window, which ends in a TooManyRequests back-off (documented in AGENTS.md section 7). I haven't tested this against live Reddit.

Folds in #93.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bi2vprJUUCtGDcGHxadEba