Repository navigation
MySQL: the race checks are locking reads - #10
Merged
Merged
Conversation
CI's MySQL job failed the three escalation race tests (removed twice, banned twice). Under MySQL's REPEATABLE READ, a plain SELECT returns the transaction's consistent snapshot, taken at its first plain read: the plain reload right after with_lock, before the content and person locks were granted. The second worker, once it got the lock, still saw no covering decision, no ban, the content still there, and enforced again. PostgreSQL and SQLite (READ COMMITTED, serialized writers) never showed it. The checks that decide "already done" are now locking reads, identical on every database: the covering decision and the open-report lookup (.lock), whether the content is gone (reload(lock: true)), the per-person prior ban (.lock), and flag_for_review!'s waiting flag (.lock). The redundant plain reloads after with_lock (which already re-reads the row FOR UPDATE) are gone from ResolveFlag, ResolveReport and ResolveAppeal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bump the version, add the CHANGELOG entry, and re-lock so Gemfile.lock records 1.0.0.beta4 (CI installs frozen; a bump without re-locking fails at setup). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…adlock The beta4 head's MySQL run deadlocked in the different-content, same-person ban race: both escalations took their content's lock, then ran the covering read (a locking read, which takes gap locks on moderate_reports under REPEATABLE READ), then one waited on the person's lock while holding the gap lock the other's report INSERT needed. ResolveFlag#enforce! now takes the flag owner's row lock right after the content's and before any read of moderate_reports (the gem's order: flag, content, person, report), so a second escalation about the same person waits before it touches the reports table at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… for a ban From Codex's confirming pass on 76c9836: - Flag.flag_for_review! locked the content and then the waiting flag (a locking read), the reverse of ResolveFlag (the flag, then the content), so a review and an enforcement of the same content could deadlock on PostgreSQL as well as MySQL. It now takes the content's lock only and reads the waiting flag with a plain read under it: the content lock already serializes reviews of that content, and no flag row lock means no inversion and no MySQL gap locks on the flags index for two reviews that insert. - Every escalation locked the owner's row, even removal-only. The person's lock is now taken only when the decision can ban (ban_user), still before any read of moderate_reports, which keeps the MySQL gap-lock fix for bans. The order is documented once, on Moderate.with_row_lock. Race tests (PostgreSQL and MySQL; SQLite has no row locks), with test-only gates forcing each interleaving: enforcement holding the flag while a review arrives, and a review holding the content while enforcement arrives (both deadlock on 76c9836); removal-only enforcement while another transaction holds the author's row (blocks on 76c9836). Version stays 1.0.0.beta4; Gemfile.lock is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex's second pass on 90d32f3: flag_for_review!'s plain "is a flag waiting?" read under the content lock could miss another transaction's committed flag on MySQL (REPEATABLE READ: a transaction that already read keeps its snapshot; the content lock refreshes only the locking read of the content row), creating two pending flags. moderate_flags gets pending_key ("<flaggable_type>/<flaggable_id>/<field>" while pending, NULL once closed, kept in sync by the model on every save, so closing frees it and reopening takes it back) with a unique index. Flag.flag! inserts in a savepoint and, on RecordNotUnique, returns the waiting flag read with a locking read (the latest committed row on every database), retrying if it was decided meanwhile. flag_for_review! takes no lock first (it holds no content while waiting on a flag, so the order flag, content, person, report holds) and no plain read decides anything. While a flag waits on a field, another flag on it (filters and ClassifyJob too) joins it. The migration ships in the gem's db/migrate, guarded and reversible, backfilling the newest pending flag per content and field; the install template has the column for fresh installs; the dummy keeps the beta3 schema so CI runs the upgrade. The engine stops appending its db/migrate when Rails' engine tasks already do (the gem's own Rakefile) or the host copied it (*.moderate.rb), either of which raised DuplicateMigrationNameError. Tests: Codex's sequence (A reads, B files and commits, A files: same flag, one row); concurrent filings; the unique index (refusal, closing frees it, reopening); the migration down/up/backfill/idempotence; the race tests rewritten for one pending flag per field, with every sleep replaced by waiting until the database shows the other connection blocked (pg_blocking_pids, innodb_trx LOCK WAIT), bounded at 5 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
information_schema.innodb_trx is served from a cache that refreshes only when it hasn't been read for 100 ms, so a tight poll never saw the second connection's wait: the test failed after 5 s, and left the first thread's transaction parked, so CI's MySQL job hung. Poll performance_schema.data_lock_waits instead, and release every gate in an ensure so a failed wait fails fast. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a deadlocked filing InnoDB keeps a flag in two places, its row and its pending_key index entry. A filing that collides takes the entry, then waits for the row; a decision locked the row, then needed the entry to free the key: a deadlock every time. Flag#lock! now takes the entry first. The one deadlock left (the duplicate check queued on the gap the closing flag's entry moves into) is retried by Flag.flag! when it owns the transaction, and raised inside a caller's, which InnoDB rolled back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MySQL refuses defaults on JSON columns whichever adapter talks to it; the migration checked only for mysql2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…plicates MySQL's default collation ignores case, so the unique index made "Post/1/body" and "POST/1/body" one key: the second type's Flag.flag! returned the first type's flag, and the backfill, comparing keys as Ruby strings, could leave rows the index refused. The column is now utf8mb4_bin on MySQL (migration, install template; an existing column is converted), and before adding the index the migration asks the database itself for keys it would treat as one, stopping with their names and flag ids. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After three deadlocks it raised RecordNotUnique, though nothing was duplicated. It now re-raises the last ActiveRecord::Deadlocked, and RecordNotUnique only when each collision's waiting flag was decided before it could be joined. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MySQL's utf8mb4_bin ignores trailing spaces (PAD SPACE), so a field that differed only by them made a pending_key the database treats as another flag's. Flag normalizes `field` (assignment, insert_all, where), and the collision lookup in Flag.flag! builds its key from the stripped field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tten The backfill compared keys as Ruby strings, so on MySQL a legacy flag whose field ends in a space (equal under PAD SPACE to a key already held) made the UPDATE raise RecordNotUnique before the diagnostic. The keys to write and those already written now go to a temporary table whose key column has pending_key's collation; any key the database groups twice stops the migration, naming the flags. No column existed before beta4, so the conversion path is gone: an existing column is only verified (on MySQL a case-insensitive one stops the migration with the ALTER TABLE to run), and down drops the index and the column. README and CHANGELOG note PAD SPACE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The install template created pending_key and its index, so on a fresh install the upgrade migration's up left them alone yet its down dropped both, taking a feature the install migration still needed. The template is back to the beta3 table shape, and moderate:install copies the gem's migrations after it, in order, as moderate:install:migrations names them (*.moderate.rb); an app holding copies runs those and the engine leaves the gem's path out, an app installed without them (beta3) keeps getting it. Engine.migration_paths_for makes that choice testable. Tests: the generator writes both migrations in order and copies nothing twice; a fresh app and a beta3 app each find the migrations they need; on a new SQLite database, install + upgrade end with pending_key and rolling back the upgrade leaves the install's schema as it was (fails against the previous template). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rades The migration's comment, the CHANGELOG and the README upgrade note now say how a fresh install and an upgraded app each get it, that its down leaves create_moderate_tables' schema intact, and that an app holding copies runs bin/rails moderate:install:migrations after each upgrade. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rameerez
marked this pull request as ready for review
October 18, 2026 22:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI's MySQL job (the one leg that can't run locally) fails three escalation race tests on main: two flags on one content remove it twice, and two flags on different fields or different content ban the person twice. PostgreSQL and SQLite pass.
Why
Under MySQL's default REPEATABLE READ, a plain
SELECTreturns the transaction's consistent snapshot. That snapshot is taken at the first plain read, which here was the plainreloadright afterwith_lock, before the content's and the person's locks were granted. The second worker, once it had the lock, still saw no covering decision, no prior ban and the content still there, so it enforced again. PostgreSQL's READ COMMITTED (and SQLite's serialized writers) never showed this.What
Every check that decides "already done" under a lock is now a locking read, the same on all three databases, with nothing special-cased for MySQL:
ResolveFlag:.lock;reload(lock: true);ResolveReport:.lock;Flag.flag_for_review!'s waiting flag:.lock.The redundant plain reloads after
with_lock(which already re-reads the rowFOR UPDATE) are gone fromResolveFlag,ResolveReportandResolveAppeal. Lock order is unchanged: flag, content, person, report.Second fix, found by CI. The first push passed on MySQL, but the beta4 head then deadlocked once, in the same-person, different-content ban race. The new locking reads on
moderate_reportstake gap locks under REPEATABLE READ. Both escalations ran their covering read before the person's lock, so each held a gap lock that the other's report INSERT needed, while waiting on the person's lock.ResolveFlag#enforce!now takes the flag owner's row lock right after the content's, before any read of the reports table. A second escalation about the same person waits before it touches that table. This stays within the gem's lock order (flag, content, person, report).I deliberately added no automatic deadlock retry: it would re-run host hooks such as
ban_handler. Two concurrent escalations about unrelated content and people can still, rarely, deadlock on MySQL. The database then rolls one back whole, nothing is applied, and its caller getsActiveRecord::Deadlockedto retry, as with any Rails app on InnoDB.The version goes to 1.0.0.beta4, with
Gemfile.lockre-locked so CI's frozen install sees it.Third commit, from a confirming review. There is now one lock order everywhere: flag, then content, then person, then report. It is documented once, on
Moderate.with_row_lock.Flag.flag_for_review!used to lock the content and then the waiting flag, the reverse of a flag decision, so a review and an enforcement of the same content could deadlock on any database. It now takes the content's lock only and reads the waiting flag with a plain read under it. That lock already serializes reviews of the same content.lock_order_race_test.rbforce each interleaving with test-only gates. Both orders of arrival deadlock on 76c9836, and the removal-only case blocks there; all three pass now.Tests
test/services/moderate/escalation_race_test.rbare the regression tests; they failed on MySQL.mysql (3.4)job passed three consecutive runs on 90d32f3, with 350 runs and 0 skips each, so the race tests run there.🤖 Generated with Claude Code