Skip to content

MySQL: the race checks are locking reads - #10

Merged
rameerez merged 14 commits into
mainfrom
fix/mysql-race
Oct 18, 2026
Merged

rameerez merged 14 commits into
mainfrom
fix/mysql-race

Conversation

@rameerez

@rameerez rameerez commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

CI's MySQL job (the one leg that can't run locally) fails three escalation race tests on main: two flags on one content remove it twice, and two flags on different fields or different content ban the person twice. PostgreSQL and SQLite pass.

Why

Under MySQL's default REPEATABLE READ, a plain SELECT returns the transaction's consistent snapshot. That snapshot is taken at the first plain read, which here was the plain reload right after with_lock, before the content's and the person's locks were granted. The second worker, once it had the lock, still saw no covering decision, no prior ban and the content still there, so it enforced again. PostgreSQL's READ COMMITTED (and SQLite's serialized writers) never showed this.

What

Every check that decides "already done" under a lock is now a locking read, the same on all three databases, with nothing special-cased for MySQL:

  • the covering decision and the open-report lookup in ResolveFlag: .lock;
  • whether the content is gone: reload(lock: true);
  • the per-person prior ban in ResolveReport: .lock;
  • Flag.flag_for_review!'s waiting flag: .lock.

The redundant plain reloads after with_lock (which already re-reads the row FOR UPDATE) are gone from ResolveFlag, ResolveReport and ResolveAppeal. Lock order is unchanged: flag, content, person, report.

Second fix, found by CI. The first push passed on MySQL, but the beta4 head then deadlocked once, in the same-person, different-content ban race. The new locking reads on moderate_reports take gap locks under REPEATABLE READ. Both escalations ran their covering read before the person's lock, so each held a gap lock that the other's report INSERT needed, while waiting on the person's lock.

ResolveFlag#enforce! now takes the flag owner's row lock right after the content's, before any read of the reports table. A second escalation about the same person waits before it touches that table. This stays within the gem's lock order (flag, content, person, report).

I deliberately added no automatic deadlock retry: it would re-run host hooks such as ban_handler. Two concurrent escalations about unrelated content and people can still, rarely, deadlock on MySQL. The database then rolls one back whole, nothing is applied, and its caller gets ActiveRecord::Deadlocked to retry, as with any Rails app on InnoDB.

The version goes to 1.0.0.beta4, with Gemfile.lock re-locked so CI's frozen install sees it.

Third commit, from a confirming review. There is now one lock order everywhere: flag, then content, then person, then report. It is documented once, on Moderate.with_row_lock.

  • Flag.flag_for_review! used to lock the content and then the waiting flag, the reverse of a flag decision, so a review and an enforcement of the same content could deadlock on any database. It now takes the content's lock only and reads the waiting flag with a plain read under it. That lock already serializes reviews of the same content.
  • Only a decision that can ban takes the person's lock. Removal-only enforcement never touches the account's row.
  • New race tests in lock_order_race_test.rb force each interleaving with test-only gates. Both orders of arrival deadlock on 76c9836, and the removal-only case blocks there; all three pass now.

Tests

  • The existing race tests in test/services/moderate/escalation_race_test.rb are the regression tests; they failed on MySQL.
  • Local runs are green: SQLite, PostgreSQL 16, and the Rails 7.1, 7.2 and 8.1 appraisals on both.
  • MySQL is verified by this PR's CI: the mysql (3.4) job passed three consecutive runs on 90d32f3, with 350 runs and 0 skips each, so the race tests run there.

🤖 Generated with Claude Code

rameerez and others added 14 commits October 18, 2026 21:33
CI's MySQL job failed the three escalation race tests (removed twice, banned
twice). Under MySQL's REPEATABLE READ, a plain SELECT returns the transaction's
consistent snapshot, taken at its first plain read: the plain reload right after
with_lock, before the content and person locks were granted. The second worker,
once it got the lock, still saw no covering decision, no ban, the content still
there, and enforced again. PostgreSQL and SQLite (READ COMMITTED, serialized
writers) never showed it.

The checks that decide "already done" are now locking reads, identical on every
database: the covering decision and the open-report lookup (.lock), whether the
content is gone (reload(lock: true)), the per-person prior ban (.lock), and
flag_for_review!'s waiting flag (.lock). The redundant plain reloads after
with_lock (which already re-reads the row FOR UPDATE) are gone from ResolveFlag,
ResolveReport and ResolveAppeal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bump the version, add the CHANGELOG entry, and re-lock so Gemfile.lock records
1.0.0.beta4 (CI installs frozen; a bump without re-locking fails at setup).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…adlock

The beta4 head's MySQL run deadlocked in the different-content, same-person ban
race: both escalations took their content's lock, then ran the covering read
(a locking read, which takes gap locks on moderate_reports under REPEATABLE
READ), then one waited on the person's lock while holding the gap lock the
other's report INSERT needed.

ResolveFlag#enforce! now takes the flag owner's row lock right after the
content's and before any read of moderate_reports (the gem's order: flag,
content, person, report), so a second escalation about the same person waits
before it touches the reports table at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… for a ban

From Codex's confirming pass on 76c9836:

- Flag.flag_for_review! locked the content and then the waiting flag (a locking
  read), the reverse of ResolveFlag (the flag, then the content), so a review and
  an enforcement of the same content could deadlock on PostgreSQL as well as
  MySQL. It now takes the content's lock only and reads the waiting flag with a
  plain read under it: the content lock already serializes reviews of that
  content, and no flag row lock means no inversion and no MySQL gap locks on the
  flags index for two reviews that insert.
- Every escalation locked the owner's row, even removal-only. The person's lock
  is now taken only when the decision can ban (ban_user), still before any read
  of moderate_reports, which keeps the MySQL gap-lock fix for bans.

The order is documented once, on Moderate.with_row_lock. Race tests (PostgreSQL
and MySQL; SQLite has no row locks), with test-only gates forcing each
interleaving: enforcement holding the flag while a review arrives, and a review
holding the content while enforcement arrives (both deadlock on 76c9836);
removal-only enforcement while another transaction holds the author's row
(blocks on 76c9836). Version stays 1.0.0.beta4; Gemfile.lock is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex's second pass on 90d32f3: flag_for_review!'s plain "is a flag waiting?"
read under the content lock could miss another transaction's committed flag on
MySQL (REPEATABLE READ: a transaction that already read keeps its snapshot; the
content lock refreshes only the locking read of the content row), creating two
pending flags.

moderate_flags gets pending_key ("<flaggable_type>/<flaggable_id>/<field>" while
pending, NULL once closed, kept in sync by the model on every save, so closing
frees it and reopening takes it back) with a unique index. Flag.flag! inserts in a
savepoint and, on RecordNotUnique, returns the waiting flag read with a locking
read (the latest committed row on every database), retrying if it was decided
meanwhile. flag_for_review! takes no lock first (it holds no content while
waiting on a flag, so the order flag, content, person, report holds) and no plain
read decides anything. While a flag waits on a field, another flag on it (filters
and ClassifyJob too) joins it.

The migration ships in the gem's db/migrate, guarded and reversible, backfilling
the newest pending flag per content and field; the install template has the
column for fresh installs; the dummy keeps the beta3 schema so CI runs the
upgrade. The engine stops appending its db/migrate when Rails' engine tasks
already do (the gem's own Rakefile) or the host copied it (*.moderate.rb), either
of which raised DuplicateMigrationNameError.

Tests: Codex's sequence (A reads, B files and commits, A files: same flag, one
row); concurrent filings; the unique index (refusal, closing frees it, reopening);
the migration down/up/backfill/idempotence; the race tests rewritten for one
pending flag per field, with every sleep replaced by waiting until the database
shows the other connection blocked (pg_blocking_pids, innodb_trx LOCK WAIT),
bounded at 5 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
information_schema.innodb_trx is served from a cache that refreshes only
when it hasn't been read for 100 ms, so a tight poll never saw the second
connection's wait: the test failed after 5 s, and left the first thread's
transaction parked, so CI's MySQL job hung. Poll
performance_schema.data_lock_waits instead, and release every gate in an
ensure so a failed wait fails fast.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a deadlocked filing

InnoDB keeps a flag in two places, its row and its pending_key index
entry. A filing that collides takes the entry, then waits for the row;
a decision locked the row, then needed the entry to free the key: a
deadlock every time. Flag#lock! now takes the entry first. The one
deadlock left (the duplicate check queued on the gap the closing flag's
entry moves into) is retried by Flag.flag! when it owns the transaction,
and raised inside a caller's, which InnoDB rolled back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MySQL refuses defaults on JSON columns whichever adapter talks to it;
the migration checked only for mysql2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…plicates

MySQL's default collation ignores case, so the unique index made
"Post/1/body" and "POST/1/body" one key: the second type's Flag.flag!
returned the first type's flag, and the backfill, comparing keys as Ruby
strings, could leave rows the index refused. The column is now
utf8mb4_bin on MySQL (migration, install template; an existing column is
converted), and before adding the index the migration asks the database
itself for keys it would treat as one, stopping with their names and
flag ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After three deadlocks it raised RecordNotUnique, though nothing was
duplicated. It now re-raises the last ActiveRecord::Deadlocked, and
RecordNotUnique only when each collision's waiting flag was decided
before it could be joined.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MySQL's utf8mb4_bin ignores trailing spaces (PAD SPACE), so a field that
differed only by them made a pending_key the database treats as another
flag's. Flag normalizes `field` (assignment, insert_all, where), and the
collision lookup in Flag.flag! builds its key from the stripped field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tten

The backfill compared keys as Ruby strings, so on MySQL a legacy flag
whose field ends in a space (equal under PAD SPACE to a key already
held) made the UPDATE raise RecordNotUnique before the diagnostic. The
keys to write and those already written now go to a temporary table
whose key column has pending_key's collation; any key the database
groups twice stops the migration, naming the flags.

No column existed before beta4, so the conversion path is gone: an
existing column is only verified (on MySQL a case-insensitive one stops
the migration with the ALTER TABLE to run), and down drops the index and
the column. README and CHANGELOG note PAD SPACE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The install template created pending_key and its index, so on a fresh
install the upgrade migration's up left them alone yet its down dropped
both, taking a feature the install migration still needed. The template
is back to the beta3 table shape, and moderate:install copies the gem's
migrations after it, in order, as moderate:install:migrations names them
(*.moderate.rb); an app holding copies runs those and the engine leaves
the gem's path out, an app installed without them (beta3) keeps getting
it. Engine.migration_paths_for makes that choice testable.

Tests: the generator writes both migrations in order and copies nothing
twice; a fresh app and a beta3 app each find the migrations they need;
on a new SQLite database, install + upgrade end with pending_key and
rolling back the upgrade leaves the install's schema as it was (fails
against the previous template).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rades

The migration's comment, the CHANGELOG and the README upgrade note now
say how a fresh install and an upgraded app each get it, that its down
leaves create_moderate_tables' schema intact, and that an app holding
copies runs bin/rails moderate:install:migrations after each upgrade.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rameerez
rameerez marked this pull request as ready for review October 18, 2026 22:48
@rameerez
rameerez merged commit 328d92a into main Oct 18, 2026
28 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant