Skip to content

compiler: Fix stale reads and lost writes in buffering - #3063

Draft
ggorman wants to merge 7 commits into
fix-buffering-skip-rulefrom
fix-buffering-stale-reads
Draft

ggorman wants to merge 7 commits into
fix-buffering-skip-rulefrom
fix-buffering-stale-reads

Conversation

@ggorman

@ggorman ggorman commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Stacked on #3062 (which is on #3061); the base switches as they merge.

Bugs predating #3050, found against opt='noop'. Every test program has at most one firing write per step, slot and point, except the last one, which says so.

  1. ir: Let GuardBoundNext be negated. BaseGuardBoundNext._new did not accept evaluate, so Guard.negated raised a TypeError for bound guards, and so did any And over guards containing one (SymPy negates the relational arguments of an And to detect contradictions). Latent on main; commit 4 builds such an And for every buffered Function, which meets the bound guards of prefetches. Test test_ir.py::TestGuards::test_guard_bound_next_negated.
  2. Read-before-write and read-ahead slots were stale. g = f; f = T read the slot left by the previous iteration: with f.data[:] = 10 and eqs = [Eq(T, time), Eq(g, f), Eq(f, T)], main gives g = [10, 0, 1], noop [10, 10, 10]. Such slots are now loaded from the Function and the leading written slot is copied back. Tests test_read_before_write, test_read_ahead.
  3. Non-consecutive Clusters with equal guards formed distinct groups (itertools.groupby); the buffer was sized from the last group only. Test test_non_consecutive_guards.
  4. Reads after writes under other guards now share the writers' buffer, loaded (slots behind the written index included) before the first access. A load keeps only the guard conjuncts that read nothing written in the loop; an access under a guard reading a Function does not count as having loaded or written the slot; no sharing at a non-constant slot distance or across different spatial Dimensions/SubDomains (separate buffers instead of a CompilationError). Main-wrong examples: test_mutable_read_guards[repeated] gives h = [0, 2, 20, 30] vs noop [1, 2, 20, 30]; test_guard_changes_between_accesses[write-read|read-read] (with buf-async-degree=2 main reads uninitialised memory). Tests test_read_write_different_guards, test_read_write_shifted_slot, test_read_behind_guarded_write, test_read_write_different_subdomains, test_guard_changes_between_accesses, test_mutable_read_guards.
  5. Guards mutated between writes are refused with a CompilationError. Test test_mutable_guards[stable|mutable|indirect].
  6. Several written indices per iteration (f[time+1] and f[time]) are refused with a CompilationError. Test test_write_multi_index.
  7. A copy-back is not skipped if the slot is read in between. The _overwritten docstring example, a subsampled usave save plus a last-step save with g = usave under the last-step guard between them, reads the slot from storage (commit 4), which the skipped copy-back left unwritten: g = 0 instead of 6. This slot has two firing writes at that step; the test asserts the read between them and the final value, which both follow program order. Test test_slot_read_between_writes.

Each commit's tests fail with only its production change reverted to its parent. The buffering, dimension, iet, operator and ir suites pass (679 passed, 1 xfail).

Depends on #3062: with its production change reverted on this head, test_mutable_read_guards[after] gives f = [0, 2, 20, 30] vs noop [1, 2, 20, 30].

Validation beyond the unit tests: a differential review of 1,521 generated programs (each under noop, buffering and buf-async-degree=2) found no violation and no case where this branch is worse than main. Two follow-ups are left out: 8 programs with two syntactic writers under a guard reading a Function written between them are refused although a baseline computes them, and 80 programs are wrong on main as well. On an H200 node the downstream GPU test suites (serial and 2 MPI ranks) give failure sets identical to main.

🤖 Generated with Claude Code

ggorman and others added 7 commits October 16, 2026 22:19
`Guard.negated` rebuilds the opposite relation with `evaluate=False`, but
`BaseGuardBoundNext._new` did not accept the keyword, so negating a
GuardBoundNext raised a TypeError. SymPy negates the relational arguments of
an `And` to detect contradictions, so an `And` over Cluster guards that
include a GuardBoundNext (e.g., the bound guards of a prefetch) could not be
built either
…ritten ones

A buffered Function read at an index written later in the same iteration
(e.g. `Eq(f, f + 1)`), or read ahead of all written indices (e.g.
`g[time] = f[time+1]; f[time] = ...`), read a stale buffer slot left by a
previous iteration instead of the value stored in the Function. Load such
slots from the Function, and copy back the leading written slot rather than
the leading accessed one
`itertools.groupby` only merges consecutive Clusters, so Clusters with the
same guards separated by a differently guarded one formed distinct groups,
and the buffer was sized from the last group only
…er guards

Since multi-condition buffering, each set of guards gets its own buffer, so
a read missed the writes performed earlier in the same iteration into
another buffer, e.g. a read under a guard reading `T` after a write under
`time < 2` and an update to `T`. Such Clusters now share one buffer, whose
slots are loaded before the first access, as a guarded write may not be
performed; likewise for slots behind the written ones, whose write in an
earlier iteration may not have been performed. Loads drop the guards
reading a Function written in the iteration, as these may change before
the accesses they serve, and an earlier access under such guards no
longer counts as loading the slot. Clusters with slots at an unknown
distance from each other, e.g. a CondEq-shifted one, or iterating over
different spatial Dimensions keep separate buffers
The copy-back of a buffer follows the last write under a given set of
guards. If these guards read a Function, also through an index, that is
modified between two such writes, they may no longer hold at the copy-back,
and the earlier write is silently lost. Raise a CompilationError instead
…iteration

Only one slot per iteration is copied back from a buffer, so a Function
written at e.g. both `f[time+1]` and `f[time]` in the same iteration
silently lost all writes but one
A read of the buffered Function between two copy-backs into the same slot
is served from its storage, which the skipped copy-back would have left
unwritten

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant