test(two-node): simulator of the two-node HA campaign in CI

[schmidt-scaled]

The AWS two-node campaign (c1–c7b, 2026-10) found 18 bugs (F1–F18) that no unit or integration test caught. They sat in the interplay of the arbiter, the node's HA state machine, SPDK's fence/ANA/port handling and the control-plane flows, and in their timing. This PR adds tests/integration/two_node_sim: the bed's scenarios and one regression scenario per finding, run against the real code on simulated nodes and a real FoundationDB, in CI. Test code only; no product change.

What runs for real, what is modelled

Real (production code)	Modelled
arbitration/arbiter.py, decision.py, events.py, planned.py	the two storage nodes: the JC HA state machine ported rule by rule from ultra alg_two_node_ha.cpp (hold, lease, epochs/ESTALE, EBUSY/EAGAIN, abort-on-wake, the remote-JM watchdog), and SPDK's lvol side (fence/unfence with port ownership, takeover latch, ANA groups, devices)
the arbiter's default helpers (leadership restore, device check, rebuild bookkeeping, leaderless reconcile and its probe)	network: storage link, management link, client link, lossy link
every RPCClient wrapper; only RPCClient._request is routed	the NVMe multipath client (KATO 4 s, fast_io_fail 8 s, ANA-driven path choice)
the guards the flows call: _single_peer_jm_is_complete, _two_node_fence_holds_ports, _may_unblock_peer_port, _jc_compression_status_with_retry, _failover/_failback_primary_ana, _set_lvol_ana_on_node	the Go event collector (same ordering/resync contract)
tasks_controller task records, set_node_status, all FDB reads/writes (real FDB, as the integration tier requires)	the shells of the restart, port-allow, migration, monitor and shutdown flows around those real guards

Virtual time: no real sleeps or timeouts. RPCs on the decision loop delay its next verdict, which is exactly the F16 failure. Pool and recovery calls run on cooperative threads parked on the virtual clock, so their side effects land when the call returns. Runs are deterministic.

Invariants

• one writer per LVS;
• no node serving an LVS it misses writes for while the up-to-date node is down (F10);
• survivor granted within lease TTL + margin;
• after a grant, no bdev_lvol_update_lvstore/set_leader reaches the survivor while its peer is gone (F17, fix(two-node): remove promote-at-grant (c7b F17) #1485);
• an IO-triggered takeover finishes within 1 s (F7);
• client stall ≤ 3 s (node failure) / ≤ 5 s (client path), measured as the bed does (tn2 it/analyze.py max_stall_after_inject_s): the longest run of whole seconds of fio's 1 s IOPS log without a completion, from the injection. Faults are injected 0.5 s into a second (the bed's alignment was random);
• no IO errors outside partitions;
• no unexpected abort;
• healthy again without a manual step: arbiter steady, LVS normal, remote JMs healthy, no port blocked, devices present, primary leads, ANA restored, no open task, no pending rebuild.

Scenarios and findings

Test	Covers
bed 01a 01b 02a 02b 03 05 05b 07 08 10b	the AWS bed list, on main
bed 04-shutdown-sn2	passes on main since #1485
F19-long-freeze-primary	xfail (signature "client stall"): new finding F19, see below
F18-ana-groups-restored	xfail (signature "ANA"): F18, the fence restore puts back only the namespaces' groups (simnode.RESTORE_ALL_ANA_GROUPS)
F1 F2 F3 F4 F5 F6 F7 F8 F10 F12 F13 F14 F15 F16 F17	per finding: test_finding_is_fixed passes on main; test_finding_is_detected fails with the finding's signature when its bug is put back (a node switch, or the real function patched back to its pre-fix behaviour, e.g. F16 = the probe inline on the decision thread with 5 s + retry; F17 = #1476's promote-at-grant sent after a degraded grant). F1/F2/F3/F4/F10/F16 also check that the fixed run reached its critical step.
F9, F11	skipped, not modelled: F9 is below the RPC level; F11 never had a root cause and did not recur

Proof that the detection works: every test_finding_is_detected[*] passes, i.e. each scenario fails for its own reason with the bug back in. For layered fixes the switch removes every layer: F5 = node watchdog + lease-expiry grant; F12 = abort-on-wake + device check + stuck-rebuild recovery; F13 = abort-on-wake (fence stall).

New finding F19 (not fixed here)

Since #1485 removed promote-at-grant, nothing moves a frozen primary's LVS to the granted survivor. Its volume's IO resumes only when the survivor's hublvol redirect to the frozen peer fails at the SPDK-to-SPDK keep-alive (constants.KATO, 5 s), or when the frozen node aborts on wake. Repro: pytest tests/integration/two_node_sim -k F19 (sn2 hung 6 s, grant at 1.93 s): stall 4 s (gap 5.15 s), no IO errors; a 10 s freeze gives the same. On the bed, c7b (with promote-at-grant) stalled 2 s and c5 02b 3 s. 02b itself still passes because its freeze is shorter than the keep-alive. Fix direction: on a solo grant stop redirecting to the lost peer, without a leadership RPC on the takeover path that F17 forbids.

CI and runtime

• "Two-node simulation" (tox -e two-node-sim): the fast tier, meant to be required on every PR. It runs smoke, 01b, 02b, 07 and the F3/F16 proofs.
• "Two-node simulation (full)" (tox -e two-node-sim-full): everything.
• The default integration run excludes the simulator (-m "not slow and not two_node_sim").

No real waiting remains. The wall time is FDB round trips of the real code: the arbiter alone does about 13 reads per 100 ms tick, about 6 of them of the same ClusterArbitration record, which is worth a look on its own. Measured locally on WSL2 with about 2 ms per FDB round trip:

Tier	Result	Time
fast	6 passed	28 s
full	42 passed, 2 xfailed, 2 skipped	8 min 24 s

Making the fast job required is a branch-protection setting, which needs a repo admin.

See tests/integration/two_node_sim/AGENTS.md for the model, its limits and how to add a finding.

🤖 Generated with Claude Code