Repository navigation
fix(two-node): remove promote-at-grant (c7b F17) - #1485
Merged
Merged
Conversation
In c7b/04 (graceful shutdown of the primary sn2) the survivor sn1 aborted and the client got 160 EIO. Core analysis: promote-at-grant's bdev_lvol_update_lvstore made sn1 (follower of LVS_6) take over right after the grant, while sn2's shutdown had just detached the remote JM; the JC reader restart hit the detached JM -> JC error -> spdk_abort_node (alg_journal.cpp:8801; ha_jc_error_expected covers holding/fenced, not solo). The JC-side robustness goes to ultra separately. Promote-at-grant (#1476, F13) is removed, which removes the trigger: abort-on-wake covers a frozen primary, the leadership restoration after an unfence covers partitions, and the shutdown/restart flow hands leadership over on a planned loss. Removed: Arbiter.promote_after_grant, default_promote_survivor, the promote_survivor hook, the record field `promoted` (old records still load). TWO_NODE_PROMOTE_VERIFY_S stays (used by the leadership restoration). Design doc 7.12 rewritten. F13 tests replaced by F17 tests (no leadership RPC to the survivor after the grant). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
schmidt-scaled
pushed a commit
that referenced
this pull request
Oct 15, 2026
… bed stall metric, F19 xfail Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
schmidt-scaled
added a commit
that referenced
this pull request
Oct 15, 2026
* test(two-node): simulator of the two-node HA campaign in CI The AWS two-node campaign (c1-c7b) found 18 bugs that no unit or integration test caught: interaction and timing bugs between the arbiter, the node's HA state machine, SPDK's fence/ANA/port handling and the control-plane flows. tests/integration/two_node_sim runs the bed's scenarios and one regression scenario per finding against the real code. - Real: the arbiter (decision, events, planned loss), its default helpers, every RPCClient wrapper (only _request is routed), the port guards the flows consult, the ANA helpers, task records, real FDB. - Modelled: the two nodes (JC HA state machine ported from ultra alg_two_node_ha.cpp, SPDK fence/unfence/port ownership/ANA/takeover latch), the network, the multipath client, the Go collector, and the shells of the restart/port-allow/migration/monitor/shutdown flows around the real guards. - Virtual time: no real sleeps or timeouts. RPCs on the decision loop delay the next verdict (F16); pool and recovery calls run on cooperative threads parked on the virtual clock, so their side effects land when the call returns. Deterministic. - Invariants: one writer per LVS, no stale node serving while the up-to-date one is down, grant within lease TTL + margin, client stall limits, no IO errors outside partitions, no unexpected abort, healthy again without a manual step. - Proof: per finding, the scenario passes on main and fails, with the finding's signature, when its bug is put back (node switch, or the real function patched to its pre-fix behaviour). F17/F18 (open) are strict xfails checked against their failure signature; F9/F11 not modelled. CI: "Two-node simulation" runs the fast tier (tox -e two-node-sim), and "Two-node simulation (full)" everything (tox -e two-node-sim-full). The default integration run excludes the simulator. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(two-node): rebase on #1485, F17 as no-leadership-move invariant, bed stall metric, F19 xfail Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: michael <michael.schmidt1379@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Campaign c7b, F17: in the graceful-shutdown scenario (04), the survivor sn1 aborted right after the grant and the client got 160 EIO.
Root cause (core analysis)
bdev_lvol_update_lvstoremade sn1, the follower of LVS_6, take over right after the grant.spdk_abort_node(alg_journal.cpp:8801;ha_jc_error_expectedcovers holding/fenced, not solo).The JC-side robustness is an ultra change, handled separately.
Fix
Remove promote-at-grant (#1476, F13). This removes the trigger, and it isn't needed any more:
Removed:
Arbiter.promote_after_grant,default_promote_survivor, thepromote_survivorhook, and the record fieldpromoted(old records still load; unknown keys are ignored).TWO_NODE_PROMOTE_VERIFY_Sstays, because the leadership restoration uses it. Design doc 7.12 rewritten.Arbiter-only and gated on
two_node_arbitration; regular clusters are unaffected.Tests
Not merged
Per the new process, merge comes after verification: the two-node simulator scenarios (F17/F18, flipped from xfail once the simulator PR lands) and the bed gate.
F18 (ANA groups) is on the SPDK side: simplyblock/spdk PR (fix/ha-ana-restore-all-groups → new_md_journal).
🤖 Generated with Claude Code