Repository navigation
fix(two-node): the arbiter's decision thread never blocks on a node (c7, F16) - #1482
Merged
Merged
Conversation
…c7, F16) Campaign c7, 02b: the leadership reconciliation (B6) probed a node with get_lvstore (5 s timeout, 1 retry) on the decision thread just as that node froze. The call waited out its timeout, the grant to the healthy peer on the frozen node's expired lease never came, the peer fenced itself at its hold deadline and the client got 160 EIO. Rule: the decision thread makes no blocking RPC to a node. A second bounded pool (TWO_NODE_WORK_WORKERS) now runs jc_set_dual_node (configure_due), the device check + jc_unfence of a heal or self-fence re-admission, and the leadership reconciliation pass; one call per node (per cluster for the reconciliation) in flight, applied on the decision thread when finished (_drain_work), never waited for. A separate pool keeps calls blocked on a frozen node from starving the healthy peer's renewals. The reconciliation pass is skipped while a node is unreachable, lost, stopped by the control plane or holding, and the arbiter's probe uses 1 s, no retry (reconcile_lvs_leadership/_lvs_leader_probe gain parameters whose defaults keep the old behaviour). Best-effort fences (planned loss, operator override) use a 1 s timeout and never hold the grant up. Arbiter-only, gated on two_node_arbitration; regular clusters unchanged. Tests: test_two_node_f16_decision_loop.py (13): with real pools and a node whose every RPC blocks 2 s, no tick exceeds 0.5 s and the grant to the peer lands on the first tick after lease TTL + margin for a B6 probe in flight, configure, heal unfence, device check and rebuild progress; the four blocking cases fail without the fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Campaign c7, 02b (F16): the arbiter's leadership reconciliation (B6) probed a node with
get_lvstore(5 s timeout, 1 retry) on the decision thread just as that node froze. The grant to the healthy peer on the frozen node's expired lease never came, the peer fenced itself at its hold deadline, and the client got 160 EIO. Evidence:s3://tn2-mschmidt-xfer-565979732541/runs/c7/RESULTS.md.Rule
The decision thread makes no blocking RPC to a node.
Changes (arbiter only, gated on
two_node_arbitration)TWO_NODE_WORK_WORKERS=4), separate from the renewal/status pool so calls blocked on a frozen node cannot starve the healthy peer's renewals._drain_work), never waited for:jc_set_dual_node(configure_due)jc_unfenceof a heal and of a self-fence re-admissionreconcile_lvs_leadership/_lvs_leader_probeget parameters whose defaults keep today's behaviour for any other caller.Audit: node RPCs reachable from the decision thread
get_lvstore(B6 probe, viareconcile_lvs_leadership)jc_set_dual_node(configure)bdev_get+jc_unfence(heal)jc_unfence(self-fence re-admit)jc_fencebest effort (planned loss) insendjc_fence(operator override)jc_fencefence_all (partition)jc_grant_soloto the survivorjc_lease_renew,jc_ha_status, capability probe, returning-node fenceTests
tests/unit/test_two_node_f16_decision_loop.py(13): real thread pools, a node whose every RPC blocks 2 s; no tick exceeds 0.5 s, and the grant to the peer lands on the first tick after lease TTL + margin for: a B6 probe in flight, configure, heal unfence, device check, rebuild progress. The four blocking cases fail without the fix (tick blocked 2.00 s). Two-node suites: 235 passed. Full unit run: only the 36 known Windows-only failures. ruff, mypy clean.🤖 Generated with Claude Code