Repository navigation
fix: save failover votes and send them directly - #413
Merged
Merged
Conversation
The epoch of the last vote a primary granted lived only in memory, so a primary that restarted during an election could vote twice in one epoch. It is now part of the cluster state, written to nodes.conf as lastVoteEpoch, and saved before the vote is sent. Files without it still load.
Vote requests and grants were piggybacked and relayed like other gossip, and a grant named its voter in a field nothing checked, so any node could hand a candidate a quorum. They now travel in Election messages sent straight to the node they are for, and only count when the sender is the voter or candidate and the message came from its known gossip address. Primaries also refuse vote requests far past the current epoch, which would use up the epochs later elections need.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three problems with failover votes.
VoteGrantedupdate named its voter in afromfield that nothing checked, so any node that can send gossip could give a candidate a quorum by sending grants in the name of each primary. The cluster secret only proves the sender is some member of the cluster.VoteRequestwith a huge epoch made every primary that granted it record that epoch as its last vote. None of them could vote in a real election afterwards.The first commit moves the last vote epoch into
ClusterState:nodes.confaslastVoteEpochon thevarsline.The second commit sends votes directly:
GossipMessage::Electioncarries oneVoteRequestorVoteGranted. It is sent straight to the node it is for and never relayed.MAX_VOTE_EPOCH_AHEAD(100) past its current epoch. Since fix: resolve conflicting slot claims by config epoch #412, nodes see each other's epochs in slot claims, so a real candidate is at mostELECTION_ROUNDSahead.This adds a gossip message type, so, as with #412, the nodes of a cluster need to be upgraded together. The automatic failover integration test passes with direct votes. New unit tests cover the address and sender checks, piggybacked votes, the epoch bound, and saving the vote epoch.