Skip to content

fix: save failover votes and send them directly - #413

Merged
kacy merged 2 commits into
mainfrom
fix/election-votes
Sep 24, 2026
Merged

kacy merged 2 commits into
mainfrom
fix/election-votes

Conversation

@kacy

@kacy kacy commented Sep 24, 2026

Copy link
Copy Markdown
Owner

Three problems with failover votes.

  1. A primary kept the epoch of its last vote only in memory. If it restarted during an election, it could vote a second time in the same epoch.
  2. Vote requests and grants rode on regular gossip and were relayed from node to node. A VoteGranted update named its voter in a from field that nothing checked, so any node that can send gossip could give a candidate a quorum by sending grants in the name of each primary. The cluster secret only proves the sender is some member of the cluster.
  3. A VoteRequest with a huge epoch made every primary that granted it record that epoch as its last vote. None of them could vote in a real election afterwards.

The first commit moves the last vote epoch into ClusterState:

  • It is saved in nodes.conf as lastVoteEpoch on the vars line.
  • A primary saves it before granting the vote.
  • Files without the field still load.
  • The in-memory counter is gone.

The second commit sends votes directly:

  • A new GossipMessage::Election carries one VoteRequest or VoteGranted. It is sent straight to the node it is for and never relayed.
  • The receiver accepts it only when the named candidate or voter is the message's sender, and the UDP packet came from the gossip address it knows for that node. Vote updates that arrive piggybacked are ignored.
  • The candidate sends its request to every voter in the round's voter set. The voter sends its grant straight back.
  • A primary refuses a request more than MAX_VOTE_EPOCH_AHEAD (100) past its current epoch. Since fix: resolve conflicting slot claims by config epoch #412, nodes see each other's epochs in slot claims, so a real candidate is at most ELECTION_ROUNDS ahead.

This adds a gossip message type, so, as with #412, the nodes of a cluster need to be upgraded together. The automatic failover integration test passes with direct votes. New unit tests cover the address and sender checks, piggybacked votes, the epoch bound, and saving the vote epoch.

The epoch of the last vote a primary granted lived only in memory, so a
primary that restarted during an election could vote twice in one
epoch. It is now part of the cluster state, written to nodes.conf as
lastVoteEpoch, and saved before the vote is sent. Files without it
still load.
Vote requests and grants were piggybacked and relayed like other gossip,
and a grant named its voter in a field nothing checked, so any node
could hand a candidate a quorum. They now travel in Election messages
sent straight to the node they are for, and only count when the sender
is the voter or candidate and the message came from its known gossip
address. Primaries also refuse vote requests far past the current
epoch, which would use up the epochs later elections need.
@kacy
kacy merged commit d252618 into main Sep 24, 2026
12 checks passed
@kacy
kacy deleted the fix/election-votes branch September 24, 2026 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant