Skip to content

fix: resolve conflicting slot claims by config epoch - #412

Merged
kacy merged 1 commit into
mainfrom
fix/slot-claim-epochs
Sep 24, 2026
Merged

kacy merged 1 commit into
mainfrom
fix/slot-claim-epochs

Conversation

@kacy

@kacy kacy commented Sep 24, 2026

Copy link
Copy Markdown
Owner

Every gossip ping carries the sender's slot claim, and receivers applied whichever claim they saw last. The claim's incarnation only orders updates from one node. It can't settle two nodes claiming the same slot. After a failover, the old primary coming back kept announcing its old slots, so peers moved them back and forth between it and the promoted replica.

Claims now carry the claiming node's config epoch, as in Redis Cluster.

Wire format:

  • SlotsChanged, SlotsAnnounce and MemberInfo gain a config_epoch.
  • The gossip engine tracks each member's epoch and sends the local node's epoch with its slots.

Conflict rule, in ClusterState::apply_slot_claim:

  • A claimed slot goes to the claimant when it is unowned, or when the claim beats the owner's: a higher epoch, or an equal epoch and a higher node ID, so every node picks the same winner.
  • Slots the claimant owned and no longer claims become unowned.
  • The cluster's current epoch rises to the claim's epoch. A replica has therefore seen its primary's epoch, and the epoch it takes when promoted is newer.

Server:

  • Gossip SlotsChanged and MemberJoined go through apply_slot_claim instead of assigning slots directly.
  • When a newer claim takes slots from the local node, it logs a warning and re-announces its reduced slot list, so it stops claiming them.
  • CLUSTER SETSLOT <slot> NODE <self> bumps the local epoch (bump_local_epoch), as Redis does. Without this, the previous owner's claim would win until it heard about the move.

This changes the gossip wire format, like #388 did, so the nodes of a cluster need to be upgraded together.

Tests cover a newer claim taking slots and raising the epoch, a stale claim losing, slots dropped from a claim, the tie-break, and the epoch bump.

Gossip slot claims carried no epoch, so each node took whichever claim
arrived last. A primary that came back after a failover kept
announcing its old slots on every ping, and peers flipped them back to
it. Claims now carry the sender's config epoch: a claim takes a slot
from its owner only with a newer epoch, or an equal epoch and a higher
node ID. Nodes raise their current epoch to the newest they see, a node
that takes a slot with SETSLOT NODE claims it at a new epoch, and a
node that loses slots to a newer claim stops announcing them.
@kacy
kacy force-pushed the fix/slot-claim-epochs branch from 6cab36d to 7c329fb Compare September 24, 2026 20:45
@kacy
kacy merged commit b38d009 into main Sep 24, 2026
12 checks passed
@kacy
kacy deleted the fix/slot-claim-epochs branch September 24, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant