Who is the leader? Elections, fencing, and why RF=2 can't fail over in Squirix preview.9

A month ago I wrote a post-mortem about a failover test that stopped the wrong node: preview.8 shipped election code that nothing in the server called. I ended it with a promise. The article about elections would wait until elections actually run.

In 0.1.0-preview.9 they run. Stop the leader of an RF=3 group and the other two elect a new one, the client keeps working, and the leader-stop tests kill the node that matters. This is that article.

Part 1 covered the replicated write path: synchronous mutations, majority commit, write-ahead idempotency intent, and the honest CommitOutcomeUnknown answer when the owner dies mid-commit. All of that assumes something it never proves: that a node may serve current reads and acknowledge writes only under the group's quorum-backed authority.

Without that guarantee, two survivors of a partition can both believe they lead, accept different writes for the same key, and return success for histories that can never be reconciled. Preventing that requires majority intersection, durable voting and log-freshness rules, plus current-term authority checks on reads and writes.

Squirix is still an experimental preview, not a production-ready cache. Its APIs, wire format, and storage formats may change during 0.x. This article describes the current design, not a compatibility promise.

Opt-in, and only for three or more

Automatic failover is off by default. Two settings under Squirix:Cluster turn it on, and they come together:

{
  "Squirix": {
    "Cluster": {
      "ReplicaCount": 3,
      "PersistenceEnabled": true,
      "ReplicationEnabled": true,
      "AutomaticFailoverEnabled": true,
      "QuorumReadsEnabled": true
    }
  }
}

Validation refuses one without the other, and refuses both for groups of one or two replicas. Every node needs the same values, and they are set on empty data directories: the switches are part of the topology fingerprint. This is a settings fragment: node identity, peers, and inter-node mTLS are also required.

With failover off, nothing changes from preview.8: the owner of each key leads its group in term one, reads are local, and losing the owner makes its keys Unavailable until it comes back. Everything below is about the switched-on mode.

The failure mode to avoid

Imagine five nodes split 3–2 by a network cut. The minority side bumps its own term number and declares itself leader. It accepts a write for key K. Meanwhile the majority side elects a real leader and accepts a different write for K. The partition heals. Which write won? Both clients got success, and the log has a fork nothing can merge.

To prevent this, Squirix requires a candidate to win votes from a majority of the configured members. A member with an ineligible log cannot vote, and its absence does not reduce the number of votes needed. A leader that stops hearing from a majority steps down.

A lone node can inflate any number it likes. Without a majority behind it, the number means nothing.

When an election starts

A follower campaigns when it has heard neither from a leader nor granted a vote for one election timeout: one second plus up to one second of seeded jitter. The leader sends a heartbeat to every idle follower every 100 ms, so a reachable leader keeps its followers quiet. The first surviving member whose timer expires can start a campaign; winning still requires a majority, durable votes, and a current-term commit. Timer settings alone do not establish an end-to-end failover latency.

A campaign starts with a pre-vote for the next term, which changes no term anywhere. A member refuses it with leader-contact if it heard from a live leader within the last election timeout. That one rule matters more than it looks: a node cut off from the leader alone would otherwise keep raising the term and depose a perfectly healthy leader every time it reconnects. With the rule, it campaigns in vain until it can hear the leader again.

Only with a majority of pre-votes does the candidate persist the new term and its own vote, then ask for real votes. Each vote request has a 250 ms timeout, and a round ends as soon as its outcome is decided, canceling and awaiting unanswered calls. A silent member therefore need not consume its full vote timeout after a majority has already answered.

Term one always belongs to the owner of a group, which leads it without votes after the group's first start. The first election is for term two.

Terms that survive the process

Every replica group stores a durable current_term and at most one voted_for per term in a two-slot metadata file. Term advances and votes are mirrored, but the vote reply waits only for the first copy to become durable. The same work item then writes the second copy; the next metadata publish and shutdown wait for that pending mirror. Recovery selects the newest valid slot. A process restart after a granted vote therefore remembers that vote; the reply does not imply both slots were already flushed.

Voting adds a second gate: log freshness. The voter compares the candidate's last log term, then its last log index, with its own, and refuses a stale candidate with stale-log. A new leader that never saw a committed entry cannot win, so it cannot overwrite that entry — the classic "elected but amnesiac" failure.

For example, a voter whose last log entry is from term 5 rejects a candidate whose last entry is from term 4, even if the candidate has reached index 120. The voter returns stale-log.

Some members cannot vote at all. A copy that came back on an empty disk is unseeded: it grants no vote and takes no entries until the group repairs it. A copy whose log is not ready refuses every ballot. A copy waiting for a state transfer still votes, because its log is real, but it can never stand for election.

Authority must be earned, then kept

Winning an election is a moment. Serving traffic is a stream.

A winner does not serve anything yet. It first appends a leader-noop record in its own term and waits until a majority commits it. Until then every write is refused before anything is appended, with Unavailable and "nothing was read or written". Only that current-term commit lets the leader treat older entries as committed. This is Raft's leader-completeness rule stated as a startup step, and it closes the window where a new leader serves history it does not yet own.

Keeping authority is the other half. A leader steps down once fewer than a majority, itself included, answered within one election timeout, and at once when any reply carries a higher term. A deposed leader's messages carry a stale term, and followers refuse them with stale-term instead of merging them.

Some leaders are alive but unable to serve. The initial authority deadline is 30 seconds; repeated failures to obtain authority double that window, up to 240 seconds, and recent state-transfer progress holds the deadline. A 30-second apply timeout applies while committed work remains unapplied without progress. A local append failure or stall has a separate 30-second window and causes step-down only while the other electable followers can form a majority with recent contact. These are conditional progress checks, not a universal 30-second failover guarantee. On shutdown, the election driver revokes local authority; surviving members still acquire leadership through the election protocol.

Reads that refuse to guess

With quorum reads on, a read on the leader goes through five steps:

1. check authority in the current term
2. read_index = commit_index
3. confirm with a majority in that term (requests sent after step 2)
4. wait until applied_index >= read_index
5. recheck authority in that term, then return the value

A minority side or a former leader cannot pass step 1 or step 3. It answers Unavailable (read_quorum_unconfirmed, read_index_unapplied) or a stale-term refusal — never an old value presented as current. There are no time-based lease reads: authority without a quorum is never enough, even when clocks agree.

Finding the leader

A client should not have to know who leads. Any node accepts a single-key call, including a node outside the key's group, and forwards it to the leader it knows. If the answer is stale-owner or stale-term, it reroutes once to the leader the answer names. All the leader waits of one call share a three-second attempt budget, and the client's retries share one 15-second deadline in the default SDK policy. A no-leader refusal before admission means nothing was written by that attempt. An interrupted attempt that already appended can instead have an unknown commit outcome: keep the same operation_id when retrying within the supported retention window.

client --> any node --> leader it knows (term 7, current is 9)
                        |
                        +--> stale-term: one reroute to the named leader, then fail visibly

Bounded matters. "Try the next node forever" turns every leadership change into a thundering herd from the clients.

Every replication RPC also carries the term, the topology fingerprint and the configuration generation in its header. A message from another world — wrong group, older configuration, a node below the minimum cluster version — is refused before it touches the log.

Why RF=2 stays a mirror

The same majority rule that saves RF>=3 condemns RF=2 to honesty.

With two members the majority is two. Lose either node and the survivor holds one vote out of two. It cannot elect itself, cannot commit, cannot promote. That is not a missing feature; it is the protocol refusing to invent authority it does not have. Validation does not even let you switch failover on for RF=2.

RF=2, both alive:    synchronous mirror, every write on both nodes
RF=2, follower lost: writes stop; the surviving owner can still serve local reads
RF=2, owner lost:    its keys are unavailable; the surviving follower never promotes
RF>=3, one lost:     with failover on, a surviving majority can elect and keep serving

Two-node majority systems that promote on peer loss are single-node systems with extra steps — and extra data loss.

Coming back

A node that returns after an outage catches up from the leader's log. If the leader's log no longer holds what it needs — it was compacted, or the node came back on an empty disk — the leader sends a state transfer: group snapshot metadata and idempotency outcomes plus the group's cache entries. The group snapshot alone is not a full cache image. A wiped copy also forgot whom it voted for, so the leader checks a term ceiling with a majority that includes itself and excludes the wiped member before repairing it. The required peer answers come from members that retained their durable term, preventing the forgotten vote from silently creating authority.

A returning node does not take the group back. It stays a follower: its peers refuse its pre-vote while they hear the current leader, and nobody votes for a stale log. Membership is not readiness to lead.

A model that is allowed to say no

Before any election code landed, I froze the protocol in an ADR with an executable C# model (src/squirix.protocol-model, never shipped, never referenced by product code). It has immutable state, pure transitions, a deterministic BFS explorer, and invariants that report minimal counterexample traces.

The explorer covers elections, commit, and quorum reads for RF=2 through RF=5, with crash points before and after every durable write of term, vote, log, and commit index, and with loss, duplication, reorder, and one-way isolation of a node. Newer profiles add disk loss and repair. A profile that hits its state cap without a violation exits differently from a clean fixed point, so residual risk stays visible instead of silently passing.

Just as important are the broken variants that must produce counterexamples:

• grant a second vote in the same term;
• compare log length before the last term when voting;
• commit an old-term entry without a current-term commit;
• serve a read without a majority confirm, or before apply catches up;
• repair a wiped copy without the term ceiling.

If the explorer ever stops finding those violations, the harness is broken, not the protocol sound. Conformance tests run real product components and ask the model whether it can reach the same trace.

The post-mortem was about the gap between this model and the server. The model checks invariants within its explored bounds; it never said the server runs it. What closes the gap now are the E2E tests: an RF=3 and an RF=5 cluster whose leader is stopped, a leader isolated by a partition, and a five-node 2/3 split with the leader in the minority. Other scenarios repair a wiped follower by state transfer and restart a former leader that stays a follower. The tests that stop a node assert that it was the leader.

What preview.9 does not promise

Membership is static: no dynamic membership, rebalancing, or joint consensus. Election timings are fixed per member; the group's detection still shortens with more replicas. The model covers single-node isolation; the five-node split is covered by a test, not by the model, and richer partitions remain residual risk. A group that lost the disks of a majority of its members at once is not repaired.

Closing thought

Commit protocols ask: do enough nodes hold this write? Election protocols ask the harder question first: who is allowed to ask?

The hard part is refusing authority precisely:

• no majority, no leadership — even if the node feels healthy;
• stale log, no votes — even with supporters;
• no current-term commit, no serving old history;
• no quorum confirm, no current read;
• a live leader, no pre-vote for anyone else.

Preview.9 makes each refusal durable, fenced, and observable — and, this time, switched on. Part 3 covers what operators see: the opt-in, metrics, readiness, and an RF=3 compose example you can run.

If you build leader election or quorum reads in .NET, comments are welcome on where you draw the same lines — especially vote persistence, pre-vote, and rerouting budgets.

Links

Squirix docs:

• Release v0.1.0-preview.9
• Replication consensus ADR
• Clustering
• Configuration (automatic failover)
• Consistency

Related article: The failover test that stopped the wrong node.

Try it:

dotnet add package squirix --version 0.1.0-preview.9
dotnet add package squirix.server --version 0.1.0-preview.9
dotnet tool install --global squirix.server.tool --version 0.1.0-preview.9