The Raft Consensus Protocol: Leader Election, Log Replication, and Safety Guarantees in Distributed Systems

Distributed consensus is the problem of getting a cluster of nodes to agree on a sequence of values, even when some nodes crash or messages are delayed. If you have ever built a system that needs a single authoritative state across multiple machines, you have run into this problem: who gets to write, what happens when the current writer crashes, and how do you guarantee you never return stale or conflicting data to clients?

Paxos has been the canonical answer since Lamport published it in 1989. But Paxos is notoriously hard to implement correctly. The original paper describes single-decree consensus (agreeing on one value), not the multi-decree log replication that distributed databases actually need. Multi-Paxos, which is what systems actually implement, requires filling in significant design gaps yourself: leader election, log compaction, membership changes. Different implementations make different choices, and those choices are rarely documented.

Raft was designed explicitly to be understandable. Diego Ongaro and John Ousterhout published it in 2014 with the subtitle “a consensus algorithm for understandable systems.” That framing is not marketing. The paper describes a complete algorithm: leader election, log replication, safety properties, cluster membership changes, and log compaction. There is one correct way to do each piece, which is why etcd, CockroachDB, TiKV, and Consul all implement recognizably the same thing.

The Core Decomposition

Raft separates the consensus problem into three mostly independent subproblems:

1. Leader election: at any given time, exactly one server is the leader. The leader accepts all client writes.
2. Log replication: the leader receives log entries from clients, replicates them to followers, and tells followers when entries are safe to apply.
3. Safety: if any server has applied a log entry at a given index, no other server will ever apply a different entry at that index.

Everything else (membership changes, log compaction via snapshots) builds on this foundation.

Terms: The Logical Clock

Raft uses terms to detect stale leaders. A term is a monotonically increasing integer. Each term begins with an election. If a candidate wins, it serves as leader for the rest of that term. Terms act as a logical clock: any RPC that carries a lower term number is rejected, and the receiver immediately reverts to follower state if it sees a higher term.

type NodeRole = "follower" | "candidate" | "leader";

interface RaftState {
  // Persistent state (must survive crashes)
  currentTerm: number;
  votedFor: string | null;  // candidateId or null
  log: LogEntry[];

  // Volatile state
  commitIndex: number;
  lastApplied: number;

  // Leader-only volatile state (re-initialized after election)
  nextIndex: Map<string, number>;   // per-follower: next log index to send
  matchIndex: Map<string, number>;  // per-follower: highest known replicated index
}

interface LogEntry {
  term: number;
  index: number;
  command: unknown;
}

The votedFor field is critical: it must be written to durable storage before responding to any RequestVote RPC. If a node crashes after granting a vote but before persisting votedFor, it must not grant a second vote in the same term after restarting. Without persistence, two candidates could each collect a majority of votes in the same term, violating the safety guarantee.

Leader Election

Servers start as followers. A follower expects to receive heartbeats (empty AppendEntries RPCs) from the current leader within a bounded time called the election timeout. If the timeout fires without a heartbeat, the follower concludes the leader is dead and starts an election.

To start an election, a follower:

1. Increments its currentTerm
2. Transitions to candidate state
3. Votes for itself
4. Sends RequestVote RPCs to all other nodes in parallel

interface RequestVoteArgs {
  term: number;
  candidateId: string;
  lastLogIndex: number;
  lastLogTerm: number;
}

interface RequestVoteReply {
  term: number;
  voteGranted: boolean;
}

function handleRequestVote(
  state: RaftState,
  args: RequestVoteArgs
): RequestVoteReply {
  if (args.term < state.currentTerm) {
    return { term: state.currentTerm, voteGranted: false };
  }

  if (args.term > state.currentTerm) {
    state.currentTerm = args.term;
    state.votedFor = null;
    // transition to follower
  }

  const alreadyVoted =
    state.votedFor !== null && state.votedFor !== args.candidateId;

  if (alreadyVoted) {
    return { term: state.currentTerm, voteGranted: false };
  }

  // Safety: only grant vote if candidate's log is at least as up-to-date
  const myLastLog = state.log[state.log.length - 1];
  const candidateIsUpToDate =
    args.lastLogTerm > (myLastLog?.term ?? 0) ||
    (args.lastLogTerm === (myLastLog?.term ?? 0) &&
      args.lastLogIndex >= (myLastLog?.index ?? 0));

  if (!candidateIsUpToDate) {
    return { term: state.currentTerm, voteGranted: false };
  }

  state.votedFor = args.candidateId;
  persistState(state); // must flush to disk before replying
  return { term: state.currentTerm, voteGranted: true };
}

A candidate wins if it receives votes from a majority of the cluster (including itself). In a five-node cluster, three votes win. This majority requirement is what prevents split-brain: two nodes cannot both win elections in the same term because they would need overlapping votes, and a node can only vote once per term.

Election Timeouts and Split Votes

If two followers time out simultaneously, they both become candidates and may split the votes so neither reaches a majority. Raft handles this by randomizing election timeouts. Each node picks a timeout from a range (the paper suggests 150-300ms). The node that times out first usually wins the election before the others even start.

If a split vote does occur, each candidate waits for its next randomized timeout before starting a new election. In practice, split votes are rare and self-resolve in one or two additional election rounds.

Log Replication

Once a leader is elected, it handles all client requests. Each request becomes a log entry. The leader appends the entry to its own log and then sends AppendEntries RPCs to followers in parallel.

interface AppendEntriesArgs {
  term: number;
  leaderId: string;
  prevLogIndex: number;
  prevLogTerm: number;
  entries: LogEntry[];
  leaderCommit: number;
}

interface AppendEntriesReply {
  term: number;
  success: boolean;
  // Optimization: follower's conflicting term and first index for that term
  conflictTerm?: number;
  conflictIndex?: number;
}

function handleAppendEntries(
  state: RaftState,
  args: AppendEntriesArgs
): AppendEntriesReply {
  if (args.term < state.currentTerm) {
    return { term: state.currentTerm, success: false };
  }

  // Reset election timeout — we have a valid leader
  resetElectionTimer();

  if (args.term > state.currentTerm) {
    state.currentTerm = args.term;
    state.votedFor = null;
  }

  // Consistency check: verify the entry before prevLogIndex matches
  const prevEntry = state.log[args.prevLogIndex];
  if (args.prevLogIndex > 0 && prevEntry?.term !== args.prevLogTerm) {
    // Log is inconsistent — tell leader which term conflicts
    const conflictTerm = prevEntry?.term;
    const conflictIndex = state.log.findIndex(
      (e) => e.term === conflictTerm
    );
    return {
      term: state.currentTerm,
      success: false,
      conflictTerm,
      conflictIndex: conflictIndex === -1 ? args.prevLogIndex : conflictIndex,
    };
  }

  // Append new entries, overwriting any conflicting entries
  for (let i = 0; i < args.entries.length; i++) {
    const logIndex = args.prevLogIndex + 1 + i;
    if (state.log[logIndex]?.term !== args.entries[i].term) {
      state.log.splice(logIndex, state.log.length - logIndex, ...args.entries.slice(i));
      break;
    }
  }

  persistLog(state.log);

  if (args.leaderCommit > state.commitIndex) {
    state.commitIndex = Math.min(
      args.leaderCommit,
      state.log[state.log.length - 1]?.index ?? 0
    );
    applyCommittedEntries(state);
  }

  return { term: state.currentTerm, success: true };
}

The prevLogIndex and prevLogTerm fields implement a consistency check. Before appending new entries, the follower verifies that its log contains an entry at prevLogIndex with the matching term. This is the inductive invariant that keeps logs consistent: if the consistency check passes, the follower’s log up to prevLogIndex is identical to the leader’s log up to that point.

An entry is committed once the leader has replicated it to a majority of the cluster. The leader tracks matchIndex for each follower (the highest log index known to be replicated on that follower). When a majority have reported a given index, the leader advances its commitIndex and notifies followers via the leaderCommit field in subsequent AppendEntries RPCs.

Safety Guarantees

The Raft safety property is the Log Matching Property: if two logs contain an entry with the same index and term, then the logs are identical in all entries up to that index. This follows from two invariants:

1. A leader only ever creates one entry with a given index in a given term.
2. The consistency check in AppendEntries ensures that a follower only appends entries if its log matches the leader’s up to that point.

The stronger guarantee is the Leader Completeness Property: if a log entry is committed in a given term, it will be present in the logs of all future leaders. This is enforced by the vote-granting rule: a node only votes for a candidate whose log is at least as up-to-date as its own. “Up-to-date” is determined first by the last entry’s term (higher wins), then by log length (longer wins).

The practical consequence: a newly elected leader always has every committed entry. It may have uncommitted entries from a previous leader, but it will never be missing a committed one.

The Tricky Case: Old Entries from Previous Terms

A subtle safety issue: a leader cannot commit an entry from a previous term by counting replicas. Consider this scenario:

1. Leader A writes an entry at index 5 with term 2
2. A replicates to two followers but crashes before committing
3. Leader B is elected in term 3, has the entry at index 5 term 2
4. B replicates some new entries in term 3
5. B cannot declare index 5 committed just because it is now on a majority — if B crashes before committing its own term-3 entries, a new leader could be elected without the term-2 entry and overwrite it

Raft’s solution: a leader only directly commits entries from its own term. Old entries from previous terms are committed indirectly, as a side effect of committing a later entry from the current term. The leaderCommit field carries the leader’s current commit index, which propagates the commit of older entries without ever counting replicas for entries with stale term numbers.

Handling Network Partitions and Split-Brain

Network partitions are the hardest failure mode. Suppose a five-node cluster splits into a group of two and a group of three. The group of three can elect a leader and continue making progress (it has a majority). The group of two cannot elect a leader: no candidate can get three votes from only two nodes. Any leader in the minority partition will eventually detect it has no quorum: its AppendEntries RPCs will time out, it will stop getting acknowledgments, and it will be unable to commit new entries.

When the partition heals, the old minority leader (now with a lower term) will receive an RPC from the majority’s current leader (with a higher term). It immediately reverts to follower and its uncommitted entries are overwritten.

The key guarantee: entries that were committed (replicated to a majority) before the partition cannot be lost. The majority partition’s new leader must have those entries, because it collected votes from a majority that overlaps with the commit quorum.

Log Compaction via Snapshots

Raft logs grow indefinitely. In production, you need snapshots. The state machine periodically takes a snapshot of its current state and records the last log index and term included in the snapshot. All log entries up to that point can then be discarded.

interface Snapshot {
  lastIncludedIndex: number;
  lastIncludedTerm: number;
  data: Uint8Array; // serialized state machine state
}

async function installSnapshot(
  state: RaftState,
  snapshot: Snapshot
): Promise<void> {
  if (snapshot.lastIncludedIndex <= state.commitIndex) {
    return; // already have this or newer state
  }

  // Discard log entries covered by snapshot
  const retainFrom = state.log.findIndex(
    (e) =>
      e.index === snapshot.lastIncludedIndex &&
      e.term === snapshot.lastIncludedTerm
  );

  if (retainFrom !== -1) {
    // Keep entries after the snapshot point
    state.log = state.log.slice(retainFrom + 1);
  } else {
    // Snapshot covers entries we don't have — discard entire log
    state.log = [];
  }

  state.commitIndex = snapshot.lastIncludedIndex;
  state.lastApplied = snapshot.lastIncludedIndex;

  await persistSnapshot(snapshot);
  await applySnapshot(snapshot.data); // restore state machine
}

Leaders send InstallSnapshot RPCs to lagging followers that have fallen too far behind for normal AppendEntries catch-up. This happens when the follower’s next required log entry has already been compacted. The follower replaces its state machine state with the snapshot contents.

Cluster Membership Changes

Adding or removing nodes from a live cluster is where many Raft implementations cut corners. A naive approach (just update the config on each node) risks two disjoint majorities existing simultaneously if nodes see the new config at different times.

Raft uses joint consensus or single-server changes to avoid this. Single-server changes (adding or removing exactly one node at a time) are the simpler approach: when you add one node, the old majority and new majority always overlap, so you cannot have two independent majorities.

The rule: a membership change is itself a log entry. It takes effect when committed. A server uses whichever config is latest in its log, whether committed or not.

Raft Tradeoffs vs. Paxos and Other Approaches

Property	Raft	Multi-Paxos	Viewstamped Replication
Algorithm completeness	Full spec (election, log, snapshots, membership)	Core only; gaps require implementation decisions	Full spec
Implementation complexity	Moderate	High (many design choices)	Moderate
Leader bottleneck	Yes, single writer	Yes, single proposer	Yes, single primary
Read scalability	Linearizable reads require quorum or lease	Same	Same
Reconfiguration	Single-server changes or joint consensus	Varies	Epoch-based
Understandability	Designed to be understandable	Notoriously difficult	Moderate
Production deployments	etcd, CockroachDB, TiKV, Consul	Chubby, Zookeeper (ZAB variant)	VR original, some databases

The single-leader design is both a strength and a constraint. Writes always go through one node, which simplifies reasoning but creates a throughput ceiling and a latency dependency on leader proximity. Systems like CockroachDB layer range-level Raft groups on top of each other so that different key ranges have different leaders distributed across nodes, effectively sharding the write path.

How Production Systems Use Raft

etcd is the canonical Raft implementation outside the original paper. It uses Raft to store Kubernetes cluster state. etcd exposes a linearizable key-value API: reads that must be linearizable go through the leader (or use ReadIndex, where the leader confirms it is still the leader by getting a majority acknowledgment before serving the read, without writing a log entry). etcd also implements leases, which are heartbeat-based time-bounded locks used by Kubernetes to elect controllers.

CockroachDB partitions its key space into ranges (default 512MB). Each range is independently replicated using Raft. The range’s Raft group has a leader, and that leader serves as the leaseholder for reads and writes on that range. When a range splits (because it grew too large), two new Raft groups are created. This architecture distributes Raft leadership across the cluster proportionally to data volume.

Consul uses Raft for its service catalog and distributed lock primitives. A notable operational detail: Consul separates server nodes (participate in Raft) from client nodes (forward requests to servers, do not vote). This limits the Raft cluster to 3-5 nodes regardless of fleet size, keeping election and replication latency bounded.

Practical Operational Considerations

Election timeout tuning: the election timeout must be significantly larger than the typical round-trip time between nodes, but small enough to recover quickly from leader failure. For a cluster in a single datacenter (1-5ms RTT), 150-300ms is reasonable. For geo-distributed clusters, you need to account for cross-region latency. An election timeout that is too small causes unnecessary elections under normal network jitter.

Leader stickiness: when a leader steps down (due to a configuration change or intentional shutdown), it is useful to transfer leadership to a specific follower rather than letting election timeouts fire. Most production implementations add a TimeoutNow RPC that tells a follower to start an election immediately. This avoids the 150-300ms delay before the cluster recovers.

Linearizable reads without log entries: having every read go through AppendEntries is expensive. Two alternatives exist. ReadIndex: the leader records its current commit index, confirms it still has a quorum (by waiting for a heartbeat round), and then serves the read when the state machine has applied up to that index. Leases: the leader maintains a time-bounded lease (shorter than the election timeout) during which it knows it is the only leader, and serves reads directly without a quorum check. Leases trade correctness (they assume bounded clock skew) for performance.

Follower reads with bounded staleness: some systems allow follower reads if the application can tolerate some staleness. Followers know their commitIndex and can serve reads consistent with that point. The staleness bound is roughly the heartbeat interval.

Snapshot frequency: taking snapshots too rarely means slow follower catch-up after a crash (replaying a long log). Taking them too often wastes I/O. A common heuristic is to snapshot when the log size exceeds a threshold (e.g., 64MB or a fixed number of entries). The snapshot operation itself should not block the state machine: copy-on-write or serializing to a separate process keeps the log replication path responsive.

The Fundamental Latency Constraint

Raft requires two round trips for a write to be committed and durable: one to replicate to followers, and one to acknowledge to the client after the majority has confirmed. In practice, pipeling helps: a leader can send the next batch of entries without waiting for the previous batch’s acknowledgments, keeping the network saturated. But the minimum latency is one network round trip to the majority, which is why geo-distributed Raft clusters with cross-region quorums have higher write latency than single-region deployments. That is not a Raft limitation; it is physics.

---

The reason Raft became dominant is not that it is theoretically superior to Paxos. It is that it makes the design space explicit and constrained. When you implement Raft you are not filling in gaps with local decisions; the algorithm specifies the gaps. That property is what makes Raft implementations interoperable, auditable, and debuggable. etcd’s Raft library can be embedded in your own system because the invariants are unambiguous. That is a harder property to achieve than it looks.