How etcd Works Internally: Raft Consensus, MVCC Storage, and the Watch API Behind Kubernetes State Management

Kubernetes treats etcd as the ground truth for all cluster state. Every Pod spec, every ConfigMap, every Service endpoint lives there. If etcd is wrong, the cluster is wrong. If etcd is unavailable, the API server goes read-only and controllers stop reconciling.

Yet most engineers treat etcd as a black box. They know it stores keys and values. They know it uses Raft. They’ve seen the recommendation to run three or five nodes. But the mechanics underneath matter when you’re tuning a large cluster, debugging watch latency, or trying to understand why compaction pauses cause cascading timeouts.

This article goes through the internals: how Raft actually works in etcd, how MVCC storage is layered on top of bbolt, how the Watch API avoids polling, and where the real performance tradeoffs live.

The Raft Consensus Layer

etcd implements Raft from the ground up, in Go. The key insight behind Raft is that it decomposes consensus into two well-understood sub-problems: leader election and log replication. Once a leader exists, all writes go through it, which makes reasoning about consistency tractable.

Leader Election

Every node in the cluster is in one of three states: follower, candidate, or leader. Followers expect a heartbeat from the leader within the election timeout window (default: 1000ms, configurable). If a follower times out without hearing from the leader, it increments its current term, transitions to candidate, and sends RequestVote RPCs to peers.

A candidate wins the election when it receives votes from a majority of nodes. The vote is granted only if:

1. The candidate’s term is at least as large as the voter’s current term.
2. The candidate’s log is at least as up-to-date as the voter’s log (compared by last log term, then last log index).

This “log completeness” check is what prevents a stale node from becoming leader and overwriting committed entries.

// Simplified RequestVote logic inside etcd's raft package
func (r *raft) handleRequestVote(m pb.Message) {
    canVote := r.Vote == m.From ||
        (r.Vote == None && r.lead == None) ||
        (m.Type == pb.MsgPreVote && m.Term > r.Term)

    logOK := m.LogTerm > r.raftLog.lastTerm() ||
        (m.LogTerm == r.raftLog.lastTerm() && m.Index >= r.raftLog.lastIndex())

    if canVote && logOK {
        r.send(pb.Message{
            To:   m.From,
            Type: voteRespMsgType(m.Type),
        })
        r.electionElapsed = 0
        r.Vote = m.From
    }
}

etcd also implements PreVote, an optimization where a candidate first checks that it can win before incrementing its term. This prevents network partitions from inflating term numbers and causing unnecessary elections when the partition heals.

Log Replication

Once a leader is elected, all writes go through it. The leader appends the entry to its local log, then sends AppendEntries RPCs to all followers. An entry is “committed” once a majority of nodes have acknowledged it. The leader then applies the committed entry to its state machine (the key-value store) and responds to the client.

The critical point is that committed entries are durable. A committed entry will survive any future leader change, because any new leader must have been up-to-date enough to win an election, meaning it has all committed entries.

// After majority acknowledgment, leader advances commit index
func (r *raft) maybeCommit() bool {
    mis := make(uint64Slice, 0, len(r.prs.Voters))
    for _, pr := range r.prs.Voters {
        mis = append(mis, pr.Match)
    }
    sort.Sort(sort.Reverse(mis))
    mci := mis[len(mis)/2] // majority match index
    return r.raftLog.maybeCommit(mci, r.Term)
}

The majority index (median of sorted match indices) is what the leader uses to advance the commit index. Followers learn about the new commit index via the next AppendEntries heartbeat.

MVCC Storage on bbolt

etcd’s storage layer implements multi-version concurrency control (MVCC) on top of bbolt, a pure-Go embedded B+tree database. The design allows concurrent readers to never block writers, and enables the Watch API to stream historical changes.

The Revision Model

Every write operation in etcd increments a global revision counter. The revision is a monotonically increasing 64-bit integer attached to the entire keyspace, not to individual keys. This is the central concept in etcd’s storage model.

Each key has its own version (reset to 1 on each creation after deletion) and its own create revision and mod revision. The tuple (main_revision, sub_revision) forms the unique key in the underlying bbolt store, where sub_revision handles batched writes in a single Raft log entry.

bbolt key format:
<key_bytes> + <revision_bytes> -> <value_bytes>

For key "/registry/pods/default/my-pod" at revision 42:
\x00/registry/pods/default/my-pod\x00\x00\x00\x00\x00\x00\x00\x2a\x00\x00\x00\x00\x00\x00\x00\x00

There are two bbolt buckets that matter:

• key bucket: stores the actual data, keyed by (user_key, main_rev, sub_rev).
• meta bucket: stores the current revision, consistent index (last applied Raft log index), and other metadata.

A separate in-memory structure called the treeIndex maps each user-facing key to its revision history. The treeIndex is a B-tree (using the google/btree library) that stores, for each key, the full list of revisions at which it was modified. This index is rebuilt from bbolt on startup.

// keyIndex holds the revision history for a single key
type keyIndex struct {
    key         []byte
    modified    revision  // last modification revision
    generations []generation
}

type generation struct {
    ver     int64
    created revision
    revs    []revision  // all revisions for this generation
}

When you do a Get with a specific revision (a historical read), etcd consults the treeIndex to find the correct revision entry, then does a point lookup in bbolt. This is O(log n) in the number of distinct keys, not in the total number of versions.

Write Path

A write from a client goes through this pipeline:

1. Client sends Put(key, value) to the leader’s gRPC server.
2. The leader serializes the request into a Raft log entry and proposes it.
3. After the majority commits it, the apply goroutine picks it up.
4. The apply goroutine increments the current revision, writes to bbolt, and updates the treeIndex.
5. The response is sent back to the client after apply (not just after commit).

The distinction between “committed” and “applied” matters for the Watch API. Watches fire when entries are applied, not when they are committed. Applied means the state machine has been updated and reads will reflect the new value.

Linearizable vs. Serializable Reads

etcd exposes two read consistency levels, and the difference has real performance implications.

Linearizable reads (the default for most operations) guarantee that the read reflects any write that completed before the read started. To satisfy this, the leader cannot just read from its local state, because it might be a stale leader that was partitioned away. Instead, it sends a heartbeat round to verify that it still holds the majority, then reads. This adds one network round trip, typically 1-5ms in a local cluster.

Serializable reads skip the quorum check and read directly from the local node’s state. The data may be slightly stale if the node is a follower or if the leader hasn’t applied the latest entries. Serializable reads are fine for Kubernetes controllers that tolerate eventual consistency, and they remove the extra network round trip.

// In etcd client v3: controlling consistency
import clientv3 "go.etcd.io/etcd/client/v3"

// Linearizable (default): guaranteed up-to-date
resp, err := client.Get(ctx, "/registry/pods/default/my-pod")

// Serializable: potentially stale, but cheaper
resp, err := client.Get(ctx, "/registry/pods/default/my-pod",
    clientv3.WithSerializable(),
)

In practice, the Kubernetes API server uses linearizable reads for anything that feeds into admission or scheduling decisions, and serializable reads where staleness is acceptable (some informer cache refreshes, for example).

The Watch API

The Watch API is what makes etcd practical as a control plane store. Rather than polling for changes, clients open a watch stream and receive events as they happen. The implementation is non-trivial.

WatchableStore and Event Fanout

etcd wraps its regular MVCC store in a watchableStore. When a write is applied, the watchableStore computes which watches are affected and fans out events to those watch streams.

Internally, watches are divided into two groups:

• Synced watches: watches that have received all events up to the current revision. New events are delivered immediately when they are applied.
• Unsynced watches: watches that are behind (created with a start revision older than the current compacted revision, or that haven’t caught up yet). These are processed in a background goroutine that replays history from bbolt.

// Simplified watch event delivery in watchableStore.notify
func (s *watchableStore) notify(rev int64, evs []mvccpb.Event) {
    for w, eb := range s.synced.watcherSetByKey(evs) {
        // non-blocking send; if the channel is full, move to victim group
        select {
        case w.ch <- WatchResponse{WatchID: w.id, Events: eb}:
            eventsCounter.Add(float64(len(eb)))
        default:
            // watcher is slow; mark as victim, will be replayed
            w.victim = true
        }
    }
}

If a watcher’s channel is full (the client is consuming events too slowly), the watcher is demoted to the “victim” group and will be replayed from the MVCC log. This backpressure mechanism prevents one slow client from blocking event delivery to everyone else.

Watch on a Range

Kubernetes uses range watches extensively: watch all pods under /registry/pods/, watch all endpoints under /registry/services/endpoints/. etcd represents these as prefix watches internally, stored in an interval tree indexed by the key range. When an event fires, the interval tree is queried for all watches whose range contains the affected key.

Bookmarks and Resume Tokens

When a client disconnects and reconnects, it wants to resume watching from where it left off. The watch request includes a start_revision field. etcd replays events from that revision by scanning bbolt from the given revision forward.

A client that doesn’t know its last revision can request a bookmark event, which is a synthetic event containing only the current revision. The client can persist this and use it to resume on reconnect.

// Resuming a watch after reconnect
watchChan := client.Watch(ctx, "/registry/pods/",
    clientv3.WithPrefix(),
    clientv3.WithRev(lastSeenRevision),
)

for resp := range watchChan {
    for _, event := range resp.Events {
        lastSeenRevision = event.Kv.ModRevision
        // process event
    }
}

Lease Management

Leases are TTL timers attached to keys. A client creates a lease with a TTL, then attaches keys to it. When the lease expires, all attached keys are deleted atomically. This is how Kubernetes implements node heartbeats: each node holds a lease and renews it every 10 seconds.

The leader tracks all leases and advances their expiry using a min-heap. The expiry check runs every 500ms. When a lease expires, the leader proposes a revoke entry through Raft, which is applied like any other write.

Lease renewal sends a LeaseKeepAlive RPC to the leader. If the leader changes, in-flight keepalives will fail, but the etcd client library handles retrying against the new leader. The TTL is typically set to 3x the renewal interval to absorb transient failures.

// Creating and attaching a lease
lease := clientv3.NewLease(client)
grantResp, err := lease.Grant(ctx, 30) // 30 second TTL

// Attach a key to the lease
_, err = client.Put(ctx, "/nodes/node-1/heartbeat", "alive",
    clientv3.WithLease(grantResp.ID),
)

// Keep the lease alive (runs in background)
keepAliveChan, err := lease.KeepAlive(ctx, grantResp.ID)
go func() {
    for range keepAliveChan {
        // renewal confirmed; reset your watchdog here if needed
    }
    // channel closed: lease expired or context cancelled
}()

Compaction and Defragmentation

Every write creates a new revision. Without cleanup, bbolt grows unboundedly. Two maintenance operations address this.

Compaction removes old revisions from the treeIndex and bbolt, keeping only the most recent version of each key (or up to a specified historical revision). Kubernetes runs compaction roughly every 5 minutes via the API server’s --etcd-compaction-interval flag. After compaction, watches started at a revision older than the compaction point will receive an ErrCompacted error, forcing clients to re-list and re-watch.

Compaction is a Raft-replicated operation: the leader proposes the compaction revision, and all nodes compact their local stores. The compaction itself is done under a write lock on bbolt, which is why large compaction runs can cause brief latency spikes.

Defragmentation reclaims free space in the bbolt file. bbolt never shrinks the file on its own; it only reuses freed pages internally. Defragmentation rewrites the bbolt database from scratch into a new file, then replaces the old one. This takes an exclusive write lock for the duration and must be done one node at a time, so it should be scheduled during low-traffic windows.

Tradeoffs Comparison

System	Consensus	Storage Model	Watch/Notify	Linearizable Reads	Operational Complexity
etcd	Raft	MVCC on bbolt	Native streaming watch API	Yes, with quorum read	Low: single binary, embedded storage
ZooKeeper	ZAB (Zookeeper Atomic Broadcast)	Tree of znodes, no MVCC	Ephemeral watches (one-shot, must re-register)	Yes	High: JVM tuning, separate ZK ensemble ops
Consul	Raft (HashiCorp)	Key-value + service catalog	Blocking queries (long-poll)	Yes	Medium: agent model, multi-datacenter setup
NATS JetStream	Raft per stream	Log-structured (message streams)	Push consumers, pull consumers	No native compare-and-set	Low: single binary, broader messaging model
FoundationDB	Paxos-based	MVCC, multi-key transactions	No native watch (use triggers layer)	Yes, strict serializable	High: requires dedicated cluster, complex tuning

The main differentiators for etcd:

• MVCC with revision history enables Watch from a specific point in time, which ZooKeeper’s one-shot watches cannot do.
• Native Kubernetes integration means the client library, authentication model, and operational playbooks are all standardized.
• The embedded bbolt model keeps operational overhead minimal but also limits write throughput relative to systems with more sophisticated storage engines.

Production Considerations

Cluster sizing. Three nodes tolerate one failure. Five nodes tolerate two. More than five rarely makes sense: write latency scales with quorum round trips, and more nodes means more messages per write. Large Kubernetes clusters should run a five-node etcd cluster with dedicated SSDs, not etcd on shared nodes.

Disk I/O is everything. bbolt uses fsync on every commit to guarantee durability. On spinning disks or overloaded SSDs, fsync latency directly sets the floor for write latency. The etcd documentation recommends fewer than 10ms for disk commit duration, measured via the etcd_disk_wal_fsync_duration_seconds Prometheus metric. Anything above 25ms will start causing Raft heartbeat timeouts.

Watch stream overload. If clients receive events faster than they process them, their watch channels back up. etcd will eventually close slow watch streams. Design watch consumers to be non-blocking: push events onto a local queue and process asynchronously. The Kubernetes informer framework does this correctly.

Compaction tuning. Too-frequent compaction keeps the database small but adds CPU and lock contention. Too-infrequent compaction causes bbolt to grow large and makes defragmentation take longer. A 5-minute compaction interval with a 1-hour history retention is the Kubernetes default, and it works for most clusters.

Snapshot and restore. etcd takes periodic Raft snapshots to bound log replay time on restart. The snapshot is a serialized copy of the in-memory treeIndex plus the bbolt state. The etcdctl snapshot save command creates a portable snapshot for disaster recovery. Test your restore procedure before you need it.

Quota enforcement. The default storage quota is 2GB. When the quota is exceeded, etcd enters a read-only alarm state and the Kubernetes API server starts rejecting writes. Monitor etcd_mvcc_db_total_size_in_bytes and defragment before hitting the quota, not after.

Certificate rotation. etcd uses mutual TLS for both peer and client communication. Certificate expiry on a live cluster is a serious operational event. Automate rotation with cert-manager or a similar tool, and monitor expiry with ssl_certificate_expiry_seconds.

Understanding etcd at this level makes the operational decisions less arbitrary. The 10ms fsync threshold isn’t a magic number: it’s derived from the Raft heartbeat interval and the number of network round trips a write requires. The compaction pause that kills your controllers is directly traceable to bbolt’s exclusive write lock. The behavior all follows from the design.

The design is deliberately conservative: a single embedded store, Raft with straightforward leader semantics, MVCC with a simple revision counter. That conservatism is what makes etcd reliable enough to hold cluster state for critical infrastructure.