How ZooKeeper Works Internally: ZAB Protocol, Znodes, Watches, and the Coordination Engine Behind Distributed Systems

Distributed systems need a place to park small, critical state: which node is the current leader, what shards belong to which worker, whether a named lock is held. This coordination state is different from application data. It is read far more than it is written, it must be consistent across all readers, and losing it is catastrophic. You cannot put it in a database that itself depends on coordination to function.

Apache ZooKeeper was built specifically for this problem. Kafka used it for broker metadata and controller election until 3.x. HBase uses it for region server tracking and master election. Hadoop YARN uses it for ResourceManager high availability. If you have operated any of these systems, you have interacted with ZooKeeper, usually at 3am when it misbehaved. Understanding what it actually does internally makes those 3am incidents considerably shorter.

The Data Model: Znodes

ZooKeeper exposes a hierarchical namespace that looks like a filesystem tree. Each node in the tree is called a znode. A znode can hold a small byte payload (up to 1 MB by default, practically useful below 100 KB) and has children. The tree is fully in-memory on every server in the ensemble.

Znodes come in four flavors:

• Persistent: survives client disconnection and reconnection. Deleted only on explicit delete.
• Ephemeral: tied to the session that created it. When the session expires, the znode is deleted automatically. This is the mechanism behind distributed locks and leader election.
• Persistent sequential: like persistent, but ZooKeeper appends a monotonically increasing 10-digit counter to the name. Two clients creating /queue/item- get /queue/item-0000000001 and /queue/item-0000000002.
• Ephemeral sequential: combines both properties. Essential for barrier patterns and lock queues.

// Typical ZooKeeper client usage via the `node-zookeeper-client` library
import * as zookeeper from "node-zookeeper-client";

const client = zookeeper.createClient("zk1:2181,zk2:2181,zk3:2181", {
  sessionTimeout: 30000,
  retries: 3,
});

client.connect();

// Create an ephemeral node to register as a service instance
function registerServiceInstance(
  serviceName: string,
  address: string
): Promise<string> {
  return new Promise((resolve, reject) => {
    const path = `/services/${serviceName}/instance-`;
    const data = Buffer.from(JSON.stringify({ address, startedAt: Date.now() }));

    client.create(
      path,
      data,
      zookeeper.CreateMode.EPHEMERAL_SEQUENTIAL,
      (err, createdPath) => {
        if (err) return reject(err);
        resolve(createdPath);
      }
    );
  });
}

Each znode has a stat structure that tracks version numbers for the data (version), children (cversion), and ACL (aversion), plus creation and modification transaction IDs (czxid, mzxid), timestamps, and the ephemeral owner session ID. The version numbers are used for conditional updates, the ZooKeeper equivalent of compare-and-swap.

function conditionalUpdate(
  path: string,
  newData: Buffer,
  expectedVersion: number
): Promise<void> {
  return new Promise((resolve, reject) => {
    // setData with version -1 means "overwrite unconditionally"
    // Passing the actual version means "fail if someone else changed this"
    client.setData(path, newData, expectedVersion, (err) => {
      if (err) {
        if (err.getCode() === zookeeper.Exception.BAD_VERSION) {
          reject(new Error("Concurrent modification detected, retry"));
        } else {
          reject(err);
        }
      } else {
        resolve();
      }
    });
  });
}

Watch Notifications

Clients can set a watch on a znode when they read it. A watch fires exactly once when the watched znode changes: data updated, children added or removed, or the node deleted. After firing, the watch is gone. To maintain continuous notification, the client must re-register after each event.

This single-trigger semantic is not a bug. It is a deliberate design choice to prevent watch storms. If watches were persistent, a popular znode changing rapidly could generate an unbounded flood of notifications to many clients simultaneously.

function watchChildren(
  path: string,
  onChildren: (children: string[]) => void
): void {
  function refresh() {
    client.getChildren(
      path,
      (event) => {
        // Watch fired — re-register and fetch again
        refresh();
      },
      (err, children) => {
        if (err) {
          console.error(`Failed to get children of ${path}:`, err);
          return;
        }
        onChildren(children);
      }
    );
  }
  refresh();
}

There is a subtle timing issue here. Between the watch firing and the client re-registering with getChildren, another change could occur. The client would receive data consistent with the moment it re-read, but it would have missed an intermediate change event. For most coordination use cases (leader election, service discovery) this is fine: the client learns the current state, which is what matters. For cases where every change must be observed, watches are the wrong primitive.

Watch delivery is ordered: a client will always see the watch event before it sees any subsequent state that postdates the change. This is guaranteed by the server delivering events over the same connection as responses, in the same order.

Session Management

Every client connection to ZooKeeper creates a session, identified by a 64-bit session ID. The session has a configurable timeout (typically 10-30 seconds in production). The client must send a heartbeat (any request, or an explicit ping) within the session timeout window. If it does not, the server expires the session.

Session expiry is consequential: all ephemeral znodes owned by that session are deleted. This is the mechanism that makes distributed locks safe. If a lock holder crashes without releasing the lock, ZooKeeper expires its session and deletes the ephemeral lock node, automatically unblocking the next candidate.

client.on("state", (state) => {
  if (state === zookeeper.State.SYNC_CONNECTED) {
    console.log("Connected, session active");
  } else if (state === zookeeper.State.EXPIRED) {
    // Session expired: all your ephemeral nodes are gone.
    // You must re-create them from scratch.
    console.error("Session expired — re-registering all ephemeral state");
    reregisterEverything();
  } else if (state === zookeeper.State.DISCONNECTED) {
    // Still within session timeout — ZooKeeper will reconnect automatically.
    // Do NOT re-create ephemeral nodes yet.
    console.warn("Disconnected, waiting for reconnect");
  }
});

The distinction between DISCONNECTED and EXPIRED is critical. In DISCONNECTED state, the session is still alive on the server side. The client will reconnect, and all ephemeral nodes remain. If the client incorrectly treats a disconnection as expiry and re-creates its ephemeral nodes, it can create duplicates. This is a common bug in ZooKeeper client code.

Sessions are tied to the ensemble, not to a single server. When a client reconnects after a network partition, it can connect to any server in the ensemble and recover its session, as long as the session timeout has not elapsed.

The ZAB Protocol

ZooKeeper’s consistency guarantees come from ZAB (ZooKeeper Atomic Broadcast), a protocol purpose-built for ZooKeeper rather than adapted from Paxos or Raft. ZAB provides two key properties:

1. Total order: all servers apply state changes in the same order.
2. Causal order: if a client sees change A before change B, every client will see A before B.

ZAB operates in two modes: recovery mode (leader election and state sync) and broadcast mode (normal operation).

Leader Election

ZooKeeper uses a fast leader election algorithm based on epoch numbers and log positions, not a Paxos-style ballot protocol. Each server knows its own zxid (ZooKeeper transaction ID), a 64-bit value where the high 32 bits are the epoch and the low 32 bits are a counter within that epoch.

During election, each server broadcasts a vote: (myId, myZxid). Servers update their vote to favor the server with the highest zxid, breaking ties by server ID. A server wins when it receives votes from a majority of the ensemble (including itself).

The server with the highest zxid wins because it has the most complete transaction history. This guarantees that no committed transaction is lost when a new leader takes over.

Recovery Mode

After election, the new leader enters recovery mode. It must ensure all followers have the same state before accepting new client writes.

The leader fetches the latest committed zxid from each follower. For followers that are only slightly behind (within the leader’s in-memory transaction log), it sends the missing transactions directly (DIFF sync). For followers that are far behind or diverged, it sends a full snapshot (SNAP sync). Followers that have transactions the leader does not have committed must truncate those transactions (TRUNC sync).

Leader state after election:
  epoch: 5, lastCommittedZxid: 5:1042

Follower A: epoch 5, zxid 5:1040  → DIFF sync (send txns 1041, 1042)
Follower B: epoch 4, zxid 4:9900  → SNAP sync (full snapshot)
Follower C: epoch 5, zxid 5:1044  → TRUNC sync (roll back to 1042)

Follower C’s situation is the interesting case. Transactions 1043 and 1044 were proposed by the previous leader but never committed (the leader crashed before getting a majority acknowledgment). They must be discarded. ZAB’s safety guarantee is that only committed transactions survive leader transitions.

Broadcast Mode

Once a quorum of followers has synchronized, the leader starts accepting client writes. Each write becomes a proposal, assigned a new zxid. The leader sends the proposal to all followers. Followers write it to their transaction log and send an ACK. When the leader receives ACKs from a quorum (including itself), it sends a COMMIT message to all followers. The write is now applied to the in-memory tree and visible to readers.

Client → Leader: setData("/config/timeout", "5000")
  Leader → Followers: PROPOSE zxid=5:1043, setData("/config/timeout", "5000")
  Followers → Leader: ACK zxid=5:1043
  [quorum reached]
  Leader → Followers: COMMIT zxid=5:1043
  Leader → Client: OK

All reads in ZooKeeper are served locally by whichever server the client is connected to. This is a deliberate performance choice: reads are cheap, but they can return slightly stale data. If a client needs a linearizable read, it issues a sync request first, which forces the server to catch up to the current leader’s commit point before responding.

Request Processing Pipeline

On the leader, a write request travels through a chain of RequestProcessors:

1. PrepRequestProcessor: validates the request, checks ACLs, acquires any necessary locks on parent znodes, and builds the transaction record.
2. ProposalRequestProcessor: assigns a zxid, writes the proposal to the transaction log, and sends it to followers in parallel. Also hands the request to the CommitProcessor.
3. CommitProcessor: holds the request until the proposal is committed (quorum ACK received). This is where the write actually blocks waiting for durability.
4. ToBeAppliedRequestProcessor: applies the committed transaction to the in-memory data tree.
5. FinalRequestProcessor: sends the response to the client.

On followers, reads are handled by a simpler pipeline ending at FinalRequestProcessor. Write requests are forwarded to the leader, which processes them through the full pipeline above and sends the response back through the follower to the client.

Persistence: Snapshots and Transaction Logs

ZooKeeper stores state in two forms on disk:

Transaction log: an append-only log of every committed transaction. Each entry contains the zxid, the transaction type, and the full diff. Entries are padded to 64-byte boundaries and written with a sync() call before ACKing the leader. The log is the source of truth for recovery. ZooKeeper pre-allocates log files in 64 MB chunks to avoid filesystem metadata updates on every write.

Snapshots: periodic full serializations of the in-memory tree. Taking a snapshot does not pause writes. The snapshot process walks the tree concurrently with ongoing transactions, which means the snapshot is a fuzzy snapshot: it is consistent with some point in time, but not necessarily a single zxid. ZooKeeper handles this by replaying the transaction log from a point slightly before the snapshot was started, which makes the final state consistent.

Recovery sequence:
1. Find the most recent snapshot file
2. Deserialize it into memory
3. Find the transaction log entry corresponding to the snapshot's starting zxid
4. Replay all log entries from that point forward
5. In-memory tree is now at the latest committed state

ZooKeeper retains a configurable number of snapshots and their corresponding log segments (default: 3). Old files are cleaned up automatically. In practice you want enough history to allow a follower that was partitioned for a few minutes to rejoin via DIFF sync rather than forcing a full SNAP sync.

Production Considerations

Ensemble sizing: always use an odd number. A 3-node ensemble tolerates 1 failure. A 5-node ensemble tolerates 2 failures. The write quorum is floor(n/2) + 1, so a 4-node ensemble also tolerates only 1 failure while adding latency (it still needs 3 ACKs). There is no practical reason to run 4 nodes instead of 3 or 5.

JVM heap and GC: ZooKeeper holds the entire data tree in memory. With large datasets (tens of millions of znodes), GC pauses become the dominant source of latency spikes. Use G1GC with explicit MaxGCPauseMillis targets. Keep the heap at or below 8 GB. If you need more than 8 GB of coordination state, ZooKeeper is likely the wrong tool.

Transaction log on a dedicated disk: the transaction log must be fsynced before every ACK. Sharing a disk with application data, snapshots, or OS logging causes write latency spikes that directly increase client latency. Put the transaction log on a dedicated low-latency SSD.

Watch storms: a thundering herd of watches firing simultaneously because one znode changed is a real failure mode. Design your watch topology so that a single change does not trigger thousands of re-reads. The canonical pattern is a barrier znode: clients watch a parent node, and the server writes a single signal value rather than creating hundreds of individual znodes.

Session timeout tuning: the session timeout must be greater than the GC pause duration plus the maximum network round-trip latency. In practice, 10 seconds is a reasonable floor. Lower values cause false session expirations during GC. Higher values mean distributed locks take longer to release after a crash.

Leader election latency: ZooKeeper’s fast leader election typically completes in under 200 ms on a healthy LAN. But during leader election, ZooKeeper is unavailable for writes. Design your clients to retry with exponential backoff and surface this as a degraded state to upstream systems rather than treating it as a fatal error.

async function zkWriteWithRetry(
  path: string,
  data: Buffer,
  maxAttempts = 5
): Promise<void> {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      await setData(client, path, data);
      return;
    } catch (err: any) {
      const isRetryable =
        err.getCode() === zookeeper.Exception.CONNECTION_LOSS ||
        err.getCode() === zookeeper.Exception.OPERATION_TIMEOUT;

      if (!isRetryable || attempt === maxAttempts) throw err;

      const backoffMs = Math.min(100 * Math.pow(2, attempt), 5000);
      await new Promise((r) => setTimeout(r, backoffMs));
    }
  }
}

Do not store large payloads: ZooKeeper is not a key-value store. Each znode is fully loaded into every server’s heap. Storing megabyte-scale payloads in znodes inflates memory pressure ensemble-wide and slows snapshot serialization. Store a pointer (S3 URL, database row ID) in ZooKeeper and keep the payload elsewhere.

Tradeoffs: ZooKeeper vs etcd vs Consul vs Chubby

Dimension	ZooKeeper	etcd	Consul	Chubby
Consensus protocol	ZAB (custom)	Raft	Raft	Paxos (custom)
Data model	Hierarchical tree (znodes)	Flat key-value with prefix scans	Key-value with services/health	File-like with advisory locks
Watch semantics	Single-trigger per registration	Streaming watch (persistent until cancelled)	Blocking poll on index	Event-based callbacks
Read model	Local reads (stale by default), sync() for linearizable	Linearizable by default (quorum read)	Stale by default, consistent param for quorum	Linearizable
Client session	Yes, with ephemeral nodes	Leases (TTL-based keys)	Sessions with TTL	Sessions with locks
Operational complexity	High (JVM, GC tuning, separate cluster)	Low (single binary, built into Kubernetes)	Medium (agent model, DNS interface)	N/A (internal Google service)
Write throughput	~10K-50K ops/sec (5-node, fsync on)	~10K-100K ops/sec (depends on disk)	Similar to etcd	Not published
Horizontal reads	No (all servers serve reads)	Yes (linearizable via quorum)	Yes (with caveats)	No
Best fit	Legacy Hadoop/Kafka ecosystem, complex coordination patterns	Kubernetes, modern microservices, cloud-native	Service mesh, health checks, DNS-based discovery	Google internal systems
Maturity	Very high (15+ years, stable API)	High (CNCF graduated)	High (HashiCorp, widely deployed)	Very high (internal only)

ZAB versus Raft is often cited as the key technical difference, but in practice both protocols provide the same fundamental guarantee: a total order of committed writes with majority-quorum durability. The more material operational difference is that etcd ships as a single statically-linked Go binary with no external dependencies, while ZooKeeper requires a JVM, careful heap tuning, and a dedicated transaction log disk to run reliably at production latencies.

ZooKeeper’s watch model is richer than etcd’s key-value watches for hierarchical coordination patterns. Ephemeral sequential znodes make implementing fair distributed locks and barriers straightforward without any application-side sequencing logic. etcd achieves the same with leases and prefix scans, but the client code is more verbose.

For new systems not bound to the Hadoop ecosystem, etcd or Consul are almost always the better operational choice. For systems already running ZooKeeper, the migration cost rarely justifies switching: ZooKeeper’s API is stable, its failure modes are well understood, and its throughput is sufficient for coordination workloads.

The Fundamental Insight

ZooKeeper’s design embodies one specific bet: coordination state is small, access patterns are read-heavy and watch-heavy, and the guarantees need to be strong enough that you can build higher-level primitives on top without worrying about the underlying consistency model. The ZAB protocol, ephemeral nodes, and single-trigger watches are all consequences of that bet.

Understanding the full pipeline from client write to ZAB proposal to quorum commit to watch notification lets you reason accurately about what ZooKeeper can and cannot tolerate. A leader election that appears to take 10 seconds is probably a GC pause extending past the session timeout, not a network partition. A watch that fires twice for a single change is probably a client reconnect mid-flight, not a ZooKeeper bug. The semantics are precise once you know where to look.