How Apache Pulsar Works Internally: Tiered Storage, BookKeeper-Backed Persistence, and the Multi-Tenant Messaging Engine That Separates Compute from Storage

Most messaging systems treat the broker as a single unit: it receives messages, stores them, and serves consumers. That design is simple and it works until you need to scale storage independently of throughput, or until you need one cluster to serve dozens of teams with hard isolation between them. Apache Pulsar was built around a different assumption: compute and storage are separate concerns and the architecture should reflect that.

The result is a system with more moving parts than Kafka or RabbitMQ, but one that handles use cases those systems make awkward: near-infinite retention backed by object storage, zero-copy broker failover without rebalancing, and first-class multi-tenancy without external bolted-on layers. Understanding what actually happens inside Pulsar when a message is published and consumed makes it possible to reason about capacity, latency, and failure modes with confidence.

The Core Separation: Brokers and BookKeeper

The first thing to understand about Pulsar is that brokers do not store data. They are stateless routing and serving processes. All persistence is handled by Apache BookKeeper, a separate distributed log storage system that Pulsar runs as a dependency.

This separation is the architectural bet Pulsar makes. Because brokers own no data, a broker failure triggers no data migration. Another broker picks up ownership of a topic bundle and starts serving immediately, using the same underlying BookKeeper ledgers the failed broker was writing to. Compare that to Kafka, where a leader failure requires a new leader election among replicas that each hold a copy of the partition log, and where adding storage capacity means adding broker nodes that then participate in potentially expensive rebalances.

The tradeoff is operational complexity. You are running two distributed systems: the Pulsar broker tier and the BookKeeper bookie tier. Each has its own scaling, configuration, and failure modes.

BookKeeper Internals: Journal and EntryLog

BookKeeper persistence happens in two phases on each bookie node.

The journal is a write-ahead log. Every append is fsynced to the journal before the bookie acknowledges it. The journal is purely sequential and is never read during normal operation. Its only purpose is durability: if a bookie crashes, the journal replays to recover any entries that did not make it to the entrylog.

The entrylog is where entries actually live. BookKeeper multiplexes entries from many ledgers into a small number of large entrylog files. This is intentional. Writing many small files per ledger would fragment the disk; multiplexing into shared files keeps I/O sequential. Reads use an in-memory ledger index to locate specific entries within the entrylog by offset.

A write to BookKeeper quorum looks like this: the client (in Pulsar’s case, the broker’s managed ledger layer) sends the entry to Wq bookies (the write quorum), and waits for acknowledgment from Aq bookies (the ack quorum). For a typical production configuration of ensemble size 3, write quorum 2, ack quorum 2, the broker gets a durable ack after two of three bookies have fsynced to their journals.

// Illustrative: how a Pulsar producer configures persistence guarantees
import Pulsar from "pulsar-client";

const client = new Pulsar.Client({ serviceUrl: "pulsar://broker:6650" });

const producer = await client.createProducer({
  topic: "persistent://payments/prod/transactions",
  // sendTimeoutMs controls how long to wait for the broker ack,
  // which itself waits for the BookKeeper ack quorum
  sendTimeoutMs: 5000,
  // batchingEnabled groups entries before sending to the broker,
  // reducing per-message journal fsync overhead at the cost of latency
  batchingEnabled: true,
  batchingMaxPublishDelayMs: 10,
  batchingMaxMessages: 1000,
});

const msgId = await producer.send({
  data: Buffer.from(JSON.stringify({ orderId: "ord_9kX2", amount: 4999 })),
  properties: { region: "us-east-1" },
});

console.log(`Published with id: ${msgId.toString()}`);
await producer.close();
await client.close();

Managed Ledgers: The Bridge Between Broker and BookKeeper

Pulsar topics do not talk to BookKeeper directly. The managed ledger is the abstraction layer that sits between them.

Each topic partition maps to one managed ledger. The managed ledger is responsible for: creating new BookKeeper ledgers when the current one rolls over (by size or by time), tracking cursor positions for all subscriptions, and coordinating reads for both tailing and catch-up consumers.

A ledger rollover happens when the current ledger exceeds its configured size (default 2 GB) or age (default 4 hours). When a ledger rolls over, the broker creates a new ledger in BookKeeper and continues appending. The old ledger stays open for reads. The managed ledger keeps a manifest of all ledgers that make up the topic’s log, stored in ZooKeeper (or etcd in newer Pulsar versions using the Oxia metadata store).

This ledger-per-segment structure is what makes tiered storage possible and what enables broker failover without data movement.

The Write Path

When a producer publishes a message:

1. The client connects to the broker that owns the topic bundle. Ownership is tracked in the metadata store; any broker can answer a lookup request and redirect.
2. The broker appends the message to the managed ledger.
3. The managed ledger writes the entry to the current BookKeeper ledger using its configured ensemble and quorum.
4. BookKeeper fans the write out to Wq bookies, which each append to their journal and fsync.
5. Once Aq bookies acknowledge, BookKeeper returns success to the managed ledger.
6. The broker sends the producer acknowledgment.

The critical path for publish latency is the journal fsync on Aq bookies. On commodity NVMe storage, this is typically sub-millisecond. On spinning disk it can be 5-10 ms. This is why BookKeeper journal disks are usually provisioned separately from entrylog disks in high-throughput deployments.

The Read Path: Tailing vs Catch-Up Consumers

Pulsar maintains a strict distinction between consumers reading at the tip of the log and consumers that are behind.

A tailing consumer receives messages via the broker’s in-memory dispatch queue. When the managed ledger receives a new entry and the entry is acknowledged by the write quorum, the broker dispatches it directly from memory to any connected tailing consumers without a BookKeeper read. This is the fast path.

A catch-up consumer is one whose cursor is behind the last-acknowledged entry by more than a configurable threshold. The broker detects this and routes reads through the managed ledger’s read cache (backed by a Netty-allocated off-heap cache) or directly to BookKeeper bookies. BookKeeper reads look up the entry’s position in the ledger index, seek to the appropriate entrylog offset, and return the bytes.

// Tailing consumer: low-latency path, broker dispatches from memory
const tailingConsumer = await client.subscribe({
  topic: "persistent://payments/prod/transactions",
  subscription: "fraud-detection",
  subscriptionType: Pulsar.SubscriptionType.KeyShared,
  // Start from the latest message (tailing)
  subscriptionInitialPosition: Pulsar.InitialPosition.Latest,
  receiverQueueSize: 1000,
});

// Catch-up consumer: reads are served from BookKeeper
const catchUpConsumer = await client.subscribe({
  topic: "persistent://payments/prod/transactions",
  subscription: "audit-log",
  subscriptionType: Pulsar.SubscriptionType.Exclusive,
  // Start from earliest stored message (catch-up)
  subscriptionInitialPosition: Pulsar.InitialPosition.Earliest,
  receiverQueueSize: 200, // smaller queue to avoid overwhelming the broker read path
});

Cursor positions are tracked per subscription in the managed ledger’s cursor ledger, which is itself a BookKeeper ledger. When a consumer sends a cumulative acknowledgment, the cursor advances in memory and is periodically checkpointed to the cursor ledger. Individual (non-cumulative) acknowledgments are tracked in a bitset that is also checkpointed.

Topic Partitioning and Bundle-Based Load Balancing

Pulsar supports partitioned topics, where a single logical topic is split across N partitions, each a full managed ledger owned by a (possibly different) broker. The client-side router assigns messages to partitions by round-robin, by message key hash, or by a custom routing function.

What makes Pulsar’s load balancing distinctive is that topic ownership is managed at the bundle level, not the individual topic level. A namespace is divided into a configurable number of bundles (default 16, often 64-256 in large deployments). Each bundle is a hash range, and each topic partition is assigned to a bundle based on its name hash. Brokers own bundles, not individual topics.

When a broker is overloaded, the load balancer splits a bundle at its midpoint and transfers one half to a less-loaded broker. This is a transfer of metadata ownership only: the broker that takes over a bundle starts serving requests using the same BookKeeper ledgers immediately. No data moves. This is the key advantage of the compute/storage separation in practice.

The load balancer runs as a leader-elected process among brokers and makes decisions based on CPU, memory, throughput, and message rate metrics reported by each broker to the metadata store.

Native Multi-Tenancy: Tenant/Namespace/Topic

Pulsar has a three-level hierarchy baked into the topic naming convention:

persistent://<tenant>/<namespace>/<topic>

Tenants represent organizations or teams. Namespaces represent environments, services, or workload types within a tenant. Topics live inside namespaces.

This is not cosmetic. Each namespace has its own:

• Retention policies: how long or how much data to keep after all subscriptions have acknowledged.
• Backlog quotas: maximum unacknowledged data before producers are throttled or messages are dropped.
• Persistence policies: per-namespace override of the ensemble and quorum configuration.
• Authentication and authorization: role-based access control scoped to the namespace level.
• Rate limits: per-producer and per-consumer message rate limits.

// Pulsar admin API: namespace configuration
// (typically done via CLI or HTTP API, shown here as HTTP for clarity)
const namespaceConfig = {
  retention_policies: {
    retentionTimeInMinutes: 10080,   // 7 days
    retentionSizeInMB: 51200,        // 50 GB
  },
  backlog_quota_map: {
    destination_storage: {
      limitSize: 10737418240,        // 10 GB
      policy: "producer_request_hold",
    },
  },
  persistence: {
    bookkeeperEnsemble: 3,
    bookkeeperWriteQuorum: 2,
    bookkeeperAckQuorum: 2,
    managedLedgerMaxMarkDeleteRate: 0,
  },
};

This model means that a single Pulsar cluster can safely serve dozens of teams with hard quotas and access controls between them, without the namespace-as-convention workarounds you would need in Kafka.

Tiered Storage: Offloading Cold Data to Object Storage

Every Pulsar managed ledger knows which of its BookKeeper ledgers are old enough to offload. When a ledger rolls over and its data is no longer needed by any active subscription cursor (or after a configured offload threshold), Pulsar can copy that ledger’s data to an object storage backend: S3, GCS, Azure Blob Storage, or HDFS.

The offload process reads entries from BookKeeper in order and writes them as objects in a format that maps offsets back to object boundaries. After offload, the BookKeeper ledger can be deleted, reclaiming bookie disk space. The managed ledger manifest in metadata is updated to reflect that a given ledger range now lives in object storage.

Reads for offloaded data are transparently served via the offloader’s object storage client. From the consumer’s perspective, there is no difference. Tail latency for catch-up reads against S3 is higher than reads from BookKeeper (typically 20-100 ms versus under 5 ms), but for analytics consumers replaying months of history this is acceptable.

This makes Pulsar a viable choice for workloads that need streaming and long-term data lake access from the same system, eliminating a separate pipeline from the broker to an archive store.

Geo-Replication via Replication Cursors

Pulsar geo-replication works at the namespace level. When you enable replication from cluster A to cluster B for a namespace, Pulsar creates a special subscription on each topic in that namespace: the replication cursor for cluster B.

The replication cursor works like a consumer subscription. It tracks the position of messages that have been replicated to cluster B. The replication producer on cluster A reads messages ahead of the replication cursor and publishes them to the corresponding topic on cluster B. On success, the cursor advances. On failure, it retries with backoff.

This design has important properties. Replication is asynchronous by default, which means cluster A acknowledges producers without waiting for cluster B. You can enable synchronous replication (waiting for the remote ack before the local ack) for strong cross-region consistency at the cost of latency that now includes the inter-region round trip. The replication cursor also means that data in cluster A is retained until all configured replication destinations have consumed it, regardless of the local subscription backlog.

Production Considerations

Bookie disk layout. Separate the journal and entrylog onto different physical devices. The journal’s workload is pure sequential fsync. The entrylog’s workload is sequential writes with random reads. Mixing them on the same device degrades both. NVMe for the journal, larger spinning or NVMe for the entrylog.

Managed ledger cache sizing. The managed ledger cache (managedLedgerCacheSizeMB) is an off-heap Netty buffer that serves tailing reads without hitting BookKeeper. Undersizing it forces catch-up read paths even for consumers that are only slightly behind. Size it to hold at least the last few minutes of retained data for your highest-throughput topics.

Bundle count. Too few bundles means that a hot topic cannot be spread across brokers. Too many means metadata operations become expensive. 64-256 bundles per namespace is a reasonable starting point for namespaces with more than a few dozen partitioned topics.

Subscription type selection. Exclusive and Failover guarantee ordered consumption. Shared sacrifices ordering for throughput. KeyShared gives per-key ordering with horizontal consumer scaling. Choose before deploying: changing subscription type requires recreating the subscription and losing cursor position.

ZooKeeper (or Oxia) health. Pulsar’s metadata store is on the critical path for broker startup, topic lookup, and ownership transfer. An unhealthy ZooKeeper ensemble causes cascading slowness across the cluster. Run it on dedicated nodes, size it for the number of topics (not just the number of messages), and monitor znode depth for large topic counts.

// Production consumer pattern: explicit ack with error handling
const consumer = await client.subscribe({
  topic: "persistent://payments/prod/transactions",
  subscription: "reconciliation",
  subscriptionType: Pulsar.SubscriptionType.KeyShared,
  ackTimeoutMs: 30000,
  nAckRedeliveryDelayMs: 5000,
});

while (true) {
  const msg = await consumer.receive();
  try {
    await processTransaction(JSON.parse(msg.getData().toString()));
    await consumer.acknowledge(msg);
  } catch (err) {
    // negative ack triggers redelivery after nAckRedeliveryDelayMs
    await consumer.negativeAcknowledge(msg);
  }
}

Tradeoffs Comparison

Dimension	Pulsar	Kafka	RabbitMQ	NATS JetStream	Redpanda
Storage model	Compute/storage separated (BookKeeper)	Storage on broker (partition logs)	Storage on broker (queues)	Storage on server (JetStream)	Storage on broker (partition logs, no ZK)
Broker failover	Instant (no data migration)	Leader election + catchup	Queue mirroring	Raft-based leader election	Raft-based, fast
Multi-tenancy	Native (tenant/namespace/topic)	Convention-based (topic naming, ACLs)	vhosts (limited)	Accounts (limited quotas)	Convention-based
Tiered storage	Native S3/GCS offload	Via Tiered Storage plugin (Confluent/community)	Not native	Not native	Native S3 offload
Geo-replication	Native via replication cursors	MirrorMaker 2 (operational overhead)	Shovel/federation (limited)	Leaf nodes (different model)	Not built-in
Operational complexity	High (Pulsar + BookKeeper + ZK/Oxia)	Medium (Kafka + ZK, or KRaft)	Low	Low	Low-Medium
Ordering guarantee	Per partition, per key with KeyShared	Per partition	Per queue	Per stream/subject	Per partition
Throughput ceiling	Very high (bookie tier scales independently)	Very high	Moderate	High	Very high
Latency floor	~1-5 ms (journal fsync on NVMe)	~1-5 ms	Sub-millisecond	Sub-millisecond	~1-5 ms
Sweet spot	Multi-tenant streaming with long retention and tiered storage needs	High-throughput event streaming, established ecosystem	Task queues, RPC, low-latency routing	Lightweight pub/sub, IoT, edge	Kafka-compatible without ZooKeeper complexity

Closing Thoughts

Pulsar’s architecture answers a specific question: what does a messaging system look like if you assume storage and compute will scale at different rates and that multi-tenancy is a first-class requirement rather than an afterthought? The answers, BookKeeper-backed managed ledgers, stateless brokers, bundle ownership transfer, namespace-scoped policies, and transparent tiered offload, are coherent and each one follows from the central premise.

The cost is real. Operating Pulsar means operating three distributed systems simultaneously. The bundle-based load balancer adds a layer of indirection that requires understanding before you can debug routing anomalies. Tiered storage reads have higher tail latency than BookKeeper reads. These are not reasons to avoid Pulsar; they are the terms of the tradeoff. For teams that need the capabilities Pulsar provides, the architecture earns its complexity.