How Redpanda Works Internally: Thread-Per-Core Architecture, Raft-Based Replication, and the C++ Streaming Engine That Replaces Kafka Without the JVM

Kafka is the default answer for high-throughput event streaming. It works, it scales, and most engineers know how to operate it. The cost is real, though: JVM garbage collection pauses, ZooKeeper (or KRaft) operational burden, separate Schema Registry and REST Proxy deployments, and a tuning surface that rewards specialization. Redpanda removes that cost not by building a thin wrapper around Kafka’s protocol, but by rebuilding the streaming engine from scratch in C++.

Understanding how Redpanda achieves Kafka API compatibility while delivering lower tail latencies and simpler operations requires examining its architecture at each layer: the execution model, the replication protocol, the storage engine, and the built-in components that replace standalone Kafka ecosystem services.

The Problem with JVM-Based Streaming

Before getting into what Redpanda does differently, it is worth being precise about what the JVM costs in a streaming context.

Kafka brokers handle millions of messages per second. The JVM’s garbage collector periodically stops or slows all application threads to reclaim memory. Stop-the-world GC pauses in modern collectors (G1, ZGC) have improved significantly, but p99.9 latency spikes remain a real concern under high-throughput conditions. Operators tune heap sizes, GC strategies, and pause targets, not because they want to, but because the default behavior is not acceptable in production.

Beyond GC, the JVM’s threading model uses OS threads, which means context switching overhead, shared mutable state protected by locks, and kernel scheduling decisions interleaved with application logic. For a system that lives or dies on I/O throughput, this is not the right foundation.

Redpanda is written in C++ and built on Seastar, a high-performance asynchronous framework developed originally for ScyllaDB. No JVM, no GC, no ZooKeeper dependency.

Seastar and the Thread-Per-Core Execution Model

Seastar’s central idea is simple: assign exactly one application thread per CPU core and never allow that thread to be interrupted by another core’s work.

In a conventional multi-threaded system, threads communicate through shared memory protected by mutexes. Lock contention is the limiting factor at scale. Seastar eliminates this by giving each core exclusive ownership of a subset of data and a queue of tasks. Threads do not share state. Communication between cores happens via explicit message passing through lock-free MPSC (multi-producer, single-consumer) queues.

Each Redpanda shard owns a portion of the partition space. A producer request destined for partition 7 is routed to whichever shard owns partition 7. That shard processes the entire request, writes to its log, triggers replication, and responds, all without acquiring a cross-core lock.

Within a shard, Seastar uses a cooperative task scheduler driven by futures and promises (similar in structure to JavaScript’s event loop, but in C++):

// Conceptual model of Seastar's execution on one core:
// - One OS thread, pinned to one CPU core
// - Tasks are queued as continuations of futures
// - No preemption: a task runs until it explicitly yields

async function handleProduceRequest(request: ProduceRequest): Promise<ProduceResponse> {
  // All of this runs on a single shard without locks
  const validated = await validateRequest(request);
  const offset = await appendToLog(validated);  // async I/O via io_uring
  await replicateToFollowers(offset);           // cross-shard message pass
  return buildResponse(offset);
}

The real implementation uses seastar::future<T> and coroutines, but the mental model maps directly. No thread pool. No context switches between tasks on the same core. The kernel scheduler is bypassed for almost all hot-path work.

io_uring for Storage I/O

Seastar uses io_uring (on Linux 5.1+) for asynchronous disk I/O. Traditional async I/O interfaces (epoll, libaio) require syscall overhead per operation. io_uring submits batches of I/O operations to a shared ring buffer in userspace, reducing syscall count significantly for write-heavy workloads. Redpanda’s log append path benefits directly: a high-throughput producer can submit hundreds of writes per polling cycle without proportional syscall overhead.

Raft Consensus for Partition Replication

Kafka uses a custom ISR (In-Sync Replica) replication protocol managed by ZooKeeper (or KRaft in newer versions). Every Kafka partition has a leader elected through the cluster metadata layer. When a follower falls behind, it drops out of the ISR. Writes are acknowledged when the required number of ISR members confirm.

Redpanda replaces this with Raft, the well-studied consensus protocol from the Ongaro-Ousterhout paper. Each Redpanda partition is a Raft group. The partition leader is the Raft leader. Followers are Raft followers. Leader election, log replication, and membership changes are handled entirely by the Raft protocol without an external coordination service.

The practical differences matter:

No ZooKeeper dependency. Redpanda brokers coordinate entirely among themselves. The metadata store is internal, distributed across the cluster using Raft groups. Deploying Redpanda is a single binary deployment.

Raft log entries map directly to Kafka records. The Raft log is the Kafka partition log. There is no separate replication log that gets applied to a storage layer. A record committed in Raft is a record in the partition log.

Quorum-based acknowledgment. A produce request with acks=-1 waits for the Raft quorum (majority of replicas) to confirm the write before responding. This is semantically identical to Kafka’s acks=-1 with a full ISR, but the durability guarantee is stronger: Raft ensures committed entries can never be overwritten by a new leader election, while Kafka’s ISR protocol has edge cases under certain network partition scenarios.

Leader election is fast. Raft elections complete in milliseconds. Kafka leader elections historically required ZooKeeper coordination. In practice, both are fast enough that the difference only surfaces during chaos engineering.

The Raft implementation in Redpanda is custom (called consensus) and is tightly integrated with Seastar’s async model. Each Raft group runs on the shard that owns its partition data. Heartbeats, vote requests, and append entries RPCs all flow through Seastar’s inter-shard messaging.

The Log Storage Engine

Redpanda’s log storage format is compatible with Kafka at the network protocol level (Kafka Record Batch format), but the on-disk format and I/O path are entirely custom.

Each partition log is a sequence of segments. A segment is a single file. The active segment accepts appends; older segments are read-only. Segment rotation is triggered by size (default 1 GiB) or time.

The storage engine manages its own page cache using a userspace allocator. Rather than relying on the OS page cache, Redpanda reserves a configurable portion of RAM for its data cache and manages eviction explicitly. This eliminates double-buffering (writing to OS cache and application cache simultaneously) and gives Redpanda predictable memory behavior without GC tuning.

Index files track byte offsets and timestamps for efficient position lookups, enabling offset-based and timestamp-based seeks without full segment scans.

Shadow Indexing: Tiered Storage Without a Second System

Redpanda’s shadow indexing is how tiered storage works. The name comes from the design: a lightweight “shadow” index in object storage (S3, GCS, Azure Blob) that maps segment metadata to remote locations, allowing brokers to serve reads for data that is no longer on local disk.

The upload path works like this:

1. When a segment is sealed (rotated), the broker uploads it to object storage.
2. A shadow index entry is created: a small manifest file containing the segment’s offset range, size, and remote path.
3. The local segment can then be deleted to reclaim disk space, subject to configurable retention policies.

The read path is where shadow indexing earns its name. When a consumer requests an offset that is not in local storage, the broker consults the shadow index, downloads the relevant segment chunk from object storage, and serves the data. From the consumer’s perspective, this is transparent: the broker responds with valid records regardless of whether they came from local disk or remote storage.

// Conceptual producer configuration targeting Redpanda
// (uses standard Kafka client - Redpanda is Kafka API compatible)
import { Kafka } from "kafkajs";

const kafka = new Kafka({
  clientId: "order-service",
  brokers: ["redpanda-0:9092", "redpanda-1:9092", "redpanda-2:9092"],
});

const producer = kafka.producer({
  allowAutoTopicCreation: false,
  transactionTimeout: 30000,
});

await producer.connect();

await producer.send({
  topic: "orders",
  messages: [
    {
      key: Buffer.from(order.customerId),
      value: Buffer.from(JSON.stringify(order)),
      headers: {
        "content-type": "application/json",
        "schema-id": Buffer.from("42"),
      },
    },
  ],
});

Shadow indexing is configured per-topic with two key settings:

• redpanda.remote.write: enables segment upload to object storage when true.
• redpanda.remote.read: enables transparent reads from object storage when true.

Setting both to true effectively makes local disk a write buffer and cache, with object storage as the durable long-term store. Retention on local disk can be set to minutes while maintaining days or weeks of accessible history in the shadow index.

Built-In Schema Registry and HTTP Proxy

Kafka deployments typically involve Confluent Schema Registry and Kafka REST Proxy as separate services. Redpanda bundles both into the broker binary.

Schema Registry is exposed on port 8081 by default and implements the Confluent Schema Registry API exactly. Producers register schemas; consumers fetch them. No additional service to deploy, no separate ZooKeeper path for schema storage. Schemas are stored in a dedicated Redpanda topic (_schemas) using the same replication guarantees as any other topic.

HTTP Proxy (Pandaproxy) is exposed on port 8082 and implements the Confluent REST Proxy API. It allows HTTP clients to produce and consume records without a Kafka client library:

// Producing via HTTP Proxy - no Kafka client library required
const response = await fetch("http://redpanda:8082/topics/orders", {
  method: "POST",
  headers: {
    "Content-Type": "application/vnd.kafka.json.v2+json",
  },
  body: JSON.stringify({
    records: [
      {
        key: "customer-123",
        value: { orderId: "ord-789", amount: 149.99, currency: "USD" },
      },
    ],
  }),
});

const result = await response.json();
// result.offsets[0].offset: committed offset

This is particularly useful for edge services or serverless functions where deploying a Kafka client library is impractical.

Wasm-Based Data Transforms

Redpanda 23.2 introduced inline data transforms using WebAssembly. Transforms are user-defined functions that consume records from a source topic, process them, and write results to a destination topic. They run inside the broker process in a Wasm sandbox.

The execution model is straightforward: a transform function is uploaded as a compiled Wasm binary, configured with a source and destination topic, and deployed to the cluster. Redpanda executes the Wasm function within the broker’s data path, co-located with the partition leader for the source topic.

// Transform function compiled to Wasm via TinyGo or other Wasm-target compiler
// Redpanda provides a Go SDK that compiles to Wasm

// Conceptual Go transform (compiled to Wasm and deployed to Redpanda)
// import "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform"
//
// func main() {
//   transform.OnRecordWritten(filterHighValueOrders)
// }
//
// func filterHighValueOrders(e transform.WriteEvent, w transform.RecordWriter) error {
//   var order Order
//   if err := json.Unmarshal(e.Record().Value, &order); err != nil {
//     return err
//   }
//   if order.Amount > 1000 {
//     return w.Write(e.Record())
//   }
//   return nil
// }

The advantage over a separate stream processing layer is co-location: the transform runs on the same machine as the data, avoiding an extra network hop. The constraint is also co-location: Wasm transforms are not suitable for stateful aggregations or joins across topics. They are best for stateless filtering, projection, enrichment from an in-process lookup, or format conversion.

Production Sizing and Tuning

Redpanda’s architecture simplifies many operational concerns but introduces its own sizing considerations.

CPU allocation. Because Redpanda pins one shard to one core, the number of cores directly determines the degree of parallelism. A single-socket server with 16 physical cores runs 16 shards. Partitions are distributed across shards; more cores mean more partitions can be served with full parallelism. Hyperthreading provides diminishing returns for Redpanda because I/O-bound shards spend significant time waiting even without contention.

Memory. Redpanda’s internal cache replaces the OS page cache for hot data. The recommended configuration reserves 25-30% of total RAM for Redpanda, with the kernel managing the rest. The redpanda.memory configuration option sets this limit. Under-allocating causes more reads to fall through to disk; over-allocating starves the kernel’s network and I/O buffers.

Disk. Write-ahead log (Raft log entries) and data segments are written to the same disk by default. For high-throughput workloads, separating the Raft log directory (redpanda.data_directory) onto a dedicated NVMe reduces write amplification and contention. SSDs are effectively required for production; spinning disks produce unacceptable tail latencies at Redpanda’s target throughput levels.

Replication factor. A replication factor of 3 with acks=-1 means a quorum of 2 must acknowledge a write. For the highest durability with minimum latency, all three replicas should be on distinct availability zones with low inter-zone latency. Redpanda does not automatically rack-aware distribute replicas; redpanda.rack annotations must be set and a placement constraint policy must be configured.

Compaction and tiered storage together. Log compaction and shadow indexing can be combined, but the interaction requires care. Compacted topics with shadow indexing enabled will upload uncompacted segments before compaction runs. The shadow index will contain both pre-compaction and post-compaction segments during the transition window. Consumer reads during this window may download pre-compaction segments from object storage even if a newer, compacted version exists locally. Monitor vectorized_storage_compaction_ratio to ensure compaction is keeping up with production rate.

Tradeoffs Comparison

Dimension	Redpanda	Apache Kafka	Apache Pulsar	NATS JetStream	RabbitMQ
Runtime	C++ (no JVM)	JVM (Java)	JVM (Java)	Go	Erlang
Replication protocol	Raft per partition	Custom ISR + KRaft	BookKeeper (quorum ledger)	Raft (NATS cluster)	Quorum queues (Ra / Raft)
External dependencies	None	None (KRaft) or ZooKeeper	ZooKeeper + BookKeeper	None	None
Kafka API compatibility	Native	Reference	Via Kafka protocol proxy	No	No
Tiered storage	Shadow indexing (built-in)	Tiered Storage plugin	Native (BookKeeper offload)	S3-backed (JetStream)	No
Schema registry	Built-in	Separate service	Separate service	No	No
HTTP proxy	Built-in	Separate service	Separate service (Pulsar proxy)	HTTP API built-in	AMQP + HTTP plugin
Data transforms	Wasm (built-in)	Kafka Streams (client-side)	Pulsar Functions	No native	Shovel/Federation plugins
GC pauses	None	Yes (tunable)	Yes (tunable)	None	None
Operational complexity	Low	Medium (KRaft) to High (ZooKeeper)	High	Low	Low
p99 latency at 1M msg/s	2-5ms (reported)	10-50ms (GC dependent)	5-15ms	1-3ms	10-30ms
Best fit	High-throughput Kafka workloads without JVM overhead	Large existing Kafka ecosystems	Multi-tenancy, geo-replication, pub/sub + queue mix	Low-latency internal services, microservices	AMQP workloads, request/reply patterns, task queues

Latency figures are indicative ranges from vendor benchmarks and community reports, not independent measurements. Results vary significantly with hardware, message size, replication factor, and producer configuration.

The Operational Reality

Redpanda’s single-binary deployment is the most immediate operational benefit for teams running small streaming clusters. A three-node Redpanda cluster (for replication factor 3) replaces a Kafka cluster plus ZooKeeper ensemble plus Schema Registry plus REST Proxy. That is a meaningful reduction in service count, networking surface, and monitoring complexity.

The tradeoffs worth tracking in production are:

Wasm transform limitations. The Wasm transform SDK covers stateless, per-record processing well. Anything requiring cross-partition joins, windowed aggregations, or external state must still run in a separate stream processing layer (Flink, Kafka Streams, or similar). Do not architect transforms as a general-purpose stream processing replacement.

Shadow indexing read latency. Object storage reads are slower than local disk reads by an order of magnitude. Consumer applications that routinely seek to old offsets (analytics backfills, audit log queries) will experience higher latency on remote reads. Partition-level prefetch hints and caching are improving across Redpanda versions, but deep historical reads remain slower than a Kafka setup with large local disk allocations.

Seastar’s memory model and debugging. Crashes inside Redpanda produce C++ stack traces, not JVM heap dumps. Operators familiar with Kafka’s JMX metrics and JVM observability tooling will need to adapt to Redpanda’s Prometheus metrics and its own diagnostic tools (rpk). The rpk CLI covers most operational tasks (cluster health, partition leadership, consumer group offsets, topic configuration), but the mental model is different from kafka-topics.sh and kafka-consumer-groups.sh.

Migration path from Kafka. Because Redpanda is Kafka API compatible, migrating producers and consumers requires no code changes. The migration procedure mirrors a rolling broker replacement: add Redpanda brokers to the cluster, migrate partitions, remove Kafka brokers. In practice, most teams run a parallel cluster with consumer groups reading from both, validating parity before cutting over.

Redpanda is not a drop-in replacement in every context. Kafka’s mature ecosystem, extensive connector library, Kafka Streams, and deep tooling integration in managed services (MSK, Confluent Cloud) remain real advantages. But for teams building new streaming infrastructure who want Kafka compatibility without the JVM overhead, Redpanda is a coherent, well-engineered choice.

The core bet is this: if you understand that Kafka’s operational complexity comes largely from its JVM and its ecosystem of satellite services, and you are willing to trade ecosystem depth for architectural simplicity and lower tail latencies, Redpanda’s design pays off in production.