How ScyllaDB Works Internally: Shard-Per-Core Architecture, LSM Storage, and the C++ Engine That Outperforms Cassandra

Cassandra’s architecture was a landmark contribution: a leaderless, peer-to-peer, wide-column store that scale-out horizontally without a single point of failure. But it was designed in the Java era, and it shows. JVM garbage collection pauses, coarse-grained thread pools, and kernel-managed I/O scheduling combine to produce latency distributions that look fine at the p50 but spike at the p99 under real production load.

ScyllaDB reimplements the same CQL API and data model entirely in C++, built on top of Seastar, an asynchronous framework that takes control of CPU scheduling, I/O, memory allocation, and networking at a level the JVM cannot reach. The result is a system that processes more requests on fewer nodes, with tighter tail latencies, because it removes most of the software overhead Cassandra inherits from the Java ecosystem.

This article covers how that works: the threading model, storage engine, replication protocol, consensus mechanism, and what actually happens when you run a query.

The Shard-Per-Core Model

Most servers today have 32, 64, or 128 CPU cores. Traditional database architectures treat all those cores as a shared pool: a thread pool picks up work, acquires locks on shared data structures, does its job, releases the locks, and goes back to the pool. When many cores compete for the same lock, throughput degrades and latency spikes.

ScyllaDB takes the opposite approach. On startup, it detects how many hardware threads are available and spawns exactly one application thread per logical CPU core. Each thread owns a non-overlapping partition of the data on that node. No core touches another core’s data. There are no mutexes protecting shared state, because there is no shared state.

This is the shard-per-core model. A shard in ScyllaDB is the pairing of a CPU core with its exclusive slice of data, memory arena, and I/O queues. When a request arrives, it is routed to the shard that owns the partition key the request targets. That shard handles the entire request lifecycle without ever coordinating with another shard on the same node.

The framework enabling all of this is Seastar. Seastar is a C++ library for building asynchronous, event-driven server applications that run at kernel-bypass speeds. It provides:

• A per-shard event loop backed by io_uring (or epoll on older kernels)
• A futures/promises based concurrency model with continuations rather than threads
• A per-shard memory allocator that eliminates cross-NUMA-node allocation
• User-space networking support (DPDK) for workloads where kernel networking is the bottleneck
• An I/O scheduler that enforces latency and bandwidth priorities

When cross-shard communication is necessary (for example, a multi-partition batch), ScyllaDB uses inter-shard message passing over a lock-free single-producer, single-consumer queue between cores. This is cheap compared to acquiring a mutex, but the system is designed to minimize how often it happens.

// Conceptual illustration of how a client driver routes to a shard.
// In practice, the driver computes the token from the partition key
// and the cluster's token map to pick the owning node, then the
// owning node maps the token to an internal shard.

function tokenForKey(partitionKey: Buffer): bigint {
  // ScyllaDB uses the Murmur3 hash (same as Cassandra) for token computation
  return murmur3Token(partitionKey);
}

function shardForToken(token: bigint, shardCount: number): number {
  // Token space is [-2^63, 2^63). Map linearly to [0, shardCount).
  const normalized = token + BigInt("9223372036854775808"); // shift to [0, 2^64)
  return Number((normalized * BigInt(shardCount)) >> BigInt(64));
}

LSM-Tree Storage Engine

ScyllaDB’s storage engine is an LSM-tree, the same class of structure used by RocksDB, Cassandra, and LevelDB. Writes are fast because they are always sequential: data lands in an in-memory buffer called the memtable, and once the memtable reaches capacity it is flushed to disk as an SSTable (Sorted String Table). SSTables are immutable once written.

Each shard manages its own memtable and its own set of SSTables independently. There is no cross-shard coordination for writes to different partition keys. For a write to a given partition, the path is:

1. Write the mutation to the commit log (a per-shard append-only file for crash recovery)
2. Apply the mutation to the in-memory memtable
3. Return success to the client

The commit log and the memtable write happen concurrently. If the node crashes before the memtable is flushed, the commit log replays the mutations on restart.

SSTables

When a memtable flushes, it produces several files that together form one SSTable:

• Data file: the actual rows, sorted by partition token then clustering key
• Index file: a sparse index mapping partition keys to byte offsets in the data file
• Summary file: a coarser index into the index file, kept in memory
• Bloom filter: a probabilistic structure that answers “does this partition key exist in this SSTable?” without a disk read for the negative case
• Statistics file: metadata including min/max timestamps and tombstone information

Reads require consulting the bloom filter first, then the summary, then the index, then the data file. SSTables accumulate over time, so compaction is necessary to bound read amplification and reclaim space from overwritten and deleted data.

Compaction Strategies

ScyllaDB supports two primary compaction strategies, matching Cassandra’s approach but executing them with less I/O overhead due to the shard-local scheduler.

Size-Tiered Compaction (STCS) groups SSTables by size and merges groups of four (by default) when enough similarly-sized tables accumulate. It produces larger and larger tables over time. Write amplification is low, but space amplification can be high because you temporarily need twice the space of the input during a compaction run. STCS is a good fit for write-heavy workloads where reads hit recent data most of the time.

Leveled Compaction (LCS) organizes SSTables into levels. Level 0 accepts flushes directly. Each level has a size limit 10x larger than the previous level. A compaction job takes one SSTable from a given level and merges it with the SSTables in the next level that overlap its key range. This bounds the total number of SSTables a read must consult (no more than one per level), which significantly reduces read amplification. The cost is higher write amplification because data is rewritten multiple times as it moves through levels.

ScyllaDB also offers Incremental Compaction Strategy (ICS), a proprietary strategy that compacts SSTables in smaller increments to reduce space amplification compared to STCS while retaining reasonable write throughput. ICS is the recommended default for most production workloads on ScyllaDB.

The I/O scheduler mediates compaction’s disk access so that background compaction does not starve foreground queries. Each shard has a configurable I/O weight for compaction versus reads versus writes, and Seastar enforces these weights through a token-bucket-style controller.

CQL Compatibility and the Coordinator Layer

ScyllaDB speaks CQL (Cassandra Query Language), the same wire protocol as Apache Cassandra. Existing drivers, ORMs, and application code work without changes. ScyllaDB passes the Cassandra Query Language compatibility test suite.

When a CQL request arrives, any node in the cluster can act as the coordinator for that request. The coordinator:

1. Parses the CQL statement and resolves the partition key
2. Computes the token for the partition key using Murmur3
3. Consults the token ring to find the replica nodes responsible for that token
4. Forwards the request to the appropriate replicas according to the replication factor and consistency level
5. Waits for enough acknowledgments to satisfy the consistency level (e.g., QUORUM requires floor(RF/2) + 1 responses)
6. Returns the result to the client

If the coordinator node is also a replica for the request’s partition, it handles that replica locally (on the correct shard) without a network round trip to itself.

// Pseudocode: how a coordinator resolves replicas for a token
interface ReplicaSet {
  primary: string;
  replicas: string[];
}

function resolveReplicas(
  token: bigint,
  tokenRing: Map<bigint, string>,
  replicationFactor: number
): ReplicaSet {
  // Walk the ring clockwise from the token to collect RF nodes
  const sorted = [...tokenRing.keys()].sort((a, b) => (a < b ? -1 : 1));
  const startIdx = sorted.findIndex((t) => t >= token) ?? 0;
  const replicas: string[] = [];

  for (let i = 0; i < replicationFactor; i++) {
    const idx = (startIdx + i) % sorted.length;
    const node = tokenRing.get(sorted[idx])!;
    if (!replicas.includes(node)) replicas.push(node);
  }

  return { primary: replicas[0], replicas };
}

Gossip and Cluster Membership

ScyllaDB uses the same gossip protocol as Cassandra for cluster membership and failure detection. Each node periodically (once per second) exchanges state with a small random set of peers. The exchanged state includes:

• Node status (NORMAL, LEAVING, REMOVING, JOINING)
• Token ownership
• Schema version
• Load information
• Generation number (a timestamp used to detect restarts)

The gossip messages propagate information epidemically. A new node’s presence reaches all nodes in O(log N) gossip rounds, where N is the cluster size.

Failure detection uses Phi Accrual, a continuous failure detector that outputs a phi value instead of a binary up/down. The phi value increases as the time since the last heartbeat grows relative to historical heartbeat intervals. Applications set a threshold (the phi_convict_threshold) above which a node is considered down. This approach adapts to network jitter rather than using a fixed timeout.

When a node is detected as down, its responsibilities are temporarily absorbed by its replicas. If the failure is confirmed as permanent, the cluster initiates repair to restore the replication factor.

Lightweight Transactions and Raft Consensus

Cassandra’s lightweight transactions (LWT) were implemented using Paxos, specifically a variant called “single-decree Paxos.” The implementation is expensive: four round trips per transaction, and each round trip goes to all replicas in the Paxos group. Performance degrades significantly under contention.

ScyllaDB replaced the Paxos-based LWT with Raft consensus starting in version 5.0 (for schema changes) and extended Raft to data-level lightweight transactions. Raft is operationally simpler than Paxos and has better-understood failure behavior.

In ScyllaDB’s Raft-based LWT:

• A Raft group is formed per partition (or per table, for schema changes)
• One member of the group is elected leader via Raft’s leader election mechanism
• Write transactions go through the leader, which appends them to its log and replicates to followers
• Linearizability is guaranteed: every read sees all writes that committed before it

Raft requires a majority quorum to commit a write. For a replication factor of 3, that means 2 out of 3 replicas must acknowledge. This is the same requirement as QUORUM consistency, so the performance cost of Raft-based LWT is primarily the serialization overhead and the leader election on first use, not additional replica coordination.

Schema changes in ScyllaDB also go through Raft, which eliminates the schema disagreement bugs that have historically plagued Cassandra clusters during rolling upgrades.

I/O Scheduler and Memory Allocator

Seastar’s I/O scheduler is a priority-based dispatcher that sits between the application and the kernel’s I/O submission queue. Each shard maintains separate I/O queues for:

• Compaction I/O
• Memtable flush I/O
• Foreground read I/O
• Commitlog write I/O

Each queue has a configured share of the disk’s IOPS budget. Under load, the scheduler throttles low-priority queues to preserve IOPS for foreground reads. This is why ScyllaDB’s read latency degrades gracefully during compaction, a scenario where Cassandra often shows sudden p99 spikes.

Memory allocation uses a per-shard allocator based on the “seastar allocator,” which avoids the cross-core cache-line bouncing that the glibc allocator causes when different threads free each other’s allocations. Each shard allocates from its own memory pool, and the allocator tracks per-shard memory usage so that the system can apply backpressure before a shard exhausts its budget.

Streaming and Repair

When a new node joins the cluster, it must receive the data it is now responsible for from existing nodes. This process is called streaming. ScyllaDB streams SSTables over TCP connections between nodes, using the same shard-per-core model: the streaming sender and receiver dedicate individual shards to the transfer to avoid disrupting the request-handling shards.

Repair is the process of reconciling differences between replicas that may have diverged due to node downtime or network partitions. ScyllaDB uses Merkle tree-based repair, the same approach as Cassandra. Each replica computes a Merkle tree over its data for a given token range, and replicas exchange tree hashes to identify the ranges where they disagree. Only the divergent ranges are synchronized, which bounds the data transferred during repair.

ScyllaDB’s repair is significantly faster than Cassandra’s because it parallelizes across shards. Each shard repairs its own token range independently, saturating disk and network bandwidth uniformly across all cores.

Production Considerations

Replication factor and consistency level selection. For most production workloads, RF=3 with QUORUM consistency gives the right balance: one node can fail without losing availability, and reads always see the most recent write because the quorum overlaps with the write quorum. QUORUM requires floor(3/2) + 1 = 2 acknowledgments, which means one replica node can be down before operations start failing.

Compaction strategy selection. Use STCS for write-heavy time-series or append-only workloads. Use LCS for read-heavy workloads with random access patterns. Use ICS (ScyllaDB-specific) as the general-purpose default when you are unsure. Monitor compaction lag with the compaction_manager_completed_tasks metric; sustained lag indicates disk throughput is undersized for the write rate.

Token distribution. Use the default virtual node (vnode) configuration with 256 vnodes per node. Unequal token distribution causes hot nodes. Validate the distribution with nodetool ring or the ScyllaDB Manager UI.

JVM-free capacity planning. ScyllaDB’s memory usage is predictable because there is no GC heap. A common starting point: allocate 50% of RAM to the row cache (if reads are cache-friendly), leave 20% for OS page cache, and let ScyllaDB manage the rest. Monitor cache_hit_rate to validate the allocation.

Tablet-based replication (ScyllaDB 6.0+). ScyllaDB introduced tablets as a replacement for the traditional vnode model. Tablets are finer-grained, movable units of data that allow the cluster to rebalance automatically as nodes are added or removed without manual token assignment. Production clusters on 6.0+ should evaluate enabling tablets for workloads where horizontal scaling is frequent.

Timeouts and backpressure. ScyllaDB applies admission control at the shard level. When a shard’s queue depth exceeds its limit, new requests receive an overloaded error rather than queuing indefinitely. Configure client retry logic with exponential backoff and jitter. Monitor reactor_utilization per shard; sustained values above 0.9 indicate the shard is saturated.

Tombstone accumulation. Deletes in LSM-tree systems are implemented as tombstones, markers that suppress the deleted data. Tombstones accumulate in SSTables and slow down reads until compaction removes them. For workloads with high delete rates, tune gc_grace_seconds to the minimum safe value (the maximum expected node downtime), and ensure repair runs within that window.

Tradeoffs Comparison

Dimension	ScyllaDB	Apache Cassandra	Amazon DynamoDB	CockroachDB	FoundationDB
Architecture	Shard-per-core, C++, Seastar event loop	Thread-pool, JVM, OS-scheduled I/O	Fully managed, opaque internals	Distributed SQL, RocksDB storage	Ordered key-value core, layers on top
Storage model	LSM-tree (memtable + SSTable), shard-local	LSM-tree (memtable + SSTable), JVM-managed	LSM-based (B-tree hybrid, not disclosed)	RocksDB LSM per range	Custom LSM, strictly ordered
Consistency model	Tunable (ONE to ALL), Raft for LWT	Tunable (ONE to ALL), Paxos for LWT	Eventual by default, optional strong per-item	Serializable (SQL transactions)	Strict serializable (ACID)
Latency profile	Sub-millisecond p50, low p99 jitter due to C++ runtime	Low p50, high p99 under GC pressure or compaction	Single-digit ms p50, variable p99	Higher latency due to consensus overhead	Low latency for key-value, higher for complex transactions
Operational complexity	Moderate: capacity planning without GC guesswork, tablet rebalancing in 6.0+	Moderate: GC tuning, repair scheduling, schema migration fragility	Low: fully managed, no ops	Moderate: Kubernetes-friendly but cluster sizing takes care	High: layer model requires deep expertise, no managed offering
Sweet spot	High-throughput, low-latency wide-column workloads needing Cassandra compatibility with better hardware utilization	Existing Cassandra workloads, teams with operational expertise	AWS-native teams needing zero-ops NoSQL	Relational workloads needing horizontal scale and ACID	Systems requiring strict serializability and custom data models built as layers

The core decision between these systems comes down to what you are optimizing for. If you need Cassandra’s data model and API with better tail latency on the same hardware, ScyllaDB is the direct upgrade path. If you need SQL semantics and ACID transactions across distributed data, CockroachDB is the right trade. If operational overhead is the constraint and you are already in AWS, DynamoDB removes most of it at the cost of portability and query expressiveness.

ScyllaDB’s architecture demonstrates what becomes possible when you build a database around hardware primitives instead of around a language runtime. The shard-per-core model is not a micro-optimization; it is a fundamentally different approach to how a server process uses a modern multi-core machine, and the latency numbers show it.