How Cassandra Works: Wide-Column Storage, Gossip Protocol, and Tunable Consistency From Write Path to Compaction

Apache Cassandra is one of the few databases that genuinely delivers on “no single point of failure.” Every node is equal. Writes go anywhere. Reads come back from multiple replicas. The tradeoffs are real, but so is the operational model once you understand what the system is actually doing under the hood.

This article walks through the internals that matter: the data model, write path, read path, compaction, cluster membership, consistency, and repair. The goal is not to repeat the documentation but to explain why each mechanism exists and what breaks when you misuse it.

The Wide-Column Data Model

Cassandra is a wide-column store, not a relational database and not a key-value store. The distinction is precise.

A table has a partition key and an optional set of clustering keys. Together they form the primary key. Within a partition, rows are sorted on disk by the clustering key. This layout enables fast range scans within a partition but makes cross-partition queries expensive by design.

// Cassandra CQL schema
// CREATE TABLE events (
//   tenant_id  uuid,
//   event_time timestamp,
//   event_id   uuid,
//   payload    text,
//   PRIMARY KEY ((tenant_id), event_time, event_id)
// ) WITH CLUSTERING ORDER BY (event_time DESC);

interface EventRow {
  tenantId: string;    // partition key
  eventTime: Date;     // clustering key 1
  eventId: string;     // clustering key 2
  payload: string;
}

All rows sharing the same tenant_id land on the same set of replica nodes. The clustering columns event_time and event_id determine the on-disk sort order within that partition. A query for “last 100 events for tenant X” is a sequential read from the partition head. A query for “events across all tenants in the last hour” is a full cluster scan.

This is not a limitation to work around. It is the design contract. Model your data around your access patterns before you write a single row. If you query by partition key and clustering key, Cassandra is fast. If you scan arbitrarily, you pay the full cost of a distributed scatter-gather.

Internally, each partition maps to a cell structure: a sorted list of column name plus value pairs. The wide-column name comes from this: a single partition can hold millions of cells (clustered rows). There is no fixed row width. This is what separates it from a relational model where every row has the same schema.

The Write Path

Cassandra’s write path is optimized for throughput. Writes never go directly to disk in a random-access pattern. Instead, every write passes through two structures: the commit log and the memtable.

Client write
     |
     v
Commit Log (sequential, append-only, per-node)
     |
     v
Memtable (in-memory sorted structure, per-table)
     |
     v (when memtable is full or flush threshold hit)
SSTable (immutable, sorted, written to disk)

The commit log is a sequential append-only file on disk. Its only job is durability: if the node crashes before the memtable flushes, the commit log replays on restart. Once a memtable has been flushed to an SSTable, the corresponding commit log segment is discarded.

The memtable is an in-memory sorted map keyed by partition key, then by clustering columns. Writes accumulate here until the memtable reaches a configured size threshold (default: 256 MB per table) or a commit log size limit triggers a flush. The flush produces an immutable SSTable on disk and the memtable is cleared.

This means a single write involves: one sequential disk write (commit log) and one in-memory update. That is why Cassandra’s write latency is consistently low regardless of dataset size. There is no read-before-write for most operations. You do not need to look up the current state of a row to write to it (with the exception of lightweight transactions, which use Paxos and are expensive).

SSTables (Sorted String Tables) are immutable files on disk. Once written, they are never modified. An update is a new cell with a higher timestamp. A delete is a tombstone: a special cell that records “this data was deleted at time T.” The actual data is reclaimed during compaction.

Each SSTable consists of several files on disk:

• Data.db: the actual cell data
• Index.db: a sparse index of partition keys to byte offsets in Data.db
• Summary.db: an in-memory index of the index (a sample of Index.db)
• Filter.db: a Bloom filter over partition keys
• Statistics.db: metadata including min/max clustering keys, tombstone counts, compression info

The Read Path

Reading from Cassandra is more complex than writing because data for a single partition may be spread across multiple SSTables on disk plus the current memtable.

Client read (partition key: tenant_id = "abc-123")
     |
     v
Check memtable (in-memory, most recent writes)
     |
     v
Check row cache (if enabled, rarely recommended)
     |
     v
For each SSTable (newest to oldest):
  1. Check Bloom filter -- partition definitely absent? Skip.
  2. Check key cache (cached byte offset in Data.db)
  3. If cache miss: consult Summary.db -> Index.db -> seek Data.db
     |
     v
Merge results from all sources, resolve by timestamp
     |
     v
Apply tombstones, return surviving cells

The Bloom filter is the critical optimization here. A Bloom filter is a probabilistic structure: it can tell you with certainty that a key does not exist in an SSTable (no false negatives), but it has a configurable false positive rate (the filter says the key might be present, so you do a disk seek, and the key is not there). The default false positive rate in Cassandra is 0.1% (1 in 1000 lookups triggers an unnecessary seek). Tuning bloom_filter_fp_chance per table is one of the highest-leverage performance knobs you have.

Once cells are read from memtable and all SSTables, they are merged by partition key and clustering key. When the same cell exists in multiple SSTables (because of updates), the one with the highest timestamp wins. This is last-write-wins (LWW) conflict resolution. Cassandra does not use vector clocks for this. It uses wall clock timestamps from the client. If your clients have clock skew, you can lose writes silently.

Read repair happens during the read path when consistency levels above ONE are used. If a coordinator reads from two replicas and they disagree, it sends the newer data to the stale replica in the background (or synchronously with CL=QUORUM). This is a passive form of anti-entropy but is not a substitute for full repair.

Compaction Strategies

Because writes produce new SSTables and never modify old ones, SSTables accumulate over time. Compaction merges SSTables, eliminates superseded cells, and purges tombstones that have exceeded the gc_grace_seconds window (default: 10 days).

Choosing the wrong compaction strategy is one of the most common Cassandra misconfigurations in production. There are three strategies that matter:

STCS: Size-Tiered Compaction Strategy

The default. Cassandra groups SSTables into tiers by size. When four or more SSTables of similar size accumulate, they are merged into one larger SSTable.

Good for: write-heavy workloads where you rarely read old data (time series, event logs, append-only patterns).

Bad for: read-heavy or update-heavy workloads. Because each tier can still hold multiple SSTables, reads may scan many files. Also, compaction itself is expensive: it requires reading all input SSTables and writing the output, temporarily doubling disk usage.

LCS: Leveled Compaction Strategy

Organizes SSTables into levels (L0, L1, L2, …). Each level contains non-overlapping SSTables of a fixed size (default: 160 MB). When L0 grows, it is merged into L1. When L1 overflows, SSTables are promoted to L2, and so on.

Good for: read-heavy workloads where you want predictable read amplification. At any given level, a partition exists in at most one SSTable. A read touches at most one SSTable per level.

Bad for: write-heavy workloads. Write amplification is high because moving an SSTable from L0 to L5 means rewriting it multiple times. Disk space usage is more predictable (around 10x the SSTable size), but the compaction I/O budget is significant.

TWCS: Time-Window Compaction Strategy

Designed specifically for time series data where old data is never updated. Partitions data into time windows (configurable: 1 day, 1 hour, etc.). Within a window, it runs STCS. Once a window closes (all writes have moved past it), that window’s SSTables are compacted once into a single SSTable and never touched again.

Good for: IoT sensor data, metrics, logs, any append-only time series.

Critical requirement: your TTLs must align with your window sizes, and all data in a closed window must have passed its TTL before tombstones are purged. If you write to old time windows, TWCS behaves poorly.

// Pseudocode: choosing a compaction strategy
function chooseCompactionStrategy(
  workload: "write-heavy" | "read-heavy" | "time-series"
): string {
  if (workload === "time-series") return "TWCS";
  if (workload === "read-heavy") return "LCS";
  return "STCS"; // write-heavy or mixed
}

Gossip Protocol and Cluster Membership

Cassandra has no central coordinator, no leader, no metadata server. Nodes discover each other and share state through the gossip protocol.

Every second, each node initiates a gossip round with one to three randomly selected peers. In each round, nodes exchange their local state (heartbeat counter, node status, load, schema version) and the states of other nodes they know about. Each piece of state carries a version number. The node with the higher version wins the merge.

The state a node gossips about includes:

• STATUS: NORMAL, LEAVING, LEFT, REMOVING, REMOVED
• LOAD: disk usage, used to inform token placement
• SCHEMA: schema version UUID, to detect schema disagreements
• DC and RACK: datacenter and rack topology for replication placement
• TOKENS: the vnode token assignments (covered next)

Because gossip is epidemic, state propagates to all N nodes in O(log N) rounds. A cluster of 1000 nodes reaches full dissemination in roughly 10 rounds (10 seconds). This is eventually consistent cluster membership. There is a window after a node failure where other nodes disagree about whether it is dead.

Failure detection uses a Phi Accrual Failure Detector (borrowed from Akka). Rather than a binary up/down judgment, each node maintains an estimate of the heartbeat arrival distribution for each peer. The phi value represents confidence that a peer has failed. When phi exceeds a threshold (default: 8), the node is marked as DOWN. This avoids false positives from transient GC pauses or network jitter.

Consistent Hashing and Virtual Nodes

Cassandra uses consistent hashing to map partition keys to nodes. A partition key is hashed using Murmur3 (the default partitioner) to produce a 64-bit token in the range [-2^63, 2^63). The hash ring represents this token space. Each node is responsible for a range of tokens.

Without virtual nodes (vnodes), each node owns one contiguous token range. Adding a node requires manually splitting one existing range and moving half its data. Removing a node requires manually merging ranges.

Vnodes (enabled by default since Cassandra 1.2) assign each physical node multiple non-contiguous token ranges, typically 256 per node. This distributes each node’s responsibility across many small arcs of the ring rather than one large arc.

Physical cluster: 4 nodes, 8 vnodes per node (simplified)

Ring: [T0 ... T_max]

Node A owns tokens: [T1, T5, T12, T17, ...]
Node B owns tokens: [T2, T7, T10, T19, ...]
Node C owns tokens: [T3, T8, T14, T21, ...]
Node D owns tokens: [T4, T9, T16, T23, ...]

When a node is added, it picks random token positions and takes ownership of small slices from many existing nodes. Data migration is automatically distributed across the cluster rather than loading a single source node. This also means a node failure distributes the load recovery across all remaining nodes instead of concentrating it on one neighbor.

The downside of vnodes is higher coordination overhead per operation. A request touching a partition must be routed to the node that owns that partition’s token. The coordinator node resolves this using its local copy of the ring state from gossip.

Replication and Consistency

The replication factor (RF) determines how many nodes store a copy of each partition. With RF=3, every partition lives on three nodes. The replication strategy determines which three nodes those are.

SimpleStrategy places replicas on the next N nodes clockwise on the ring. Used only for single-datacenter setups.

NetworkTopologyStrategy is what you use in production. It places replicas per datacenter separately, taking rack topology into account to ensure replicas are not all on the same physical rack.

CREATE KEYSPACE my_app
WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'us-east-1': 3,
  'eu-west-1': 2
};

Tunable Consistency

This is where Cassandra’s contract becomes explicit. For every read and write, you specify a consistency level. The database does not pick one for you.

Consistency Level	Behavior
ONE	Succeed after acknowledgment from 1 replica
TWO	Succeed after acknowledgment from 2 replicas
THREE	Succeed after acknowledgment from 3 replicas
QUORUM	Succeed after ceil(RF/2 + 1) replicas respond
LOCAL_QUORUM	QUORUM within the local datacenter only
EACH_QUORUM	QUORUM in every datacenter
ALL	All replicas must respond
LOCAL_ONE	1 replica in the local datacenter
ANY	Write accepted even if all replicas are down (stored as hinted handoff)

For strong consistency: write at QUORUM and read at QUORUM (or write at ALL and read at ONE). With RF=3 and QUORUM=2, every write touches at least 2 replicas and every read touches at least 2 replicas. Because 2 + 2 > 3, at least one replica in every read has seen the latest write.

For availability over consistency: write at ONE and read at ONE. You may read stale data but the cluster stays available during partial failures.

// Conceptual consistency check
function isStronglyConsistent(rf: number, writeCL: number, readCL: number): boolean {
  // writeCL + readCL > rf guarantees at least one overlap
  return writeCL + readCL > rf;
}

// RF=3, QUORUM=2: 2 + 2 = 4 > 3 → true
// RF=3, ONE=1:   1 + 1 = 2 < 3 → false
console.log(isStronglyConsistent(3, 2, 2)); // true
console.log(isStronglyConsistent(3, 1, 1)); // false

LOCAL_QUORUM is what most multi-datacenter deployments should use for reads and writes. It satisfies quorum within the local datacenter and avoids cross-datacenter round trips for every operation, while still tolerating the loss of one replica locally.

Hinted Handoff

When a write arrives and a target replica is temporarily unavailable (network partition, restart, GC pause), the coordinator stores the write as a hint locally. The hint is a record of the mutation that was not delivered. When the target replica comes back online, the coordinator replays the hints to bring it back in sync.

This is why CL=ANY exists: the coordinator can accept a write even if zero replicas are reachable, storing it as a hint. This maximizes availability at the cost of all durability guarantees (the write is only on the coordinator; if that node also fails, the write is gone).

By default, hints are stored for up to 3 hours (max_hint_window). If a node is down longer than that, its hints are discarded and it needs a full repair to resync.

Hinted handoff is not a substitute for repair. It covers short outages. For longer outages, or for detecting data divergence that accumulated without a node being explicitly down, you need anti-entropy repair.

Anti-Entropy Repair

The nodetool repair command synchronizes data between replicas by computing a Merkle tree over each partition range. Two nodes compare their Merkle trees: if a subtree hash matches, that range is in sync and requires no data transfer. If hashes differ, the differing leaf nodes identify exactly which token ranges need streaming.

Node A Merkle tree:    Node B Merkle tree:
  [root: H1]             [root: H2]  <- different, dive deeper
    [H3] [H4]              [H3] [H5] <- right subtree differs
      ...  [H6] [H7]         ...  [H8] [H7]
               ^-- H6 != H8, stream this range

Full repair rewrites every SSTable it touches and is CPU/IO intensive. The common practice is incremental repair (Cassandra 4.0+), which marks repaired SSTables with a special flag. Subsequent repair rounds only compare unrepaired SSTables, making repeated repairs much cheaper.

Repair frequency depends on your gc_grace_seconds setting. You must run repair on every node at least once within gc_grace_seconds (default 10 days) to ensure tombstones are propagated before they are eligible for purging. A tombstone that gets purged on one replica but has not propagated to another will cause deleted data to “resurrect” when the replica streams its pre-purge SSTable during future repair.

Tradeoffs Table

Dimension	Cassandra	DynamoDB	ScyllaDB	CockroachDB	MongoDB
Consistency model	Tunable (eventual to strong)	Eventually consistent (strong opt-in per-request)	Tunable (same as Cassandra)	Serializable (CockroachDB is ACID)	Tunable (snapshot isolation, causal)
Write throughput	Extremely high (append-only path)	High (managed, auto-scaled)	Higher than Cassandra (C++ runtime, no JVM)	Moderate (Raft consensus per write)	High with write concern 1
Read latency	Low with QUORUM, varies with SSTable count	Low (single-digit ms SLA)	Lower than Cassandra consistently	Higher (Raft round trip)	Low with replicated reads
Data model	Wide-column, query-by-partition	Key-value with optional sort key	Wide-column (CQL compatible)	Relational (SQL)	Document (BSON)
Horizontal scaling	Linear, masterless	Transparent (managed)	Linear, masterless	Linear (Raft-based partitioning)	Horizontal with sharding
Operational overhead	High (repair, tuning, compaction)	None (fully managed)	High (similar to Cassandra)	Moderate	Moderate
Multi-region	Strong support (NetworkTopologyStrategy)	Global Tables with eventual sync	Strong support	Strong support (global CockroachDB)	Atlas global clusters
Joins	None (denormalize)	None	None	Full SQL joins	Limited ($lookup)
Transactions	Lightweight transactions (Paxos, slow)	Transactions (up to 100 items)	Lightweight transactions	Full ACID transactions	Multi-document transactions
Sweet spot	High-throughput append-heavy workloads with known access patterns	Serverless, unpredictable scale, AWS-native	Cassandra workloads with lower latency requirement	OLTP requiring SQL and ACID	Semi-structured data, flexible schema

Production Considerations

Schema design is not flexible after the fact. Changing a partition key requires rewriting the entire table. Model your data around queries before you write anything.

Monitor SSTable count per table. More than 20 SSTables on a read-heavy table means your compaction is not keeping up. Check nodetool tpstats and nodetool tablestats. Uncontrolled SSTable growth degrades read latency quadratically.

Tombstone accumulation kills read performance. A partition with millions of tombstones requires scanning all of them on every read before compaction purges them. If your workload involves frequent deletes, track tombstone counts with nodetool tablestats and tune gc_grace_seconds down if you can guarantee repair runs more frequently.

Clock skew causes silent data loss. Cassandra uses LWW with client-supplied timestamps. If a client writes with a timestamp 5 seconds in the future, a subsequent write from a different client with an accurate clock will be silently discarded. Use NTP or PTP and monitor clock drift across nodes and application servers.

Run repair before gc_grace_seconds expires, not after. The 10-day default means you need at least one successful repair cycle per node every 10 days. Use a repair scheduler (reaper.io is the standard tool) rather than cron jobs. Missed repair cycles lead to data inconsistency that is hard to detect and expensive to fix.

LOCAL_QUORUM is the right default for most multi-datacenter deployments. QUORUM works but adds cross-datacenter latency to every request. ALL is almost never correct in production because a single unavailable replica blocks every write.

JVM tuning matters for Cassandra, not for ScyllaDB. If you are running Cassandra on the JVM, G1GC with a 8-16 GB heap is the current recommendation. Large heaps cause long GC pauses that trigger false failure detection. ScyllaDB’s C++ runtime eliminates this class of problem entirely but introduces its own operational surface.

Cassandra is a strong fit for workloads where you write far more than you read, where access patterns are known at schema design time, and where multi-datacenter active-active replication is a requirement. Its operability ceiling is high but so is the floor: you need to understand the internals to run it well. The abstractions here are deliberately thin. That is the design.