How RocksDB Works Internally: LSM-Tree Storage, Compaction Strategies, and the Embedded Engine Behind Modern Distributed Databases

If you use CockroachDB, TiKV, MyRocks, Yugabyte, or any of a dozen other distributed databases, you are already using RocksDB. It is the storage engine those systems chose precisely because of the tradeoffs it makes: high write throughput, predictable tail latency, and an embeddable C++ library with no separate process to manage. Understanding what happens inside RocksDB makes a meaningful difference when you are diagnosing a write stall at 3 AM, choosing a compaction strategy for a new workload, or deciding whether to use RocksDB at all.

This article traces the execution path from a single Put call to durable bytes on disk, explains how reads are served across multiple levels, and works through the compaction strategies that determine whether you are optimizing for writes, reads, or space.

The Core Architecture: Log-Structured Merge Trees

RocksDB is an LSM-tree (log-structured merge tree) implementation. The central insight of the LSM design is to convert random writes into sequential writes by buffering mutations in memory and flushing them to disk in sorted order. Disk seeks are expensive; sequential writes are cheap. Everything in RocksDB’s architecture flows from that observation.

The trade RocksDB makes is: write performance is excellent, but reads may need to check multiple levels, and background compaction consumes CPU and I/O to keep those levels organized. The three numbers to keep in mind at all times are write amplification, read amplification, and space amplification. Every tuning decision moves one of these at the cost of the others.

The Write Path

Write-Ahead Log

Every Put, Delete, or Merge operation is first appended to the write-ahead log (WAL). The WAL is a sequential log file on disk. Its sole purpose is crash recovery: if the process dies before the MemTable is flushed, the WAL lets RocksDB replay any unflushed writes on restart.

By default, RocksDB groups writes into a batch and issues a single fsync per batch via sync_file_range. You can disable per-write syncing (sync: false) for maximum throughput if you are willing to lose the last few milliseconds of writes on a crash, or enable sync: true to flush after every write for hard durability guarantees. Most production deployments accept a small sync interval (every 1 ms or every 1000 writes) rather than per-write fsync.

import RocksDB from "rocksdb";
import { promisify } from "util";

const db = new RocksDB("/data/mydb");
const open = promisify(db.open.bind(db));
const put = promisify(db.put.bind(db));
const get = promisify(db.get.bind(db));

await open({ sync: false, createIfMissing: true });

// Write is first appended to WAL, then written to MemTable
await put("user:01HX4K9M", JSON.stringify({ name: "Alice", plan: "pro" }));

MemTable

After the WAL append, the write is applied to the active MemTable. The MemTable is an in-memory data structure sorted by key. RocksDB uses a skip list by default (offering O(log n) insertions and lookups), but also supports a hash skip list and a hash linked list for more specific access patterns.

When the MemTable reaches its size limit (default 64 MB), it becomes immutable. A new empty MemTable becomes active, and the old immutable MemTable is scheduled for flushing to disk. Multiple immutable MemTables can accumulate while the background flush thread is busy, up to the configured limit. When that limit is hit, incoming writes stall.

// RocksDB with tuned MemTable configuration
await open({
  createIfMissing: true,
  writeBufferSize: 128 * 1024 * 1024,     // 128 MB MemTable
  maxWriteBufferNumber: 4,                  // allow up to 4 MemTables before stalling
  minWriteBufferNumberToMerge: 2,           // merge 2 immutable MemTables per flush
});

Flushing to L0 SST Files

When an immutable MemTable is flushed, RocksDB writes its contents as a Sorted String Table (SST) file to Level 0 (L0). An SST file is an immutable, sorted, binary file divided into data blocks, index blocks, and a Bloom filter section. The data blocks hold the actual key-value pairs, compressed. The index block holds one entry per data block pointing to the starting key of that block, enabling binary search. The Bloom filter encodes which keys might exist in the file.

L0 SST files are special: they may have overlapping key ranges because each is an independent MemTable flush. This means reads at L0 may need to check every L0 file. Levels 1 and above maintain non-overlapping key ranges, which makes reads there faster.

The Read Path

Lookup Order

A Get for a key in RocksDB checks data sources in recency order:

1. Active MemTable
2. Immutable MemTables (newest first)
3. L0 SST files (newest first, all of them because ranges overlap)
4. L1 and above (binary search within each level because ranges are non-overlapping)

A key found at any layer stops the search. This ordering guarantees that the most recent version is returned. Deleted keys are represented by tombstones (a record with the key and a deletion marker). A tombstone found at any layer masks older versions of that key.

// Demonstrating point lookup behavior
const value = await get("user:01HX4K9M");
// RocksDB checks: active MemTable -> immutable MemTables ->
//   L0 files (all of them) -> L1 binary search -> L2 binary search...
// First match wins.

Bloom Filters

Without Bloom filters, every point lookup on a miss would require reading SST file data blocks at every level until the key is confirmed absent. Bloom filters short-circuit this. Each SST file has a Bloom filter in its metadata section. Before reading any data block from a file, RocksDB queries the Bloom filter. If the filter returns negative, RocksDB skips that file entirely with no disk I/O. Bloom filters have a small false positive rate (typically 1% with 10 bits per key) but no false negatives.

In practice, Bloom filters eliminate most unnecessary disk reads for point lookups. An eight-level database that would otherwise require checking multiple files at each level reduces its I/O to at most one file per level for a real hit, and near-zero I/O for a miss.

// Bloom filters are on by default for block-based table format.
// Configuring filter bits per key affects accuracy vs memory:
await open({
  createIfMissing: true,
  filterPolicy: { bloomBitsPerKey: 10 },   // ~1% false positive rate
  // use 15+ bits for read-heavy workloads with aggressive caching
});

Block Cache

SST data blocks are cached in the block cache, an LRU in-memory cache shared across all SST files. Frequently accessed key ranges stay in the cache across reads without re-reading from disk. The block cache is the single most impactful memory tuning parameter for read-heavy workloads.

RocksDB provides two block cache implementations: LRUCache (standard LRU with sharded locking) and HyperClockCache (a lock-free clock eviction algorithm with lower tail latency under high parallelism).

// Allocate 2 GB block cache shared across the database
await open({
  createIfMissing: true,
  blockCacheSize: 2 * 1024 * 1024 * 1024,
  // pin L0 and L1 index and filter blocks in cache to avoid
  // evicting the data structures you need for every lookup
  cacheIndexAndFilterBlocks: true,
  pinL0FilterAndIndexBlocksInCache: true,
});

Compaction Strategies

L0 SST files accumulate with every MemTable flush. Without compaction, reads would eventually need to check an unbounded number of L0 files. Compaction is the background process that merges SST files, eliminates stale versions and tombstones, and reorganizes data into sorted levels. Choosing the right compaction strategy is the most consequential RocksDB configuration decision.

Leveled Compaction

Leveled compaction is the default and the right choice for most production workloads. Each level beyond L0 has a target size. When a level exceeds its target, RocksDB picks one SST file from that level and merges it with all overlapping files in the next level, writing new SST files. The result is that L1 and above maintain non-overlapping key ranges within each level.

The key property: a point lookup at any level from L1 onward touches exactly one SST file (found via binary search on key ranges). This keeps read amplification bounded regardless of the number of levels.

The cost: leveled compaction has significant write amplification. A key that starts in L0 gets rewritten when compacted to L1, again when L1 compacts into L2, and so on. With a level multiplier of 10, writing to six levels produces write amplification of roughly 30x to 60x. For write-heavy workloads on SSDs with finite write endurance, this matters.

// Leveled compaction configuration
await open({
  createIfMissing: true,
  levelCompactionDynamicLevelBytes: true,    // auto-adjust level targets
  maxBytesForLevelBase: 256 * 1024 * 1024,  // 256 MB L1 target
  maxBytesForLevelMultiplier: 10,            // each level 10x larger
  numLevels: 7,
  targetFileSizeBase: 64 * 1024 * 1024,     // 64 MB SST file target
});

levelCompactionDynamicLevelBytes is worth enabling. Without it, RocksDB fills levels from L1 upward, and your actual data ends up spread across all configured levels even if the dataset is small. With it, RocksDB pins the bulk of data at the deepest level and expands upward only when needed, dramatically reducing space amplification for small to medium datasets.

Universal Compaction

Universal compaction uses a different strategy: instead of maintaining sorted levels with non-overlapping ranges, it keeps a sorted sequence of “sorted runs” (each run can be a single SST or a group). Compaction picks multiple adjacent runs and merges them into one.

The tradeoff is reversed compared to leveled: write amplification is significantly lower (each byte is compacted fewer times), but read amplification and space amplification are higher (multiple overlapping runs means more files to check per read, and two copies of the dataset may coexist during compaction).

Universal compaction makes sense when:

• Writes dominate and write amplification is a bottleneck (NVMe endurance or CPU saturation on compaction threads)
• The working dataset fits in memory so that higher read amplification is hidden by the block cache
• Space is not constrained (you can tolerate up to 2x the raw data size temporarily)

For time-series workloads where writes are always newer than existing keys and old data is rarely read, universal compaction with FIFO deletion can be a strong fit.

FIFO Compaction

FIFO compaction does almost nothing: it accumulates SST files chronologically and deletes the oldest file when total size exceeds the configured limit. There is no merging. Write amplification is 1x (the lowest possible). Read amplification grows linearly with the number of SST files. Space amplification is bounded by the configured size limit.

FIFO is appropriate for append-only time-series data where you want a fixed-size sliding window of recent data and accept that older data is simply evicted. Cache implementations and short-term telemetry storage are the most common fits.

Compaction strategy	Write amplification	Read amplification	Space amplification	Best for
Leveled	High (30-60x)	Low (1 file per level)	Low (1.1x)	Mixed read/write, point lookups
Universal	Low (10x)	Medium	High (up to 2x)	Write-heavy, cache-warmed reads
FIFO	1x	High (grows with files)	Fixed limit	Time-series sliding window

Column Families

RocksDB organizes data into column families, which are logically separate keyspaces within a single database file. Each column family has its own MemTable, SST files, compaction configuration, and bloom filter settings. The WAL, however, is shared across all column families (by default), providing atomicity for writes that span multiple column families.

Column families are the right abstraction when you have data with different access patterns in a single application: a “metadata” column family with leveled compaction and aggressive bloom filters alongside a “events” column family with universal compaction and FIFO cleanup.

import RocksDB from "rocksdb";

// Open database with multiple column families
const db = new RocksDB("/data/mydb");

// Each column family can have distinct compaction, cache, and filter settings.
// In the rocksdb node binding, column families are accessed via separate handles.
// The default column family always exists.

// Batch writes across column families are atomic:
const batch = db.batch();
batch.put("metadata:user:01HX4K9M", JSON.stringify({ plan: "pro" }));
batch.put("events:01HX4K9M:1717900800", JSON.stringify({ type: "login" }));
batch.write((err) => {
  if (err) throw err;
  // both writes committed atomically via WAL
});

Write Stalls and Rate Limiting

Write stalls are one of the most disruptive production behaviors in RocksDB. They occur when background compaction cannot keep pace with write throughput, and RocksDB applies backpressure to protect itself. The specific triggers:

• L0 file count reaches level0_slowdown_writes_trigger (default 20): RocksDB slows writes
• L0 file count reaches level0_stop_writes_trigger (default 36): RocksDB stops writes completely
• Immutable MemTable count reaches max_write_buffer_number: writes stall waiting for flush
• Estimated bytes pending compaction exceed soft_pending_compaction_bytes_limit: write slowdown

Write stalls surface as sudden latency spikes in your application. The correct diagnostic is to check RocksDB’s own statistics and the compaction stats log, not application-level metrics alone.

// Rate limiting prevents compaction from monopolizing I/O,
// but also prevents it from falling behind. Tune based on disk throughput.
await open({
  createIfMissing: true,
  // Allow compaction to use up to 100 MB/s of I/O
  rateLimiter: {
    rateBytesPerSec: 100 * 1024 * 1024,
    refillPeriodUs: 100_000,   // refill token bucket every 100 ms
  },
  // Increase compaction concurrency to help compaction keep pace
  maxBackgroundCompactions: 4,
  maxBackgroundFlushes: 2,
});

The most common write stall cause in practice is insufficient compaction thread budget. The L0 file count grows because the compaction thread cannot process files as fast as MemTable flushes produce them. Increasing maxBackgroundCompactions and ensuring the rate limiter ceiling is high enough to handle peak ingestion resolves most sustained stall scenarios.

Production Considerations

Block size tuning. The default block size is 4 KB. For workloads dominated by large range scans, increasing to 16 KB or 32 KB reduces index block size and improves sequential read efficiency. For point-lookup-heavy workloads, smaller blocks reduce read amplification within a block. Do not change this without measuring; the default is a good starting point.

Compression per level. L0 and L1 are frequently rewritten during compaction, so applying heavyweight compression there wastes CPU without lasting benefit. A typical production configuration uses lz4 for L0 and L1 (fast, low CPU, modest ratio) and zstd for L2 and below (higher CPU at write time, but that data is rarely rewritten). With zstd at level 6, storage requirements for deep levels often drop by 60-70% compared to uncompressed, which matters when the dataset is hundreds of gigabytes.

Monitoring compaction lag. RocksDB exposes compaction statistics via its GetProperty API. The key metric is rocksdb.estimate-pending-compaction-bytes. If this grows monotonically during a write burst and does not recover, you have a compaction capacity problem. Add compaction threads, raise the rate limiter ceiling, or reduce maxBytesForLevelBase to trigger more frequent but smaller compactions.

Write batch grouping. Every write goes through a batch group leader mechanism. The thread that arrives first while the WAL is being synced becomes the leader and absorbs writes from other threads that arrive during the sync. This amortizes WAL fsync cost across concurrent writers automatically. Applications that issue many small single-key writes benefit from this without any code change. Applications that already batch writes manually should continue to do so.

Iterator and snapshot lifecycle. RocksDB iterators and snapshots hold references to SST files and MemTable versions. An iterator held open across a long operation prevents compaction from reclaiming those files. This is the RocksDB equivalent of a WiredTiger long-running transaction holding update chains: it inflates file count and memory usage. Always close iterators and release snapshots as quickly as possible, and never hold them across network calls.

Column family isolation for hot paths. If one column family is write-heavy and another is read-heavy, giving each its own MemTable configuration and compaction strategy is cheaper than trying to find a single configuration that satisfies both. The WAL is still shared, so cross-column-family atomic writes remain available.

Comparison with Related Storage Engines

Engine	Architecture	Write throughput	Read throughput	Space efficiency	Embeddable	Notable users
RocksDB	LSM-tree, multi-level	High	Medium (bloom filters help)	High (with zstd)	Yes (C++)	CockroachDB, TiKV, MyRocks
LevelDB	LSM-tree, two-level	Medium	Medium	Medium	Yes (C++)	Chrome (IndexedDB), early Bigtable clients
Pebble	LSM-tree, RocksDB-compatible	High	Medium	High	Yes (Go)	CockroachDB (replaced RocksDB)
WiredTiger	B-tree + LSM hybrid	Medium	High	Medium	Yes (C)	MongoDB
BoltDB	B-tree, LMDB-style	Low	High	Low	Yes (Go)	etcd (pre-bbolt), Consul

A few observations on this table:

LevelDB is RocksDB’s predecessor. Meta (then Facebook) forked it in 2012 because LevelDB lacked multi-threaded compaction, rate limiting, column families, and serious production tooling. LevelDB is fine for embedded use cases with modest write rates, but it plateaus at sustained high-throughput ingestion.

Pebble is written in Go by the CockroachDB team as a drop-in replacement for RocksDB with the same on-disk format. Its primary advantage is that it avoids the cgo boundary overhead present when a Go application embeds RocksDB. CockroachDB completed its migration from RocksDB to Pebble in 2021 and reports improved tail latency as a result.

WiredTiger (MongoDB’s engine) uses a B-tree primary structure. B-trees have better random read performance than LSM trees by default because a key lives at a stable location in the tree. The cost is write amplification from page splits and random writes to maintain the B-tree structure. WiredTiger adds an LSM mode, but MongoDB uses the B-tree mode.

BoltDB uses a copy-on-write B-tree (LMDB-style). It is effectively read-optimized: reads are cheap, writes require copying pages. etcd moved from plain bolt to bbolt (a maintained fork) and is considering Pebble for future versions. BoltDB is the right choice when reads dominate heavily and write throughput requirements are modest.

Closing Thoughts

RocksDB’s design decisions are visible in the workloads it powers. The WAL-plus-MemTable write path lets distributed databases like TiKV accept writes at line rate and replicate through Raft without waiting for compaction. The multi-level structure with bloom filters makes point lookups fast enough for OLTP-adjacent workloads. Column families let a single embedded engine host metadata, primary data, and secondary indexes with different access patterns.

The hard part of operating RocksDB is not understanding what it does: it is calibrating compaction throughput to match ingestion throughput. A RocksDB instance under-provisioned on compaction threads is one write burst away from a stall. Monitor estimate-pending-compaction-bytes, give the background threads enough CPU and I/O budget, and choose the compaction strategy based on your actual workload amplification tolerance rather than the default. The defaults are a reasonable starting point, not an optimized configuration.