How YugabyteDB Works Internally: DocDB Storage, Tablet Splitting, and the Dual API Architecture That Scales PostgreSQL Horizontally

YugabyteDB takes a different approach to distributed SQL than CockroachDB or TiDB. Where CockroachDB builds its own KV engine and puts PostgreSQL compatibility on top as an optional layer, and TiDB ports the MySQL dialect onto a separate storage tier, YugabyteDB forks PostgreSQL’s actual query layer (the full pg source tree) and replaces only the storage engine beneath it. The result is a system where YSQL is not a compatibility shim but a genuine PostgreSQL instance, with PostgreSQL’s parser, planner, and executor intact, routing storage calls down into a distributed engine called DocDB.

Understanding how YugabyteDB works requires understanding three independent systems: DocDB (the storage layer), the tablet abstraction (the unit of distribution and replication), and the dual API surface that exposes the same data through both a PostgreSQL and a Cassandra-compatible interface.

The DocDB Storage Engine

DocDB is the foundation of YugabyteDB. It is a distributed document-oriented key-value store built on top of RocksDB, with Raft consensus layered at the tablet level for replication and fault tolerance.

Every piece of data in DocDB lives in a structured key-value encoding. For YSQL, a row in a table maps to one or more DocDB keys, where the key encodes the table UUID, the primary key column values in a byte-comparable format, and a column ID. The value contains the column data. For tables with a single-column primary key, a row produces one KV entry per non-primary column. For composite keys, the encoding preserves sort order across all key components so range scans work correctly over the RocksDB LSM tree.

The LSM tree is central to DocDB’s write path. Every write lands in RocksDB’s MemTable first, gets flushed to an L0 SSTable on disk, and then moves through compaction levels. YugabyteDB inherits all of RocksDB’s compaction strategies but applies one important modification: it replaces RocksDB’s WAL with Raft’s log. The Raft log serves the same durability purpose the WAL does in single-node RocksDB: before a write is acknowledged, it is committed to the Raft log on a quorum of replicas. RocksDB’s internal WAL is disabled per tablet because it would be redundant.

DocDB also introduces a document model over the raw KV encoding. A single logical document (a row in YSQL, a partition row in YCQL) is stored as a set of keys that share a common key prefix, called the document key. This allows reads of individual columns without deserializing the entire row, and it enables sparse column storage: columns that have never been written do not consume space.

// Conceptual DocDB key encoding for a YSQL row
// Table: orders(order_id UUID PK, customer_id UUID, amount NUMERIC, status TEXT)

interface DocDBKey {
  tableId: string;          // table UUID, 16 bytes
  primaryKey: Uint8Array;   // encoded primary key columns (byte-comparable)
  columnId: number;         // column ID, identifies which column this KV holds
  timestamp: HybridLogicalTimestamp; // MVCC version
  writeType: "regular" | "tombstone" | "merge";
}

// A single row produces N KV entries: one per non-PK column
// The document key is (tableId + primaryKey), shared by all column entries
const documentKey = Buffer.concat([tableIdBytes, encodedOrderId]);

MVCC is implemented at the DocDB layer, not in the SQL engine. Every write stamps the KV entry with a hybrid logical clock (HLC) timestamp. Older versions accumulate below the current version and are cleaned up by compaction. Reads specify a read timestamp and see only versions at or below that timestamp, giving snapshot isolation without read locks.

Tablets: The Unit of Distribution

A tablet is YugabyteDB’s equivalent of a CockroachDB range or a TiDB region. It is a contiguous interval of the sorted key space for a given table, stored in a RocksDB instance on a specific node, and replicated to a configurable number of peers (default: 3) via a per-tablet Raft group.

Tables are hash-partitioned or range-partitioned into tablets at creation time. Hash partitioning (the default in YCQL, optional in YSQL) distributes tablet assignment using a 16-bit hash of the partition key, spreading writes evenly at the cost of range scan efficiency. Range partitioning preserves sort order for range queries but concentrates sequential inserts on the final tablet, requiring explicit pre-splitting or relying on automatic splitting.

-- YSQL: create a hash-partitioned table (distributes writes evenly)
CREATE TABLE orders (
  order_id  UUID    PRIMARY KEY,
  customer_id UUID  NOT NULL,
  amount    NUMERIC NOT NULL,
  created_at TIMESTAMPTZ NOT NULL
) SPLIT INTO 12 TABLETS;

-- YSQL: range-partitioned with explicit pre-split boundaries
CREATE TABLE time_series_events (
  tenant_id  UUID,
  event_time TIMESTAMPTZ,
  payload    JSONB,
  PRIMARY KEY (tenant_id, event_time)
) SPLIT AT VALUES (
  ('00000000-0000-0000-0000-000000000000', '2024-01-01'),
  ('55555555-5555-5555-5555-555555555555', '2024-01-01')
);

Automatic Tablet Splitting

YugabyteDB can split tablets automatically when they exceed a size threshold (512 MB by default, configurable with tablet_split_size_threshold_bytes). The split process is coordinated by the master server:

1. A tablet reaches the split threshold.
2. The master server selects a split key, typically the median key of the tablet’s key space.
3. The leader replica of the Raft group proposes a split operation through the Raft log.
4. After commit, the original tablet’s data is divided at the split key. Two child tablets are created, each inheriting a subset of the parent’s SSTable files.
5. The master server schedules rebalancing: if one child ends up on an overloaded node, it is moved to a node with spare capacity.

Splitting is non-blocking for reads and writes. Clients that were talking to the parent tablet get redirected to the appropriate child by the tablet server, which holds the routing metadata in memory until the master server propagates the updated tablet map.

Per-Tablet Raft Consensus

Each tablet forms an independent Raft group. There is no global consensus step, which is the same architectural choice CockroachDB makes with ranges and TiDB makes with regions. Horizontal scalability comes from the fact that adding nodes adds capacity for more Raft groups without increasing the coordination load on any existing group.

The Raft leader for a tablet is also the tablet’s primary read/write server. Reads at strong consistency go to the leader. Follower reads (available under snapshot isolation with a configurable staleness bound) serve reads from any replica, useful for geographically distributed deployments where you want to route reads to the closest replica.

// Conceptual write path for a single-shard YSQL INSERT
// Application -> TiDB server... wait, wrong article.
// Application -> YSQL node -> DocDB tablet leader -> Raft quorum

interface RaftEntry {
  index: bigint;
  term: number;
  operation: DocDBOperation;
  hybridTime: HybridLogicalTimestamp; // encoded into every entry
}

interface DocDBOperation {
  type: "write" | "delete" | "transactionApply";
  keys: DocDBKeyValue[];
  transactionId?: string; // set for distributed transactions
}

// The YSQL node calls into DocDB via an internal RPC
// DocDB's tablet leader:
// 1. Assigns an HLC timestamp (max of local clock and peer clocks seen recently)
// 2. Proposes the entry to the Raft group
// 3. Waits for quorum acknowledgement (n/2 + 1 replicas wrote to their WAL)
// 4. Applies to RocksDB MemTable
// 5. Returns the commit timestamp to the YSQL layer

Raft log entries include an HLC timestamp set by the leader. Followers advance their own HLC when they process entries, ensuring all replicas maintain a causally consistent view of time. This is how YugabyteDB achieves linearizability per-tablet without relying on GPS receivers or atomic clocks.

Hybrid Logical Clocks and Distributed MVCC

YugabyteDB uses hybrid logical clocks for all versioning, the same mechanism CockroachDB uses. An HLC combines a physical wall-clock component with a logical counter. The physical component tracks real time; the logical component establishes ordering within the same physical millisecond. Every node updates its HLC when it sends or receives an RPC, ensuring that causally related events always have increasing HLC values even across nodes.

For multi-shard transactions, a transaction coordinator (one of the YSQL nodes or a dedicated participant) picks a read timestamp from the local HLC and a provisional commit timestamp. Writes go to each involved tablet’s transaction status tablet first as provisional records, which are visible only to the writing transaction. On commit, the coordinator sends an APPLY record to each involved tablet, converting provisional records into regular MVCC versions stamped with the commit timestamp.

// Simplified distributed transaction flow
interface Transaction {
  id: string;          // UUID generated at transaction start
  readTimestamp: HybridLogicalTimestamp;
  status: "pending" | "committed" | "aborted";
  participantTablets: string[]; // tablet IDs with provisional writes
}

// Provisional record structure in DocDB
interface ProvisionalRecord {
  documentKey: Uint8Array;
  transactionId: string;
  value: Uint8Array;
  // Not visible to other readers until transaction status is COMMITTED
}

// On commit: the status tablet records commit timestamp
// Each participant tablet then applies provisional -> regular MVCC version
interface TransactionApplyRecord {
  transactionId: string;
  commitTimestamp: HybridLogicalTimestamp;
  // Participants resolve their provisional records against this timestamp
}

Read-committed and snapshot isolation are both available in YSQL. The default is snapshot isolation, which is stronger than what standard PostgreSQL offers by default (PostgreSQL defaults to read-committed). Serializable isolation is also available, implemented via conflict detection at commit time rather than predicate locks, with automatic retry on conflict.

The Dual API: YSQL and YCQL

YugabyteDB exposes two independent query layers over the same DocDB storage layer.

YSQL is the PostgreSQL-compatible API. YugabyteDB ships a patched version of the PostgreSQL process, with storage engine calls redirected from PostgreSQL’s buffer manager and heap access methods to DocDB. The query planner, EXPLAIN output, function catalog, extension framework, and wire protocol are all unmodified PostgreSQL. This means pg_stat_statements, pg_stat_activity, EXPLAIN (ANALYZE, BUFFERS), and most PostgreSQL extensions work as expected.

YCQL is the Cassandra-compatible API. It exposes a subset of Cassandra Query Language 3.x, with a driver-compatible wire protocol. YCQL maps natively to DocDB’s document model: partition keys map to DocDB document keys, clustering keys map to range-encoded column prefixes within the document. YCQL supports time-to-live at the row and column level, which maps to expiry metadata on DocDB KV entries handled during compaction.

The two APIs share the same underlying tablet map and Raft groups. A table created in YSQL and a table created in YCQL are both DocDB tablets; the API layer determines how keys are encoded and which query planner handles the statement.

// Connecting to YSQL (PostgreSQL wire protocol)
import { Pool } from "pg";

const ysqlPool = new Pool({
  host: "yugabyte-node-1",
  port: 5433,         // YSQL port (not 5432)
  database: "yugabyte",
  user: "yugabyte",
  password: process.env.YB_PASSWORD,
  max: 20,            // keep pool small; YB handles connection routing internally
  idleTimeoutMillis: 30_000,
  connectionTimeoutMillis: 5_000,
});

// Connecting to YCQL (Cassandra driver)
import { Client } from "cassandra-driver";

const ycqlClient = new Client({
  contactPoints: ["yugabyte-node-1", "yugabyte-node-2", "yugabyte-node-3"],
  localDataCenter: "us-east-1",
  keyspace: "production",
  pooling: { coreConnectionsPerHost: { local: 2, remote: 1 } },
});

// YCQL is better for time-series or wide-column patterns with TTL
await ycqlClient.execute(
  `INSERT INTO sensor_readings (device_id, ts, value)
   VALUES (?, ?, ?) USING TTL 2592000`, // 30-day TTL
  [deviceId, timestamp, reading],
  { prepare: true }
);

Choosing between YSQL and YCQL is primarily a data model decision. YSQL wins for relational workloads with joins, foreign keys, secondary indexes with complex predicates, and stored procedures. YCQL wins for wide-column, time-series, or high-throughput append workloads where the partition key is always known, TTL is useful, and Cassandra driver compatibility matters.

xCluster Replication for Multi-Region Deployments

YugabyteDB offers two strategies for multi-region operation.

Multi-zone single-region places the three Raft replicas across availability zones within one region. Writes commit synchronously to a quorum across zones, giving fault tolerance against a single AZ failure with no additional latency beyond intra-region RTT.

xCluster replication links two or more independent YugabyteDB clusters across regions. It is asynchronous, similar in concept to PostgreSQL streaming replication or MySQL binlog shipping. Change records flow from a source cluster’s tablet leaders to the target cluster’s tablet leaders via a replication stream. Lag is a function of inter-region latency and write throughput.

xCluster supports bidirectional replication (active-active), which allows writes to both clusters simultaneously. Conflict resolution in active-active mode uses “last writer wins” based on HLC timestamp. This means: if the same primary key is updated on both clusters before the replication stream delivers the update, the higher HLC timestamp wins and the lower one is silently discarded. Applications that cannot tolerate this must route writes for a given partition key to a designated “home” cluster and treat the other as a read replica.

// Pattern: route writes to the "home" region for a given tenant
// Prevents active-active conflicts for critical data

function getWritePool(tenantId: string): Pool {
  const homeRegion = tenantRegionMap.get(tenantId) ?? "us-east-1";
  return regionPools[homeRegion]; // pre-configured Pool per region
}

// Reads can go to the local cluster regardless of home region
function getReadPool(preferredRegion: string): Pool {
  return regionPools[preferredRegion];
}

// xCluster lag monitoring via YugabyteDB master API
async function getReplicationLagMs(masterAddr: string): Promise<number> {
  const resp = await fetch(`http://${masterAddr}:7000/api/v1/tablet-replication`);
  const data = await resp.json();
  // Find max lag across all replication streams
  return Math.max(...data.streams.map((s: { lag_ms: number }) => s.lag_ms));
}

For fully synchronous multi-region commits, YugabyteDB supports placing Raft replicas across regions using geo-partitioning and tablespace placement policies. A write to a geo-partitioned table waits for quorum across the assigned regions before returning, giving synchronous replication at the cost of cross-region round-trip latency on every write. This is the right model for regulatory data residency requirements, not for general-purpose global latency optimization.

Production Considerations

Connection pooling. YSQL uses the same process-per-connection model as PostgreSQL, which means 500+ raw connections saturate the TServer processes quickly. Deploy PgBouncer or a YugabyteDB-aware connection pool in front of the YSQL port. YugabyteDB’s own YSQL Connection Manager (built into recent versions) provides built-in connection multiplexing without a separate sidecar.

Tablet count and overhead. Each tablet has a Raft group, an in-memory data structure for the tablet map, and background compaction threads. A cluster with too many tablets (common after aggressive splitting or creating many small tables) puts memory and thread pressure on TServer processes. Monitor tserver_tablets_num per node and cap total tablets relative to node memory. A rough guideline: 2,000-5,000 tablets per TServer is manageable; above 10,000 per node, you will see elevated compaction latency and memory pressure.

Write amplification. DocDB inherits RocksDB’s write amplification. Every write to the MemTable eventually fans out through multiple compaction levels. With the default leveled compaction strategy, expect 10-30x write amplification on sustained workloads. For write-heavy time-series patterns, consider FIFO or TWCS-style compaction (YugabyteDB exposes compaction strategy configuration per table). Monitor rocksdb_bytes_written against rocksdb_actual_delayed_write_rate to detect when compaction cannot keep up with ingestion.

Hotspot detection. Range-partitioned tables with monotonic primary keys accumulate all new writes on the last tablet. Use yb-admin list_tablets to see per-tablet write rates and identify imbalanced tablets. Pre-split at creation time when insert patterns are predictable, or switch to hash partitioning when they are not.

EXPLAIN and statistics. YSQL’s query planner uses PostgreSQL’s statistics (via pg_statistic) but those statistics must be populated by ANALYZE, which does not run automatically by default in early YugabyteDB versions. Schedule ANALYZE on busy tables and watch for plan regressions after bulk loads. The yb_enable_optimizer_statistics GUC flag (available in recent versions) enables more aggressive statistics usage in the cost model.

-- Useful diagnostics in YSQL

-- Check tablet distribution across nodes
SELECT yb_server_zone(), count(*) as tablet_count
FROM yb_local_tablets
GROUP BY 1;

-- Identify tables with stale statistics
SELECT relname, last_analyze, n_live_tup, n_dead_tup
FROM pg_stat_user_tables
WHERE last_analyze < now() - interval '24 hours'
  OR last_analyze IS NULL
ORDER BY n_live_tup DESC
LIMIT 20;

-- Monitor slow queries (requires pg_stat_statements)
SELECT query, calls, mean_exec_time, stddev_exec_time, rows
FROM pg_stat_statements
WHERE mean_exec_time > 100  -- queries averaging over 100ms
ORDER BY mean_exec_time DESC
LIMIT 10;

Clock skew. HLC correctness depends on bounded clock skew between nodes. YugabyteDB enforces a maximum skew of 500ms by default and will refuse to start if NTP is too far out of sync. Keep max_clock_skew_usec in mind when sizing timeout parameters: transaction retries triggered by clock skew look identical to conflict retries in application logs without proper tracing.

Tradeoffs Comparison

Dimension	YugabyteDB	CockroachDB	TiDB	Cloud Spanner	Vitess
PostgreSQL compatibility	High (forked pg query layer)	High (wire-compatible, reimplemented)	No (MySQL wire protocol)	Low (ANSI SQL subset)	No (MySQL wire protocol)
MySQL compatibility	No	No	High (MySQL 5.7/8.0)	No	High
Storage engine	DocDB (RocksDB LSM, per-tablet Raft)	Pebble (LSM, per-range Raft)	TiKV (RocksDB LSM, per-region Raft)	Colossus + Paxos	MySQL InnoDB per shard
Replication unit	Tablet (configurable size)	Range (512 MB default)	Region (96 MB default)	Split (8 GB threshold)	MySQL shard
Isolation level	Snapshot (default), Serializable	Serializable (default)	Snapshot (default), Serializable	Serializable	MySQL-dependent
Multi-region	xCluster async + synchronous geo-partition	Multi-region survivability + follower reads	TiFlash + async replication	Native global, TrueTime sync	Manual cross-datacenter routing
Clock mechanism	HLC (NTP-bounded)	HLC (NTP-bounded)	TSO service (single-region bottleneck)	TrueTime (GPS + atomic clock)	Not applicable
HTAP support	No dedicated columnar replica	No	TiFlash columnar replica	Separate Spanner BigQuery integration	No
Operational model	Self-hosted or YugabyteDB Aeon (managed)	Self-hosted or CockroachDB Dedicated/Serverless	Self-hosted or TiDB Cloud	Fully managed (Google Cloud only)	Self-hosted (complex)
Schema changes	Online, Raft-coordinated	Online, multi-version descriptor	Online, F1-style	Online	Manual or pt-online-schema-change
Best fit	PostgreSQL apps needing horizontal write scale without rewriting SQL	Serializable transactions, global survivability, PostgreSQL dialect	MySQL apps needing horizontal scale plus real-time analytics	Google Cloud, external consistency without clock management	MySQL shard management with minimal app changes

Closing

YugabyteDB’s core design decision, keeping PostgreSQL’s query layer intact and replacing only the storage engine, pays off when your application is already PostgreSQL-shaped. You get horizontal write scalability and multi-AZ fault tolerance without rewriting queries, functions, or extension usage. The cost is a more complex storage layer (DocDB’s document encoding, tablet management, and two-layer MVCC), write amplification from the LSM compaction chain, and the operational surface area of a distributed database. If your workload genuinely fits a relational model and you are hitting PostgreSQL’s single-node write limits, YugabyteDB is one of the more faithful escape hatches available.