How TiDB Works Internally: Distributed SQL, Raft-Based Storage, and the HTAP Architecture That Scales MySQL Horizontally

MySQL works until it does not. At some point, your single primary saturates its IOPS budget, your replica lag makes cross-shard reads unsafe, and your application code is carrying more sharding logic than business logic. The usual escapes are Vitess (adds a proxy and sharding awareness to MySQL), Aurora (scales reads, not writes), or a full rewrite onto a distributed database. TiDB takes a different position: wire-compatible with MySQL 5.7/8.0, distributed writes from day one, and a built-in columnar replica so the same cluster answers both OLTP queries from your API and analytical queries from your data team without ETL.

This article walks through every layer of TiDB’s architecture, from how a SELECT becomes a distributed plan to how TiKV persists data across Raft groups, and explains what that means for operations at scale.

The Problem TiDB Solves

A typical MySQL migration path looks like this: read replicas absorb read traffic until replication lag becomes a product problem. Then you introduce sharding, either at the application layer or through a proxy. Sharding solves write throughput but breaks transactions that span shards, complicates schema changes, and means every new feature has to be shard-aware from the start. Cross-shard joins either do not happen or route through your application tier, adding latency and complexity.

The analytical problem is separate but adjacent. When your data team’s queries start blocking application queries on the same MySQL instance, you replicate to a warehouse, accept the lag, and manage the pipeline. That pipeline has SLAs, failure modes, and a cost.

TiDB’s thesis is that both problems share a root cause: a single-node storage architecture. Fix the storage layer first, and the SQL layer can stay familiar.

Architecture Overview

TiDB has four main components:

1. TiDB server: stateless SQL nodes. Handle parsing, planning, and execution coordination.
2. TiKV: distributed key-value store. Handles all row-format data.
3. PD (Placement Driver): the cluster brain. Allocates timestamps and schedules regions.
4. TiFlash: columnar replica of TiKV data. Handles analytical queries.

These communicate over gRPC. From an application’s perspective, you point your MySQL client at a TiDB server, and the rest is invisible.

Application
    │  MySQL protocol (port 4000)
    ▼
TiDB Server (stateless, N instances)
    │  gRPC: Coprocessor, KV Get/Scan, Batch KV
    ├──────────────────────────────────────┐
    ▼                                      ▼
TiKV (row storage, Raft groups)     TiFlash (columnar replicas)
    │
    ▼
PD (TSO, region scheduling, metadata)

The SQL Layer: TiDB Server

TiDB servers are stateless. You run as many as your query concurrency demands, behind a load balancer. Each instance holds the full MySQL parser, optimizer, and executor. No shared state lives in the TiDB tier; all persistent state is in TiKV.

Query Lifecycle

When your application sends a query:

1. The TiDB server parses it into an AST using a MySQL-compatible grammar.
2. The logical planner rewrites the AST into a logical plan tree, applying predicate pushdown, column pruning, and subquery decorrelation.
3. The statistics module (which samples TiKV data continuously) supplies row count estimates to the cost-based optimizer.
4. The physical planner chooses between operators: TableReader (full scan via Coprocessor), IndexReader (index scan), IndexLookup (index + row fetch), and their distributed equivalents.
5. The executor runs the physical plan. For distributed queries, it fans out Coprocessor tasks to TiKV nodes.

The Coprocessor is important. Rather than pulling raw rows back to the TiDB server and filtering there, TiDB pushes partial aggregations and filtering to TiKV. A SELECT count(*) FROM orders WHERE status = 'pending' sends a Coprocessor request to every relevant TiKV region: each region evaluates the predicate locally, returns a partial count, and the TiDB server merges the results. This is analogous to how BigQuery pushes compute to storage.

// Conceptual: what a TiDB Coprocessor task looks like at the gRPC boundary
interface CoprocessorRequest {
  context: RegionContext;       // which region and epoch
  ranges: KeyRange[];           // key ranges to scan
  data: Uint8Array;             // serialized DAG request (operators + expressions)
  startTs: bigint;              // read timestamp from TSO
}

interface CoprocessorResponse {
  data: Uint8Array;             // serialized partial result
  range: KeyRange;              // range actually covered
  remainingRows?: number;       // for pagination
}

Distributed Joins

For joins that cannot be pushed down, TiDB has two strategies:

Hash join: the smaller table is broadcast to all TiDB nodes working on the join. Each node holds the hash table in memory. Works well when one side fits in memory.

Index join: use an index on the inner table to probe row by row. Lower memory pressure, higher latency for large inner sets.

For very large joins, TiDB 7.x introduced parallel hash join with work-stealing, which distributes the build and probe phases across multiple goroutines and limits memory usage through spilling.

TiKV: Raft-Based Key-Value Storage

TiKV is where your data actually lives. The architecture has two layers: Raft consensus at the top, RocksDB at the bottom.

Data Model

TiKV stores everything as key-value pairs. The TiDB server is responsible for encoding SQL rows into TiKV keys. A row in table t with primary key 42 becomes a key like:

t{tableID}_r{rowID}  →  encoded(col1, col2, col3, ...)

Secondary indexes produce separate key-value pairs:

t{tableID}_i{indexID}_{indexValue}_{rowID}  →  null (or include columns for covering indexes)

This encoding is entirely managed by TiDB server. TiKV has no concept of tables or SQL.

Regions

TiKV splits the key space into regions, each covering a contiguous key range, defaulting to approximately 96 MB. Every region is replicated across three TiKV nodes (configurable) using the Raft consensus protocol.

Each region has one leader and two followers. All reads and writes go through the leader by default (follower reads are available as an optimization). When a region grows past the size threshold, TiKV splits it into two regions. When a TiKV node goes down, PD detects the missing replicas and instructs surviving nodes to elect a new leader and re-replicate.

// Simplified region metadata as tracked by PD
interface RegionMeta {
  id: number;
  startKey: Uint8Array;
  endKey: Uint8Array;
  peers: Peer[];              // {id, storeId, role: 'voter' | 'learner'}
  leaderPeerId: number;
  epoch: { confVer: number; version: number };
}

Raft and RocksDB

Within each region, Raft works as follows:

1. A write arrives at the leader.
2. The leader appends the entry to its Raft log and sends AppendEntries RPCs to followers.
3. Once a majority (two of three) acknowledge, the entry is committed.
4. The leader applies the committed entry to RocksDB and returns success to the client.

RocksDB is a log-structured merge-tree (LSM) engine. Writes go to a write-ahead log (WAL) and a memtable. When the memtable fills, it flushes to L0 SSTs. Background compaction merges SSTables across levels, reclaiming space and maintaining read performance.

TiKV uses RocksDB column families to separate Raft log data from applied KV data. This means Raft log compaction (truncating committed log entries no longer needed for replication) does not interfere with KV compaction.

One subtlety: TiKV runs one RocksDB instance per store (node), not per region. All regions on a node share the same RocksDB. This is intentional: each RocksDB instance has background threads, file handles, and memory buffers. Running thousands of regions as separate RocksDB instances would be ruinous for resource consumption.

Multi-Version Concurrency Control

TiDB uses MVCC to provide snapshot isolation. Every write to TiKV is tagged with a timestamp from PD. Reads specify a startTs (the transaction’s read timestamp) and retrieve the latest version of each key that is at or before that timestamp.

The transaction protocol is Percolator-style, borrowed from Google’s Bigtable paper:

1. Prewrite phase: the TiDB server picks a primary key for the transaction, writes all mutations as locked records tagged with the start timestamp.
2. Commit phase: if prewrite succeeds for all keys, PD allocates a commit timestamp. The primary lock is atomically converted to a committed record. Secondary keys are asynchronously committed.
3. Cleanup: stale locks from crashed transactions are resolved lazily by readers who encounter them.

This protocol allows TiDB to commit transactions across multiple TiKV regions atomically without a two-phase commit coordinator that holds locks.

// Simplified Percolator prewrite record layout in TiKV
// For each key in the transaction:
// Write CF:  key@commitTs  →  { kind: 'put' | 'delete', startTs }
// Lock CF:   key           →  { primary, startTs, ttl, txnSize }
// Default CF: key@startTs  →  value bytes

The Placement Driver: Timestamp Oracle and Scheduler

PD is a small cluster (typically three nodes) running etcd internally for its own consensus. It has two primary responsibilities.

Timestamp Oracle (TSO)

All transactions need a globally monotonic timestamp. PD implements this as a lease-based counter: the PD leader pre-allocates a batch of timestamps (typically 10,000 at a time) and commits the upper bound to etcd. It then serves timestamps from this batch in memory. When the batch is exhausted, it persists the next upper bound and continues. This gives sub-millisecond TSO allocation for most requests, with a brief stall only when the batch needs renewal.

TSO latency matters. A typical OLTP transaction needs at least two TSO calls (one for start, one for commit). At 100ms network round-trips to PD, you cap out at 5 transactions per second per client. In practice, TiDB batches TSO requests across concurrent transactions, but PD placement (keep it close to TiDB servers, co-region if possible) is a real production concern.

Region Scheduler

PD continuously receives heartbeats from TiKV stores and region leaders. It uses this data to:

• Detect missing replicas and schedule re-replication.
• Balance region count and leader count across stores.
• Move hot regions (high read/write traffic) away from saturated stores.
• Honor placement rules (e.g., keep one replica in each availability zone).

The scheduler emits operator commands to TiKV: AddPeer, RemovePeer, TransferLeader, SplitRegion. TiKV executes them through the normal Raft mechanism, ensuring no data loss.

TiFlash: Columnar HTAP Replica

TiFlash is TiDB’s answer to the analytical workload problem. It is a columnar storage engine that replicates TiKV data in real time via the Raft learner mechanism.

How Replication Works

A TiFlash node registers itself as a Raft learner for the regions it covers. Learners receive the same Raft log entries as voter replicas but do not participate in commit quorums. This means TiFlash replication does not add latency to OLTP writes.

TiFlash converts incoming Raft log entries (which are row-format deltas) into columnar storage. It uses Delta-Main storage: recent writes land in a row-format delta layer (for fast ingestion), while a background compaction process merges deltas into columnar MergeTree segments (similar to ClickHouse’s storage model). Reads on TiFlash merge the delta and stable layers on the fly.

TiKV Region Leader
    │  Raft AppendEntries (log entries)
    ▼
TiFlash Learner
    │  delta ingestion
    ▼
Delta Store (row format, recent writes)
    │  background compaction
    ▼
Stable Store (columnar segments, Parquet-like layout)

Intelligent Query Routing

When TiFlash replicas exist for a table, the TiDB optimizer can choose between TiKV and TiFlash for each table scan based on estimated cost. A query joining a 10-row config table with a 500-million-row events table will scan the events table from TiFlash (columnar, better for aggregations) and the config table from TiKV (row-format, fast for point lookups). The routing decision happens at the per-table-scan level within a single query.

-- Force TiFlash for a specific table in a query (useful for testing)
SELECT /*+ READ_FROM_STORAGE(tiflash[events]) */
  date_trunc('day', created_at) AS day,
  count(*) AS cnt
FROM events
WHERE tenant_id = 42
GROUP BY 1
ORDER BY 1 DESC;

TiFlash supports MPP (massively parallel processing) mode: for large analytical queries, TiDB pushes the entire join and aggregation into TiFlash nodes, which exchange intermediate results directly with each other without routing data back through the TiDB server first. This means a 100-million-row join can be processed in parallel across a TiFlash cluster without the TiDB server becoming a bottleneck.

Production Considerations

MySQL Compatibility Gaps

TiDB is MySQL-compatible but not MySQL-identical. The gaps that bite most migrations:

• AUTO_INCREMENT behavior: TiDB assigns IDs in batches per node, so IDs are monotonically increasing per node but not globally sequential. Applications assuming globally sequential IDs for ordering will break. Use AUTO_RANDOM instead, which distributes primary keys to avoid write hot spots while maintaining uniqueness.
• FOREIGN KEY constraints: supported in TiDB 6.6+, but earlier versions silently accepted the DDL and ignored enforcement. Verify your TiDB version and test constraint behavior explicitly.
• SELECT ... FOR UPDATE with range conditions: behavior differs from MySQL in some edge cases around gap locking. TiDB uses optimistic transactions by default; heavy SELECT FOR UPDATE workloads may need explicit pessimistic transaction mode (BEGIN PESSIMISTIC).
• Stored procedures and triggers: limited support. Migrate logic to application layer before switching.
• LOAD DATA INFILE: supported but paths behave differently in distributed context. Use TiDB Lightning for bulk loads instead.

Hotspot Avoidance

Because TiDB splits regions on key ranges, monotonically increasing primary keys (typical AUTO_INCREMENT) cause write hot spots: all new inserts go to the rightmost region, which receives all write traffic until a split occurs. After a split, the new rightmost region immediately becomes hot again.

The solution is AUTO_RANDOM:

CREATE TABLE orders (
  id BIGINT AUTO_RANDOM PRIMARY KEY,
  user_id BIGINT NOT NULL,
  status VARCHAR(32) NOT NULL,
  created_at DATETIME NOT NULL,
  INDEX idx_user_status (user_id, status)
);

AUTO_RANDOM sets the upper bits of the ID to a random value, distributing writes evenly across regions while keeping the ID unique. The tradeoff: IDs are no longer sortable by insertion time. If you need time-ordered IDs externally (for API cursor pagination), generate them in the application using a Snowflake-style scheme and treat TiDB’s internal ID as an implementation detail.

TiDB Lightning for Initial Load

Migrating data from MySQL with a live INSERT loop will be slow and will generate hot spots. TiDB Lightning converts source data into SSTable files and imports them directly into TiKV, bypassing the SQL and Raft layers entirely. A 500 GB MySQL dump that would take hours via INSERT can load in 20-40 minutes with Lightning in local backend mode.

# tidb-lightning.toml (simplified)
[lightning]
level = "info"

[tikv-importer]
backend = "local"
sorted-kv-dir = "/mnt/ssd/sorted-kv"

[mydumper]
data-source-dir = "/data/mysql-dump"

[tidb]
host = "tidb-server"
port = 4000
user = "root"

Observability

Key metrics to watch per layer:

• TiDB server: tidb_query_duration_seconds (p99 by query type), tidb_session_transaction_duration_seconds, TSO round-trip latency.
• TiKV: tikv_scheduler_latch_wait_duration_seconds (lock contention), tikv_raftstore_apply_log_duration_seconds (apply throughput), rocksdb_compaction_pending_bytes (compaction backlog).
• PD: pd_tso_dispatch_duration_seconds, pd_scheduler_store_status (region balance ratio), pd_region_label_iso_level (placement rule compliance).
• TiFlash: tiflash_schema_apply_count (DDL sync latency), tiflash_raft_read_index_duration_seconds (MVCC read consistency cost).

TiDB ships Grafana dashboards for all of these via its operator or helm chart. The dashboards are well-maintained and cover the metrics above at useful quantiles.

Schema Changes

TiDB uses an online DDL system based on Google’s F1 paper. Adding a column, building an index, or changing a column type does not lock the table. The DDL runs in the background in multiple phases, maintaining compatibility with concurrent reads and writes throughout. ADD INDEX on a large table will backfill in TiKV, consuming significant I/O. Throttle it with:

-- Control backfill speed to limit I/O impact
SET GLOBAL tidb_ddl_reorg_worker_cnt = 4;     -- default 4, reduce to 2 during business hours
SET GLOBAL tidb_ddl_reorg_batch_size = 256;   -- default 256, reduce to 64 for lower I/O spikes

Tradeoffs Comparison

Dimension	TiDB	CockroachDB	Vitess	PlanetScale	Aurora (MySQL)
Architecture	Distributed SQL (TiKV + TiFlash)	Distributed SQL (monolithic node)	MySQL proxy + sharding	Vitess-based DBaaS	Single-writer + read replicas
MySQL compatibility	High (5.7/8.0 wire protocol)	Low (PostgreSQL wire protocol)	High (MySQL proxy)	High (MySQL protocol)	High (MySQL fork)
HTAP	Native (TiFlash columnar replica, MPP)	None	None	None	None (Aurora ML is not HTAP)
Write scaling	Horizontal (region-based sharding)	Horizontal (range-based)	Horizontal (explicit sharding)	Horizontal (vtgate routing)	Vertical (single writer)
Cross-shard transactions	Native ACID (Percolator, no app changes)	Native ACID (two-phase commit)	Manual (application-aware)	Limited (single-shard preferred)	Not applicable
Operational complexity	High (four component types, Kubernetes operator recommended)	Medium (single binary per node)	High (vtgate, vttablet, orchestration)	Low (fully managed)	Low (fully managed)
Self-hosted option	Yes (TiDB Operator for Kubernetes)	Yes	Yes	No	No (AWS-only)
Cloud managed option	TiDB Cloud (AWS, GCP)	CockroachDB Serverless/Dedicated	No native offering	Yes (PlanetScale Scaler, PS One)	Yes (AWS RDS Aurora)
Schema change safety	Online DDL (F1-style, non-blocking)	Online schema changes	Online DDL via gh-ost integration	Non-blocking via PlanetScale DDL	Some locks, depends on operation
Replication lag (analytics)	Near-zero (TiFlash learner via Raft)	Not applicable	Depends on MySQL replica lag	Depends on PlanetScale branches	Minutes (binlog to warehouse)
Best fit	MySQL workloads needing horizontal write scale and real-time analytics on one cluster	PostgreSQL workloads needing global distribution and strong consistency	Existing MySQL with sharding expertise already in-house	MySQL SaaS product needing zero-downtime schema changes, DBaaS comfort	AWS-native apps with read-heavy workloads and limited write scale needs

Closing Thoughts

TiDB’s design makes a clear bet: most teams would rather stay on MySQL semantics and let the storage layer handle distribution than learn a new query model. That bet pays off when your write traffic is genuinely too high for a single MySQL writer and you cannot afford the application complexity of manual sharding. It pays off especially when your engineering team also wants to run analytical queries on fresh data without building a separate pipeline.

The cost is operational complexity. Running TiDB correctly means operating TiDB server, TiKV, PD, and optionally TiFlash, each with their own failure modes, tuning parameters, and capacity planning. TiDB Cloud removes most of that overhead in exchange for the usual managed-service tradeoffs. For teams comfortable with Kubernetes, the TiDB Operator is mature and the component failure modes are well-understood once you have instrumented the key metrics above.

If you are hitting MySQL write saturation and your data team is asking for real-time access to production data, TiDB is the most complete answer in the MySQL-compatible ecosystem. If you are already on PostgreSQL, CockroachDB deserves a parallel evaluation. If your write volume fits on one node but you need read scale, Aurora still has better operational ergonomics at that tier.