How Redis Works Internally: Single-Threaded Event Loop, Data Structures, Persistence, and Replication

Redis is one of those tools that almost every engineering team reaches for, often without understanding why it is fast. The conventional answer is “it stores data in memory,” which is true but incomplete. Memcached also stores data in memory. So do dozens of other caches. What makes Redis different is the combination of: a single-threaded event loop that eliminates lock contention, purpose-built in-memory data structures with adaptive encodings that minimize overhead, and a persistence and replication layer that provides durability guarantees without sacrificing throughput.

This article walks through how Redis actually works, from the moment a client sends a command to the moment data lands on disk or a replica. Every section covers the internal mechanism, not just the surface API.

The Single-Threaded Event Loop

Redis uses a single thread to process all client commands. This is not a limitation; it is an architectural choice that eliminates the need for locks on shared data structures. There is never a race condition between two commands because no two commands ever execute concurrently on the main thread.

The event loop is built on the ae (Async Events) abstraction layer, which wraps epoll on Linux, kqueue on macOS/BSD, and select as a fallback. The loop runs continuously in a tight cycle:

while (running) {
  1. Call epoll_wait() with a short timeout (1ms default)
  2. For each ready file descriptor:
     a. If readable: read bytes from socket into input buffer
     b. If writable: flush pending replies from output buffer
  3. Process all fully-buffered commands (RESP parsing)
  4. Execute time events (expiry checks, background task callbacks)
}

The key insight is that Redis never blocks waiting for I/O on a single client. epoll_wait returns all ready file descriptors at once, so one slow client does not stall others. The server makes progress on every ready connection in the same loop iteration.

This design means throughput is bounded by CPU, not by I/O multiplexing overhead. A single Redis instance on modern hardware can handle 100,000+ operations per second on a single core. The ceiling is usually the network card or the cost of command processing, not the event loop itself.

What the Single Thread Does Not Handle

Redis 6 introduced I/O threads for reading and writing socket data. The main thread still executes all commands, but multiple I/O threads can read from and write to sockets in parallel. This removes the socket I/O bottleneck without introducing lock contention on data structures. In practice, I/O threading is most beneficial when clients send large values or when there are hundreds of concurrent connections.

Background persistence (RDB and AOF rewrite) runs in forked child processes, completely outside the event loop. Cluster bus communication and the replication feed also have separate threads. The “single-threaded” label applies specifically to command execution.

In-Memory Data Structures

Each Redis type maps to one or more internal representations depending on the size and content of the value. Understanding these encodings explains both the performance characteristics and the memory usage you observe in production.

Strings: SDS

Redis strings are not plain C strings. They use SDS (Simple Dynamic Strings), a length-prefixed structure with a separate header:

struct sdshdr {
  uint32_t len;    // current length
  uint32_t alloc;  // allocated capacity
  uint8_t  flags;  // type flags for header size selection
  char     buf[];  // actual bytes
};

SDS enables O(1) length operations (no strlen scan), safe binary data with embedded null bytes, and amortized O(1) appends. The header size varies: sdshdr5, sdshdr8, sdshdr16, sdshdr32, and sdshdr64 pack smaller lengths into smaller headers to reduce overhead for short strings.

Redis further optimizes string storage with three object encodings:

• int: for strings that fit in a long integer, the pointer stores the integer value directly with no heap allocation.
• embstr: for strings up to 44 bytes, the robj header and the SDS buffer are allocated in a single contiguous block. One malloc call, better cache locality.
• raw: for strings over 44 bytes, the robj header and SDS are separate allocations.

The 44-byte threshold comes from the jemalloc allocation size class boundary. A 64-byte allocation holds a 16-byte robj header plus a 3-byte SDS header, leaving exactly 44 bytes for the string content with one byte for the null terminator. Staying within 64 bytes means one cache line and one allocator size class.

Hashes: Listpack and Dict

A Redis hash starts as a listpack (formerly called ziplist in Redis versions prior to 7.0). A listpack is a contiguous byte array where each entry stores its own length, the length of the previous entry, and the data. Traversal is sequential, which is cache-friendly. Memory overhead per entry is roughly 11 bytes, far less than a dict entry.

+--------+--------+--------+--------+
| entry1 | entry2 | entry3 | END(FF)|
+--------+--------+--------+--------+
Each entry: [prevlen][encoding+len][data]

When the hash exceeds hash-max-listpack-entries (default 128) or any field exceeds hash-max-listpack-value (default 64 bytes), Redis converts the listpack to a dict. A Redis dict is a classic chained hash table with two tables for incremental rehashing.

Incremental rehashing is the mechanism that avoids O(n) pauses during resize. When a resize is triggered, Redis allocates the new (larger) table but does not move all entries immediately. Instead, each subsequent read or write operation migrates one hash bucket from the old table to the new table. After all buckets are migrated, the old table is freed. Lookups check both tables during the migration period.

// Observing encoding via redis-cli or ioredis:
import { Redis } from "ioredis";

const client = new Redis();

// Small hash: stored as listpack
await client.hset("user:1", { name: "alice", age: "30" });
const encoding = await client.call("object", "encoding", "user:1");
console.log(encoding); // "listpack"

// Force conversion to hashtable
for (let i = 0; i < 130; i++) {
  await client.hset("user:1", `field${i}`, "value");
}
const encodingAfter = await client.call("object", "encoding", "user:1");
console.log(encodingAfter); // "hashtable"

Lists: Listpack and Quicklist

Short lists (under list-max-listpack-size entries and element size under list-max-ziplist-size bytes) use a listpack. Once a list grows beyond these thresholds, Redis converts it to a quicklist: a doubly-linked list of listpack nodes. Each node holds up to list-max-listpack-size entries (configurable). This gives O(1) head/tail operations with good memory locality per node.

Sorted Sets: Listpack and Skiplist

Small sorted sets use a listpack (under zset-max-listpack-entries entries, default 128, and element size under zset-max-listpack-value bytes, default 64). Once the threshold is crossed, Redis converts to a skiplist paired with a hash table.

The skiplist provides O(log n) rank queries and range operations (ZRANGEBYSCORE, ZRANGE by rank). A skip list is a probabilistic data structure with multiple levels of linked lists. Each node is promoted to the next level with probability 0.25. The maximum level is 64. Expected search time is O(log n) because higher levels skip large portions of the list.

The hash table provides O(1) score lookup by member name (used by ZSCORE and ZADD when updating an existing member). The two structures are kept in sync: every ZADD updates both.

Level 3: [head] --------------------> [node E] ----> [tail]
Level 2: [head] -------> [node C] --> [node E] ----> [tail]
Level 1: [head] -> [A] -> [B] -> [C] -> [D] -> [E] -> [tail]

const pipeline = client.pipeline();
pipeline.zadd("leaderboard", 9850, "alice");
pipeline.zadd("leaderboard", 9720, "bob");
pipeline.zadd("leaderboard", 10100, "carol");
await pipeline.exec();

// Range query uses the skiplist: O(log n + k)
const top3 = await client.zrevrangebyscore(
  "leaderboard",
  "+inf",
  "-inf",
  "LIMIT",
  0,
  3
);

// Score lookup uses the hash table: O(1)
const carolScore = await client.zscore("leaderboard", "carol");

Sets: Intset, Listpack, and Hashtable

Sets of small integers use an intset: a sorted array of integers with binary search for O(log n) membership tests and very compact memory. Once a non-integer is added, or the set exceeds set-max-intset-entries (default 512), Redis converts to a listpack (for small sets up to set-max-listpack-entries, default 128) or a hashtable for larger sets.

Persistence

RDB: Fork and Copy-On-Write

BGSAVE forks the Redis process. The child inherates a copy of the entire address space via the OS copy-on-write (CoW) mechanism. The child iterates the entire keyspace and writes a compact binary snapshot to a temp file, then atomically renames it to dump.rdb. The parent continues serving commands without interruption.

Parent process (serving commands)
         |
    fork()
         |
         +--------> Child process
                         |
                    Writes RDB file
                         |
                    rename(tmp, dump.rdb)
                         |
                       exit()

The CoW cost is proportional to the write rate during the snapshot. Every page the parent modifies after the fork must be copied by the OS before the child’s view is updated. Under heavy write load, a BGSAVE can double memory usage temporarily. This is the most common cause of OOM kills on Redis instances with maxmemory set close to available RAM.

RDB snapshots are triggered by BGSAVE, by the save config directives (e.g., save 900 1 means save if at least 1 key changed in 900 seconds), or at shutdown. Recovery from RDB is fast because the file is a fully materialized snapshot.

AOF: Append-Only File with Fsync Policies

AOF records every write command as it executes, appending a RESP-encoded command to the AOF file. On restart, Redis replays the file to reconstruct the dataset. AOF provides finer-grained durability than RDB, at the cost of a larger file and slower restarts.

The critical configuration is appendfsync, which controls when the OS flushes the write buffer to disk:

• always: fsync after every command. Durability: at most one command lost on crash. Throughput: limited by disk fsync latency (typically 1,000-5,000 writes/second).
• everysec: fsync once per second in a background thread. Durability: at most one second of data lost. Throughput: near-full, since the main thread does not block.
• no: leave fsync to the OS (typically 30 seconds). Durability: up to 30 seconds of data at risk. Throughput: maximum.

// Check current AOF status
const info = await client.info("persistence");
const lines = info.split("\r\n");
const aofEnabled = lines.find((l) => l.startsWith("aof_enabled"));
const aofRewrite = lines.find((l) => l.startsWith("aof_rewrite_in_progress"));
console.log(aofEnabled);  // "aof_enabled:1"
console.log(aofRewrite);  // "aof_rewrite_in_progress:0"

The AOF file grows without bound as commands accumulate. BGREWRITEAOF compacts it by forking a child process that writes the current dataset as the minimal set of commands needed to reconstruct it, eliminating redundant intermediate states. The parent buffers new commands in an AOF rewrite buffer during the rewrite; when the child finishes, the buffer is appended to the new file, which then replaces the old one.

Hybrid Persistence

Redis 4.0 introduced an AOF hybrid mode (aof-use-rdb-preamble yes). When a rewrite runs, the child writes an RDB preamble (the full dataset as a binary snapshot) followed by the AOF commands accumulated since the rewrite started. This gives fast restart (reading a binary RDB preamble is much faster than replaying thousands of text commands) while preserving the durability of AOF for recent writes.

Replication

Redis replication is asynchronous by default. The replica connects to the primary and undergoes an initial full sync: the primary forks a child to write an RDB snapshot and streams it to the replica. The replica loads the RDB, then receives a continuous stream of write commands from the primary’s replication feed.

The Replication Backlog

The primary maintains a circular replication backlog (ring buffer), defaulting to 1MB. Every write command executed by the primary is appended to this buffer with an offset. The replica tracks its own repl_offset, the position up to which it has processed the replication stream.

If a replica disconnects and reconnects within the time window covered by the backlog, it sends a PSYNC request with its replication_id and repl_offset. The primary checks whether the requested offset is still within the backlog. If yes, it sends only the missing bytes: a partial resync. If no (the offset has been overwritten), a full resync is required, which involves another RDB transfer.

Primary replication backlog (ring buffer, 1MB default):
+-------+-------+-------+-------+-------+
| cmd1  | cmd2  | cmd3  | cmd4  | cmd5  |  <-- newest
+-------+-------+-------+-------+-------+
         ^
         replica repl_offset (replica needs cmd2 onwards on reconnect)

Redis 4.0 added a secondary replication ID to handle failover. When a replica is promoted to primary, it retains its old replication ID as a secondary ID. Other replicas that were synced to the old primary can perform a partial resync against the new primary using this secondary ID, avoiding a full RDB transfer after a failover.

// Force synchronous replication acknowledgement before proceeding
// WAIT numreplicas timeout_ms
const replicas = await client.wait(1, 1000);
console.log(`${replicas} replica(s) acknowledged the latest write`);

Redis Cluster

Redis Cluster distributes data across multiple nodes using hash slots. The keyspace is divided into 16,384 slots. A key’s slot is computed as:

slot = CRC16(key) % 16384

For keys with a hash tag (a substring in {...}), only the content inside the braces is hashed. This allows related keys to be co-located on the same node, enabling multi-key commands on those keys.

Key: "user:{alice}:profile"  -> CRC16("alice") % 16384 -> slot N
Key: "user:{alice}:session"  -> CRC16("alice") % 16384 -> slot N (same node)

Each node owns a range of hash slots. When a client sends a command for a key, the node checks whether that key’s slot belongs to it. If yes, it executes the command. If no, it returns a MOVED redirection:

-MOVED 7638 192.168.1.3:6379

The client caches the slot-to-node mapping and retries the command directly against the correct node. A smart client (like ioredis in cluster mode) builds this map at startup and updates it on MOVED responses, so the redirect round-trip only happens after topology changes.

ASK redirections handle ongoing slot migrations. When slot N is being moved from node A to node B, keys may exist on either node mid-migration. Node A returns an ASK redirection to direct the client to node B for this one request without updating the slot map. The client sends ASKING before the actual command to signal it is following an ASK redirect.

import { Cluster } from "ioredis";

const cluster = new Cluster([
  { host: "redis-node-1", port: 6379 },
  { host: "redis-node-2", port: 6379 },
  { host: "redis-node-3", port: 6379 },
]);

// ioredis handles MOVED/ASK transparently
await cluster.set("user:{alice}:session", "abc123");
await cluster.set("user:{alice}:profile", JSON.stringify({ name: "Alice" }));

// Both keys land on the same slot because of the hash tag
const [session, profile] = await cluster.mget(
  "user:{alice}:session",
  "user:{alice}:profile"
);

Cluster requires a minimum of three primary nodes. Each primary can have one or more replicas. Failover is automatic: if a primary is unreachable for cluster-node-timeout milliseconds (default 15 seconds), its replicas hold an election and one is promoted. During the election window, keys on the failing primary’s slots are unavailable.

Production Considerations

Memory: maxmemory and Eviction

Always set maxmemory to leave headroom for fork CoW, replication buffers, and the client output buffers. A safe rule: set maxmemory to 60-70% of available RAM on instances that run BGSAVE. On instances that never persist (pure cache tier), 80-85% is reasonable.

When maxmemory is reached, Redis applies the configured eviction policy. The policies and their use cases:

• noeviction: return errors on writes. Use when data loss is unacceptable and you prefer backpressure.
• allkeys-lru: evict the least recently used key from the entire keyspace. Standard cache tier policy.
• volatile-lru: evict LRU keys from the set with an expiry set. Useful when some keys are permanent and others are ephemeral.
• allkeys-lfu: evict the least frequently used key. Better than LRU for Zipf-distributed access patterns.
• volatile-ttl: evict keys with the shortest remaining TTL first.

Redis LRU is approximate, not exact. Rather than maintaining a true LRU list (expensive), Redis samples maxmemory-samples random keys (default 5) and evicts the least recently used among them. Setting maxmemory-samples to 10 produces near-exact LRU at roughly double the CPU cost of the default.

Connection Pooling

Each Redis client connection consumes memory and a file descriptor on both sides. Under high concurrency, unbounded connections degrade performance significantly. Always use a connection pool:

import { Redis } from "ioredis";

const client = new Redis({
  host: "localhost",
  port: 6379,
  maxRetriesPerRequest: 3,
  // ioredis uses a single multiplexed connection by default
  // For blocking commands, use a separate client instance
});

// For blocking commands (BLPOP, BRPOP, SUBSCRIBE) that tie up a connection:
const subscriber = new Redis({ host: "localhost", port: 6379 });
await subscriber.subscribe("events");
subscriber.on("message", (channel, message) => {
  console.log(`${channel}: ${message}`);
});

SCAN Over KEYS

KEYS pattern is O(n) over the entire keyspace and blocks the event loop until it completes. On a large dataset it can cause multi-second latency spikes for all other clients. Use SCAN with a cursor:

async function scanAll(client: Redis, pattern: string): Promise<string[]> {
  const results: string[] = [];
  let cursor = "0";

  do {
    const [nextCursor, keys] = await client.scan(
      cursor,
      "MATCH",
      pattern,
      "COUNT",
      100
    );
    cursor = nextCursor;
    results.push(...keys);
  } while (cursor !== "0");

  return results;
}

SCAN is O(1) per call (it scans COUNT buckets per iteration, not COUNT matching keys). It may return more or fewer than COUNT keys depending on hash table density.

Slowlog and Latency Monitoring

// Retrieve commands slower than 10ms (10,000 microseconds)
await client.call("config", "set", "slowlog-log-slower-than", "10000");
const slowlog = await client.call("slowlog", "get", "10");
// Returns: id, timestamp, duration in microseconds, command args

Monitor latency_histogram (Redis 7+) and the redis_commands_duration_seconds Prometheus metric exported via the redis-exporter for percentile-level visibility into command latency. The slowlog captures outliers; the histogram captures the distribution.

Fork Latency

BGSAVE and BGREWRITEAOF fork the process. The fork system call on Linux copies the page table, not the data, but on a large dataset (say, 50GB) the page table itself can be multiple gigabytes. A fork on a 50GB Redis instance with Transparent Huge Pages enabled can take hundreds of milliseconds, during which the event loop is blocked.

Disable Transparent Huge Pages on Redis hosts:

echo never > /sys/kernel/mm/transparent_hugepage/enabled

This is one of the most impactful single-line fixes for fork latency in production Redis deployments.

Tradeoffs: Redis vs. Alternatives

Dimension	Redis	Memcached	DragonflyDB	KeyDB	Valkey
Threading model	Single-threaded command execution, I/O threads optional (v6+)	Multi-threaded, partitioned per core	Multi-threaded via shared-nothing fibers	Multi-threaded with lock striping	Single-threaded (forked Redis 7.2)
Data structures	Strings, hashes, lists, sets, sorted sets, streams, bitmaps, HyperLogLog, geospatial	Strings only	Redis-compatible full type set	Redis-compatible full type set	Redis-compatible full type set
Persistence	RDB + AOF, hybrid mode	None (cache only)	RDB + AOF (in progress), Tiered storage roadmap	RDB + AOF (Redis-compatible)	RDB + AOF (Redis-compatible)
Replication	Primary-replica async, Sentinel, Cluster	None built-in	Active-active multi-master planned	Active-active multi-master, subms replication	Primary-replica, Cluster (Redis-compatible)
Throughput ceiling	~100K ops/sec single core, ~1M with I/O threads	~1M ops/sec per instance	Claims 25x Redis throughput on multi-core via fiber scheduler	2-5x Redis throughput on multi-core via threading	Similar to Redis 7.x
Protocol	RESP2 / RESP3	Memcache text + binary	RESP2 / RESP3	RESP2 / RESP3	RESP2 / RESP3
Operational maturity	Very high: 15+ years, extensive tooling	Very high	Early production: founded 2022	Moderate: production-ready since 2019	High: backed by Linux Foundation, forked 2024
Sweet spot	General-purpose data structures, sessions, leaderboards, pub/sub, rate limiting	Pure ephemeral string cache, simplicity above all	Multi-core single-instance throughput with Redis compatibility	Multi-master, active-active replication with Redis compatibility	Drop-in Redis replacement under an open license

The decision between these systems usually comes down to two axes: how much of the Redis type system you need, and how much throughput you require from a single instance. If you only cache string values and want maximum simplicity, Memcached’s simpler operational model is an advantage. If you need sorted sets, streams, or Lua scripting, Redis (or a compatible fork) is the only real option. DragonflyDB and KeyDB are worth evaluating if you are hitting Redis throughput limits on a single node and cannot or do not want to shard with Redis Cluster.

Closing

The single-threaded event loop is Redis’s most counterintuitive property and its most important one. It eliminates a whole class of concurrency bugs at the server level, shifting the complexity to the client and the persistence layer. Understanding how the event loop interacts with fork-based persistence, how data structures adapt their encoding based on size thresholds, and how the replication backlog determines whether a reconnecting replica triggers a full resync or a partial one: these are the details that matter when you move from “Redis is fast” to “Redis behaves exactly as I expect it to under load.”