Designing a Real-Time Chat System: Message Ordering, Delivery Guarantees, and Presence Management at Scale

Building a chat system looks deceptively simple until you hit the first ordering anomaly in production. Two users send messages within the same millisecond, clocks on different servers disagree by 200ms, and suddenly your conversation thread reads like a shuffled deck. The real complexity in chat is not the WebSocket handshake or the React component. It is the combination of message ordering, delivery semantics, and presence state that all need to behave correctly under network partitions, server restarts, and concurrent writers.

This article works through the core subsystems of a production chat system at the level where the interesting decisions live.

Message Storage and Ordering with Hybrid Logical Clocks

The naive approach uses wall-clock timestamps for ordering. This breaks as soon as you have multiple server instances: NTP drift of even 50ms between nodes causes messages to arrive out of order, and there is no way to distinguish true simultaneity from clock skew.

Auto-increment sequences work within a single database but are a coordination bottleneck at scale. Every insert requires a lock or a round trip to a sequence generator.

Hybrid Logical Clocks (HLCs) solve both problems. An HLC timestamp has two components: a physical time component (milliseconds since epoch) and a logical counter. The physical component keeps the timestamp close to wall time. The logical counter breaks ties and handles the case where the physical component would go backwards.

interface HLCTimestamp {
  physical: number; // ms since epoch
  logical: number;  // monotonic counter within same physical ms
  nodeId: string;   // breaks ties between concurrent nodes
}

class HybridLogicalClock {
  private physical: number = 0;
  private logical: number = 0;
  private readonly nodeId: string;

  constructor(nodeId: string) {
    this.nodeId = nodeId;
  }

  tick(): HLCTimestamp {
    const now = Date.now();
    if (now > this.physical) {
      this.physical = now;
      this.logical = 0;
    } else {
      // Wall clock did not advance: increment logical counter
      this.logical += 1;
    }
    return { physical: this.physical, logical: this.logical, nodeId: this.nodeId };
  }

  // Advance clock when receiving a remote timestamp
  receive(remote: HLCTimestamp): HLCTimestamp {
    const now = Date.now();
    const maxPhysical = Math.max(now, remote.physical, this.physical);

    if (maxPhysical === this.physical && maxPhysical === remote.physical) {
      this.logical = Math.max(this.logical, remote.logical) + 1;
    } else if (maxPhysical === this.physical) {
      this.logical += 1;
    } else if (maxPhysical === remote.physical) {
      this.logical = remote.logical + 1;
    } else {
      this.logical = 0;
    }

    this.physical = maxPhysical;
    return { physical: this.physical, logical: this.logical, nodeId: this.nodeId };
  }

  compare(a: HLCTimestamp, b: HLCTimestamp): number {
    if (a.physical !== b.physical) return a.physical - b.physical;
    if (a.logical !== b.logical) return a.logical - b.logical;
    return a.nodeId.localeCompare(b.nodeId);
  }
}

Store messages partitioned by conversation ID. Each partition has a monotonically increasing sequence within it that you derive from the HLC. The table structure matters:

CREATE TABLE messages (
  conversation_id  UUID        NOT NULL,
  message_id       UUID        NOT NULL DEFAULT gen_random_uuid(),
  hlc_physical     BIGINT      NOT NULL,
  hlc_logical      INT         NOT NULL,
  hlc_node_id      TEXT        NOT NULL,
  sender_id        UUID        NOT NULL,
  content          TEXT        NOT NULL,
  content_type     TEXT        NOT NULL DEFAULT 'text',
  created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (conversation_id, hlc_physical, hlc_logical, hlc_node_id)
);

CREATE INDEX idx_messages_conversation_recency
  ON messages (conversation_id, hlc_physical DESC, hlc_logical DESC);

The composite primary key gives you total ordering within a conversation without a global sequence generator. Queries for “load messages in order” resolve entirely from the index.

Delivery Guarantees: At-Least-Once with Client-Side Deduplication

At-most-once delivery (fire and forget) is not acceptable for chat. Messages would silently disappear on network hiccups. Exactly-once delivery is theoretically possible but requires distributed transactions that introduce latency and complexity that most systems cannot justify.

At-least-once delivery with idempotent clients is the practical choice. The client generates a client_message_id (a UUID) before sending. The server stores this alongside the message. On retry, the server detects the duplicate and returns the existing message rather than inserting a second copy.

interface OutboundMessage {
  clientMessageId: string;  // UUID generated by sender client
  conversationId: string;
  content: string;
  contentType: 'text' | 'image' | 'file';
}

interface MessageAck {
  clientMessageId: string;
  serverMessageId: string;
  hlcTimestamp: HLCTimestamp;
  status: 'delivered' | 'duplicate';
}

// Server-side handler
async function handleSendMessage(
  payload: OutboundMessage,
  senderId: string,
  clock: HybridLogicalClock,
  db: DatabaseClient,
): Promise<MessageAck> {
  // Check for existing message with this client ID
  const existing = await db.queryOne(
    `SELECT message_id, hlc_physical, hlc_logical, hlc_node_id
     FROM messages
     WHERE conversation_id = $1 AND client_message_id = $2`,
    [payload.conversationId, payload.clientMessageId],
  );

  if (existing) {
    return {
      clientMessageId: payload.clientMessageId,
      serverMessageId: existing.message_id,
      hlcTimestamp: {
        physical: existing.hlc_physical,
        logical: existing.hlc_logical,
        nodeId: existing.hlc_node_id,
      },
      status: 'duplicate',
    };
  }

  const ts = clock.tick();

  await db.execute(
    `INSERT INTO messages
       (conversation_id, client_message_id, message_id, hlc_physical, hlc_logical,
        hlc_node_id, sender_id, content, content_type)
     VALUES ($1, $2, gen_random_uuid(), $3, $4, $5, $6, $7, $8)`,
    [
      payload.conversationId, payload.clientMessageId,
      ts.physical, ts.logical, ts.nodeId,
      senderId, payload.content, payload.contentType,
    ],
  );

  return {
    clientMessageId: payload.clientMessageId,
    serverMessageId: /* returned from insert */ '',
    hlcTimestamp: ts,
    status: 'delivered',
  };
}

The client keeps a send queue. Each message sits in the queue until it receives an ack with its clientMessageId. If the WebSocket disconnects before the ack arrives, the client retransmits on reconnect. The server-side dedup key makes retransmission safe.

Read Receipts and Delivery Acknowledgments

Track two separate states per message per recipient: delivered (message reached the client device) and read (user viewed it). These are high-write, low-read workloads. A separate table avoids hot rows on the messages table.

CREATE TABLE message_receipts (
  conversation_id  UUID        NOT NULL,
  message_id       UUID        NOT NULL,
  user_id          UUID        NOT NULL,
  delivered_at     TIMESTAMPTZ,
  read_at          TIMESTAMPTZ,
  PRIMARY KEY (conversation_id, message_id, user_id)
);

When the client receives a message over WebSocket, it immediately sends a delivered receipt. When the user scrolls the message into view, it sends a read receipt. Batch these writes: collect receipts for up to 500ms and write them in a single statement. Individual per-message writes to this table at scale (imagine 1000 concurrent conversations each with 10 participants) produce write amplification that saturates the database.

Presence Management: Heartbeats and Typing Indicators

Presence state (online, offline, last seen) does not belong in your relational database for high-frequency updates. Use Redis with TTL-based expiry.

class PresenceManager {
  private readonly redis: RedisClient;
  private readonly heartbeatIntervalMs = 30_000;
  private readonly presenceTtlSeconds = 65; // slightly more than 2 heartbeat intervals

  async markOnline(userId: string, connectionId: string): Promise<void> {
    const key = `presence:${userId}`;
    await this.redis.setex(key, this.presenceTtlSeconds, connectionId);
    await this.redis.publish('presence:changes', JSON.stringify({
      userId,
      status: 'online',
      timestamp: Date.now(),
    }));
  }

  async markOffline(userId: string): Promise<void> {
    await this.redis.del(`presence:${userId}`);
    await this.redis.publish('presence:changes', JSON.stringify({
      userId,
      status: 'offline',
      timestamp: Date.now(),
    }));
  }

  async isOnline(userId: string): Promise<boolean> {
    const result = await this.redis.exists(`presence:${userId}`);
    return result === 1;
  }

  async refreshHeartbeat(userId: string, connectionId: string): Promise<void> {
    // Only refresh if this connection is still the active one
    const current = await this.redis.get(`presence:${userId}`);
    if (current === connectionId) {
      await this.redis.expire(`presence:${userId}`, this.presenceTtlSeconds);
    }
  }
}

Each connected client sends a heartbeat every 30 seconds. The server refreshes the TTL. If the TTL expires (missed two consecutive heartbeats), Redis evicts the key. A keyspace notification triggers a markOffline event that propagates to subscribers.

Typing indicators are ephemeral and lossy by design. They expire in 3 seconds if not refreshed. Do not persist them. Route them through Redis pub/sub directly to the conversation’s subscribers.

async function handleTypingIndicator(
  userId: string,
  conversationId: string,
  redis: RedisClient,
): Promise<void> {
  const key = `typing:${conversationId}:${userId}`;
  // Set with 3-second TTL; refreshed while user continues typing
  await redis.setex(key, 3, '1');
  await redis.publish(`conversation:${conversationId}:typing`, JSON.stringify({
    userId,
    typing: true,
  }));
}

Fan-Out Strategies for DMs vs Group Chats

Direct messages involve two participants. When user A sends to user B, the fan-out is trivial: write the message, then push to B’s connection if online. The coordination cost is near zero.

Group chats change the math. A chat with 500 members means a single message triggers 499 WebSocket pushes. At 1000 messages per minute across a busy group, that is 499,000 push operations per minute from a single conversation. At this scale you need to think about whether to fan out at write time or read time.

Write-time fan-out (also called push-on-write): when a message arrives, immediately enqueue a push to every member’s connection. This is fast for readers but expensive for large groups. Works well for groups under 100 members.

Read-time fan-out (pull-on-read): store the message once, let clients poll or maintain a cursor. Members fetch new messages when they reconnect or when a lightweight notification arrives. More efficient for large groups but adds latency and complexity for the client.

Hybrid approach: use write-time fan-out for members who are online at the moment the message is sent, and pull-on-read for members who are offline. Offline members fetch missed messages when they reconnect using a cursor (the HLC timestamp of their last seen message).

async function fanOutMessage(
  message: StoredMessage,
  conversationId: string,
  presenceManager: PresenceManager,
  connectionRouter: ConnectionRouter,
  memberIds: string[],
): Promise<void> {
  const onlineMembers = await Promise.all(
    memberIds
      .filter(id => id !== message.senderId)
      .map(async id => ({
        id,
        online: await presenceManager.isOnline(id),
      })),
  );

  // Push to online members immediately
  const pushPromises = onlineMembers
    .filter(m => m.online)
    .map(m => connectionRouter.push(m.id, { type: 'new_message', message }));

  await Promise.all(pushPromises);
  // Offline members will pull via cursor on reconnect — no work needed here
}

WebSocket Connection Routing Across Server Instances

When you have multiple WebSocket server instances behind a load balancer, you face a routing problem: user A is connected to server 1, but a message for user A arrives at server 2. Server 2 cannot push directly to a socket it does not own.

The standard solution is a Redis pub/sub channel per user (or per conversation). Each server subscribes to channels for the connections it owns. When a message needs routing, publish to the target channel and let the owning server deliver it.

class ConnectionRouter {
  private readonly redis: RedisClient;
  private readonly subscriber: RedisClient;
  // Map of userId -> WebSocket connection (local connections only)
  private readonly localConnections = new Map<string, WebSocket>();

  async registerConnection(userId: string, ws: WebSocket): Promise<void> {
    this.localConnections.set(userId, ws);
    // Subscribe to this user's delivery channel
    await this.subscriber.subscribe(`deliver:${userId}`, (message) => {
      const ws = this.localConnections.get(userId);
      if (ws?.readyState === WebSocket.OPEN) {
        ws.send(message);
      }
    });
  }

  async deregisterConnection(userId: string): Promise<void> {
    this.localConnections.delete(userId);
    await this.subscriber.unsubscribe(`deliver:${userId}`);
  }

  async push(userId: string, payload: unknown): Promise<void> {
    // Publish to the channel — the owning server will deliver it
    await this.redis.publish(`deliver:${userId}`, JSON.stringify(payload));
  }
}

One Redis pub/sub subscription per connected user is feasible up to roughly 100K concurrent users per Redis instance. Beyond that, shard your Redis cluster by user ID range or use a consistent-hash ring to route pub/sub channels across Redis nodes.

For very high connection counts, evaluate whether you need sticky sessions at the load balancer level. Sticky sessions reduce the pub/sub overhead (the server already has the socket) but complicate deployments: a rolling restart drains all connections from a server, causing reconnection spikes. Non-sticky with Redis routing handles restarts more gracefully.

Tradeoffs Table

Concern	Option A	Option B	When to prefer B
Message ordering	Wall-clock timestamp	Hybrid Logical Clock	Any multi-node deployment; wall-clock breaks under NTP drift
Delivery semantics	At-most-once (fire and forget)	At-least-once + client dedup	Always for chat; message loss is user-visible
Presence storage	Relational DB row	Redis TTL key	Heartbeat write rate exceeds ~1K/s; relational becomes a bottleneck
Group fan-out strategy	Write-time push (all members)	Hybrid push/pull	Groups over 100 members; write amplification grows linearly
WebSocket routing	Sticky sessions	Redis pub/sub channel per user	Rolling deployments, auto-scaling; stickiness complicates draining
Read receipts	Synchronous per-message write	Batched async write	More than ~50 concurrent conversations; per-message writes saturate disk
Typing indicators	Persisted DB record	Redis ephemeral TTL + pub/sub	Always; persistence adds write load with no durable value

Production Considerations

Message gap detection on reconnect. When a client reconnects, it sends the HLC timestamp of its last received message. The server queries for all messages in the conversation with a higher timestamp. This cursor-based catch-up avoids fetching the full history and handles the case where multiple messages arrived during a brief disconnect.

async function fetchMissedMessages(
  conversationId: string,
  sinceHlc: HLCTimestamp,
  limit: number,
  db: DatabaseClient,
): Promise<StoredMessage[]> {
  return db.query(
    `SELECT * FROM messages
     WHERE conversation_id = $1
       AND (hlc_physical, hlc_logical, hlc_node_id) > ($2, $3, $4)
     ORDER BY hlc_physical, hlc_logical, hlc_node_id
     LIMIT $5`,
    [conversationId, sinceHlc.physical, sinceHlc.logical, sinceHlc.nodeId, limit],
  );
}

Conversation partitioning and hot conversations. A single high-traffic conversation (think a large public channel) can saturate a database partition. Consider capping write throughput per conversation and routing overflow to a queue with back-pressure. Alternatively, move very-high-volume conversations to an append-only log (Kafka or equivalent) and derive the relational view asynchronously.

Backpressure on WebSocket push. A slow client that is not reading from its socket will cause the server-side send buffer to fill. Set a maximum buffer size and close connections that exceed it after a grace period. Clients that fall too far behind should reconnect and fetch missed messages via the catch-up mechanism rather than receiving a stream they cannot process.

Clock skew between client and server. Do not trust client-provided timestamps for ordering. Clients set their clocks to arbitrary values. The server assigns the HLC timestamp at the moment the message is persisted. Return the server-assigned timestamp in the ack so the client can display messages in server-canonical order.

Read receipt privacy. Some users do not want senders to know they have read a message. Add a per-user setting that suppresses read receipt emission. The delivered receipt (device received the message) is acceptable to always send, but read requires explicit consent in many jurisdictions.

Pagination cursors for history load. Load conversation history using keyset pagination on the composite HLC primary key, not OFFSET. At large offsets, PostgreSQL must scan and discard rows before returning results. Keyset pagination remains O(1) regardless of how far back in history the user scrolls.

---

A chat system that handles ordering, delivery, presence, and routing correctly is genuinely complex to build. The failure modes are subtle: messages that appear in the wrong order, receipts that lie about delivery state, presence that shows a user online 10 minutes after they disconnected. Getting these subsystems right requires treating each one as its own domain with its own consistency requirements, rather than assuming a single database and a single server will carry everything cleanly.