Designing a Social Feed System: Fan-Out Strategies, Ranking, and Real-Time Updates at Scale

The social feed is one of the most deceptively complex systems to build. At first glance it looks like a list of posts sorted by time. In production it is an intersection of distributed storage, real-time delivery, personalization logic, and write amplification that breaks in ways that cost you significant engineering time if you have not thought through the tradeoffs upfront.

This walkthrough covers the architecture decisions you need to make before writing a line of feed code: fan-out strategy, storage layout, ranking, real-time push, the celebrity problem, and how these interact under real traffic patterns.

The Core Problem: Who Writes What, When

When user A follows 200 people and one of them posts, user A’s feed needs to include that post. That sounds simple until you multiply it: 10 million users, each following a median of 150 accounts, and the system absorbing 5,000 new posts per second during peak hours. The feed delivery problem is fundamentally a fan-out problem: one write event must become visible to many readers, and the tradeoff is whether you do the work at write time or at read time.

Both approaches are valid. Choosing the wrong one for your scale characteristics will cause pain.

Fan-Out on Write

Fan-out on write means that when a user publishes a post, your system immediately copies (or enqueues a reference to) that post into every follower’s pre-built feed. The feed is materialized: when a user opens the app, you read their pre-computed list from cache or storage and serve it directly.

interface Post {
  postId: string;
  authorId: string;
  content: string;
  mediaUrls: string[];
  createdAt: Date;
}

interface FeedItem {
  postId: string;
  authorId: string;
  score: number; // ranking score, pre-computed
  insertedAt: Date;
}

async function fanOutWrite(post: Post, redis: Redis): Promise<void> {
  const followers = await getFollowers(post.authorId);

  const pipeline = redis.pipeline();

  for (const followerId of followers) {
    const feedKey = `feed:${followerId}`;
    const item: FeedItem = {
      postId: post.postId,
      authorId: post.authorId,
      score: computeInitialScore(post),
      insertedAt: new Date(),
    };

    // Use a sorted set keyed by score so ranking is trivial at read time
    pipeline.zadd(feedKey, { score: item.score, member: post.postId });

    // Trim to a max depth to bound memory
    pipeline.zremrangebyrank(feedKey, 0, -(MAX_FEED_DEPTH + 1));
  }

  await pipeline.exec();
}

function computeInitialScore(post: Post): number {
  // Simple time-decay seed score; re-ranked asynchronously
  return Date.now() / 1000;
}

const MAX_FEED_DEPTH = 1000;

Where this works well: Systems with moderate follower counts (under 10K per user), high read-to-write ratios, and latency-sensitive feed loads. The read path becomes a single Redis ZRANGE call. Feed generation is O(1) regardless of how many people the viewer follows.

Where fan-out on write breaks: Celebrity accounts. A user with 5 million followers creates 5 million write operations on a single post. At 1,000 posts per second from one such account, your write pipeline becomes the bottleneck. At 100K followers the fan-out is manageable with a queue and worker pool; at 5M it requires architectural changes (covered below).

Fan-Out on Read

Fan-out on read means you do not pre-materialize feeds. When a user requests their feed, you look up who they follow, fetch recent posts from each of those authors, merge them, rank them, and return the result.

async function readFeed(
  viewerId: string,
  limit: number,
  db: Database,
  redis: Redis
): Promise<Post[]> {
  const following = await redis.smembers(`following:${viewerId}`);

  if (following.length === 0) return [];

  // Fetch recent posts from each followed account
  // In production: scatter-gather across a posts-by-author index
  const postBatches = await Promise.all(
    following.map((authorId) =>
      db.query<Post>(
        `SELECT * FROM posts WHERE author_id = $1
         ORDER BY created_at DESC LIMIT $2`,
        [authorId, 50]
      )
    )
  );

  const allPosts = postBatches.flat();

  // Merge and rank
  const ranked = rankPosts(allPosts, viewerId);

  return ranked.slice(0, limit);
}

function rankPosts(posts: Post[], viewerId: string): Post[] {
  return posts.sort((a, b) => {
    const scoreA = computeRankScore(a, viewerId);
    const scoreB = computeRankScore(b, viewerId);
    return scoreB - scoreA;
  });
}

Where this works well: Low read traffic, very high write traffic, accounts with massive follower counts (the celebrity case), and systems where feed personalization is expensive enough that pre-computing it at write time is impractical.

Where fan-out on read breaks: When the viewer follows hundreds of accounts, the scatter-gather at read time generates hundreds of parallel DB queries. Read latency becomes a function of following count. Aggressive caching per-author helps, but you still pay merge and ranking cost on every feed load. At 50ms per author query and 300 followed accounts, you cannot serve the feed in under 100ms without heroic caching.

The Hybrid Strategy

In practice, nearly every large feed system lands on a hybrid: fan-out on write for normal accounts, fan-out on read for high-follower-count celebrities. The boundary is typically set around 10K to 50K followers, tuned to your write throughput budget.

const CELEBRITY_FOLLOWER_THRESHOLD = 50_000;

async function routeFanOut(post: Post, db: Database): Promise<void> {
  const authorFollowerCount = await getFollowerCount(post.authorId, db);

  if (authorFollowerCount <= CELEBRITY_FOLLOWER_THRESHOLD) {
    // Enqueue for fan-out on write
    await writeFanOutQueue.publish({ postId: post.postId, authorId: post.authorId });
  } else {
    // Mark as celebrity post; readers will merge it at read time
    await markCelebrityPost(post.postId, post.authorId, db);
  }
}

async function hybridReadFeed(
  viewerId: string,
  redis: Redis,
  db: Database
): Promise<FeedItem[]> {
  // 1. Read pre-built fan-out feed from cache
  const fanOutItems = await redis.zrevrange(`feed:${viewerId}`, 0, 200, "WITHSCORES");

  // 2. Identify which followed accounts are celebrities
  const following = await redis.smembers(`following:${viewerId}`);
  const celebrities = await getCelebritiesAmong(following, db);

  // 3. Fetch recent celebrity posts directly
  const celebrityPosts = await Promise.all(
    celebrities.map((uid) =>
      redis.zrevrange(`posts:author:${uid}`, 0, 20, "WITHSCORES")
    )
  );

  // 4. Merge and re-rank
  const merged = mergeAndRank(fanOutItems, celebrityPosts.flat());

  return merged.slice(0, 100);
}

The key operational detail: celebrity thresholds need to be re-evaluated periodically. A user who crosses 50K followers mid-day should not cause a fan-out explosion for the posts they published before crossing the threshold.

Storage Layout

The feed storage layer has two concerns: the post store (source of truth) and the feed store (read-optimized materialized views).

Post store: A relational database (Postgres) with a posts table indexed on (author_id, created_at DESC). This index is the backbone of both the fan-out worker and the celebrity fan-out-on-read scatter-gather. A covering index on (author_id, created_at, post_id, content_preview) avoids heap fetches for the common case.

CREATE TABLE posts (
  post_id     UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  author_id   UUID NOT NULL REFERENCES users(user_id),
  content     TEXT NOT NULL,
  media_urls  JSONB DEFAULT '[]',
  created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  deleted_at  TIMESTAMPTZ
);

CREATE INDEX idx_posts_author_timeline
  ON posts (author_id, created_at DESC)
  INCLUDE (post_id, content);

CREATE INDEX idx_posts_created_at
  ON posts (created_at DESC)
  WHERE deleted_at IS NULL;

Feed store: Redis sorted sets, keyed by feed:{userId}. Members are post IDs, scores are ranking values. At read time you ZREVRANGE to get ordered post IDs, then do a batch GET against a post detail cache or the posts DB. Keep the sorted set capped at MAX_FEED_DEPTH entries to bound memory. A typical cap of 1,000 items per user with 4 bytes per post ID plus 8 bytes for the score is about 12 KB per user feed: at 10 million active users that is 120 GB of Redis memory, which is manageable on a mid-size cluster.

Feed Ranking

Chronological feeds have a known problem: bursty authors bury everyone else. Ranking by engagement signals produces better retention but at the cost of recency and real-time feel. Most production feeds use a time-decay scoring function that balances freshness against engagement.

interface RankingSignals {
  ageSeconds: number;
  likesCount: number;
  commentsCount: number;
  sharesCount: number;
  viewerAffinityScore: number; // 0.0-1.0, based on past interactions
}

function computeRankScore(signals: RankingSignals): number {
  const {
    ageSeconds,
    likesCount,
    commentsCount,
    sharesCount,
    viewerAffinityScore,
  } = signals;

  // Hacker News-style gravity: engagement / (age + 2)^gravity
  const GRAVITY = 1.8;
  const engagementScore =
    likesCount * 1.0 +
    commentsCount * 2.5 +
    sharesCount * 3.0;

  const timeDecay = Math.pow((ageSeconds / 3600) + 2, GRAVITY);

  // Affinity multiplier: boost posts from accounts the viewer interacts with
  const affinityMultiplier = 1.0 + viewerAffinityScore * 0.5;

  return (engagementScore / timeDecay) * affinityMultiplier;
}

Ranking scores stored in sorted sets are stale by design: they were computed at insert time. For a feed that truly responds to viral engagement, you need a re-ranking pass. Run this as a background job rather than inline:

async function reRankFeedItems(
  viewerId: string,
  redis: Redis,
  signalStore: SignalStore
): Promise<void> {
  const items = await redis.zrevrange(`feed:${viewerId}`, 0, 200, "WITHSCORES");

  const pipeline = redis.pipeline();

  for (const { member: postId } of items) {
    const signals = await signalStore.getSignals(postId, viewerId);
    const newScore = computeRankScore(signals);
    pipeline.zadd(`feed:${viewerId}`, { score: newScore, member: postId, xx: true });
  }

  await pipeline.exec();
}

The xx: true flag ensures you only update existing members and never accidentally insert posts that were removed by the viewer.

Real-Time Push with WebSockets

A user sitting on the feed screen should see new posts without a manual refresh. The simplest production-viable approach is a dedicated WebSocket gateway that subscribes to a Redis pub/sub channel per user, and pushes new post IDs (not full post content) to connected clients. Clients then fetch post details independently.

import { WebSocketServer, WebSocket } from "ws";

const wss = new WebSocketServer({ port: 8080 });
const connections = new Map<string, Set<WebSocket>>();

wss.on("connection", async (ws, req) => {
  const userId = await authenticateConnection(req);
  if (!userId) { ws.close(4001, "Unauthorized"); return; }

  if (!connections.has(userId)) connections.set(userId, new Set());
  connections.get(userId)!.add(ws);

  // Subscribe to this user's real-time feed channel
  const sub = redis.duplicate();
  await sub.subscribe(`feed:realtime:${userId}`);

  sub.on("message", (_channel, message) => {
    if (ws.readyState === WebSocket.OPEN) {
      ws.send(message);
    }
  });

  ws.on("close", () => {
    connections.get(userId)?.delete(ws);
    sub.unsubscribe();
    sub.quit();
  });
});

// Called by the fan-out worker after inserting into the feed sorted set
async function notifyFeedUpdate(
  userId: string,
  postId: string,
  redis: Redis
): Promise<void> {
  const payload = JSON.stringify({ type: "new_post", postId });
  await redis.publish(`feed:realtime:${userId}`, payload);
}

At scale, a single WebSocket server cannot hold connections for millions of users. You need a connection registry: each WebSocket server instance registers its connected user IDs in Redis, and the fan-out worker looks up which server holds the connection before publishing. The simpler production pattern is to use a managed pub/sub service (e.g., Ably, Pusher, or a Kafka-backed gateway) that handles connection routing for you, reserving custom infrastructure for when you have scale that justifies it.

Cache Warming

When a user loads the feed for the first time (or after a cache eviction), there is no pre-built sorted set to serve. A cold feed load triggers a read-time construction that is slow. Prevent this with proactive cache warming:

1. When a user logs in, check if their feed cache exists. If not, trigger an async warm job before the first feed request completes.
2. On a schedule (every 30 minutes for active users), backfill the feed cache to ensure it has at least N items.
3. After a fan-out write completes, check if the recipient’s feed key exists. If it does not, skip the ZADD and instead mark the user for warming on their next login.

async function warmFeedCache(
  userId: string,
  redis: Redis,
  db: Database
): Promise<void> {
  const exists = await redis.exists(`feed:${userId}`);
  if (exists) return;

  const following = await db.query<{ userId: string }>(
    `SELECT followed_id AS "userId" FROM follows WHERE follower_id = $1`,
    [userId]
  );

  const recentPosts = await db.query<Post>(
    `SELECT p.* FROM posts p
     WHERE p.author_id = ANY($1::uuid[])
       AND p.created_at > NOW() - INTERVAL '7 days'
       AND p.deleted_at IS NULL
     ORDER BY p.created_at DESC
     LIMIT 500`,
    [following.map((f) => f.userId)]
  );

  if (recentPosts.length === 0) return;

  const pipeline = redis.pipeline();
  for (const post of recentPosts) {
    const score = post.createdAt.getTime() / 1000;
    pipeline.zadd(`feed:${userId}`, { score, member: post.postId });
  }
  pipeline.expire(`feed:${userId}`, 86400 * 7); // 7-day TTL
  await pipeline.exec();
}

Tradeoffs

Dimension	Fan-Out on Write	Fan-Out on Read	Hybrid
Read latency	O(1) sorted set read	O(following count) scatter-gather	O(1) + small celebrity merge
Write amplification	High (1 post = N writes)	None at write time	Moderate (only non-celebrity followers)
Celebrity problem	Breaks at >50K followers	Handles naturally	Isolates celebrities explicitly
Feed freshness	Near-real-time (push-based)	Always fresh	Near-real-time for most; fresh for celebrities
Cache memory	High (per-user sorted sets)	Low (no materialized feeds)	Moderate (no celebrity feeds stored)
Implementation complexity	Low	Moderate	High
Re-ranking complexity	Background job updates scores	Computed at read time	Mixed: background for fan-out, read-time for celebrity posts

Production Considerations

Deletion propagation. When a post is deleted, you need to remove it from every feed sorted set it was written to. At write fan-out time, record a mapping of post_id -> [follower_ids] in a secondary store so deletion workers know which feeds to scan. For fan-out-on-read, deletion is trivially handled by the post store soft-delete.

Follow/unfollow consistency. When a user follows a new account, retroactively inject that account’s recent posts into the follower’s feed. When a user unfollows, remove that account’s posts from the feed sorted set. Both operations should happen asynchronously; the UI can optimistically update before the backend propagates.

Feed pagination. Do not use offset-based pagination for feeds. Use cursor-based: the cursor is the last-seen rank score, and the query is ZREVRANGEBYSCORE feed:{userId} (lastScore -inf LIMIT 0 20. Score ties (two posts with identical ranking) need a secondary sort by post ID to produce stable pages.

Observability. Track: fan-out queue depth and lag, feed cache hit rate by user segment, P99 feed load latency, re-ranking job staleness (how old the oldest score in a user’s feed is), and WebSocket connection drop rate. A feed cache hit rate below 90% on active users signals a warming strategy problem.

Backfill on schema changes. If you change the ranking formula, you need to backfill existing feed sorted sets. At 10 million users with 1,000 items per feed, that is 10 billion score updates. Run backfill incrementally, targeting only active users first, and use a versioned cache key (feed:v2:{userId}) so old and new feeds coexist during the rollout.

The “ghost post” problem. A post can appear in a fan-out feed after the author deletes it, if the deletion event loses a race with the fan-out worker. Solve this at read time: when hydrating post details from IDs, filter out any post that has deleted_at set. Never trust the feed sorted set as a source of truth for post existence; it is a ranking index, not a durable store.

Closing

A social feed system is not one problem; it is a scheduling problem (fan-out timing), a storage problem (materialized views versus scatter-gather), a ranking problem (time-decay versus engagement signals), and a real-time delivery problem, all layered on top of each other. The decisions compound: fan-out on write simplifies reads but makes deletion hard; fan-out on read handles celebrities easily but punishes users who follow many accounts. There is no single correct answer, only the right set of tradeoffs for your traffic shape and team operational capacity.