Designing a News Feed System: Fan-Out Strategies, Ranking Algorithms, and Real-Time Updates at Scale

A news feed is deceptively simple to prototype and genuinely hard to scale. At a few thousand users, a SQL JOIN across a follows table works fine. At a few million, that same query causes read latency that compounds with every new follower relationship. The fundamental tension is between write amplification (doing work at post time) and read amplification (doing work at read time). Getting that tradeoff wrong early means a painful rewrite under load.

This article covers the design decisions that matter: fan-out model selection, feed ranking, real-time delivery, storage layout, and the cache behaviors that turn a working prototype into a production system.

The Fan-Out Problem

When a user with 500,000 followers posts, the system needs to deliver that post to all 500,000 feeds. Two basic models exist, with a hybrid as the practical production choice.

Fan-Out on Write (Push Model)

At write time, for every follower of the posting user, insert a record into that follower’s feed. Reads are fast because each user’s feed is pre-materialized. Writes are expensive and proportional to follower count.

async function fanOutOnWrite(postId: string, authorId: string): Promise<void> {
  const followerIds = await db.query<{ follower_id: string }>(
    `SELECT follower_id FROM follows WHERE followee_id = $1`,
    [authorId]
  );

  const pipeline = redis.pipeline();
  const score = Date.now(); // Unix ms as sort key

  for (const { follower_id } of followerIds) {
    // Each user's feed is a Redis sorted set keyed by timestamp
    pipeline.zadd(`feed:${follower_id}`, score, postId);
    // Cap feed size to avoid unbounded growth
    pipeline.zremrangebyrank(`feed:${follower_id}`, 0, -1001);
  }

  await pipeline.exec();
}

A user with 1M followers triggers 1M Redis writes per post. With a Redis pipeline that costs roughly 5-10ms per 1000 operations, fanning out to 1M followers takes 5-10 seconds synchronously. This has to be async, which introduces delivery lag.

Fan-Out on Read (Pull Model)

At read time, fetch all users the reader follows, then fetch their recent posts and merge them. No write amplification, but read latency grows with follow count and post volume.

async function buildFeedOnRead(
  userId: string,
  limit: number,
  cursor?: string
): Promise<FeedItem[]> {
  const followeeIds = await db.query<{ followee_id: string }>(
    `SELECT followee_id FROM follows WHERE follower_id = $1`,
    [userId]
  );

  const ids = followeeIds.map((r) => r.followee_id);

  // Fetch recent posts from each followee, then merge-sort
  const postRows = await db.query<Post>(
    `SELECT * FROM posts
     WHERE author_id = ANY($1)
       AND ($2::text IS NULL OR created_at < $2::timestamptz)
     ORDER BY created_at DESC
     LIMIT $3`,
    [ids, cursor ?? null, limit]
  );

  return postRows;
}

For a user following 2000 accounts, this query scans potentially hundreds of thousands of rows. With proper indexing on (author_id, created_at DESC) it can stay fast, but it degrades as follow counts grow and as the number of recent posts per followee increases.

The Hybrid Approach (Celebrity Problem)

The practical answer is a hybrid: fan-out on write for normal users, fan-out on read for high-follower accounts. Define a follower threshold above which a user is treated as a celebrity. At read time, merge the pre-materialized feed with a live pull from celebrity accounts the reader follows.

const CELEBRITY_THRESHOLD = 100_000;

async function buildHybridFeed(
  userId: string,
  limit: number,
  cursor?: string
): Promise<FeedItem[]> {
  // 1. Read pre-materialized feed from Redis (covers normal users)
  const maxScore = cursor ? parseInt(cursor, 10) : '+inf';
  const precomputedPostIds = await redis.zrevrangebyscore(
    `feed:${userId}`,
    maxScore,
    '-inf',
    'LIMIT',
    0,
    limit
  );

  // 2. Fetch celebrity posts inline at read time
  const celebrityFollowees = await db.query<{ followee_id: string }>(
    `SELECT f.followee_id
     FROM follows f
     JOIN user_stats s ON s.user_id = f.followee_id
     WHERE f.follower_id = $1
       AND s.follower_count >= $2`,
    [userId, CELEBRITY_THRESHOLD]
  );

  let celebrityPosts: Post[] = [];
  if (celebrityFollowees.length > 0) {
    const ids = celebrityFollowees.map((r) => r.followee_id);
    celebrityPosts = await db.query<Post>(
      `SELECT * FROM posts
       WHERE author_id = ANY($1)
         AND ($2::text IS NULL OR created_at < $2::timestamptz)
       ORDER BY created_at DESC
       LIMIT $3`,
      [ids, cursor ?? null, limit]
    );
  }

  // 3. Fetch post details for pre-computed IDs
  const precomputedPosts =
    precomputedPostIds.length > 0
      ? await db.query<Post>(
          `SELECT * FROM posts WHERE id = ANY($1)`,
          [precomputedPostIds]
        )
      : [];

  // 4. Merge and re-sort by timestamp, return top N
  return [...precomputedPosts, ...celebrityPosts]
    .sort((a, b) => b.createdAt.getTime() - a.createdAt.getTime())
    .slice(0, limit);
}

The fan-out worker only writes to followers of non-celebrity accounts. Celebrity fan-out is skipped entirely at write time and absorbed at read time, where it’s bounded by the user’s celebrity follow count (almost always a small number).

Feed Ranking

Chronological feeds are simple and predictable, but they leave engagement on the table. Algorithmic scoring lets you surface relevant content over raw recency.

Engagement-Weighted Scoring

A basic scoring function combines recency with engagement signals. The decay factor controls how quickly older posts fall in rank.

interface PostSignals {
  createdAt: Date;
  likes: number;
  comments: number;
  shares: number;
  authorAffinityScore: number; // 0.0 to 1.0, personalized per viewer
}

function scorePost(signals: PostSignals): number {
  const ageHours =
    (Date.now() - signals.createdAt.getTime()) / (1000 * 60 * 60);

  const engagementScore =
    signals.likes * 1.0 +
    signals.comments * 3.0 + // comments signal stronger interest
    signals.shares * 5.0;

  const decayFactor = Math.pow(0.95, ageHours); // 5% decay per hour
  const affinityBoost = 1 + signals.authorAffinityScore;

  return engagementScore * decayFactor * affinityBoost;
}

Affinity score captures how often the viewer interacts with that author. It can be pre-computed and stored in a user_affinity table, updated asynchronously on each interaction.

Storing the float score as the Redis sorted set score means re-ranking requires re-writing scores, which is expensive. A common alternative: store post IDs with a timestamp score for ordering, then score at read time against a small candidate set (e.g., top 200 posts from the feed) and return the top N.

async function getRankedFeed(userId: string, limit: number): Promise<Post[]> {
  // Pull a larger candidate set than needed
  const candidateIds = await redis.zrevrange(`feed:${userId}`, 0, 199);

  const posts = await db.query<Post & PostSignals>(
    `SELECT p.*, COALESCE(ps.likes, 0) as likes, ...
     FROM posts p
     LEFT JOIN post_stats ps ON ps.post_id = p.id
     WHERE p.id = ANY($1)`,
    [candidateIds]
  );

  const affinityMap = await getAffinityMap(userId, posts.map((p) => p.authorId));

  return posts
    .map((p) => ({
      post: p,
      score: scorePost({ ...p, authorAffinityScore: affinityMap[p.authorId] ?? 0 }),
    }))
    .sort((a, b) => b.score - a.score)
    .slice(0, limit)
    .map((r) => r.post);
}

This keeps Redis writes simple (timestamp score only) while enabling rich ranking at read time against a bounded candidate set.

Real-Time Delivery

Three options for delivering new posts to open sessions: polling, SSE, and WebSockets. Each has a different complexity/resource profile.

Short Polling

The simplest approach. Client polls an endpoint every N seconds. Fine for low-volume feeds, wastes connections and backend resources at scale.

Server-Sent Events (SSE)

SSE sits between polling and WebSockets in complexity. The server pushes events over a persistent HTTP connection. The client reconnects automatically on drop. One-directional, which is exactly what a feed needs.

import { Hono } from 'hono';
import { streamSSE } from 'hono/streaming';

const app = new Hono();

app.get('/feed/stream/:userId', (c) => {
  const userId = c.req.param('userId');

  return streamSSE(c, async (stream) => {
    const subscriber = redis.duplicate();
    await subscriber.subscribe(`feed-updates:${userId}`);

    subscriber.on('message', async (_channel, postId) => {
      const post = await getPostById(postId);
      await stream.writeSSE({
        data: JSON.stringify(post),
        event: 'new-post',
        id: postId,
      });
    });

    // Keep alive ping every 30s to prevent proxy timeouts
    const keepAlive = setInterval(() => {
      stream.writeSSE({ data: '', event: 'ping' }).catch(() => {
        clearInterval(keepAlive);
      });
    }, 30_000);

    // Cleanup on disconnect
    stream.onAbort(() => {
      clearInterval(keepAlive);
      subscriber.unsubscribe();
      subscriber.quit();
    });

    // Block until client disconnects
    await stream.sleep(Infinity);
  });
});

When a fan-out worker writes to a user’s feed, it also publishes to the Redis pub/sub channel for that user. Connected SSE clients receive the post within milliseconds.

WebSockets

Use WebSockets when you need bidirectional communication: reactions, typing indicators, read receipts. For a pure feed, SSE is simpler and sufficient.

Storage Patterns for Timeline Materialization

Redis Sorted Sets

The primary feed store. Keys are feed:{userId}, scores are Unix timestamps (or composite scores), members are post IDs. This gives O(log N) insertion and O(log N + M) range queries where M is the number of results.

// Write: fan-out worker inserts post into follower feeds
await redis.zadd(`feed:${followerId}`, Date.now(), postId);
// Trim to last 1000 items
await redis.zremrangebyrank(`feed:${followerId}`, 0, -1001);

// Read: paginate by score (timestamp)
const posts = await redis.zrevrangebyscore(
  `feed:${userId}`,
  cursorScore ?? '+inf',
  '-inf',
  'WITHSCORES',
  'LIMIT',
  0,
  pageSize + 1 // fetch one extra to detect hasMore
);

Denormalized Feed Tables

For feeds that need to survive cache eviction or support complex queries (filtering by media type, showing only posts from certain lists), a denormalized SQL table works alongside Redis:

CREATE TABLE feed_items (
  user_id     UUID NOT NULL,
  post_id     UUID NOT NULL,
  author_id   UUID NOT NULL,
  created_at  TIMESTAMPTZ NOT NULL,
  score       FLOAT NOT NULL DEFAULT 0,
  PRIMARY KEY (user_id, post_id)
);

CREATE INDEX idx_feed_items_user_score
  ON feed_items (user_id, score DESC, created_at DESC);

This adds write load but enables arbitrary filtering and avoids Redis as a primary store. Most large-scale feeds use Redis as the hot path and fall back to the database when a feed is cold (user hasn’t opened the app in a while).

Cache Layer Design

Write-Through vs Lazy Loading

Write-through: on every post, update the feed cache immediately. The cache is always warm. Write costs are higher and any bug in the write path silently corrupts feeds.

Lazy loading: on cache miss, build the feed from the database, populate the cache, serve the result. The first load after eviction is slow. The cache is eventually consistent.

For feeds, write-through via the fan-out worker is standard. The fan-out is already async, so writing to Redis during that process adds minimal overhead and keeps reads fast for active users.

async function fanOutWorker(job: { postId: string; authorId: string }): Promise<void> {
  const { postId, authorId } = job;
  const post = await db.query<Post>(
    `SELECT * FROM posts WHERE id = $1`,
    [postId]
  );

  if (!post) return;

  const followers = await db.query<{ follower_id: string; is_celebrity: boolean }>(
    `SELECT f.follower_id, (s.follower_count >= $2) as is_celebrity
     FROM follows f
     JOIN user_stats s ON s.user_id = f.follower_id
     WHERE f.followee_id = $1`,
    [authorId, CELEBRITY_THRESHOLD]
  );

  const pipeline = redis.pipeline();
  const score = post.createdAt.getTime();

  for (const { follower_id } of followers) {
    pipeline.zadd(`feed:${follower_id}`, score, postId);
    pipeline.zremrangebyrank(`feed:${follower_id}`, 0, -1001);
    // Notify any open SSE connections
    pipeline.publish(`feed-updates:${follower_id}`, postId);
  }

  await pipeline.exec();
}

Tradeoffs: Fan-Out Strategy Comparison

Dimension	Fan-Out on Write	Fan-Out on Read	Hybrid
Read latency	Low (pre-materialized)	High at scale (O(follows * posts))	Low for most cases
Write latency	High for large follower counts	Negligible	Moderate (skips celebrities)
Storage overhead	High (N copies per post)	Low (one copy)	Moderate
Celebrity problem	Severe (millions of writes per post)	Not a problem (read-time fan-out)	Solved (celebrities pulled at read time)
Implementation complexity	Low	Low	Moderate
Cache invalidation on delete	Complex (remove from all feeds)	Simple (source of truth is posts table)	Partial complexity
Feed staleness	Near-zero (push is immediate)	On cache miss, may be stale	Near-zero for normal users, seconds for celebrities
Best fit	Small-to-medium follower counts	Read-heavy feeds with large follow graphs	Production at scale with mixed follower distributions

Cursor-Based Pagination

Offset-based pagination (LIMIT 20 OFFSET 400) degrades with deep pages and produces inconsistent results when new posts arrive. Use a cursor encoding the last seen score.

interface FeedPage {
  posts: Post[];
  nextCursor: string | null;
}

async function getFeedPage(
  userId: string,
  cursor: string | null,
  limit: number
): Promise<FeedPage> {
  const maxScore = cursor ? parseInt(Buffer.from(cursor, 'base64').toString(), 10) : Date.now();

  const results = await redis.zrevrangebyscore(
    `feed:${userId}`,
    maxScore,
    '-inf',
    'WITHSCORES',
    'LIMIT',
    0,
    limit + 1
  );

  const hasMore = results.length > limit * 2; // WITHSCORES returns [id, score, id, score, ...]
  const items = parseZrevrangebyscoreWithScores(results).slice(0, limit);

  const nextCursor = hasMore
    ? Buffer.from(String(items[items.length - 1].score - 1)).toString('base64')
    : null;

  const posts = await fetchPostsByIds(items.map((i) => i.id));

  return { posts, nextCursor };
}

Subtracting 1 from the last score ensures the next page starts strictly after the current page boundary without overlap.

Production Considerations

Handling deletes and edits in materialized feeds. When a post is deleted, every copy in every fan-out feed is stale. You have two options: eagerly remove it from all feeds (expensive, requires tracking which feeds received it), or tombstone it. A tombstone approach marks the post as deleted in the posts table. Feed reads check for tombstones and filter them out. The feed still contains the post ID, but it is excluded at read time. This trades a small read overhead for avoiding a massive write fanout on delete.

Edit propagation follows the same logic. The post ID remains in feeds, but the post content in the posts table is updated. Feed reads always hydrate post content from the source table, so edits are visible on the next read without any feed rewrite.

Cold feeds. Users who haven’t opened the app in days have stale or evicted Redis feeds. On first load, fall back to building the feed from the database. This is acceptable because cold users don’t expect real-time freshness. After the initial rebuild, populate Redis so subsequent reads are fast.

Fan-out queue depth. If a celebrity with 10M followers posts, the fan-out queue will absorb 10M jobs. With a hybrid model this doesn’t happen, but for normal users with tens or hundreds of thousands of followers, the async worker needs backpressure controls. Use a bounded queue with worker autoscaling and monitor queue depth as a primary SLO metric. A backed-up queue means feed delivery lag, which users notice within seconds.

Feed integrity on re-follow. When a user unfollows and re-follows someone, their pre-materialized feed may be missing posts from the gap period. Decide whether to backfill (expensive, complex) or accept the gap (simpler, most products do this). Document the choice so it doesn’t become a surprise edge case during an incident.

The hardest part of feed system design isn’t picking Redis or choosing SSE over WebSockets. It’s tracking which decisions interact: the fan-out model affects delete complexity, the ranking model affects cursor design, the real-time delivery model affects how you handle feed mutations. Map those interactions before committing to any single component choice.