Designing a Real-Time Feed System: Fan-Out Strategies, Activity Streams, and Personalized Timelines at Scale

Feed systems look trivial in interviews and brutal in production. A naïve implementation works until you hit the first high-follower account, the first ranking requirement, or the first request for real-time delivery. Then you are retrofitting architecture onto a system that was never designed to support it.

This article walks through the decisions you need to make before writing a line of feed code: activity stream storage, fan-out strategy and its failure modes, the hybrid pattern for asymmetric follower graphs, personalization and ranking layers, real-time push with WebSockets and SSE, cursor pagination, and cache warming. No hand-waving. Real tradeoffs.

The Problem Space

A feed system takes discrete events (posts, likes, comments, follows) generated by a set of actors and assembles them into a personalized, ordered timeline for each observer. The actors are the users someone follows. The observers are all of their followers.

The structural challenge is asymmetry. A typical user follows a few hundred people and is followed by a few hundred. But some accounts have millions of followers. Every write those accounts make must become visible to millions of readers. The write volume is manageable; the fan-out is not.

The second challenge is ranking. A strict reverse-chronological feed is the simplest thing to build and the fastest to serve. But most products add a relevance signal at some point: engagement weight, relationship strength, content type, freshness decay. Once you add ranking, pre-materialized feeds become stale and on-demand computation becomes expensive. You need a strategy for both.

The third challenge is delivery latency. Users expect new content to appear without refreshing. Real-time delivery via WebSockets or Server-Sent Events adds a connection management layer on top of an already complex write pipeline.

Activity Stream Storage

Before deciding how to fan-out, you need a model for the raw activity stream. This is the normalized source of truth: every event recorded once, regardless of how many people will eventually see it.

interface ActivityEvent {
  eventId: string;         // globally unique, sortable (e.g., Snowflake ID)
  actorId: string;         // who performed the action
  verb: "post" | "like" | "comment" | "follow" | "share";
  objectId: string;        // the post, comment, or user that was acted upon
  objectType: "post" | "comment" | "user";
  targetId?: string;       // for comments: the post being commented on
  metadata: Record<string, unknown>;
  createdAt: Date;
}

This schema is intentionally generic. The Activity Streams 2.0 spec (a W3C standard) uses the same actor/verb/object/target shape. Normalizing here means you never duplicate the full post body in every follower’s feed row, which matters at scale: a post with 1 million followers that stores 2 KB of content inline creates 2 GB of fan-out data per event.

Store activity events in append-only, time-ordered storage. PostgreSQL with a BRIN index on created_at handles millions of events per day well. For larger volumes, a dedicated time-series store or Kafka-backed event log gives you both storage and a built-in distribution mechanism for the fan-out pipeline.

CREATE TABLE activity_events (
  event_id     TEXT PRIMARY KEY,
  actor_id     TEXT NOT NULL,
  verb         TEXT NOT NULL,
  object_id    TEXT NOT NULL,
  object_type  TEXT NOT NULL,
  target_id    TEXT,
  metadata     JSONB,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX activity_events_actor_created
  ON activity_events (actor_id, created_at DESC);

-- BRIN is efficient for append-only time-ordered data
CREATE INDEX activity_events_created_brin
  ON activity_events USING BRIN (created_at);

The actor index supports fetching “all events by this user” when you need to backfill a follower’s feed. The BRIN index supports range scans across the full stream.

Fan-Out on Write vs. Fan-Out on Read

These are the two canonical strategies and the tradeoffs are well-understood. The choice is not about which is better in the abstract; it is about which fits your specific read-to-write ratio, follower distribution, and latency requirements.

Fan-Out on Write (Push Model)

When an actor creates a post, you immediately write a reference to that post into every follower’s materialized feed. The feed is pre-computed; reads are fast and require no joins.

async function fanOutOnWrite(
  event: ActivityEvent,
  db: Database,
  redis: Redis
): Promise<void> {
  // Fetch follower IDs in batches to avoid loading millions into memory
  const BATCH_SIZE = 1000;
  let cursor: string | undefined;

  do {
    const { followers, nextCursor } = await db.getFollowersBatch(
      event.actorId,
      { cursor, limit: BATCH_SIZE }
    );

    // Write to each follower's feed in Redis sorted set
    // Score is the ranking score (initially time-based, refined later)
    const pipeline = redis.pipeline();
    const score = computeInitialScore(event);

    for (const followerId of followers) {
      pipeline.zadd(`feed:${followerId}`, {
        score,
        member: event.eventId,
      });
      // Cap feed length to avoid unbounded growth
      pipeline.zremrangebyrank(`feed:${followerId}`, 0, -(MAX_FEED_SIZE + 1));
    }

    await pipeline.exec();
    cursor = nextCursor;
  } while (cursor);
}

function computeInitialScore(event: ActivityEvent): number {
  // Use epoch milliseconds as score for chronological ordering
  // Override with engagement-weighted score during re-ranking jobs
  return event.createdAt.getTime();
}

Read time is a single ZREVRANGE against the user’s sorted set, then a batch fetch of post bodies from a separate cache or database:

async function getFeed(
  userId: string,
  redis: Redis,
  db: Database,
  pagination: { cursor?: string; limit: number }
): Promise<FeedPage> {
  const { minScore, maxScore } = decodeCursor(pagination.cursor);

  const eventIds = await redis.zrevrangebyscore(
    `feed:${userId}`,
    maxScore,
    minScore,
    { limit: { count: pagination.limit + 1 } }
  );

  const hasMore = eventIds.length > pagination.limit;
  const pageIds = hasMore ? eventIds.slice(0, pagination.limit) : eventIds;

  const posts = await db.getPostsByIds(pageIds);
  const nextCursor = hasMore
    ? encodeCursor(posts[posts.length - 1].score)
    : null;

  return { posts, nextCursor };
}

Advantages: Reads are O(1) on the sorted set. No fan-in joins at read time. Well-suited for high-read, low-write scenarios.

Disadvantages: Write amplification is proportional to follower count. A celebrity with 5 million followers creates 5 million sorted-set writes per post. The fan-out pipeline becomes the bottleneck and must be asynchronous to avoid blocking the write path.

Fan-Out on Read (Pull Model)

Instead of materializing each follower’s feed at write time, you reconstruct the feed at read time by fetching recent events from every account the user follows and merging them.

async function fanOutOnRead(
  userId: string,
  db: Database,
  redis: Redis,
  pagination: { cursor?: string; limit: number }
): Promise<FeedPage> {
  const following = await db.getFollowing(userId);

  // Fetch recent events from each followed account (from cache or DB)
  const eventSets = await Promise.all(
    following.map((actorId) =>
      redis.zrevrangebyscore(
        `actor_events:${actorId}`,
        "+inf",
        "-inf",
        { limit: { count: 200 } }
      )
    )
  );

  // Merge and sort by score
  const allEventIds = eventSets.flat();
  const withScores = await redis.zmscore(
    `global_scores`,
    allEventIds
  );

  const merged = allEventIds
    .map((id, i) => ({ id, score: withScores[i] ?? 0 }))
    .sort((a, b) => b.score - a.score);

  const page = merged.slice(0, pagination.limit);
  const posts = await db.getPostsByIds(page.map((p) => p.id));

  return { posts, nextCursor: encodeCursor(page) };
}

Advantages: No write amplification. A celebrity posting triggers exactly one write. Well-suited for low-follower-count scenarios or cases where follower counts are uniformly small.

Disadvantages: Read latency scales with the number of accounts followed. If a user follows 500 accounts, you fan out to 500 cache reads per page load. The merge operation is proportional to the number of followed accounts times the lookback window. Response time degrades as following graph grows.

Tradeoffs at a Glance

Dimension	Fan-Out on Write	Fan-Out on Read
Read latency	O(1), sorted set lookup	O(following count)
Write amplification	O(follower count) per event	O(1) per event
Stale feeds on re-rank	Yes, requires backfill jobs	No, always recomputed
Celebrity problem	Severe	None
Suitable scale	Median follower count < 50K	Median following count < 500
Cache hit dependency	High (feed sorted sets)	High (per-actor event sets)

The Hybrid Approach: Asymmetric Fan-Out

Most production feed systems use a hybrid: fan-out-on-write for regular accounts, fan-out-on-read (or deferred fan-out) for accounts above a follower threshold.

const CELEBRITY_THRESHOLD = 100_000;

async function routeFanOut(
  event: ActivityEvent,
  db: Database,
  redis: Redis,
  queue: MessageQueue
): Promise<void> {
  const followerCount = await db.getFollowerCount(event.actorId);

  if (followerCount > CELEBRITY_THRESHOLD) {
    // Tag the event as high-reach; do not fan out now.
    // Followers will pull celebrity posts at read time.
    await redis.setex(
      `celebrity_event:${event.eventId}`,
      86400, // 24h TTL
      JSON.stringify(event)
    );
    await db.markAsCelebrity(event.eventId);
  } else {
    // Standard fan-out via job queue (async, not blocking the write path)
    await queue.enqueue("fan_out_write", {
      eventId: event.eventId,
      actorId: event.actorId,
    });
  }
}

At read time, the feed assembly merges the user’s pre-materialized sorted set with a live query for celebrity events from followed high-reach accounts:

async function assembleFeed(
  userId: string,
  db: Database,
  redis: Redis,
  limit: number,
  cursor?: string
): Promise<FeedPage> {
  const { minScore } = decodeCursor(cursor);

  // Pre-materialized feed for regular accounts
  const materializedEventIds = await redis.zrevrangebyscore(
    `feed:${userId}`,
    "+inf",
    minScore,
    { limit: { count: limit * 2 } } // over-fetch for merge headroom
  );

  // Live pull from celebrity accounts this user follows
  const followedCelebrities = await db.getFollowedCelebrities(userId);
  const celebrityEvents = await fetchCelebrityEvents(
    followedCelebrities,
    redis,
    minScore
  );

  // Merge, rank, and paginate
  const merged = mergeByScore(
    materializedEventIds,
    celebrityEvents.map((e) => e.eventId)
  );

  const page = merged.slice(0, limit);
  const posts = await db.getPostsByIds(page.map((p) => p.id));

  return {
    posts,
    nextCursor: page.length === limit ? encodeCursor(page[page.length - 1]) : null,
  };
}

The threshold value (100K here) should be tuned against your observed write throughput. If a write burst from a 5M-follower account saturates your job queue workers, lower it. If the celebrity live-pull adds measurable read latency, raise it or add a bounded cache for celebrity recent events per follower cluster.

Ranking and Personalization

Chronological feeds are easy to implement and easy to reason about. Ranked feeds require a score that captures relevance, and that score has two problems: it must be computed cheaply enough to use at insertion time, and it must be refreshed when engagement signals change.

A typical score function combines freshness decay with engagement weight:

interface ScoringContext {
  event: ActivityEvent;
  authorEngagementRate: number;  // 30-day like/view ratio for this author
  viewerAuthorAffinity: number;  // interaction frequency: 0–1
  mediaBoost: number;            // image: 1.1, video: 1.3, text: 1.0
}

function computeRankingScore(ctx: ScoringContext): number {
  const ageHours =
    (Date.now() - ctx.event.createdAt.getTime()) / (1000 * 60 * 60);

  // Gravity decay: score halves every ~7 hours
  const freshnessDecay = Math.pow(0.9, ageHours);

  const score =
    freshnessDecay *
    (1 + ctx.authorEngagementRate * 0.5) *
    (1 + ctx.viewerAuthorAffinity * 2) *
    ctx.mediaBoost;

  // Scale to millisecond-precision integer range for Redis sorted set
  return Math.round(score * 1e9);
}

The viewerAuthorAffinity signal is the hardest to compute at fan-out time because it is unique to every viewer-author pair. The practical approach is to use a coarser signal during write (author-level engagement) and apply a personalization re-rank during the read path for the top N items in the assembled feed:

async function personalizeTopN(
  events: FeedEvent[],
  userId: string,
  affinityModel: AffinityModel,
  n: number
): Promise<FeedEvent[]> {
  const topCandidates = events.slice(0, Math.min(n * 3, events.length));
  const rest = events.slice(topCandidates.length);

  const affinityScores = await affinityModel.batchScore(
    userId,
    topCandidates.map((e) => e.authorId)
  );

  const reranked = topCandidates
    .map((event, i) => ({
      ...event,
      finalScore: event.baseScore * (1 + affinityScores[i]),
    }))
    .sort((a, b) => b.finalScore - a.finalScore)
    .slice(0, n);

  return [...reranked, ...rest];
}

Keep the re-rank window bounded (top 50, not top 500). Affinity model inference over 500 events per page load will add hundreds of milliseconds. If your affinity model is expensive, pre-compute pairwise affinity scores asynchronously and cache them per viewer-author pair with a 6-hour TTL.

Real-Time Push: WebSockets vs. SSE

Once the feed is built, you need a way to push new items to open clients without polling. Two options dominate:

Server-Sent Events (SSE): Unidirectional, HTTP-based, auto-reconnects. Simpler to implement and deploy. Ideal when the client only needs to receive (not send) real-time updates.

WebSockets: Bidirectional, stateful connection. Necessary if the client needs to send events over the same channel (typing indicators, reactions). More complex to scale horizontally because connections are stateful.

For a feed system where the client only receives new posts, SSE is the right choice in most cases. The implementation is simpler and the scaling properties (standard HTTP load balancing, no sticky sessions required if you use a pub/sub layer) are better.

// Server: SSE endpoint using Hono or a raw Node.js http handler
import { Hono } from "hono";
import { streamSSE } from "hono/streaming";

const app = new Hono();

app.get("/feed/stream", async (c) => {
  const userId = c.get("userId"); // from auth middleware

  return streamSSE(c, async (stream) => {
    const subscriber = redis.duplicate();
    await subscriber.subscribe(`feed_updates:${userId}`);

    subscriber.on("message", async (_channel, message) => {
      const event = JSON.parse(message) as FeedEvent;
      await stream.writeSSE({
        data: JSON.stringify(event),
        event: "new_post",
        id: event.eventId,
      });
    });

    // Clean up on disconnect
    stream.onAbort(async () => {
      await subscriber.unsubscribe();
      await subscriber.quit();
    });
  });
});

The fan-out write pipeline publishes to Redis Pub/Sub after writing to the sorted set:

async function fanOutWithRealtimePush(
  event: ActivityEvent,
  followerIds: string[],
  redis: Redis
): Promise<void> {
  const pipeline = redis.pipeline();
  const score = computeInitialScore(event);

  for (const followerId of followerIds) {
    pipeline.zadd(`feed:${followerId}`, { score, member: event.eventId });
    // Publish real-time notification
    pipeline.publish(
      `feed_updates:${followerId}`,
      JSON.stringify({ eventId: event.eventId, score })
    );
  }

  await pipeline.exec();
}

At scale, a dedicated Redis Pub/Sub cluster prevents pub/sub traffic from competing with sorted-set operations. For extremely high connection counts (millions of concurrent SSE connections), move to a horizontally sharded WebSocket gateway (a dedicated service per shard of user IDs) backed by Pub/Sub or a message bus, and avoid relying on Redis Pub/Sub alone as the delivery mechanism.

Cursor-Based Pagination

Offset pagination (LIMIT 20 OFFSET 200) breaks on feeds because the feed is ordered by score, not by a stable row number. Scores change when re-ranking runs, so offsets shift. Cursor pagination solves this by encoding the last-seen position in the response, which the client sends with the next request.

A cursor for a score-ordered feed encodes the score of the last returned item:

interface FeedCursor {
  maxScore: number;  // exclusive upper bound for next page
  eventId: string;  // tie-breaker for items with identical scores
}

function encodeCursor(lastItem: { score: number; eventId: string }): string {
  const cursor: FeedCursor = {
    maxScore: lastItem.score,
    eventId: lastItem.eventId,
  };
  return Buffer.from(JSON.stringify(cursor)).toString("base64url");
}

function decodeCursor(encoded?: string): FeedCursor | null {
  if (!encoded) return null;
  try {
    return JSON.parse(Buffer.from(encoded, "base64url").toString("utf8"));
  } catch {
    return null;
  }
}

async function getFeedPage(
  userId: string,
  redis: Redis,
  limit: number,
  cursor?: string
): Promise<{ events: Array<{ id: string; score: number }>; nextCursor: string | null }> {
  const decoded = decodeCursor(cursor);

  // zrevrangebyscore returns items in descending score order
  // Use (maxScore syntax for exclusive bound to avoid returning the cursor item again
  const rawItems = await redis.zrevrangebyscore(
    `feed:${userId}`,
    decoded ? `(${decoded.maxScore}` : "+inf",
    "-inf",
    { withScores: true, limit: { count: limit + 1 } }
  );

  const hasMore = rawItems.length > limit;
  const page = hasMore ? rawItems.slice(0, limit) : rawItems;

  const nextCursor = hasMore
    ? encodeCursor({ score: page[page.length - 1].score, eventId: page[page.length - 1].id })
    : null;

  return { events: page, nextCursor };
}

One edge case: if two events have identical scores (common if you are using timestamp-based scoring with millisecond precision and high write throughput), you need the tie-breaker eventId in the cursor to ensure deterministic pagination. Always include it.

Cache Warming

Cold-start feeds are expensive. When a user logs in after a week away, their sorted set in Redis may have expired. Rebuilding it requires scanning their following list, fetching recent events per actor, scoring and inserting them. This can take seconds.

Two strategies reduce this cost:

Proactive warming on login: When authentication succeeds, enqueue a background job that rebuilds the feed sorted set before the user’s first feed request lands.

async function onLoginSuccess(
  userId: string,
  queue: MessageQueue
): Promise<void> {
  const feedKey = `feed:${userId}`;
  const exists = await redis.exists(feedKey);

  if (!exists) {
    // Enqueue with high priority so it completes before first page load
    await queue.enqueue(
      "warm_feed_cache",
      { userId },
      { priority: "high" }
    );
  }
}

async function warmFeedCache(
  userId: string,
  db: Database,
  redis: Redis
): Promise<void> {
  const following = await db.getFollowing(userId);
  const LOOKBACK_DAYS = 7;
  const since = new Date(Date.now() - LOOKBACK_DAYS * 86400 * 1000);

  const recentEvents = await db.getRecentEventsByActors(following, since);

  const pipeline = redis.pipeline();
  for (const event of recentEvents) {
    const score = computeInitialScore(event);
    pipeline.zadd(`feed:${userId}`, { score, member: event.eventId });
  }

  // Set TTL: if user is not active, let the key expire rather than maintain it forever
  pipeline.expire(`feed:${userId}`, 7 * 86400);
  await pipeline.exec();
}

Lazy rebuild with a loading state: Return a partial feed from the database for the first page, and rebuild the cache in the background. Show a “loading more” state to the user while the warm-up completes. This avoids the login latency spike at the cost of a slightly degraded first-load experience.

For users who log in daily, keep sorted sets warm by extending TTLs on each access. Set expiry to 3-7 days. Accounts that have not been active in a week can have their feed sorted sets evicted; re-warm on next login.

Production Considerations

The pieces above interact in ways that create production failure modes worth calling out explicitly:

Fan-out queue depth spikes on viral events. A post from a 3M-follower account will enqueue 3M fan-out tasks in seconds. Your worker pool must be sized for burst, not steady-state. Use auto-scaling workers with a max concurrency cap, and ensure your sorted-set write throughput on Redis can absorb the burst. Consider rate-limiting fan-out per account (process 10K/second, not 3M at once) with a backlog queue.

Re-ranking stale feeds. If you run a periodic job that updates scores based on engagement, you will write to potentially millions of sorted sets. Run these jobs during off-peak hours, batch updates per event rather than per follower, and use ZADD XX (update existing only) to avoid inserting expired events back into active feeds.

SSE connection limits at scale. Each open SSE connection is a persistent HTTP connection. At 1 million concurrent users, you need either very many server instances or a dedicated gateway that multiplexes many user subscriptions behind fewer upstream connections. Do not let SSE connections land on the same servers handling feed reads; isolate the connection-holding layer.

Deleted content in materialized feeds. When a user deletes a post, references to its event ID remain in all follower sorted sets. Use a deny-list at hydration time: before returning post bodies, filter out deleted post IDs. Update the deny-list synchronously on delete; it is a small set relative to the event volume.

Risk	Signal	Mitigation
Fan-out queue saturation	Queue depth > 1M, lag growing	Celebrity threshold, rate-limited worker pool
Cold feed on login	P95 login latency spike	Proactive warm on auth success
Stale ranked feeds	Engagement-based scores not reflected	Async re-rank jobs on score change
Deleted post visible	User reports seeing deleted content	Deny-list filter at hydration
SSE connections OOM	Process RSS growing with connection count	Isolated connection gateway, connection TTL

Closing Thought

The feed problem is a write amplification problem disguised as a read problem. Most teams optimize the read path first (caching, sorted sets) without thinking through what happens to the write side at scale, then hit a celebrity account that overwhelms the fan-out workers, and retrofit the hybrid model under pressure.

Design the hybrid from the start. Pick a celebrity threshold that is comfortably below the follower count that would saturate your fan-out pipeline. Build the read-time merge path for celebrity events before you need it. The additional complexity at the start is much smaller than the complexity of migrating a live feed system under traffic.

The ranking and real-time delivery layers are additive. Get the fan-out strategy right first; everything else can be layered on top without structural rearchitecting.