Designing a Recommendation System: Collaborative Filtering, Real-Time Scoring, and Feature Store Architecture

Recommendation systems are deceptively simple to prototype and genuinely hard to operate at scale. A few matrix factorizations and cosine similarity lookups will get you a demo. Serving accurate, low-latency recommendations to millions of users while handling cold start, feedback loops, and model drift is a different problem entirely.

This article walks through the full architecture: candidate generation, ML-based ranking with a feature store, the serving layer and its latency constraints, and the production concerns that determine whether the system keeps improving or slowly drifts into irrelevance.

The Two-Stage Pipeline

Every recommendation system at scale uses a two-stage architecture: retrieve a small candidate set, then rank it. The reason is math. If your catalog has 50 million items and you need a recommendation response in under 100ms, you cannot score all 50 million items per request. The retrieval stage prunes the space down from millions to hundreds. The ranking stage applies an expensive ML model to that smaller set.

User request
  └── Candidate Generation  (millions → ~500 candidates)
        ├── Collaborative Filtering (ANN over user/item embeddings)
        ├── Content-Based Filtering (item features, tags, category)
        └── Rule-Based (trending, seasonal, editorial)
  └── Ranking                (~500 candidates → top N)
        ├── Feature Store lookup (user features + item features + context)
        ├── ML ranking model inference
        └── Business rules (diversity, freshness, pacing)
  └── Response

Each stage has a different latency budget and a different data dependency. Getting those two budgets wrong is where most designs break.

Candidate Generation

Collaborative Filtering with Embeddings

Classic matrix factorization (ALS, SVD) learns a low-dimensional embedding for every user and item such that the dot product between a user vector and an item vector approximates that user’s rating or interaction signal. In production, you precompute all item embeddings and store them in an approximate nearest neighbor (ANN) index. At request time, you fetch the user embedding and run a single ANN query.

interface UserEmbedding {
  userId: string;
  vector: Float32Array;   // 128-256 dimensions is typical
  computedAt: string;
}

interface ItemEmbedding {
  itemId: string;
  vector: Float32Array;
  computedAt: string;
}

// ANN retrieval: fetch top-K items by cosine similarity
async function retrieveCandidatesByEmbedding(
  userEmbedding: Float32Array,
  index: ANNIndex,
  topK: number = 500
): Promise<string[]> {
  const results = await index.query({
    vector: userEmbedding,
    topK,
    filter: { isActive: true },
  });
  return results.map((r) => r.id);
}

The embedding model retrains on a schedule (daily or weekly, depending on catalog size and interaction volume). The ANN index rebuilds after each training run. Index reads are served from memory, which is why latency is predictable in the 5-20ms range even at scale.

Common ANN backends: Faiss (self-hosted, high throughput), Pinecone or Weaviate (managed, easier ops), pgvector (if your candidate space is small enough and you want fewer moving parts). The choice depends on your catalog size and whether you want to own the infrastructure.

Content-Based Filtering

Content-based retrieval ignores interaction history and retrieves items similar to what the user has engaged with, based on item features. This is your fallback when interaction data is thin and your primary path for new-item retrieval before those items accumulate interactions.

interface ItemFeatures {
  itemId: string;
  category: string;
  tags: string[];
  textEmbedding: Float32Array;   // from a language model encoder
  price: number;
  publishedAt: string;
}

async function retrieveCandidatesByContent(
  seedItemIds: string[],
  catalog: ItemFeatureStore,
  topK: number = 200
): Promise<string[]> {
  // Average the embeddings of seed items to get a query vector
  const seedEmbeddings = await catalog.getEmbeddings(seedItemIds);
  const queryVector = averageVectors(seedEmbeddings);
  return catalog.queryByVector(queryVector, topK);
}

Content signals age better than interaction signals. An item published today has no interactions but does have a text embedding, a category, and tags. Content-based retrieval surfaces it immediately. Collaborative filtering surfaces it once users start engaging with it. Running both in parallel and merging the candidate lists is the baseline hybrid approach.

Merging Candidate Sources

With multiple retrieval paths, you need to deduplicate and decide on pool size before ranking. A simple merge is enough here. The ranking stage will sort them correctly.

function mergeCandidates(
  sources: { candidates: string[]; weight: number }[]
): string[] {
  const seen = new Set<string>();
  const merged: string[] = [];

  // Interleave sources proportionally to their weight
  const iterators = sources.map((s) => ({
    items: s.candidates[Symbol.iterator](),
    budget: s.weight,
  }));

  for (const iter of iterators) {
    for (const itemId of iter.items) {
      if (!seen.has(itemId)) {
        seen.add(itemId);
        merged.push(itemId);
      }
      if (merged.length >= 500) break;
    }
  }

  return merged;
}

The Feature Store

The ranking model needs features for both the user and each candidate item at inference time. This is where most systems introduce latency surprises. Fetching features from a cold database on every request is not an option at 50ms budgets.

A feature store has two layers:

• Online store: low-latency key-value lookup (Redis, DynamoDB, or Bigtable). Pre-materialized feature vectors. Reads at 1-5ms p99.
• Offline store: full feature history, used for model training (data warehouse, Parquet files).

The critical constraint: the online store must be updated continuously from the offline store (or from event streams) so that the features the model sees at serving time match the distribution it was trained on. Feature skew between training and serving is one of the most common sources of silent model degradation.

interface UserFeatureVector {
  userId: string;
  recentCategoryAffinities: Record<string, number>;
  avgSessionLengthDays7: number;
  purchaseFrequency: number;
  accountAgeDays: number;
  lastActiveAt: string;
}

interface ItemFeatureVector {
  itemId: string;
  globalCtr: number;
  categoryRank: number;
  ageHours: number;
  pricePercentile: number;
  stockStatus: "in_stock" | "low_stock" | "out_of_stock";
}

class FeatureStore {
  constructor(
    private readonly redis: RedisClient,
    private readonly ttlSeconds: number = 3600
  ) {}

  async getUserFeatures(userId: string): Promise<UserFeatureVector | null> {
    const raw = await this.redis.get(`user_features:${userId}`);
    if (!raw) return null;
    return JSON.parse(raw) as UserFeatureVector;
  }

  async batchGetItemFeatures(
    itemIds: string[]
  ): Promise<Map<string, ItemFeatureVector>> {
    const keys = itemIds.map((id) => `item_features:${id}`);
    const values = await this.redis.mget(...keys);
    const result = new Map<string, ItemFeatureVector>();
    for (let i = 0; i < itemIds.length; i++) {
      if (values[i]) {
        result.set(itemIds[i], JSON.parse(values[i]) as ItemFeatureVector);
      }
    }
    return result;
  }
}

The batch read for item features matters. With 500 candidates per request and a 50ms budget, you cannot do 500 sequential reads. MGET in Redis returns all values in a single round trip, keeping the feature fetch inside 5-10ms for batches of this size.

ML Ranking

The ranking model takes the merged candidate list plus features and outputs a score per candidate. In a well-tuned system, the model is a gradient-boosted tree (XGBoost, LightGBM) or a shallow neural network, not a large transformer. The reason is latency. Gradient-boosted trees score 500 items in 5-10ms on a single CPU core. A transformer-based re-ranker is more accurate but adds 50-150ms, which breaks most serving budgets.

interface RankingInput {
  userId: string;
  userFeatures: UserFeatureVector;
  candidates: Array<{
    itemId: string;
    itemFeatures: ItemFeatureVector;
    retrievalScore: number;    // similarity score from the ANN stage
    retrievalSource: string;   // "collaborative" | "content" | "trending"
  }>;
  contextFeatures: {
    requestTime: string;
    deviceType: "mobile" | "desktop" | "tablet";
    surfaceId: string;         // which page/widget is requesting
  };
}

interface RankedCandidate {
  itemId: string;
  score: number;
  explanation: string[];      // for debugging and model transparency
}

async function rankCandidates(
  input: RankingInput,
  model: RankingModel
): Promise<RankedCandidate[]> {
  const featureMatrix = buildFeatureMatrix(input);
  const scores = await model.predict(featureMatrix);

  return input.candidates
    .map((c, i) => ({
      itemId: c.itemId,
      score: scores[i],
      explanation: extractFeatureContributions(featureMatrix[i], scores[i]),
    }))
    .sort((a, b) => b.score - a.score);
}

After ranking, apply business rules in a post-processing pass. This is separate from the model intentionally. Diversity constraints (no more than 2 items from the same category in the top 5), freshness boosts, and out-of-stock suppression are rules, not signals. Baking them into the model makes them hard to change and nearly impossible to audit.

The Serving Layer and Latency Budget

A realistic latency budget for a recommendation API endpoint:

Request received
  └── Auth/routing                  ~2ms
  └── User embedding fetch          ~3-5ms   (Redis, precomputed)
  └── ANN retrieval (CF path)       ~10-20ms (HNSW graph traversal)
  └── Content retrieval             ~10-15ms (parallel with CF)
  └── Candidate merge               ~1ms
  └── Feature batch fetch           ~5-10ms  (Redis MGET, ~500 items)
  └── Ranking model inference       ~5-15ms  (gradient boosted tree)
  └── Post-processing + serialise   ~2-5ms
Total target: p50 ~40ms, p99 ~80ms

The CF and content retrieval paths run in parallel. That is the most important performance decision in the serving stack. Running them serially doubles the latency floor for no benefit.

async function generateRecommendations(
  userId: string,
  surfaceId: string,
  count: number,
  deps: { featureStore: FeatureStore; annIndex: ANNIndex; model: RankingModel }
): Promise<RankedCandidate[]> {
  const [userFeatures, userEmbedding] = await Promise.all([
    deps.featureStore.getUserFeatures(userId),
    deps.featureStore.getUserEmbedding(userId),
  ]);

  // Parallel candidate retrieval
  const [cfCandidates, contentCandidates, trendingCandidates] =
    await Promise.all([
      userEmbedding
        ? retrieveCandidatesByEmbedding(userEmbedding, deps.annIndex, 300)
        : Promise.resolve([]),
      userFeatures
        ? retrieveCandidatesByContent(
            getRecentInteractionItems(userId),
            deps.featureStore,
            200
          )
        : Promise.resolve([]),
      fetchTrendingItems(surfaceId, 100),
    ]);

  const candidates = mergeCandidates([
    { candidates: cfCandidates, weight: 0.5 },
    { candidates: contentCandidates, weight: 0.3 },
    { candidates: trendingCandidates, weight: 0.2 },
  ]);

  const itemFeatures = await deps.featureStore.batchGetItemFeatures(candidates);

  const ranked = await rankCandidates(
    {
      userId,
      userFeatures: userFeatures ?? defaultUserFeatures(),
      candidates: candidates.map((id) => ({
        itemId: id,
        itemFeatures: itemFeatures.get(id) ?? defaultItemFeatures(id),
        retrievalScore: 1.0,
        retrievalSource: "merged",
      })),
      contextFeatures: {
        requestTime: new Date().toISOString(),
        deviceType: "desktop",
        surfaceId,
      },
    },
    deps.model
  );

  return applyBusinessRules(ranked, { maxPerCategory: 2, count });
}

Caching Recommendations

For non-personalized surfaces (homepage trending, category top-picks), cache the full ranked list and invalidate on a TTL. For personalized surfaces, precompute recommendations asynchronously and cache per user with a shorter TTL (15-30 minutes). Real-time computation on every request is expensive and often unnecessary. A user’s recommendation profile changes on the order of hours, not seconds.

The tradeoff: precomputed recommendations are stale. Real-time recommendations are fresh but expensive. Most systems serve precomputed for the first response and trigger an async refresh in the background.

Cold Start

Cold start comes in two varieties: new users and new items.

New users. You have no interaction history and no user embedding. Fallbacks in order of preference: (1) ask for explicit preferences during onboarding, (2) infer from demographics or referral source, (3) serve popularity-based recommendations in the user’s inferred segment, (4) serve global trending. Start collecting implicit feedback (clicks, dwell time, scrolling depth) from the first session. After 5-10 interactions, collaborative signals become usable.

New items. No interactions means no appearance in collaborative filtering output. Route around the problem: surface new items through the content-based retrieval path immediately. Use impression-exploration policies (similar to epsilon-greedy or Thompson sampling) to deliberately inject new items into recommendation slates for a proportion of users. Track click-through and engagement. Once the item accumulates enough interaction data, it enters the collaborative pool.

function applyNewItemExploration(
  ranked: RankedCandidate[],
  newItems: string[],
  explorationRate: number = 0.1
): RankedCandidate[] {
  const explorationSlots = Math.ceil(ranked.length * explorationRate);
  const explorationItems = newItems.slice(0, explorationSlots).map((id) => ({
    itemId: id,
    score: 0,
    explanation: ["new_item_exploration"],
  }));

  // Inject at fixed positions (e.g., position 3, 7, 12) for consistent measurement
  const result = [...ranked];
  const injectionPositions = [2, 6, 11];
  for (let i = 0; i < Math.min(explorationItems.length, injectionPositions.length); i++) {
    result.splice(injectionPositions[i], 0, explorationItems[i]);
  }

  return result.slice(0, ranked.length);
}

Feedback Loops and Drift

A recommendation system that only learns from its own outputs will develop a feedback loop. The items it recommends get more clicks, which strengthens their features, which makes them rank higher, which crowds out everything else. Popularity bias compounds over time.

Mitigations:

• Explore-exploit balance. Reserve a fraction of recommendation slots for exploration (random items, new items, serendipity injection). Track those slots separately so they do not distort your model training data.
• Counterfactual logging. Log not just what was recommended and clicked, but what was shown and not clicked (negative signal) and what was in the candidate pool but not shown. Without negative signal, your model trains on a biased dataset.
• Diversity constraints. Enforce at serving time, not in the model. Post-process the ranked list to limit repetition. Maximum 2 items from the same seller, no more than 30% from a single category.

Drift detection: monitor the distribution of served items over time. If the top 1% of items are capturing 60% of impressions, the feedback loop is compressing diversity. Set alerts on the Gini coefficient of impression distribution. A rising Gini coefficient is an early signal before quality metrics degrade visibly.

A/B Testing Recommendations

Recommendation A/B tests are harder than standard feature experiments. Users interact with recommendations over time, so a single session is not the unit of measurement. Use user-level assignment with holdout periods long enough to see repeat behavior (7-14 days minimum, 30 days for lower-traffic surfaces).

Metrics that matter: click-through rate is easy to measure but weakly correlated with business value. Measure downstream conversion (purchase, saved item, return visit) and long-term engagement (session depth in the following week). CTR optimizes for clickbait. Downstream metrics optimize for what users actually wanted.

Guard against novelty effect: a new recommendation model always shows a CTR lift in the first 48 hours because anything new looks different. Extend your experiment window beyond the novelty decay curve before drawing conclusions.

Tradeoffs

Decision	Option A	Option B	When to prefer B
Candidate retrieval	ANN on precomputed embeddings	Live item scoring	Catalog under 10K items; simplicity beats performance
Feature freshness	Precomputed hourly batch	Real-time event-driven updates	User behavior changes within the hour (news, live events)
Ranking model	Gradient boosted tree	Neural re-ranker	Latency budget above 200ms and training data above 10M events
Recommendation freshness	Precomputed per-user cache	Real-time on every request	High-velocity catalogs or real-time context (current cart)
Cold start fallback	Trending items	Segment-based	User acquisition channels are diverse and have distinct preferences

Production Concerns

Model versioning. Shadow-serve new models alongside production before promoting. Log scores from both models for offline comparison. Never promote a new model without at least 24 hours of shadow data.

Feature pipeline failures. If the feature store returns empty for a user, you need a graceful fallback. The serving layer should detect missing features and fall back to a simpler model (popularity-based ranking without personalization) rather than returning an error or serving garbage scores.

Latency percentiles. Track p50, p95, and p99 separately. p50 tells you about the typical case. p99 tells you about the user experience for your worst-served users. Recommendation latency at p99 is often driven by ANN index latency under high concurrent load. Index sharding and load balancing the ANN layer are the primary levers.

Monitoring. The metrics that matter most are not just latency and error rate. Watch engagement metrics (CTR, session depth) on a rolling 7-day window. A drop in engagement with no corresponding change in error rate usually means model drift, feature pipeline problems, or a silent change in how users are interacting with your product. Connect model monitoring to the same alerting infrastructure as infrastructure monitoring.

Closing Thought

A recommendation system’s architecture is relatively standard across companies. The candidate generation, ranking, and serving layers look similar everywhere. What differentiates production systems is how they handle the parts that are easy to ignore: cold start edge cases, feedback loop suppression, feature freshness under load, and the telemetry needed to detect drift before users notice.

Build the measurement infrastructure before you build the models. A recommendation system you cannot measure is a system you cannot improve.