Building a Recommendation Engine: Collaborative Filtering, Vector Similarity, and Real-Time Personalization in Production

Most teams build their first recommendation engine by following a textbook example: build a user-item interaction matrix, run matrix factorization, serve the output. It works in a Jupyter notebook. It fails in production.

The problems are predictable: cold start for new users and items, stale embeddings served from a batch pipeline that ran six hours ago, no way to know whether recommendations are driving clicks or just surfacing what users would have found anyway, and a serving layer that adds 400ms to every page load.

This article covers how recommendation systems actually work in production environments, where each of those failure modes costs real money.

The Three Approaches (and Where Each Breaks Down)

Collaborative Filtering

Collaborative filtering works from user behavior: users who interacted similarly in the past will likely respond similarly in the future. The implementation can be user-based (find users similar to you, recommend what they liked) or item-based (find items similar to what you’ve interacted with, recommend those).

Matrix factorization (ALS, SVD) is the standard approach. It factorizes the user-item interaction matrix into latent factor representations, then computes similarity in that latent space. The Netflix Prize popularized this. It still works well for catalogs where you have dense historical interaction data.

Where it breaks down: new users have no interaction history. New items have no interactions. This is the cold start problem. Collaborative filtering is also a closed system: it can only recommend items that exist in the training data, which creates a feedback loop where popular items get recommended more, collect more interactions, and become even more popular. This is filter bubble behavior, and it actively harms discovery for new content.

Content-Based Filtering

Content-based filtering ignores user behavior and instead models item similarity directly from item features: genre, tags, text description, visual features. Given a user’s history, find items with similar features.

This solves the item cold start problem: a new item with known features can be recommended immediately. It also avoids filter bubbles, since item similarity is computed from properties, not from what other users did.

Where it breaks down: it requires good item metadata, which is often missing or inconsistent in practice. It cannot surface items that are surprising or serendipitous, because it can only find items that look like what the user already consumed. It also requires the user to have some history (user cold start still applies).

Hybrid Systems

Production systems almost always combine both. The specific blending strategy matters: you can blend at the scoring level (weighted sum of collaborative and content scores), at the candidate retrieval level (use both sources to generate candidates, then re-rank), or at the model level (learn a unified model that consumes both interaction features and content features together).

Candidate retrieval followed by re-ranking is the most common production architecture, and the one that scales. Retrieval fetches a large candidate set (hundreds to thousands of items) efficiently from approximate nearest neighbor indexes. Re-ranking applies a heavier model over that smaller candidate set to produce final scores. This separation lets you optimize each stage independently: retrieval for recall, re-ranking for precision.

Embedding-Based Recommendations

The modern approach treats recommendation as a vector similarity problem. Items and users are embedded into a shared latent space using a learned model. At serve time, you find the nearest items to the user’s current embedding.

The embedding model can be trained in several ways:

• Two-tower architecture: separate encoder for user features and item features, trained so that user-item pairs with positive interactions are close in the shared space
• Sequence models (BERT4Rec, SASRec): model the user’s interaction sequence with transformers, then predict what comes next
• Graph neural networks: explicitly model the user-item interaction graph, where node embeddings capture structural proximity

The key advantage over classical collaborative filtering: once you have embeddings, you can do approximate nearest neighbor search at query time with sub-millisecond latency. Libraries like FAISS (Facebook AI Similarity Search), Annoy, or a vector database like Qdrant or Weaviate handle the retrieval.

Here is a minimal embedding-based recommendation service in TypeScript:

import { QdrantClient } from "@qdrant/js-client-rest";

interface UserProfile {
  userId: string;
  embedding: number[];
  updatedAt: Date;
}

interface RecommendationResult {
  itemId: string;
  score: number;
  metadata: Record<string, unknown>;
}

const qdrant = new QdrantClient({ url: process.env.QDRANT_URL });

const COLLECTION = "items";
const EMBEDDING_DIM = 256;

// Retrieve user embedding from feature store
async function getUserEmbedding(userId: string): Promise<number[] | null> {
  const res = await fetch(
    `${process.env.FEATURE_STORE_URL}/users/${userId}/embedding`
  );
  if (!res.ok) return null;
  const profile: UserProfile = await res.json();
  return profile.embedding;
}

// Fetch nearest item embeddings from vector store
async function fetchCandidates(
  userEmbedding: number[],
  limit: number,
  filters?: Record<string, unknown>
): Promise<RecommendationResult[]> {
  const results = await qdrant.search(COLLECTION, {
    vector: userEmbedding,
    limit,
    filter: filters
      ? { must: Object.entries(filters).map(([key, value]) => ({ key, match: { value } })) }
      : undefined,
    with_payload: true,
  });

  return results.map((r) => ({
    itemId: r.id as string,
    score: r.score,
    metadata: r.payload as Record<string, unknown>,
  }));
}

// Re-rank candidates with a lightweight scoring model
async function rerank(
  userId: string,
  candidates: RecommendationResult[]
): Promise<RecommendationResult[]> {
  const res = await fetch(`${process.env.RANKER_URL}/rank`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ userId, candidates }),
  });
  if (!res.ok) return candidates; // fall back to vector score if ranker fails
  return res.json();
}

export async function getRecommendations(
  userId: string,
  options: {
    limit?: number;
    filters?: Record<string, unknown>;
    fallbackToPopular?: boolean;
  } = {}
): Promise<RecommendationResult[]> {
  const { limit = 10, filters, fallbackToPopular = true } = options;

  const userEmbedding = await getUserEmbedding(userId);

  if (!userEmbedding) {
    // Cold start: return popular items if no user embedding exists
    if (fallbackToPopular) {
      return getPopularItems(limit, filters);
    }
    return [];
  }

  // Fetch more candidates than needed to give ranker room to work
  const candidates = await fetchCandidates(userEmbedding, limit * 5, filters);
  const ranked = await rerank(userId, candidates);

  return ranked.slice(0, limit);
}

async function getPopularItems(
  limit: number,
  filters?: Record<string, unknown>
): Promise<RecommendationResult[]> {
  const results = await qdrant.scroll(COLLECTION, {
    limit,
    filter: filters
      ? { must: Object.entries(filters).map(([key, value]) => ({ key, match: { value } })) }
      : undefined,
    with_payload: true,
    order_by: { key: "popularity_score", direction: "desc" },
  });

  return (results.points ?? []).map((r) => ({
    itemId: r.id as string,
    score: (r.payload?.popularity_score as number) ?? 0,
    metadata: r.payload as Record<string, unknown>,
  }));
}

The rerank function is a critical detail. Embedding similarity alone scores items by how closely they match the user’s historical representation. A re-ranker can apply business rules, freshness bonuses, diversity penalties (to avoid ten near-identical items), and user-specific contextual signals that are too expensive to encode in the embedding itself.

The Cold Start Problem in Practice

Cold start is not a single problem. There are three distinct failure modes:

User cold start: a new user has no history. The embedding model has nothing to condition on. Solutions in increasing complexity: (1) use demographic or onboarding signals to infer a starting embedding, (2) prompt the user to select preferences explicitly and use those as a signal, (3) use a content-based fallback that does not require user history.

Item cold start: a new item enters the catalog with no interactions. Collaborative filtering cannot rank it. Solution: generate an embedding from item content (text, image, metadata) using the content encoder half of your two-tower model. This item can immediately participate in similarity search even before it has interactions. You then transition gradually from content-based to interaction-based scoring as interaction data accumulates.

System cold start: you are building from scratch with no historical data at all. This is the most painful case. Start with content-based recommendations from day one. Instrument interaction events carefully. Once you have a few thousand interaction events, you can begin training a collaborative model. Set explicit thresholds: “we switch to collaborative filtering when we have X distinct users with Y or more interactions each.”

Real-Time vs Batch Pipelines

Most recommendation systems have both. The question is what each pipeline handles.

Batch pipeline: generates user embeddings and pre-computes recommendation lists for known users on a schedule (hourly, daily). Stores results in a fast key-value store (Redis, DynamoDB). Serving is just a cache lookup. Latency is very low; freshness is limited by batch cadence.

Real-time pipeline: processes interaction events as they happen (Kafka, Kinesis) and updates user state incrementally. Does not re-train the model on every event, but updates a user feature vector in the feature store. The next recommendation request uses the updated features.

Online inference: calls the ranking model at query time with fresh features. Required when you need to incorporate signals from the current session (what the user just clicked, search context, location). Adds latency; requires an inference endpoint with tight SLOs.

A common production pattern: pre-compute item embeddings daily, update user embeddings every 15-30 minutes via a streaming job, and call the re-ranker at query time with session-level features. Staleness measured in minutes, not hours, without the cost of full online inference on every request.

Feature Stores

A feature store is a system that manages the feature computation, storage, and serving pipelines for ML models. For recommendation systems, it solves a specific problem: the features your model trains on (batch-computed historical aggregates) need to be available at serving time with low latency and without re-computing everything.

Without a feature store, you commonly hit one of two failure modes. Either serving logic re-computes features from scratch (slow, expensive, inconsistent with training), or you hard-code feature computation into the serving path and it diverges from training as the code evolves. This training-serving skew is one of the most common causes of recommendation quality degradation in production.

Key features to serve for a recommendation model:

• User features: historical interaction counts, category affinities, recency scores, session context
• Item features: popularity in the last 24h/7d, content embeddings, freshness
• User-item features (cross features): has the user seen this item before, time since last interaction with this category

Managed options include Feast (open source), Tecton, and Vertex AI Feature Store. At smaller scale, Redis with structured keys and TTL-based expiry handles most use cases without the operational complexity of a dedicated feature platform.

A/B Testing Recommendations

Recommendations are notoriously hard to evaluate offline. Metrics like precision@k, recall@k, and NDCG measure how well the model predicts held-out interactions, but they do not tell you whether better recommendations drive more business value. Users may click on worse recommendations because they are more novel, or ignore better ones because they are surfaced in a less visible position.

This means online A/B testing is mandatory, not optional. But recommendation A/B tests have specific pitfalls:

Novelty effect: a new recommendation algorithm often gets a short-term click-rate boost simply because it surfaces unfamiliar items. Users click out of curiosity, not preference. Run experiments for long enough (two or more weeks) to let novelty effects decay.

Spillover effects: in social platforms, variant B can increase distribution for certain creators, which bleeds into the control group. Pure A/B testing breaks down; cluster-level randomization (by creator or community) is required.

Metric selection: click-through rate is easy to measure and often misleading. A model that recommends low-quality clickbait will win on CTR. Define metrics that proxy the business outcome you actually care about: session depth, return visit rate, purchase conversion, long-term retention. Track both.

A minimal but honest metric set for most recommendation systems: CTR (sanity check), satisfaction signal (explicit rating or completion rate), and 7-day retention delta between variants.

Production Architecture for Low-Latency Serving

A recommendation serving endpoint needs to be fast. Users tolerate 100-200ms for a page recommendation widget, not 800ms. Here is a typical architecture for hitting p95 under 150ms:

User Request
    |
    v
API Gateway (auth, rate limit)
    |
    v
Recommendation Service (Node.js / Go)
    |
    +-- Feature lookup (Redis) -------> user embedding, session features
    |
    +-- ANN search (Qdrant/FAISS) ----> top-K candidate items
    |
    +-- Re-ranker (gRPC call) --------> final scored list
    |
    v
Response (< 150ms p95)

Critical implementation details:

Cache aggressively at the right layer. User embeddings change slowly. Cache them in Redis with a 15-minute TTL. Item embeddings change even more slowly; cache them for hours. The ANN search itself can be cached for a user with a short TTL (30-60 seconds) if the same user is making rapid requests.

Parallelize independent calls. Feature lookups and candidate pre-fetching for different recommendation widgets on a page should happen in parallel, not serial.

Set hard timeouts on the re-ranker. If the re-ranker takes longer than your budget allows, fall back to the vector similarity ranking. A slightly worse ranking is far better than a timeout.

Monitor p99, not just p95. Recommendation endpoints are called on every page load for logged-in users. A fat tail in your latency distribution is a user experience problem at scale.

When to Use Off-the-Shelf vs Custom

Dimension	Off-the-Shelf (Algolia Recommend, Amazon Personalize)	Custom (two-tower + ANN search)
Time to first result	Days	Weeks to months
Cold start handling	Built-in, well-tested	Must implement explicitly
Cost at low volume	Predictable, low	Engineering overhead dominates
Cost at high volume	Can become expensive per-event	Infrastructure cost dominates
Customization depth	Limited to API surface	Full control
Data sovereignty	Vendor holds data	You control data
Business logic integration	Webhook/API-level integration	Direct code integration
ML team required	No	Yes

Use off-the-shelf if: you do not have an ML engineer, you are in the first six months of a product, you have fewer than 100k monthly active users, or your recommendation surface is not a core product differentiator.

Build custom if: you need very tight latency SLOs (sub-50ms), your item catalog has complex metadata or multimodal features (text + images + structured attributes) that off-the-shelf models cannot consume well, you need recommendations as a core competitive moat, or your interaction signal is unusual (not clicks and purchases but domain-specific behavior).

A common path: start with Algolia Recommend or Amazon Personalize to validate that recommendations drive business value, then migrate to a custom system once you have enough data, enough traffic, and a clear signal that the off-the-shelf model is the constraint.

Production Considerations

Versioning and rollout. Never flip recommendation models for all users at once. Use shadow mode first (run the new model in parallel, log outputs, compare against current model offline). Then ramp: 1%, 10%, 50%, 100%. Define rollback criteria before the ramp begins.

Embedding drift. Catalogs evolve and user tastes shift. If you do not retrain regularly, performance degrades silently. Schedule retraining weekly for catalogs that change slowly; monitor AUC on held-out interactions and recall@k on a fixed evaluation set.

Observability. Track recommendation request volume by surface, CTR per surface, null-result rate (empty candidate sets indicate cold start or index failures), and ranking latency broken down by stage: ANN search, re-rank, feature fetch.

Idempotency of interaction events. Click and purchase events arrive as duplicates. Deduplicate by event ID before updating feature values, or you will over-weight interaction signals and corrupt user embeddings.

Index memory. ANN indexes are memory-resident. FAISS HNSW loads into RAM at startup. For catalogs over 50M items, plan for sharding, disk-based indexes (Qdrant’s on-disk storage), or aggressive pre-filtering to reduce effective index size.

---

The simplest thing that works is not always the simplest system. For a product where recommendations drive core engagement, the two-tower architecture plus candidate retrieval plus re-ranking is worth building right from the start. The candidate set retrieval is fast and cheap; the re-ranker is where you encode business judgment.

The failure mode to avoid is the one where you build a pure collaborative filter, it works fine for active users, and you never fix cold start because it is always “next sprint.” New users are the ones you have not converted yet. That is where the system needs to work hardest.