Building a Semantic Search Engine: Embedding Pipelines, Approximate Nearest Neighbors, and Relevance Tuning in Production

Keyword search breaks in a specific and predictable way. A user types “how do I cancel my account” and your Elasticsearch index returns nothing useful because your documentation calls it “deactivating a subscription.” The words do not overlap. BM25 scores zero. The user sees an empty results page and opens a support ticket instead.

This is not a rare edge case. It is the normal failure mode for intent-based queries, which represent the majority of search traffic on most products. Users describe what they want to do, not what you called the feature. Keyword search does not bridge that gap.

Semantic search solves this by searching in meaning-space rather than token-space. Two sentences that mean the same thing but share no words will have similar embedding vectors and will appear close together in the vector index. That is the core idea. The engineering challenge is building a system that does this reliably, at acceptable latency, at reasonable cost, and with relevance that holds up as your corpus changes.

This article covers the full architecture: embedding selection, pipeline construction, ANN index types, hybrid scoring, re-ranking, and the production concerns that are easy to underestimate.

Embedding Model Selection

Not all embedding models are equivalent. The two axes that matter most for production are: embedding quality on your domain, and inference throughput.

For English text, all-MiniLM-L6-v2 from sentence-transformers is a reasonable starting point. It produces 384-dimensional vectors, runs fast on CPU, and has decent general-purpose quality. If your corpus is technical or domain-specific, bge-large-en-v1.5 or e5-large-v2 will outperform it at the cost of higher inference time and larger index size (1024 dimensions vs 384).

OpenAI’s text-embedding-3-small is worth benchmarking against. It is competitive on MTEB benchmarks and removes the infrastructure burden of running your own model, but you pay per token and your pipeline has an external latency dependency. At scale, self-hosted wins on cost. At lower volume, managed wins on operational simplicity.

One decision that is often deferred too long: context length. all-MiniLM-L6-v2 has a 256-token limit. If you feed it 2000-word documents without chunking, the later content is simply truncated and not represented in the embedding. This is one of the most common causes of poor recall in production systems.

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("BAAI/bge-large-en-v1.5")

def embed_batch(texts: list[str], batch_size: int = 64) -> np.ndarray:
    """
    Embed a list of texts in batches.
    BGE models expect a query prefix for retrieval tasks.
    Documents do NOT get the prefix. Queries DO.
    """
    return model.encode(
        texts,
        batch_size=batch_size,
        normalize_embeddings=True,  # required for cosine similarity via dot product
        show_progress_bar=False,
    )

def embed_query(query: str) -> np.ndarray:
    # BGE instruction prefix improves retrieval quality
    prefixed = f"Represent this sentence for searching relevant passages: {query}"
    return model.encode(prefixed, normalize_embeddings=True)

The normalize_embeddings=True flag matters. Normalized vectors allow you to use dot product instead of cosine similarity, which is faster in most ANN implementations and produces equivalent results when both vectors are unit-normalized.

The Embedding Pipeline

Documents do not embed themselves. You need a pipeline that handles chunking, enriches chunks with metadata, and keeps the index current as your corpus changes.

Chunking Strategy

The goal is to produce chunks where each chunk contains one coherent idea and fits within the model’s context window. There is no universally correct chunk size. Larger chunks preserve more context but dilute the embedding signal. Smaller chunks are more precise but lose surrounding context.

A pragmatic default: 512 tokens with 64-token overlap. The overlap prevents ideas that span chunk boundaries from disappearing entirely.

For structured content (FAQs, documentation with headers, product descriptions), chunking at the document or section boundary is better than fixed-token chunking. The structure already tells you where ideas begin and end.

from dataclasses import dataclass
from typing import Optional
import tiktoken

@dataclass
class Chunk:
    text: str
    doc_id: str
    chunk_index: int
    metadata: dict
    token_count: int

enc = tiktoken.get_encoding("cl100k_base")

def chunk_document(
    text: str,
    doc_id: str,
    metadata: dict,
    chunk_size: int = 512,
    overlap: int = 64,
) -> list[Chunk]:
    tokens = enc.encode(text)
    chunks = []
    start = 0
    index = 0

    while start < len(tokens):
        end = min(start + chunk_size, len(tokens))
        chunk_tokens = tokens[start:end]
        chunk_text = enc.decode(chunk_tokens)

        chunks.append(Chunk(
            text=chunk_text,
            doc_id=doc_id,
            chunk_index=index,
            metadata=metadata,
            token_count=len(chunk_tokens),
        ))

        start += chunk_size - overlap
        index += 1

    return chunks

Metadata Enrichment

Metadata is not optional. It is how you filter results without re-ranking everything. Every chunk should carry at minimum: document ID, source URL or path, creation date, content type (article, FAQ, product), and any domain-specific facets (category, language, tenant ID for multi-tenant systems).

You will use these for pre-filtering in the ANN query: “find the 20 nearest neighbors, but only from chunks created after 2025-01-01 in category=billing.” Most vector databases support metadata filtering natively. Design your metadata schema before you build the pipeline, because changing it later means re-indexing.

Async Batched Ingestion (TypeScript API Layer)

The TypeScript service layer handles ingestion requests from the application, fans them out to the embedding workers, and writes results to the vector store.

import { QdrantClient } from "@qdrant/js-client-rest";

interface Document {
  id: string;
  text: string;
  metadata: Record<string, unknown>;
}

interface EmbedResponse {
  embeddings: number[][];
}

const qdrant = new QdrantClient({ url: process.env.QDRANT_URL });
const COLLECTION = "docs";
const EMBED_BATCH_SIZE = 64;

async function fetchEmbeddings(texts: string[]): Promise<number[][]> {
  const res = await fetch(`${process.env.EMBED_SERVICE_URL}/embed`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ texts }),
  });
  if (!res.ok) throw new Error(`Embed service error: ${res.status}`);
  const data = (await res.json()) as EmbedResponse;
  return data.embeddings;
}

export async function ingestDocuments(documents: Document[]): Promise<void> {
  for (let i = 0; i < documents.length; i += EMBED_BATCH_SIZE) {
    const batch = documents.slice(i, i + EMBED_BATCH_SIZE);
    const texts = batch.map((d) => d.text);
    const embeddings = await fetchEmbeddings(texts);

    const points = batch.map((doc, idx) => ({
      id: doc.id,
      vector: embeddings[idx],
      payload: doc.metadata,
    }));

    await qdrant.upsert(COLLECTION, { points, wait: true });
  }
}

The wait: true flag tells Qdrant to block until the write is acknowledged by the index. Remove it for higher-throughput bulk ingestion where you can tolerate eventual visibility.

ANN Index Types

Exact nearest neighbor search over millions of vectors is too slow for interactive queries. Approximate nearest neighbor algorithms trade a small amount of recall for orders-of-magnitude faster search.

HNSW (Hierarchical Navigable Small World) is the default choice for most workloads. It builds a multi-layer graph where higher layers connect distant nodes and lower layers connect nearby nodes. At query time, it navigates from coarse to fine, which gives O(log n) search. It has excellent recall at low latency (typically 10-30ms for millions of vectors) and supports incremental inserts without full rebuild. The cost is higher memory usage: HNSW stores the full graph in RAM.

IVF (Inverted File Index) clusters vectors into Voronoi cells using k-means. At query time, only a subset of cells (the nprobe parameter) are searched. IVF uses less memory than HNSW and is faster to build, but recall degrades at lower nprobe values. It also requires a full rebuild to add new vectors to the cluster assignment. Use IVF when your index is largely static and memory is constrained.

Product Quantization (PQ) compresses vectors by splitting them into sub-vectors and quantizing each independently. A 1024-dimension float32 vector (4KB) can be compressed to 64 bytes or less. The compression ratio is real and significant at scale: 10 million 1024-dim vectors at float32 is 40GB. With PQ, that drops to 640MB. Recall takes a hit, typically 5-10% at the same nprobe. In practice, PQ is used alongside IVF (IVF-PQ) for large-scale deployments where memory is the binding constraint.

Index Type	Recall	Memory	Build Time	Supports Inserts	Best For
HNSW	High (95-99%)	High	Medium	Yes	Most workloads, interactive search
IVF-Flat	Medium-High	Medium	Fast	No (rebuild needed)	Static or slowly-changing corpora
IVF-PQ	Medium (85-95%)	Low	Fast	No	Scale >50M vectors, memory-constrained
Flat (exact)	100%	High	None	Yes	Small corpora (<100K), ground truth eval

Qdrant uses HNSW by default. The two tuning parameters that matter most are m (number of connections per node, default 16) and ef_construct (search width during build, default 100). Higher values improve recall but increase build time and memory. Leave them at defaults until you have a recall regression to investigate.

Hybrid Scoring with BM25

Pure semantic search has a known failure mode: exact match. If a user searches for a specific error code, model name, or product SKU, BM25 will find it reliably. Semantic search may not. The cosine similarity between “ERR_SSL_PROTOCOL_ERROR” and a document containing that exact string may be surprisingly low if the model has not seen that string in training.

The solution is hybrid search: run both BM25 and vector search in parallel, then combine the scores.

The standard combination is Reciprocal Rank Fusion (RRF):

RRF_score(d) = sum_over_k(1 / (k + rank_in_list_k(d)))

where k is a constant (typically 60) and rank_in_list_k(d) is the document’s rank in each result list. RRF is rank-based rather than score-based, which avoids the problem of BM25 and cosine similarity being on incomparable scales.

interface SearchResult {
  id: string;
  score: number;
  payload: Record<string, unknown>;
}

function reciprocalRankFusion(
  lists: SearchResult[][],
  k = 60
): SearchResult[] {
  const scores = new Map<string, number>();
  const payloads = new Map<string, Record<string, unknown>>();

  for (const list of lists) {
    list.forEach((result, rank) => {
      const current = scores.get(result.id) ?? 0;
      scores.set(result.id, current + 1 / (k + rank + 1));
      payloads.set(result.id, result.payload);
    });
  }

  return Array.from(scores.entries())
    .sort((a, b) => b[1] - a[1])
    .map(([id, score]) => ({
      id,
      score,
      payload: payloads.get(id)!,
    }));
}

export async function hybridSearch(
  query: string,
  limit = 20
): Promise<SearchResult[]> {
  const [vectorResults, bm25Results] = await Promise.all([
    vectorSearch(query, limit),
    bm25Search(query, limit),
  ]);

  return reciprocalRankFusion([vectorResults, bm25Results], 60).slice(0, limit);
}

Running both searches in parallel keeps latency bounded by the slower of the two, not the sum.

Relevance Tuning

The first version of your search will have gaps. Some query types will rank wrong results highly. Relevance tuning is the process of closing those gaps systematically.

Cross-Encoder Re-Ranking

Bi-encoder models (like sentence-transformers) encode the query and document independently. This is what makes them fast: you can pre-compute document embeddings. The tradeoff is that the query and document never interact during encoding. Cross-encoder models take the query and document together as a pair and produce a single relevance score. They are much more accurate but cannot pre-compute anything.

The standard pattern: retrieve 50-100 candidates with ANN (cheap, fast), then re-rank the top N with a cross-encoder (accurate, slower), then return the top 10 to the user.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank(query: str, candidates: list[dict], top_k: int = 10) -> list[dict]:
    pairs = [(query, c["text"]) for c in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(
        zip(candidates, scores),
        key=lambda x: x[1],
        reverse=True,
    )
    return [c for c, _ in ranked[:top_k]]

ms-marco-MiniLM-L-6-v2 is trained on the MS MARCO passage ranking dataset and generalizes well to most English text retrieval tasks. It runs in under 50ms for 50 pairs on a modern CPU.

User Feedback Loops

Click data is the cheapest signal you have. If you log which result the user clicked after a query, you can build a ground truth dataset for offline evaluation and model fine-tuning.

The minimum viable feedback loop:

1. Log (query, result_id, rank, clicked) for every search session.
2. Compute NDCG (Normalized Discounted Cumulative Gain) weekly over clicked results. A falling NDCG is your early warning signal.
3. Identify low-performing query clusters (queries with zero clicks, queries where rank-1 is never clicked). These are your priority test cases.
4. Use implicit feedback to fine-tune the bi-encoder via contrastive learning (query + clicked doc as positive, non-clicked docs as negatives).

Do not try to build the fine-tuning loop in the first iteration. Get the logging right first. Logging is cheap and you can always train on it later. Missing the logging is unrecoverable.

Production Concerns

Index Updates

HNSW supports incremental inserts without rebuild, but deletions are handled differently across implementations. Qdrant marks deleted vectors as tombstones and excludes them from results, then cleans them up during optimization. The index does not shrink immediately. If your delete rate is high (e.g., a document management system with frequent content removal), monitor your index size and tombstone ratio.

For bulk updates (re-embedding after a model upgrade), use a blue-green approach: build the new index in a shadow collection, validate recall against your offline eval set, then atomically swap the collection alias. Most vector databases support collection aliases for exactly this reason.

Latency Budget

A reasonable latency budget for a search API returning results to a user:

Stage	Target
BM25 (Elasticsearch/OpenSearch)	10-20ms
Vector ANN query	15-30ms
Both in parallel	25-35ms
Cross-encoder re-rank (50 candidates)	40-60ms
API overhead (serialization, network)	10-15ms
Total p95	< 120ms

If you are over budget, the first place to optimize is the re-ranker candidate count. Reducing from 100 candidates to 30 cuts re-rank time proportionally with modest recall cost. The second lever is the ANN ef search parameter: lower ef is faster but reduces recall.

Cost

The two primary cost drivers are embedding inference and vector storage.

Embedding inference: one inference call per search request. At 1024 dimensions, bge-large on CPU takes roughly 20-50ms. A single GPU instance handles most small-to-medium products.

Vector storage: 1 million vectors at 1024-dim float32 = 4GB. With HNSW graph overhead, budget 2-3x the raw vector size. At 100M vectors, switch to IVF-PQ. Most products do not reach that threshold in their first version.

Managed vector databases add operational simplicity at a cost premium. Self-hosted Qdrant on a well-provisioned instance handles tens of millions of vectors at lower per-query cost at scale. The operational burden is yours to carry either way.

The Offline Evaluation Harness

You cannot tune what you cannot measure. Build this before shipping.

interface EvalCase {
  query: string;
  relevantIds: string[];  // ground truth: which documents are relevant
}

interface EvalResult {
  query: string;
  ndcg: number;
  recallAt10: number;
}

function dcg(relevances: number[]): number {
  return relevances.reduce(
    (sum, rel, i) => sum + rel / Math.log2(i + 2),
    0
  );
}

async function evaluateSearch(cases: EvalCase[]): Promise<EvalResult[]> {
  return Promise.all(
    cases.map(async (c) => {
      const results = await hybridSearch(c.query, 10);
      const resultIds = results.map((r) => r.id);

      const relevances = resultIds.map((id) =>
        c.relevantIds.includes(id) ? 1 : 0
      );
      const idealRelevances = Array(Math.min(c.relevantIds.length, 10))
        .fill(1)
        .concat(Array(Math.max(0, 10 - c.relevantIds.length)).fill(0));

      const ndcg = dcg(relevances) / dcg(idealRelevances);
      const recallAt10 =
        relevances.filter(Boolean).length / c.relevantIds.length;

      return { query: c.query, ndcg, recallAt10 };
    })
  );
}

Even 50 hand-labeled query/relevant-document pairs is enough to catch regressions. Run this before every model upgrade, chunking change, or index parameter change. Recall at 10 tells you whether the right documents surface at all. NDCG tells you whether they surface at the top.

What the Architecture Looks Like End to End

The layers, in dependency order:

1. Ingestion pipeline: chunk documents, enrich with metadata, batch-embed, write to vector store and BM25 index concurrently.
2. Query layer: embed the query, run ANN and BM25 in parallel, fuse with RRF.
3. Re-ranking layer: pass top 50 candidates to cross-encoder, return top 10.
4. Feedback layer: log click events, compute NDCG weekly, flag query clusters with low engagement.
5. Eval harness: offline labeled dataset, automated recall/NDCG measurement, gates for index and model changes.

Most teams skip the eval harness. They iterate on models and chunking without a measurement baseline, then cannot tell whether a change helped or hurt. Build it first.

Keyword search is a solved problem for exact-match retrieval. Semantic search solves the intent gap, but only if the pipeline is built correctly: right chunking, right model for your domain, hybrid scoring to preserve exact-match recall, re-ranking to push the most relevant result to position one. None of those decisions are hard individually. Getting them all right together, at production latency, with a corpus that keeps changing, is the actual engineering work.