Vector Search at the Edge: Building Low-Latency RAG with Cloudflare Vectorize, Workers AI, and D1

Most RAG implementations look the same: embed a query, call a hosted vector database in us-east-1, fetch chunks, call a hosted LLM, return a response. That pattern works, but you’re accumulating round-trip latency from every geographic hop. When your users are globally distributed and you need sub-300ms answers, centralized RAG starts to look like the wrong architectural choice.

Cloudflare’s edge platform gives you a credible alternative. Workers run in 300+ locations, Vectorize stores your vector index with globally distributed reads, D1 handles relational metadata at the edge, and Workers AI runs inference on Cloudflare’s GPU network. You can build a complete RAG pipeline that never routes through a central cloud region.

This article walks through building that pipeline end to end. The tradeoffs are real: model size limits, vector index caps, and cold start behavior are all constraints you’ll hit in production. We’ll cover them honestly.

The Architecture

The full pipeline spans four Cloudflare primitives:

• Workers: orchestration layer, handles HTTP, runs the query pipeline
• Vectorize: managed vector index, stores embeddings with metadata filters
• D1: SQLite at the edge, stores chunk metadata (source, timestamps, arbitrary fields)
• Workers AI: runs @cf/baai/bge-base-en-v1.5 for embeddings and a small LLM for generation

At query time, the flow is: receive query, embed it via Workers AI, search Vectorize, hydrate results from D1, assemble context, generate an answer. At ingest time: receive document, chunk it, embed each chunk, upsert into Vectorize, write metadata to D1.

A Wrangler config that wires these together:

# wrangler.toml
name = "rag-edge"
main = "src/index.ts"
compatibility_date = "2024-09-23"

[[vectorize]]
binding = "VECTORIZE"
index_name = "knowledge-base"

[[d1_databases]]
binding = "DB"
database_name = "rag-metadata"
database_id = "your-database-id"

[ai]
binding = "AI"

The Worker’s Env type becomes:

interface Env {
  VECTORIZE: Vectorize;
  DB: D1Database;
  AI: Ai;
}

Document Ingestion and Chunking

Edge constraints shape your chunking strategy in ways that centralized systems don’t. Workers have a 128MB memory limit and a 30-second CPU time limit on paid plans. You cannot load a 50MB PDF into memory and process it inline.

The practical approach: ingest documents via a dedicated Worker that streams chunks into the pipeline rather than holding the full document.

Chunking Strategy

Fixed-size chunking with overlap is the simplest approach that works well enough for most corpora. For structured content (code, docs with clear headers), recursive splitting on semantic boundaries produces better retrieval.

interface Chunk {
  id: string;
  text: string;
  metadata: {
    sourceId: string;
    chunkIndex: number;
    charStart: number;
    charEnd: number;
  };
}

function chunkText(
  text: string,
  sourceId: string,
  chunkSize = 512,
  overlap = 64
): Chunk[] {
  const chunks: Chunk[] = [];
  let start = 0;
  let index = 0;

  while (start < text.length) {
    const end = Math.min(start + chunkSize, text.length);
    const chunkText = text.slice(start, end);

    chunks.push({
      id: `${sourceId}-${index}`,
      text: chunkText,
      metadata: {
        sourceId,
        chunkIndex: index,
        charStart: start,
        charEnd: end,
      },
    });

    start += chunkSize - overlap;
    index++;
  }

  return chunks;
}

For the bge-base-en-v1.5 model, 512-token chunks (roughly 400 words) hit the sweet spot between context density and embedding quality. Going larger degrades retrieval precision. Going smaller increases the number of index vectors and raises Vectorize costs.

Embedding and Upsert

Workers AI’s embedding endpoint is synchronous and returns a float32 array per input. Batch your chunks to stay within request limits.

async function ingestDocument(
  text: string,
  sourceId: string,
  env: Env
): Promise<void> {
  const chunks = chunkText(text, sourceId);

  // Workers AI embedding: max 100 inputs per call
  const BATCH_SIZE = 50;

  for (let i = 0; i < chunks.length; i += BATCH_SIZE) {
    const batch = chunks.slice(i, i + BATCH_SIZE);
    const texts = batch.map((c) => c.text);

    const embedResult = await env.AI.run("@cf/baai/bge-base-en-v1.5", {
      text: texts,
    });

    // Upsert into Vectorize
    const vectors: VectorizeVector[] = batch.map((chunk, j) => ({
      id: chunk.id,
      values: embedResult.data[j],
      metadata: {
        sourceId: chunk.metadata.sourceId,
        chunkIndex: chunk.metadata.chunkIndex,
        text: chunk.text, // store text inline for retrieval
      },
    }));

    await env.VECTORIZE.upsert(vectors);

    // Write metadata to D1 for rich filtering
    const stmt = env.DB.prepare(`
      INSERT OR REPLACE INTO chunks
        (id, source_id, chunk_index, char_start, char_end, text, ingested_at)
      VALUES (?, ?, ?, ?, ?, ?, ?)
    `);

    const batch_stmts = batch.map((chunk) =>
      stmt.bind(
        chunk.id,
        chunk.metadata.sourceId,
        chunk.metadata.chunkIndex,
        chunk.metadata.charStart,
        chunk.metadata.charEnd,
        chunk.text,
        new Date().toISOString()
      )
    );

    await env.DB.batch(batch_stmts);
  }
}

Two things worth noting: you’re storing chunk text both in Vectorize metadata and in D1. Vectorize metadata has a size limit (roughly 10KB per vector), and you cannot run SQL-style range queries on it. D1 gives you full SQL for filtering by date ranges, source IDs, or arbitrary metadata fields. The Vectorize metadata copy is a cache for the common case; D1 is the source of truth for complex queries.

Vector Index Configuration

When creating a Vectorize index, the two decisions that matter most are dimensions and distance metric.

bge-base-en-v1.5 outputs 768-dimensional vectors. Create the index to match:

npx wrangler vectorize create knowledge-base \
  --dimensions=768 \
  --metric=cosine

Cosine similarity works well for semantic search over text. If you’re doing exact nearest-neighbor lookups for deduplication or clustering, Euclidean distance is more appropriate. Dot product is faster but requires normalized vectors.

Vectorize currently caps at 5 million vectors per index and 200,000 vectors per namespace. For most production RAG use cases at the edge, that’s sufficient. A corpus of 10,000 documents at 20 chunks per document puts you at 200,000 vectors, right at the namespace limit. Plan your chunking granularity around this ceiling.

Metadata filtering lets you scope searches without a post-filter step:

const searchResults = await env.VECTORIZE.query(queryVector, {
  topK: 10,
  filter: {
    sourceId: { $eq: "product-docs-v2" },
  },
  returnMetadata: "indexed",
});

Only fields declared as indexed at index creation time are filterable. You cannot add indexed fields to an existing index without recreating it.

The Query Pipeline

A complete query pass: embed the query, search Vectorize, optionally rerank, fetch from D1, assemble context, generate.

interface RAGResponse {
  answer: string;
  sources: Array<{ id: string; sourceId: string; score: number }>;
  latencyMs: number;
}

async function query(
  userQuery: string,
  env: Env,
  sourceFilter?: string
): Promise<RAGResponse> {
  const start = Date.now();

  // Step 1: Embed the query
  const queryEmbedding = await env.AI.run("@cf/baai/bge-base-en-v1.5", {
    text: [userQuery],
  });

  const queryVector = queryEmbedding.data[0];

  // Step 2: Search Vectorize
  const searchOptions: VectorizeQueryOptions = {
    topK: 8,
    returnMetadata: "indexed",
  };

  if (sourceFilter) {
    searchOptions.filter = { sourceId: { $eq: sourceFilter } };
  }

  const searchResults = await env.VECTORIZE.query(queryVector, searchOptions);

  if (!searchResults.matches.length) {
    return {
      answer: "No relevant context found.",
      sources: [],
      latencyMs: Date.now() - start,
    };
  }

  // Step 3: Hydrate from D1 for accurate text (Vectorize metadata may be truncated)
  const chunkIds = searchResults.matches.map((m) => m.id);
  const placeholders = chunkIds.map(() => "?").join(", ");

  const rows = await env.DB.prepare(
    `SELECT id, source_id, text FROM chunks WHERE id IN (${placeholders})`
  )
    .bind(...chunkIds)
    .all<{ id: string; source_id: string; text: string }>();

  const chunkMap = new Map(rows.results.map((r) => [r.id, r]));

  // Step 4: Assemble context, respecting Workers AI prompt limits
  const MAX_CONTEXT_CHARS = 3000;
  let contextText = "";
  const usedSources: Array<{ id: string; sourceId: string; score: number }> =
    [];

  for (const match of searchResults.matches) {
    const row = chunkMap.get(match.id);
    if (!row) continue;

    const candidate = `[Source: ${row.source_id}]\n${row.text}\n\n`;
    if (contextText.length + candidate.length > MAX_CONTEXT_CHARS) break;

    contextText += candidate;
    usedSources.push({
      id: match.id,
      sourceId: row.source_id,
      score: match.score,
    });
  }

  // Step 5: Generate with a small model available on Workers AI
  const prompt = `You are a helpful assistant. Answer the question using only the provided context.

Context:
${contextText}

Question: ${userQuery}

Answer:`;

  const generated = await env.AI.run("@cf/meta/llama-3.1-8b-instruct", {
    prompt,
    max_tokens: 512,
  });

  return {
    answer: generated.response ?? "",
    sources: usedSources,
    latencyMs: Date.now() - start,
  };
}

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    if (request.method !== "POST") {
      return new Response("Method not allowed", { status: 405 });
    }

    const body = await request.json<{ query: string; sourceId?: string }>();
    const result = await query(body.query, env, body.sourceId);

    return Response.json(result);
  },
};

The D1 hydration step (Step 3) adds a round trip but keeps your Vectorize metadata lean. If your chunks are small (under 1KB of text), you can skip D1 hydration and read text directly from Vectorize metadata. For larger chunks, relying on Vectorize metadata risks truncation.

Latency Characteristics

Running this pipeline from a Worker co-located with a user in Frankfurt, against a centralized RAG stack in us-east-1, the breakdown looks like this:

Step	Edge (Cloudflare)	Centralized (us-east-1)
Query embedding	40-80ms	60-120ms (network + compute)
Vector search	20-50ms	80-200ms (network + Pinecone/Weaviate)
D1 metadata fetch	5-20ms	N/A or 80ms (RDS proxy)
LLM generation (8B model)	800-2000ms	500-1500ms (larger GPU)
Total P50	~1000ms	~1200ms
Total P99	~2500ms	~3000ms

The honest conclusion: for most use cases, the edge pipeline is roughly the same latency or modestly faster on embedding and retrieval, but slower on generation because Workers AI’s GPU fleet is smaller and the available models are smaller (8B vs 70B+). If you need a 70B model for generation quality, the edge is not the right place for that step.

Where edge RAG wins clearly is tail latency for globally distributed users. A user in Singapore hitting a centralized us-east-1 stack sees 200-300ms of extra network latency before any compute starts. With Cloudflare, that user’s request runs in the Singapore data center.

Cost Model

Resource	Unit	Price
Workers AI embeddings (bge-base)	per 1K tokens	$0.011
Workers AI inference (Llama 3.1 8B)	per 1M tokens	$0.11
Vectorize queries	per 1M queries	$0.01
Vectorize storage	per 1M vector dimensions/month	$0.05
D1 reads	per 1M rows	$0.001
Workers requests	per 1M requests	$0.30

For a knowledge base of 100,000 chunks at 768 dimensions, monthly storage is roughly $3.84. At 10,000 queries per day (each with one embedding call and one vector search), monthly inference cost is around $15-20. D1 reads are negligible.

Compare this to a self-managed stack: Pinecone Serverless at similar scale runs $5-15/month for storage alone, plus separate embedding API costs, plus separate LLM API costs. The Cloudflare stack is competitive on price and eliminates cross-service networking costs entirely.

Tradeoffs

Dimension	Cloudflare Edge RAG	Centralized RAG
Global tail latency	Excellent (local compute)	Poor (single-region network hops)
Model selection	Limited (8B max today)	Broad (any hosted or self-hosted model)
Generation quality	Lower (smaller models)	Higher (70B+ accessible)
Vector index size	5M vectors/index	Effectively unlimited (Pinecone, Weaviate)
Operational complexity	Low (managed, no infra)	Medium to high
Cold start penalty	50-200ms on first request	Minimal (always-warm)
Reranking options	None native; manual or skip	Cross-encoder rerankers available
Cost at scale	Competitive	Variable, can be lower at high volume
Customization	Constrained by Workers API	Full control

Production Considerations

Cold starts. Workers are “warm” in most popular locations when your deployment is active, but niche PoPs may cold-start. Workers AI adds its own initialization time on first call. The mitigation is a KEEP_ALIVE scheduled Worker that pings your AI bindings every few minutes, or accepting that P99 latency includes a cold-start tail.

Vectorize eventual consistency. After an upsert, new vectors are not immediately queryable. The propagation delay is typically under 5 minutes but can be longer during index rebuilds. Do not assert consistency in tests without a retry loop. For real-time ingestion (user uploads a doc, immediately queries it), this lag is a UX problem that requires a separate cache or a “pending” state in your D1 metadata.

D1 write limits. D1 on the paid Workers plan supports up to 50,000 row writes per day on the free tier, but the paid tier allows up to 25 billion row reads per month and 50 million row writes per month. For high-throughput ingestion, batch your D1 writes using db.batch() as shown above. Avoid single-statement inserts in a loop.

Context window constraints. Llama 3.1 8B on Workers AI has a 128K context window, but the practical limit for good generation quality is closer to 4K-8K tokens of context. Retrieve more chunks than you need (topK: 10-15), then trim to fit. Sending 8K tokens of retrieved context to an 8B model degrades answer quality noticeably compared to a focused 2K-3K context.

Workers AI rate limits. The platform imposes per-model rate limits that vary by tier. At scale, you will hit these before you hit cost limits. Plan for HTTP 429 handling with exponential backoff in your embedding and generation calls. The queue pattern (batch ingest via Workers Queue) is more resilient for ingestion workloads than synchronous upsert.

Streaming responses. Workers support the TransformStream API. If you want streaming generation, env.AI.run does not yet support streaming responses for all models. Check the Workers AI changelog before committing to a streaming UX.

When This Architecture Makes Sense

Build edge RAG when: your users are globally distributed and tail latency matters, your corpus fits within Vectorize limits (under 2 million vectors is comfortable), you can accept 8B model quality for generation, and you want zero infrastructure to manage.

Stick with centralized RAG when: you need 70B+ model quality, your corpus exceeds 5 million vectors, you require sub-second cold start guarantees, or you need cross-encoder reranking.

The edge is not a universal upgrade to your RAG stack. It is a specific architectural choice with specific tradeoffs. If your primary bottleneck is retrieval latency for global users and your generation quality requirements are moderate, it is the right choice.