Multimodal RAG in Production: Retrieving and Reasoning Over Images, Tables, and Code Alongside Text

Most production RAG systems are text-only pipelines wrapped around a vector store. That works until the knowledge base is not just text: PDFs with embedded charts, documentation sites with code snippets, spreadsheets, or slide decks where half the information lives in diagrams. Text-only retrieval then returns partial context at best and confidently wrong answers at worst.

Multimodal RAG does not mean “also accept image queries.” It means the pipeline can index, retrieve, and reason over content where the modality matters for meaning. A pricing table embedded in a PDF is not the same as the prose description of those tiers. A code snippet in documentation is not equivalent to its surrounding explanation. Treating them as interchangeable degrades retrieval quality in ways that are hard to debug.

This article covers the architecture decisions for a unified multimodal RAG pipeline: parsing and chunking non-text content, embedding model selection, vector store design for mixed modalities, context assembly for LLM reasoning, and retrieval quality measurement.

The Core Problem With Text-Only Pipelines on Mixed Documents

Naive PDF extraction loses structure. Tables become whitespace-delimited strings. Charts become empty space or alt-text fragments. Code blocks lose indentation semantics.

The downstream effect: a retrieval query for “Q3 revenue by region” fails to surface the relevant table because the extracted text looks like North America 1.2 EMEA 0.9 APAC 0.7 with no headers and no document context. The fix is upstream: parse with structural awareness, embed with modality semantics, and route retrieval to match query intent to content type.

Document Parsing: Structure Before Chunks

The parsing layer is where most teams take on the most technical debt. The two tools with the most production mileage are pdfplumber (Python) and unstructured (Python, with a hosted API option).

import pdfplumber
from dataclasses import dataclass
from typing import Literal

@dataclass
class DocumentChunk:
    chunk_id: str
    doc_id: str
    content_type: Literal["text", "table", "image", "code"]
    content: str          # raw text, markdown table, base64 image, or code string
    page: int
    bbox: tuple[float, float, float, float] | None
    metadata: dict

def extract_chunks(pdf_path: str, doc_id: str) -> list[DocumentChunk]:
    chunks: list[DocumentChunk] = []

    with pdfplumber.open(pdf_path) as pdf:
        for page_num, page in enumerate(pdf.pages):
            # Extract tables with structure preserved
            for table in page.extract_tables():
                if not table or not any(table):
                    continue
                headers = table[0]
                rows = table[1:]
                # Convert to markdown for embedding
                md_rows = ["| " + " | ".join(str(c or "") for c in headers) + " |"]
                md_rows.append("| " + " | ".join("---" for _ in headers) + " |")
                for row in rows:
                    md_rows.append("| " + " | ".join(str(c or "") for c in row) + " |")
                chunks.append(DocumentChunk(
                    chunk_id=f"{doc_id}-table-p{page_num}-{len(chunks)}",
                    doc_id=doc_id,
                    content_type="table",
                    content="\n".join(md_rows),
                    page=page_num,
                    bbox=None,
                    metadata={"source": pdf_path}
                ))

            # Extract images
            for img in page.images:
                # Crop and encode image region
                cropped = page.within_bbox((img["x0"], img["top"], img["x1"], img["bottom"]))
                img_obj = cropped.to_image(resolution=150)
                import io, base64
                buf = io.BytesIO()
                img_obj.save(buf, format="PNG")
                b64 = base64.b64encode(buf.getvalue()).decode()
                chunks.append(DocumentChunk(
                    chunk_id=f"{doc_id}-img-p{page_num}-{len(chunks)}",
                    doc_id=doc_id,
                    content_type="image",
                    content=b64,
                    page=page_num,
                    bbox=(img["x0"], img["top"], img["x1"], img["bottom"]),
                    metadata={"source": pdf_path}
                ))

            # Extract remaining text (excluding table regions)
            text = page.extract_text(x_tolerance=3, y_tolerance=3)
            if text and text.strip():
                # Split into paragraphs
                for para in text.split("\n\n"):
                    if len(para.strip()) > 50:
                        chunks.append(DocumentChunk(
                            chunk_id=f"{doc_id}-text-p{page_num}-{len(chunks)}",
                            doc_id=doc_id,
                            content_type="text",
                            content=para.strip(),
                            page=page_num,
                            bbox=None,
                            metadata={"source": pdf_path}
                        ))

    return chunks

For code snippets in documentation or markdown files, detect fenced blocks and preserve them as distinct chunks with language metadata:

import re

def extract_code_blocks(markdown: str, doc_id: str) -> list[DocumentChunk]:
    pattern = re.compile(r"```(\w+)?\n(.*?)```", re.DOTALL)
    chunks = []
    for match in pattern.finditer(markdown):
        lang = match.group(1) or "unknown"
        code = match.group(2).strip()
        if len(code) > 30:
            chunks.append(DocumentChunk(
                chunk_id=f"{doc_id}-code-{len(chunks)}",
                doc_id=doc_id,
                content_type="code",
                content=code,
                page=0,
                bbox=None,
                metadata={"language": lang, "source": doc_id}
            ))
    return chunks

Keep content types as separate chunks with explicit content_type labels rather than merging them. This enables per-modality embedding and per-modality retrieval tuning.

Multimodal Embedding Models

The embedding layer is where modality diverges the most. Three models see the most production use today.

CLIP embeds images and text into the same space (512 or 768 dims). Good for image retrieval via text queries; its text encoder is weaker than purpose-built text models. Do not use for text-only retrieval.

Nomic Embed Multimodal (nomic-embed-multimodal-7b) extends the Nomic text model to images with stronger text quality than CLIP. Available via the Nomic API and locally via sentence-transformers.

Cohere Embed v3 Multimodal supports text and images with explicit input_type params (image, search_document, search_query). The asymmetric document/query distinction improves retrieval precision.

For code, voyage-code-2 (Voyage AI) outperforms CLIP significantly. Code has syntactic structure that visual-semantic models do not encode well.

The practical architecture: route embedding model selection by content_type.

import Anthropic from "@anthropic-ai/sdk";
import { CohereClient } from "cohere-ai";

type ContentType = "text" | "table" | "image" | "code";

interface EmbedRequest {
  contentType: ContentType;
  content: string; // text/table/code as string, image as base64
}

interface EmbedResult {
  embedding: number[];
  model: string;
  contentType: ContentType;
}

const cohere = new CohereClient({ token: process.env.COHERE_API_KEY! });

async function embedChunk(req: EmbedRequest): Promise<EmbedResult> {
  switch (req.contentType) {
    case "image": {
      const response = await cohere.embed({
        model: "embed-english-v3.0",
        inputType: "image",
        embeddingTypes: ["float"],
        images: [`data:image/png;base64,${req.content}`],
      });
      const embedding = (response.embeddings as { float: number[][] }).float[0];
      return { embedding, model: "cohere-embed-v3", contentType: req.contentType };
    }

    case "code": {
      // Voyage code model via fetch (no official TS SDK)
      const res = await fetch("https://api.voyageai.com/v1/embeddings", {
        method: "POST",
        headers: {
          Authorization: `Bearer ${process.env.VOYAGE_API_KEY}`,
          "Content-Type": "application/json",
        },
        body: JSON.stringify({
          model: "voyage-code-2",
          input: req.content,
          input_type: "document",
        }),
      });
      const data = await res.json() as { data: { embedding: number[] }[] };
      return { embedding: data.data[0].embedding, model: "voyage-code-2", contentType: req.contentType };
    }

    case "text":
    case "table":
    default: {
      const response = await cohere.embed({
        model: "embed-english-v3.0",
        inputType: "search_document",
        embeddingTypes: ["float"],
        texts: [req.content],
      });
      const embedding = (response.embeddings as { float: number[][] }).float[0];
      return { embedding, model: "cohere-embed-v3", contentType: req.contentType };
    }
  }
}

The dimension mismatch between models (Cohere: 1024, Voyage code-2: 1536) means you cannot store all embeddings in a single flat namespace and do cross-modal ANN search. This leads to the vector store design question.

Unified Vector Store Design

There are two patterns for handling mixed-modality embeddings in a vector store.

Pattern 1: Separate namespaces per modality. Each content type gets its own namespace or collection. At retrieval time, fan out the query to all namespaces with the appropriate query embedding for each, then merge and re-rank results. This is the most common production pattern because it avoids dimension mismatch and lets you tune retrieval parameters per modality.

Pattern 2: Projection to a shared embedding space. Use a single multimodal model (Cohere or Nomic) that embeds all content types into the same dimensional space. Store everything in one namespace. This simplifies architecture but ties you to one embedding model for all modalities, which is usually a quality tradeoff.

For most teams, Pattern 1 with Pinecone, Weaviate, or Qdrant namespaces is the better starting point. Here is a TypeScript index and retrieval layer using Qdrant:

import { QdrantClient } from "@qdrant/js-client-rest";

const qdrant = new QdrantClient({ url: process.env.QDRANT_URL });

const COLLECTIONS: Record<ContentType, { name: string; size: number }> = {
  text:  { name: "docs_text",   size: 1024 },
  table: { name: "docs_table",  size: 1024 },
  image: { name: "docs_image",  size: 1024 },
  code:  { name: "docs_code",   size: 1536 },
};

async function ensureCollections(): Promise<void> {
  for (const [, { name, size }] of Object.entries(COLLECTIONS)) {
    const exists = await qdrant.collectionExists(name);
    if (!exists) {
      await qdrant.createCollection(name, {
        vectors: { size, distance: "Cosine" },
      });
    }
  }
}

async function indexChunk(chunk: DocumentChunk, embedding: EmbedResult): Promise<void> {
  const { name } = COLLECTIONS[chunk.content_type];
  await qdrant.upsert(name, {
    wait: true,
    points: [
      {
        id: chunk.chunk_id,
        vector: embedding.embedding,
        payload: {
          doc_id: chunk.doc_id,
          content_type: chunk.content_type,
          content: chunk.content,
          page: chunk.page,
          metadata: chunk.metadata,
        },
      },
    ],
  });
}

interface RetrievalResult {
  chunk_id: string;
  content_type: ContentType;
  content: string;
  score: number;
  page: number;
  metadata: Record<string, unknown>;
}

async function retrieveMultimodal(
  queryEmbeddings: Partial<Record<ContentType, number[]>>,
  topKPerModality: number = 3
): Promise<RetrievalResult[]> {
  const results: RetrievalResult[] = [];

  await Promise.all(
    (Object.entries(queryEmbeddings) as [ContentType, number[]][]).map(
      async ([modality, vector]) => {
        const { name } = COLLECTIONS[modality];
        const hits = await qdrant.search(name, {
          vector,
          limit: topKPerModality,
          with_payload: true,
        });
        for (const hit of hits) {
          results.push({
            chunk_id: hit.id as string,
            content_type: modality,
            content: hit.payload?.content as string,
            score: hit.score,
            page: hit.payload?.page as number,
            metadata: hit.payload?.metadata as Record<string, unknown>,
          });
        }
      }
    )
  );

  // Sort by score descending
  return results.sort((a, b) => b.score - a.score);
}

Cross-Modal Query Routing

When a user submits a text query, you need to decide which namespaces to search. Three options: always fan out to all (simple but slower), classify query intent first (adds latency, improves precision), or fan out with a modality prior using heuristics. The third is the right starting point for most pipelines.

function routeQuery(query: string): ContentType[] {
  const lower = query.toLowerCase();
  const modalities: ContentType[] = ["text"]; // always search text

  if (/chart|diagram|figure|image|graph|visual/.test(lower)) {
    modalities.push("image");
  }
  if (/table|row|column|comparison|breakdown|vs\.|versus/.test(lower)) {
    modalities.push("table");
  }
  if (/function|class|method|snippet|code|implement|example|`/.test(lower)) {
    modalities.push("code");
  }

  return [...new Set(modalities)];
}

Context Assembly for LLM Reasoning

Once you have retrieved chunks across modalities, assembling them into a prompt requires modality-specific handling. LLMs with vision support (GPT-4o, Claude 3.5 Sonnet) can receive base64 images directly. Tables work best as markdown. Code blocks need language fencing. Text chunks are straightforward.

interface AssembledContext {
  systemPrompt: string;
  userMessage: string | { type: "text" | "image_url"; text?: string; image_url?: { url: string } }[];
}

function assembleContext(
  query: string,
  results: RetrievalResult[]
): AssembledContext {
  const textChunks: string[] = [];
  const imageUrls: string[] = [];

  for (const result of results) {
    switch (result.content_type) {
      case "text":
        textChunks.push(`[Text, page ${result.page}]\n${result.content}`);
        break;
      case "table":
        textChunks.push(`[Table, page ${result.page}]\n${result.content}`);
        break;
      case "code":
        textChunks.push(`[Code, page ${result.page}]\n\`\`\`\n${result.content}\n\`\`\``);
        break;
      case "image":
        imageUrls.push(`data:image/png;base64,${result.content}`);
        break;
    }
  }

  const systemPrompt = `You are answering questions based on retrieved document context.
The context may include text, tables, code snippets, and images.
Base your answer only on the provided context. If the context is insufficient, say so.`;

  if (imageUrls.length === 0) {
    return {
      systemPrompt,
      userMessage: `Context:\n${textChunks.join("\n\n---\n\n")}\n\nQuestion: ${query}`,
    };
  }

  // Vision-capable model: send images as content parts
  const messageParts: AssembledContext["userMessage"] = [
    { type: "text", text: `Context:\n${textChunks.join("\n\n---\n\n")}\n\nQuestion: ${query}` },
    ...imageUrls.map((url) => ({
      type: "image_url" as const,
      image_url: { url },
    })),
  ];

  return { systemPrompt, userMessage: messageParts };
}

One practical constraint: images are expensive in token terms. A 150-DPI PNG crops to roughly 500-800 tokens with GPT-4o’s tile pricing. Cap image results at 1-2 per query, or implement an image captioning step during indexing and use captions for cheap retrieval, fetching original images only on high-confidence caption matches.

Tradeoffs Table

Dimension	Text-only RAG	Multimodal RAG (shared embedding space)	Multimodal RAG (separate namespaces)
Setup complexity	Low	Medium	High
Retrieval quality for text	High	Medium (model compromise)	High (dedicated model per type)
Retrieval quality for images	None	Medium	Medium to High
Retrieval quality for code	Low	Low	High (voyage-code-2)
Table retrieval	Poor (raw text)	Medium	High (structured markdown)
Query latency	Low	Low	Higher (fan-out)
Embedding cost	Low	Low	Higher (multiple API calls)
Modality tuning flexibility	None	None	Per-collection
LLM context cost	Low	Low	Medium (images expensive)

Evaluation Metrics for Multimodal Retrieval

Standard RAG metrics (hit rate, MRR, NDCG) apply per modality with a few extra failure modes to track.

Cross-modal false positives: a text query about a pricing formula can retrieve a same-keyword table from a different context. Per-modality precision is a better signal than aggregate precision.

Structural fidelity: for tables, the downstream metric is whether the LLM extracted the right cell value. A badly parsed table (whitespace-delimited) fails even if retrieved.

Image caption drift: if you caption images during indexing, compare captions against ground-truth descriptions after any model upgrade.

A minimal offline evaluation loop:

from dataclasses import dataclass
from typing import Callable

@dataclass
class EvalSample:
    query: str
    expected_chunk_ids: list[str]
    expected_content_types: list[str]

def evaluate_retrieval(
    samples: list[EvalSample],
    retrieve_fn: Callable[[str], list[RetrievalResult]],
    top_k: int = 5
) -> dict:
    hit_rates: dict[str, list[float]] = {"text": [], "table": [], "image": [], "code": []}
    overall_hits = []

    for sample in samples:
        results = retrieve_fn(sample.query)
        retrieved_ids = {r.chunk_id for r in results[:top_k]}
        hit = any(eid in retrieved_ids for eid in sample.expected_chunk_ids)
        overall_hits.append(float(hit))

        for content_type in sample.expected_content_types:
            relevant = [r for r in results[:top_k] if r.content_type == content_type]
            ct_hit = any(r.chunk_id in sample.expected_chunk_ids for r in relevant)
            hit_rates[content_type].append(float(ct_hit))

    return {
        "overall_hit_rate": sum(overall_hits) / len(overall_hits),
        "hit_rate_by_modality": {
            ct: (sum(vals) / len(vals) if vals else None)
            for ct, vals in hit_rates.items()
        },
    }

Run this against 50-100 labeled queries before and after any changes to parsing, chunking, or embedding models. The per-modality breakdown reveals whether a retrieval regression is in image handling or text handling, which is faster to diagnose than an aggregate score drop.

Production Considerations

Parsing failures are silent. PDFs with scanned pages or complex multi-column layouts produce garbage chunks. Add a quality check that flags chunks below a minimum character threshold or with high non-ASCII character ratios. Route flagged documents to manual review.

Table extraction is fragile. pdfplumber handles simple tables well, but merged cells and rotated tables produce incorrect structure. Validate consistent column counts per row and fall back to raw text with content_type: "text" rather than indexing a malformed markdown table.

Embedding model upgrades invalidate existing indexes. Old and new embeddings are not comparable across model versions. Plan for full re-indexing on upgrades. Store the embedding model name in each point’s payload to detect stale vectors at query time.

Image storage costs add up. Base64 images in vector store payloads are expensive and slow to retrieve. Store images in object storage (S3, R2) with a reference URL in the payload, and fetch on demand during context assembly.

Dense diagrams need sub-image chunking. A single chunk per image is too coarse for detailed figures. A fixed-grid crop producing 4-9 sub-image chunks per page meaningfully improves recall for region-specific queries.

The Actual Payoff

Text-only RAG on mixed documents does not fail loudly. It returns plausible-sounding answers that omit the table data, misquote the chart, or miss the code example that would have answered the question precisely. Users stop trusting the system incrementally, not all at once.

The multimodal pipeline is more expensive to build and maintain, but the reliability of retrieval on real enterprise document collections is meaningfully better. The parsing layer and the per-modality evaluation loop are the two pieces most teams skip. Both matter more than the choice of embedding model.