Multimodal AI Pipelines in Production: Processing Images, Audio, and Documents in a Unified Architecture

Most AI tutorials show you a single modality. You feed an image to a vision model, or transcribe audio with Whisper, or parse a PDF and ask questions about it. That is the demo. The real product is the thing your user drops a mixed bag of files into: a scanned contract, a voice memo, and three screenshots, all at once, expecting a coherent answer.

Building a system that handles that reliably is not just a matter of wiring three API calls together. You need a normalized input representation, routing logic that picks the right specialist model per modality, a strategy for fusing the outputs into a unified context, and a cost and latency model that does not blow up the moment traffic picks up. This article covers each of those layers with concrete TypeScript code you can adapt directly.

---

The Core Problem: Heterogeneous Inputs, One Reasoning Surface

Every multimodal system eventually converges on the same architectural shape: convert raw inputs into a form the reasoning layer can consume, then let the reasoning layer produce the output. The challenge is the conversion step. Images, audio, and documents each carry different information density, require different preprocessing, and route to different specialist APIs with different latency and cost profiles.

The naive approach is to handle each modality ad hoc inside a single request handler: if the input has an audio file, call Whisper; if it has an image, call GPT-4o vision; if it has a PDF, extract text. That works until you need retry logic, fallback providers, parallel processing, cost tracking, or quality evaluation. Then the ad hoc approach collapses.

The better approach is a pipeline orchestrator that normalizes all inputs into a typed intermediate representation, routes each input through the right processor, assembles the results into a unified context object, and hands that context to the reasoning model. The orchestrator is the seam where all the operational concerns live.

---

Input Normalization: A Typed Intermediate Representation

Before routing, every incoming file needs to be classified and assigned a processing path. Define a discriminated union up front:

type ModalityType = "image" | "audio" | "document" | "text";

interface NormalizedInput {
  id: string;
  modality: ModalityType;
  mimeType: string;
  sizeBytes: number;
  sourceUrl?: string;
  rawBuffer?: Buffer;
  metadata: Record<string, string>;
}

interface ProcessedModality {
  inputId: string;
  modality: ModalityType;
  textContent?: string;       // transcription or extracted text
  embeddingVector?: number[]; // for semantic retrieval
  visionDescription?: string; // structured image analysis
  processingMs: number;
  tokensUsed: number;
  model: string;
  provider: string;
}

Normalization runs before any API call. It classifies the MIME type, validates the size, and rejects inputs that are too large to process within your latency budget before you spend money on them:

const MODALITY_LIMITS: Record<ModalityType, number> = {
  image: 20 * 1024 * 1024,    // 20MB
  audio: 25 * 1024 * 1024,    // 25MB (Whisper API limit)
  document: 50 * 1024 * 1024, // 50MB
  text: 1 * 1024 * 1024,      // 1MB
};

function normalizeInput(file: {
  id: string;
  mimeType: string;
  buffer: Buffer;
  metadata?: Record<string, string>;
}): NormalizedInput {
  const modality = classifyMimeType(file.mimeType);

  if (file.buffer.length > MODALITY_LIMITS[modality]) {
    throw new Error(
      `Input ${file.id} exceeds size limit for modality ${modality}: ` +
      `${file.buffer.length} > ${MODALITY_LIMITS[modality]}`
    );
  }

  return {
    id: file.id,
    modality,
    mimeType: file.mimeType,
    sizeBytes: file.buffer.length,
    rawBuffer: file.buffer,
    metadata: file.metadata ?? {},
  };
}

function classifyMimeType(mimeType: string): ModalityType {
  if (mimeType.startsWith("image/")) return "image";
  if (mimeType.startsWith("audio/")) return "audio";
  if (["application/pdf", "application/vnd.openxmlformats-officedocument.wordprocessingml.document"].includes(mimeType)) {
    return "document";
  }
  if (mimeType.startsWith("text/")) return "text";
  throw new Error(`Unsupported MIME type: ${mimeType}`);
}

---

Modality Routing: Specialist Models per Input Type

Each modality routes to a different processor. The processors run in parallel where dependencies allow. The orchestrator tracks latency budgets per modality and aborts slow processors to avoid cascading delays.

Image Processor: Vision Models with Fallback

interface VisionConfig {
  primary: "gpt-4o" | "claude-3-5-sonnet";
  fallback: "gpt-4o-mini" | "claude-3-haiku";
  timeoutMs: number;
  systemPrompt: string;
}

async function processImage(
  input: NormalizedInput,
  config: VisionConfig
): Promise<ProcessedModality> {
  const start = Date.now();
  const base64 = input.rawBuffer!.toString("base64");
  const dataUrl = `data:${input.mimeType};base64,${base64}`;

  try {
    const result = await callVisionModel(
      config.primary,
      dataUrl,
      config.systemPrompt,
      config.timeoutMs
    );

    return {
      inputId: input.id,
      modality: "image",
      visionDescription: result.description,
      textContent: result.description, // unified text surface for context assembly
      tokensUsed: result.tokensUsed,
      processingMs: Date.now() - start,
      model: config.primary,
      provider: config.primary.startsWith("gpt") ? "openai" : "anthropic",
    };
  } catch (err) {
    if (isTimeoutOrRateLimit(err)) {
      // fall through to fallback
      return callVisionModel(config.fallback, dataUrl, config.systemPrompt, config.timeoutMs * 2)
        .then((result) => ({
          inputId: input.id,
          modality: "image",
          visionDescription: result.description,
          textContent: result.description,
          tokensUsed: result.tokensUsed,
          processingMs: Date.now() - start,
          model: config.fallback,
          provider: config.fallback.startsWith("gpt") ? "openai" : "anthropic",
        }));
    }
    throw err;
  }
}

The fallback chain matters in production. GPT-4o vision and Claude Sonnet both have rate limits that you will hit during traffic spikes. When the primary provider returns a 429 or times out, routing to the fallback adds latency but keeps the pipeline alive. Track which fallback path fired in your metrics so you know when to raise primary quotas.

Audio Processor: Transcription with Provider Comparison

type AudioProvider = "openai-whisper" | "deepgram";

interface AudioConfig {
  primary: AudioProvider;
  fallback: AudioProvider;
  language?: string;
  timeoutMs: number;
}

async function processAudio(
  input: NormalizedInput,
  config: AudioConfig
): Promise<ProcessedModality> {
  const start = Date.now();

  const transcribe = async (provider: AudioProvider): Promise<{ text: string; durationS: number }> => {
    if (provider === "openai-whisper") {
      return transcribeWithWhisper(input.rawBuffer!, input.mimeType, config.language);
    }
    return transcribeWithDeepgram(input.rawBuffer!, input.mimeType, config.language);
  };

  try {
    const result = await withTimeout(transcribe(config.primary), config.timeoutMs);

    // Whisper charges per second of audio, not per token
    const estimatedTokens = Math.ceil(result.durationS * 7); // ~7 tokens/s average

    return {
      inputId: input.id,
      modality: "audio",
      textContent: result.text,
      tokensUsed: estimatedTokens,
      processingMs: Date.now() - start,
      model: config.primary === "openai-whisper" ? "whisper-1" : "nova-3",
      provider: config.primary === "openai-whisper" ? "openai" : "deepgram",
    };
  } catch (err) {
    const result = await transcribe(config.fallback);
    return {
      inputId: input.id,
      modality: "audio",
      textContent: result.text,
      tokensUsed: Math.ceil(result.durationS * 7),
      processingMs: Date.now() - start,
      model: config.fallback === "openai-whisper" ? "whisper-1" : "nova-3",
      provider: config.fallback === "openai-whisper" ? "openai" : "deepgram",
    };
  }
}

Whisper and Deepgram have different latency profiles. Whisper with the OpenAI API processes audio in roughly 1-2x real time on average, meaning a 30-second clip takes 30-60 seconds. Deepgram’s Nova model is faster on short clips (under 10 seconds), making it better for real-time voice inputs. For batch document processing where audio clips can be long, the latency difference matters more than the cost difference.

Document Processor: Text Extraction with Structure Preservation

PDF extraction is where most pipelines go wrong. Naive extraction flattens the document and loses table structure, column layout, and header hierarchy. For contracts, invoices, or research papers, that structure is the signal.

async function processDocument(
  input: NormalizedInput
): Promise<ProcessedModality> {
  const start = Date.now();

  let extractedText: string;
  let pageCount: number;

  if (input.mimeType === "application/pdf") {
    const result = await extractPdfText(input.rawBuffer!);
    extractedText = result.text;
    pageCount = result.pageCount;
  } else {
    // DOCX and similar
    const result = await extractDocxText(input.rawBuffer!);
    extractedText = result.text;
    pageCount = result.sectionCount;
  }

  // Chunk long documents: GPT-4o context window is 128k tokens
  // Aim to stay under 80k tokens of document content to leave room for
  // system prompt, other modalities, and the response
  const MAX_DOC_CHARS = 300_000; // ~80k tokens approximate
  const truncated = extractedText.length > MAX_DOC_CHARS
    ? extractedText.slice(0, MAX_DOC_CHARS) + "\n\n[Document truncated at processing limit]"
    : extractedText;

  return {
    inputId: input.id,
    modality: "document",
    textContent: truncated,
    tokensUsed: Math.ceil(truncated.length / 3.7), // rough approximation
    processingMs: Date.now() - start,
    model: "pdfjs-extract",
    provider: "local",
  };
}

For very long documents, you have two options: truncate and tell the model what you did, or chunk the document and run multiple inference calls. Chunking gives complete coverage but multiplies cost and latency. Truncation with an explicit notice works surprisingly well because the reasoning model can acknowledge it cannot answer questions about content it was not given, rather than hallucinating an answer.

---

The Orchestrator: Parallel Processing with Budget Enforcement

The orchestrator fans out inputs to processors, tracks cumulative latency and cost, and assembles the results:

interface PipelineConfig {
  vision: VisionConfig;
  audio: AudioConfig;
  maxTotalTokenBudget: number;  // across all modalities + reasoning
  maxPipelineMs: number;        // wall-clock budget for preprocessing
  reasoningModel: "gpt-4o" | "claude-3-5-sonnet";
}

interface PipelineResult {
  processedInputs: ProcessedModality[];
  unifiedContext: string;
  reasoningResponse: string;
  totalTokensUsed: number;
  totalMs: number;
  costUsd: number;
}

async function runMultimodalPipeline(
  inputs: NormalizedInput[],
  userQuery: string,
  config: PipelineConfig
): Promise<PipelineResult> {
  const start = Date.now();

  // Group by modality for parallel processing
  const byModality = groupBy(inputs, (i) => i.modality);

  // Kick off all processors in parallel with a shared deadline
  const processingTasks = inputs.map(async (input): Promise<ProcessedModality | null> => {
    try {
      switch (input.modality) {
        case "image":
          return await withTimeout(
            processImage(input, config.vision),
            config.maxPipelineMs
          );
        case "audio":
          return await withTimeout(
            processAudio(input, config.audio),
            config.maxPipelineMs
          );
        case "document":
          return await withTimeout(
            processDocument(input),
            config.maxPipelineMs
          );
        case "text":
          return {
            inputId: input.id,
            modality: "text",
            textContent: input.rawBuffer!.toString("utf-8"),
            tokensUsed: 0,
            processingMs: 0,
            model: "passthrough",
            provider: "local",
          };
      }
    } catch (err) {
      // Log and continue: a failed modality should not block the whole pipeline
      console.error(`Processing failed for input ${input.id}:`, err);
      return null;
    }
  });

  const results = await Promise.all(processingTasks);
  const processedInputs = results.filter((r): r is ProcessedModality => r !== null);

  const preprocessingMs = Date.now() - start;

  // Assemble unified context
  const unifiedContext = assembleContext(processedInputs, userQuery);

  // Check token budget before calling the reasoning model
  const preprocessTokens = processedInputs.reduce((sum, p) => sum + p.tokensUsed, 0);
  const contextTokens = Math.ceil(unifiedContext.length / 3.7);
  const remainingBudget = config.maxTotalTokenBudget - preprocessTokens - contextTokens;

  if (remainingBudget < 1000) {
    throw new Error(`Token budget exhausted after preprocessing: ${preprocessTokens} tokens used`);
  }

  // Call the reasoning model
  const reasoningStart = Date.now();
  const reasoningResponse = await callReasoningModel(
    config.reasoningModel,
    unifiedContext,
    userQuery,
    Math.min(remainingBudget, 4096) // cap response tokens
  );
  const reasoningMs = Date.now() - reasoningStart;

  const totalTokensUsed = preprocessTokens + contextTokens + reasoningResponse.tokensUsed;

  return {
    processedInputs,
    unifiedContext,
    reasoningResponse: reasoningResponse.text,
    totalTokensUsed,
    totalMs: preprocessingMs + reasoningMs,
    costUsd: calculateCost(processedInputs, reasoningResponse, config.reasoningModel),
  };
}

The key decision here is that a failed modality processor does not abort the pipeline. If image processing fails for one of three inputs, you still want to reason over the audio and document. Logging the failure and returning null keeps the pipeline alive. The reasoning model will reason over whatever context it receives, and if you annotate missing inputs in the assembled context, it can acknowledge the gap.

---

Embedding Fusion: When You Need Semantic Search

Not every multimodal pipeline needs embedding fusion. If you are assembling context for a single synchronous inference call, text assembly is sufficient. Embedding fusion becomes necessary when you need to retrieve relevant context from a large corpus of previously processed multimodal inputs, or when you want to find similar items across modalities (find all audio clips that describe the same concept as this image).

The strategy that works in practice is late fusion: embed each modality independently using a modality-specific embedding model, then combine the embeddings at query time rather than at ingestion time.

interface FusedEmbedding {
  inputId: string;
  modality: ModalityType;
  vector: number[];
  dimensions: 1536; // text-embedding-3-small dimensions
  model: string;
}

async function embedProcessedModality(
  processed: ProcessedModality
): Promise<FusedEmbedding> {
  // All modalities converge to text before embedding
  // This gives you cross-modal semantic search at the cost of losing
  // raw signal from image patches or audio waveforms
  const textToEmbed = processed.textContent ?? processed.visionDescription ?? "";

  if (!textToEmbed) {
    throw new Error(`No text content available for embedding input ${processed.inputId}`);
  }

  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: textToEmbed,
    dimensions: 1536,
  });

  return {
    inputId: processed.inputId,
    modality: processed.modality,
    vector: response.data[0].embedding,
    dimensions: 1536,
    model: "text-embedding-3-small",
  };
}

The tradeoff with late fusion through text is real: you lose perceptual signal. An image embedding model (like CLIP) captures visual similarity that a text description will miss. Two photos of the same building will be close in CLIP embedding space but may have different text descriptions and thus end up far apart in text embedding space. For most product use cases (document QA, meeting summarization, contract analysis), text-mediated late fusion is acceptable. For image similarity search or audio fingerprinting, you need native modality embeddings and a vector store that supports multi-vector queries.

---

Context Assembly: Building the Reasoning Input

How you structure the unified context directly affects reasoning quality. Flat concatenation works for simple cases but loses the signal about which content came from which input:

function assembleContext(
  processedInputs: ProcessedModality[],
  userQuery: string
): string {
  const sections: string[] = [];

  // Group by modality for clarity in the context window
  const images = processedInputs.filter((p) => p.modality === "image");
  const audio = processedInputs.filter((p) => p.modality === "audio");
  const documents = processedInputs.filter((p) => p.modality === "document");
  const text = processedInputs.filter((p) => p.modality === "text");

  if (images.length > 0) {
    sections.push("## Images Provided\n");
    images.forEach((img, i) => {
      sections.push(
        `### Image ${i + 1} (ID: ${img.inputId})\n` +
        `${img.visionDescription ?? img.textContent}\n`
      );
    });
  }

  if (audio.length > 0) {
    sections.push("## Audio Transcriptions\n");
    audio.forEach((aud, i) => {
      sections.push(
        `### Audio ${i + 1} (ID: ${aud.inputId})\n` +
        `${aud.textContent}\n`
      );
    });
  }

  if (documents.length > 0) {
    sections.push("## Documents\n");
    documents.forEach((doc, i) => {
      sections.push(
        `### Document ${i + 1} (ID: ${doc.inputId})\n` +
        `${doc.textContent}\n`
      );
    });
  }

  if (text.length > 0) {
    sections.push("## Additional Text\n");
    text.forEach((t) => {
      sections.push(`${t.textContent}\n`);
    });
  }

  const failedCount = processedInputs.length; // compare to total inputs for gap annotation
  // In a real system, track original input count and annotate gaps here

  return sections.join("\n");
}

Labeling sections by input ID is not just documentation hygiene. When the reasoning model produces an answer that references the wrong input, you need to know which section it misread. Structured section headers make the model’s attribution errors traceable.

---

Production Tradeoffs

Dimension	Approach A	Approach B	Notes
Image processing	GPT-4o vision (native)	Extract to text, pass as text	Native: better accuracy, 5-10x more expensive
Audio provider	OpenAI Whisper	Deepgram Nova	Deepgram faster on short clips; Whisper better on accented speech
Document extraction	Local pdfjs	Cloud OCR (Google Vision)	Local: free, fast; Cloud: handles scanned PDFs with no embedded text
Embedding strategy	Text-mediated late fusion	Native modality embeddings (CLIP)	Text: cross-modal search works; Native: better perceptual similarity
Failed modality	Abort pipeline	Continue with annotation	Continue wins: partial answers are almost always better than errors
Context assembly	Flat concatenation	Structured sections with IDs	Structured: better attribution in model responses

---

Production Considerations

Latency budgets per modality. Set per-modality timeouts, not just a global pipeline timeout. A stalled Whisper call should not eat the entire budget and leave no time for the reasoning step. A reasonable starting point: 15s for images, 60s for audio (per minute of audio), 10s for document extraction, 30s for the reasoning call. Measure your p95 for each modality separately in production.

Cost attribution. Track cost at the modality level, not just per pipeline run. When you see a cost spike, you need to know whether it came from a surge in long audio files or from a batch of high-resolution images hitting the vision model. Log tokensUsed, model, and provider for every ProcessedModality and aggregate by modality in your cost dashboard.

Quality evaluation for multimodal outputs. Evaluating reasoning quality over mixed inputs is harder than single-modality eval. The failure modes are different: the model may correctly transcribe audio but ignore it in favor of a contradicting document, or it may hallucinate details from an image that were not in the description. Build an eval set that covers cross-modality contradiction cases, not just single-modality accuracy. A small set of 50-100 labeled examples with known ground truth across mixed inputs is more useful than aggregate metrics over thousands of single-modality examples.

Fallback chain discipline. Define fallback chains explicitly in config, not as scattered try/catch blocks. When the primary provider is down, you want to route all traffic to the fallback without touching the code. The config-driven approach also makes it easy to run experiments: swap the primary for a new model, observe quality and cost, revert with a config change.

Input size validation before billing. Reject oversized inputs at the normalization step, before any API call. A 200MB video uploaded by mistake will hit your audio processor and generate a billing event even if it fails partway through. Size gates at normalization are free; API errors after calling the provider are not.

Idempotency for async pipelines. If you run the multimodal pipeline asynchronously (the common case for large documents), make it idempotent. Assign a pipelineRunId at the start and store results keyed by that ID. If the pipeline retries due to a transient failure, the completed modality processors should not re-run. A simple approach: write each ProcessedModality result to a store as it completes, and skip processors whose result is already stored on retry.

---

When This Architecture is Overkill

Not every multimodal use case needs a full orchestrator with parallel processing, fallback chains, and cost tracking. If you have a single modality, or if your inputs are always the same type (say, always a PDF and always the same questions), a simpler linear handler is easier to operate and debug.

The orchestrator pattern pays off when: inputs are user-provided and unpredictable in type and size, you have more than one modality in production, you need to swap providers without code changes, or you are processing enough volume that cost attribution per modality actually changes your decisions.

---

Closing

The real challenge in multimodal pipelines is not any single API call. It is the interface between all of them: the normalized representation that lets you process heterogeneous inputs uniformly, the context assembly that lets the reasoning model see all the evidence at once, and the operational layer that keeps the pipeline alive when one provider is slow or a user uploads something unexpected. Get those three things right and the individual model integrations are largely mechanical.

Start with the types. NormalizedInput and ProcessedModality as discriminated unions force the routing logic to be explicit and make the compiler catch gaps when you add a new modality. The orchestrator emerges naturally from those types, and the production concerns follow from the orchestrator.