Building a Multimodal AI Pipeline: Processing Images, Audio, and Text with LLMs in Production

Multimodal AI in production is not “send an image to GPT-4o and call it done.” That works for demos. Production systems deal with inputs that arrive in different formats at different times, models that have different latency and cost profiles, fusion decisions that affect accuracy, and failure modes where one modality is corrupt or unavailable. Building a pipeline that handles all of this reliably is a different problem than calling a single API.

This article walks through the engineering of a production multimodal pipeline end to end: preprocessing each modality, routing inputs to the right models, fusing representations, managing latency and cost, and failing gracefully when something goes wrong.

What Multimodal Actually Means in Practice

A multimodal AI system accepts inputs from more than one modality, where a modality is a distinct type of signal: text, image, audio, video, structured data. In production, inputs rarely arrive clean or well-formed. You get:

• Images at arbitrary resolutions with inconsistent color spaces
• Audio at various sample rates, with background noise and channel configurations
• Text that may be raw OCR output, user-typed input, or structured document content

Each modality has its own preprocessing requirements, its own model family, and its own failure modes. The orchestration layer above them needs to handle all three in a way that is observable, cost-efficient, and fault-tolerant.

Input Preprocessing

Preprocessing is where most teams underinvest. Model APIs are not forgiving about malformed inputs.

Image Preprocessing

Vision models have strict requirements: input size limits, expected aspect ratios, base64 or URL encoding. Sending a 12MP raw image to a vision API will either fail, hit a size limit, or cost more tokens than necessary.

import sharp from "sharp";

interface ImagePreprocessResult {
  base64: string;
  mimeType: "image/jpeg" | "image/png" | "image/webp";
  originalWidth: number;
  originalHeight: number;
  processedWidth: number;
  processedHeight: number;
  sizeBytes: number;
}

async function preprocessImage(
  inputBuffer: Buffer,
  maxDimension = 1568 // GPT-4o's effective limit before tiling kicks in
): Promise<ImagePreprocessResult> {
  const image = sharp(inputBuffer);
  const metadata = await image.metadata();

  const originalWidth = metadata.width ?? 0;
  const originalHeight = metadata.height ?? 0;

  // Compute scaled dimensions preserving aspect ratio
  const scale = Math.min(
    maxDimension / originalWidth,
    maxDimension / originalHeight,
    1 // never upscale
  );

  const processedWidth = Math.round(originalWidth * scale);
  const processedHeight = Math.round(originalHeight * scale);

  const outputBuffer = await image
    .resize(processedWidth, processedHeight, { fit: "inside" })
    .jpeg({ quality: 85, progressive: false })
    .toBuffer();

  return {
    base64: outputBuffer.toString("base64"),
    mimeType: "image/jpeg",
    originalWidth,
    originalHeight,
    processedWidth,
    processedHeight,
    sizeBytes: outputBuffer.length,
  };
}

One practical note: vision model token cost scales with image area, not image complexity. A 1568x1568 image costs the same whether it is a blank page or a dense diagram. Build in downscaling logic early, and log the before/after dimensions so you can audit cost.

Audio Preprocessing and Transcription

Audio inputs almost always need transcription before any LLM sees them. The pipeline is: normalize audio format, send to a speech-to-text model, get a transcript with timestamps, then feed the transcript as text.

import { createReadStream, writeFileSync, unlinkSync } from "fs";
import { tmpdir } from "os";
import { join } from "path";
import OpenAI from "openai";

interface AudioPreprocessResult {
  transcript: string;
  durationSeconds: number;
  language: string | null;
  segments: Array<{
    start: number;
    end: number;
    text: string;
  }>;
}

const openai = new OpenAI();

async function preprocessAudio(
  audioBuffer: Buffer,
  mimeType: string
): Promise<AudioPreprocessResult> {
  // Write to a temp file because Whisper API requires a file stream
  const ext = mimeType.split("/")[1] ?? "mp3";
  const tmpPath = join(tmpdir(), `audio-${Date.now()}.${ext}`);

  try {
    writeFileSync(tmpPath, audioBuffer);

    const response = await openai.audio.transcriptions.create({
      file: createReadStream(tmpPath),
      model: "whisper-1",
      response_format: "verbose_json",
      timestamp_granularities: ["segment"],
    });

    return {
      transcript: response.text,
      durationSeconds: response.duration ?? 0,
      language: response.language ?? null,
      segments: (response.segments ?? []).map((s) => ({
        start: s.start,
        end: s.end,
        text: s.text,
      })),
    };
  } finally {
    unlinkSync(tmpPath);
  }
}

Transcription is where latency accumulates fast. A 5-minute audio clip takes 8-15 seconds to transcribe. If you are running multiple modalities in parallel, audio is usually on the critical path. Structure your pipeline so audio transcription starts immediately when the file arrives, not after image processing completes.

Text Preprocessing and Chunking

Text inputs need normalization and chunking before they can be included in a context window alongside image descriptions or transcripts.

interface TextChunk {
  text: string;
  tokenEstimate: number;
  chunkIndex: number;
  totalChunks: number;
}

function chunkText(
  text: string,
  maxTokens = 2000,
  overlapTokens = 100
): TextChunk[] {
  // Rough token estimate: 1 token ~= 4 chars for English
  const charsPerToken = 4;
  const maxChars = maxTokens * charsPerToken;
  const overlapChars = overlapTokens * charsPerToken;

  const chunks: string[] = [];
  let start = 0;

  while (start < text.length) {
    const end = Math.min(start + maxChars, text.length);
    chunks.push(text.slice(start, end));
    start += maxChars - overlapChars;
    if (start >= text.length) break;
  }

  return chunks.map((chunk, i) => ({
    text: chunk,
    tokenEstimate: Math.ceil(chunk.length / charsPerToken),
    chunkIndex: i,
    totalChunks: chunks.length,
  }));
}

Modality Routing

Not every input needs every model. A routing layer decides which modalities are present, which models handle them, and whether they run in parallel or sequentially.

type ModalityType = "image" | "audio" | "text" | "structured";

interface ModalityInput {
  type: ModalityType;
  data: Buffer | string;
  mimeType?: string;
  metadata?: Record<string, unknown>;
}

interface RoutingDecision {
  modalities: ModalityType[];
  runParallel: boolean;
  skipIfFailed: ModalityType[];
  requireAll: boolean;
}

function routeInput(inputs: ModalityInput[]): RoutingDecision {
  const types = inputs.map((i) => i.type);
  const hasAudio = types.includes("audio");
  const hasImage = types.includes("image");
  const hasText = types.includes("text");

  return {
    modalities: types,
    // Audio transcription and image description can run in parallel
    runParallel: hasAudio && hasImage,
    // If audio fails, fall back to image + text only
    skipIfFailed: ["audio"],
    // Require at least one text signal (transcript or direct text)
    requireAll: !(hasText || hasAudio),
  };
}

This is a simple version. Real routing logic often needs to account for cost budgets per request, whether the query is latency-sensitive, and which model has the most relevant capability for the content type.

Orchestrating Multiple Models

The core of the pipeline: run preprocessing in parallel, collect results, assemble a unified context, then call the LLM.

import Anthropic from "@anthropic-ai/sdk";

interface MultimodalResult {
  response: string;
  modalitiiesProcessed: ModalityType[];
  failedModalities: ModalityType[];
  totalLatencyMs: number;
  costs: Record<string, number>;
}

const anthropic = new Anthropic();

async function runMultimodalPipeline(
  inputs: ModalityInput[],
  systemPrompt: string
): Promise<MultimodalResult> {
  const start = Date.now();
  const routing = routeInput(inputs);
  const processed = new Map<ModalityType, unknown>();
  const failed: ModalityType[] = [];

  // Run preprocessing in parallel where possible
  const preprocessTasks = inputs.map(async (input) => {
    try {
      if (input.type === "image") {
        const result = await preprocessImage(input.data as Buffer);
        processed.set("image", result);
      } else if (input.type === "audio") {
        const result = await preprocessAudio(
          input.data as Buffer,
          input.mimeType ?? "audio/mp3"
        );
        processed.set("audio", result);
      } else if (input.type === "text") {
        const chunks = chunkText(input.data as string);
        processed.set("text", chunks);
      }
    } catch (err) {
      if (routing.skipIfFailed.includes(input.type)) {
        failed.push(input.type);
      } else {
        throw err;
      }
    }
  });

  await Promise.allSettled(preprocessTasks);

  // Assemble the message content for the vision/multimodal LLM
  const messageContent: Anthropic.ContentBlockParam[] = [];

  // Add images first (vision models attend better to early context)
  if (processed.has("image")) {
    const img = processed.get("image") as ImagePreprocessResult;
    messageContent.push({
      type: "image",
      source: {
        type: "base64",
        media_type: img.mimeType,
        data: img.base64,
      },
    });
  }

  // Add transcript from audio
  if (processed.has("audio")) {
    const audio = processed.get("audio") as AudioPreprocessResult;
    messageContent.push({
      type: "text",
      text: `[Audio transcript, ${audio.durationSeconds.toFixed(1)}s, language: ${audio.language ?? "unknown"}]\n${audio.transcript}`,
    });
  } else if (failed.includes("audio")) {
    messageContent.push({
      type: "text",
      text: "[Audio input was unavailable or failed to process]",
    });
  }

  // Add text input
  if (processed.has("text")) {
    const chunks = processed.get("text") as TextChunk[];
    const combined = chunks.map((c) => c.text).join("\n\n");
    messageContent.push({ type: "text", text: combined });
  }

  // Final query
  messageContent.push({ type: "text", text: "Analyze all provided inputs." });

  const response = await anthropic.messages.create({
    model: "claude-opus-4-5",
    max_tokens: 1024,
    system: systemPrompt,
    messages: [{ role: "user", content: messageContent }],
  });

  const responseText =
    response.content.find((b) => b.type === "text")?.text ?? "";

  return {
    response: responseText,
    modalitiiesProcessed: Array.from(processed.keys()),
    failedModalities: failed,
    totalLatencyMs: Date.now() - start,
    costs: estimateCosts(processed, response.usage),
  };
}

function estimateCosts(
  processed: Map<ModalityType, unknown>,
  usage: { input_tokens: number; output_tokens: number }
): Record<string, number> {
  const costs: Record<string, number> = {};

  if (processed.has("audio")) {
    const audio = processed.get("audio") as AudioPreprocessResult;
    // Whisper pricing: $0.006/minute
    costs.audio_transcription = (audio.durationSeconds / 60) * 0.006;
  }

  // Claude claude-opus-4-5: $15/M input, $75/M output (as of early 2026)
  costs.llm_input = (usage.input_tokens / 1_000_000) * 15;
  costs.llm_output = (usage.output_tokens / 1_000_000) * 75;

  return costs;
}

Fusion Strategies

Fusion is how the system combines signals from different modalities. There are three main approaches, and each is appropriate in different contexts.

Early Fusion

All modalities are combined into a single input representation before any model sees them. In practice with LLMs, this means assembling a single prompt that contains the image, the transcript, and the text, then sending everything to one model. This is what the orchestration above does.

When it works well: The query requires cross-modal reasoning (“does the speaker’s tone match the document’s claims?”). One powerful model can handle all modalities natively. Latency budget allows one round trip.

Where it breaks: The combined input exceeds the context window. One modality is very large and you are paying for context you do not need for a given query.

Late Fusion

Each modality is processed independently by its own model. The outputs are then combined, either by a second-pass LLM or by aggregation logic.

interface ModalityOutput {
  modality: ModalityType;
  summary: string;
  confidence: number;
  metadata: Record<string, unknown>;
}

async function lateFusion(
  modalityOutputs: ModalityOutput[],
  query: string
): Promise<string> {
  const fusionContext = modalityOutputs
    .map(
      (o) =>
        `[${o.modality.toUpperCase()} analysis, confidence ${o.confidence.toFixed(2)}]\n${o.summary}`
    )
    .join("\n\n");

  const response = await anthropic.messages.create({
    model: "claude-haiku-4-5",
    max_tokens: 512,
    messages: [
      {
        role: "user",
        content: `Given these modality analyses:\n\n${fusionContext}\n\nAnswer: ${query}`,
      },
    ],
  });

  return response.content.find((b) => b.type === "text")?.text ?? "";
}

When it works well: Each modality can be independently summarized without the others. You want to use specialized smaller models per modality and a cheap aggregator. You need to parallelize and merge.

Where it breaks: Cross-modal relationships are lost. If the image shows something that contradicts the transcript, late fusion may not surface that contradiction unless you explicitly prompt for it.

Cross-Attention (Model-Level Fusion)

This happens inside purpose-built multimodal models (Gemini, Claude with vision, GPT-4o) where attention layers can attend across modality tokens. You do not control this from the API; you enable it by using a model that supports it natively.

The practical implication: if cross-modal reasoning is important for your use case, use a model that natively ingests both modalities simultaneously rather than building your own fusion layer. For simpler cases, late fusion with a cheap aggregator is often more cost-effective.

Fusion Strategy Comparison

Strategy	Reasoning Quality	Cost	Latency	Failure Isolation	Best For
Early fusion (one model)	Highest for cross-modal	High (large context)	One round trip	Low (all or nothing)	Complex cross-modal queries
Late fusion (aggregator)	Good for independent signals	Lower (smaller inputs)	Parallelizable	High (per-modality)	Independent analysis + merge
Specialized models per modality	Best per-modality	Varies	Parallelizable	High	Domain-specific pipelines
Prompt chaining	Moderate	Low	Sequential	Moderate	When one output feeds the next

Latency Management

Multimodal pipelines are slow by default. Audio transcription takes 8-15 seconds. Vision inference adds 2-5 seconds. A sequential pipeline stacks these. The only real mitigation is parallelism.

Two principles:

Start preprocessing immediately on receipt. Do not wait for all inputs to arrive before starting any processing. If audio arrives first, start transcription. When the image arrives, start resizing in parallel with transcription.

Separate the critical path from enrichment. Identify the minimum modalities needed for a usable response. Process those first, return a result, then run enrichment (e.g., detailed image analysis) asynchronously and update the response.

async function streamingMultimodalResponse(
  inputs: ModalityInput[],
  onPartialResult: (text: string) => void
): Promise<void> {
  // Start all preprocessing concurrently
  const preprocessPromises = {
    image: inputs.find((i) => i.type === "image")
      ? preprocessImage(inputs.find((i) => i.type === "image")!.data as Buffer)
      : Promise.resolve(null),
    audio: inputs.find((i) => i.type === "audio")
      ? preprocessAudio(
          inputs.find((i) => i.type === "audio")!.data as Buffer,
          inputs.find((i) => i.type === "audio")!.mimeType ?? "audio/mp3"
        )
      : Promise.resolve(null),
  };

  const [imageResult, audioResult] = await Promise.all([
    preprocessPromises.image,
    preprocessPromises.audio,
  ]);

  const content: Anthropic.ContentBlockParam[] = [];
  if (imageResult) {
    content.push({
      type: "image",
      source: {
        type: "base64",
        media_type: imageResult.mimeType,
        data: imageResult.base64,
      },
    });
  }
  if (audioResult) {
    content.push({ type: "text", text: audioResult.transcript });
  }
  content.push({ type: "text", text: "Analyze the provided inputs." });

  const stream = anthropic.messages.stream({
    model: "claude-opus-4-5",
    max_tokens: 1024,
    messages: [{ role: "user", content }],
  });

  for await (const event of stream) {
    if (
      event.type === "content_block_delta" &&
      event.delta.type === "text_delta"
    ) {
      onPartialResult(event.delta.text);
    }
  }
}

Cost Management

Multimodal requests are expensive. A single request with an image, a 2-minute audio clip, and some text can cost 10-50x more than a pure text call. Budget and throttle accordingly.

Key cost levers:

Downscale images aggressively. Most queries do not benefit from high resolution. A 512x512 image costs a fraction of a 1568x1568 image in vision tokens. Default to lower resolution and only use high resolution when the content demands it (dense schematics, fine text in images).

Transcribe audio once, cache the transcript. If the same audio file is processed multiple times, cache the transcript keyed on a hash of the audio buffer. Whisper is not expensive, but it adds up at scale.

Route to cheaper models when modalities are not present. If a request has only text, do not send it to a vision model. Build routing that selects the cheapest capable model for the given input set.

Track per-modality cost in your telemetry. Without this, cost anomalies are invisible until the bill arrives. Log input tokens, output tokens, audio seconds, and image resolution per request.

Error Handling When a Modality Fails

Partial failure is the normal operating mode for multimodal systems. Audio files corrupt. Images exceed size limits. Network calls to transcription APIs time out. The pipeline should degrade gracefully, not fail completely.

The pattern: define which modalities are required and which are optional. Required failures propagate. Optional failures are captured and communicated to the model so it can answer with appropriate caveats.

interface ModalityProcessingResult {
  modality: ModalityType;
  status: "success" | "failed" | "skipped";
  result?: unknown;
  error?: string;
}

async function safePreprocess(
  input: ModalityInput,
  required: boolean
): Promise<ModalityProcessingResult> {
  try {
    let result: unknown;

    if (input.type === "image") {
      result = await preprocessImage(input.data as Buffer);
    } else if (input.type === "audio") {
      result = await preprocessAudio(
        input.data as Buffer,
        input.mimeType ?? "audio/mp3"
      );
    } else if (input.type === "text") {
      result = chunkText(input.data as string);
    }

    return { modality: input.type, status: "success", result };
  } catch (err) {
    const error = err instanceof Error ? err.message : String(err);

    if (required) {
      throw new Error(
        `Required modality ${input.type} failed to process: ${error}`
      );
    }

    return { modality: input.type, status: "failed", error };
  }
}

function buildFailureContext(failed: ModalityProcessingResult[]): string {
  if (failed.length === 0) return "";

  const descriptions = failed.map((f) => {
    if (f.status === "failed") {
      return `The ${f.modality} input could not be processed (${f.error ?? "unknown error"}). Do not make assumptions about its content.`;
    }
    return "";
  });

  return descriptions.filter(Boolean).join("\n");
}

Passing failure context to the model is underappreciated. If the model knows the audio failed, it can say “based on the image alone” rather than hallucinating what the audio might have contained. That difference matters in production.

Production Considerations

Rate limiting per modality. Vision APIs and transcription APIs have separate rate limits. Track them independently and implement per-API backpressure. A spike in audio requests should not block image processing.

Input validation before preprocessing. Reject inputs that will obviously fail: images over 20MB before even calling sharp, audio files over 25MB before calling Whisper, text inputs with null bytes or encoding issues. Fail fast with clear error messages rather than burning compute on bad inputs.

Timeouts per stage. Set independent timeouts for image preprocessing, audio transcription, and LLM inference. A hung transcription job should not hold the entire request open indefinitely. Use Promise.race with a timeout wrapper around each stage.

Observability. Log the full modality mix per request, preprocessing duration per modality, which modalities succeeded or failed, and per-request cost broken out by stage. Without this, debugging production issues is a guessing game.

Content safety checks. Multimodal inputs expand the attack surface. Images can embed adversarial prompts (printed text in an image that attempts prompt injection). Audio can contain jailbreak attempts. Run content filtering on preprocessed outputs, not just raw inputs.

Closing

Multimodal AI pipelines are not conceptually hard, but they are operationally complex. The preprocessing, routing, fusion, and failure handling decisions compound. Getting them right early means you can extend the pipeline to new modalities or swap models without rebuilding everything. The pattern above is opinionated but it reflects what actually holds up under load: parallel preprocessing, explicit routing, per-modality failure isolation, and cost tracked from day one.