Building a Document Processing Pipeline with AI: OCR, Classification, and Extraction for Production Systems

Most document processing tutorials show you how to run Tesseract on a single image or call a cloud OCR API and print the text. That covers about 10% of the problem. The other 90% is what happens between “raw file arrives” and “structured data reaches your application database”: classifying the document type, handling multi-page PDFs, scoring extraction confidence, routing uncertain results to a human reviewer, and doing all of this at scale without losing documents or double-processing them.

This article builds the full pipeline. The target reader has already shipped something and knows why the naive approach breaks in production.

---

The Pipeline Shape

Before getting into components, here is what the full pipeline looks like:

Ingest (S3/queue) → Normalize → OCR → Classify → Extract → Score → Route → Store
                                                                   ↓
                                                          Human Review Queue

Each stage is a separate concern. The failure modes are different at each step, and you want to be able to replay any stage independently when models change or bugs surface.

A concrete TypeScript interface for a document as it moves through the pipeline:

type DocumentStage =
  | "ingested"
  | "normalized"
  | "ocr_complete"
  | "classified"
  | "extracted"
  | "scored"
  | "routed"
  | "complete"
  | "human_review"
  | "failed";

interface PipelineDocument {
  id: string;
  sourceKey: string;       // S3 key or equivalent
  mimeType: string;
  pageCount: number;
  stage: DocumentStage;
  ocrText?: string;
  ocrEngine?: "tesseract" | "google_vision" | "azure_form" | "llm";
  classification?: DocumentClassification;
  extraction?: StructuredExtraction;
  confidence?: ConfidenceScore;
  reviewReason?: string;
  createdAt: Date;
  updatedAt: Date;
}

interface DocumentClassification {
  type: string;            // "invoice", "contract", "id_document", etc.
  confidence: number;      // 0-1
  alternates: Array<{ type: string; confidence: number }>;
}

interface StructuredExtraction {
  schema: string;          // schema version used
  data: Record<string, unknown>;
  rawLLMResponse?: string;
  parseErrors: string[];
}

interface ConfidenceScore {
  overall: number;
  byField: Record<string, number>;
  flags: string[];         // "low_ocr_quality", "schema_mismatch", "unusual_value"
}

Carry this document record through every stage. Every step writes its output back to the same record and updates the stage field. This makes replay and debugging straightforward: pick up a document at any stage and run forward from there.

---

OCR Engine Selection

The choice of OCR engine has downstream consequences on extraction quality that most teams underestimate. The main options in 2026 are:

Tesseract (open source, self-hosted): Good enough for clean, typed text on white backgrounds. Degrades significantly on rotated pages, handwriting, low-resolution scans, and documents with complex layouts. Useful when you need zero marginal cost per page and the documents are reliably high quality. Not useful for HealthTech intake forms or legal discovery scans.

Google Cloud Vision / AWS Textract / Azure Form Recognizer: Cloud APIs that handle layout detection, table extraction, and form field identification natively. Textract specifically models forms as key-value pairs, which means you can skip a layer of LLM extraction for structured forms like W-9s or standard contracts. Cost scales with page count; at high volume (millions of pages/month) it becomes the dominant cost driver.

Multimodal LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro): Pass the document image directly to a vision model and ask it to transcribe or extract in one shot. For poor-quality scans, handwritten text, or mixed content (charts + text + tables), multimodal models outperform dedicated OCR pipelines by a meaningful margin. The tradeoff is cost: processing a dense 20-page PDF image-by-image through a vision API costs orders of magnitude more than a cloud OCR API call.

The practical answer is a tiered routing strategy based on document characteristics:

async function selectOCREngine(doc: PipelineDocument): Promise<OcrEngine> {
  const meta = await getDocumentMetadata(doc.sourceKey);

  // Native PDF with embedded text: no OCR needed
  if (meta.hasEmbeddedText && meta.textCoverage > 0.9) {
    return "extract_text_direct";
  }

  // High-resolution scan, simple layout: Tesseract is fast and free
  if (meta.dpi >= 300 && meta.estimatedComplexity === "low") {
    return "tesseract";
  }

  // Standard business documents: cloud API handles layout and tables well
  if (meta.pageCount <= 50 && meta.estimatedComplexity === "medium") {
    return "google_vision";
  }

  // Handwriting, poor scan quality, or complex mixed content
  if (meta.hasHandwriting || meta.dpi < 150 || meta.estimatedComplexity === "high") {
    return "llm_vision";
  }

  return "google_vision"; // safe default
}

Compute estimatedComplexity from a cheap preprocessing pass: page dimensions, image entropy, color distribution, and whether the document has embedded text via PDF metadata. This routing decision saves significant cost at scale.

---

Document Classification with Embeddings

Once you have text, you need to know what kind of document it is before you know which extraction schema to apply. There are two approaches worth comparing: classifier models and embedding similarity.

Fine-tuned classifier: Train a text classification model on labeled examples per document type. Accurate within distribution, brittle outside it. Adding a new document type requires retraining. Not recommended for systems where the document type space grows frequently.

Embedding similarity: Embed a page or chunk of the document text, then find the nearest labeled centroid or k-nearest examples in embedding space. Adding a new document type means labeling 10-20 examples, not retraining a model.

The embedding approach with a small labeled set per type works well in production:

import OpenAI from "openai";

const openai = new OpenAI();

interface DocumentTypeTemplate {
  type: string;
  centroid: number[];       // precomputed average embedding of examples
  examples: string[];       // kept for debugging
}

async function classifyDocument(
  ocrText: string,
  templates: DocumentTypeTemplate[]
): Promise<DocumentClassification> {
  // Use first 1000 tokens to avoid context limits and reduce cost
  const sample = ocrText.slice(0, 4000);

  const embeddingResponse = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: sample,
  });

  const docEmbedding = embeddingResponse.data[0].embedding;

  const scores = templates.map((template) => ({
    type: template.type,
    confidence: cosineSimilarity(docEmbedding, template.centroid),
  }));

  scores.sort((a, b) => b.confidence - a.confidence);

  return {
    type: scores[0].type,
    confidence: scores[0].confidence,
    alternates: scores.slice(1, 3),
  };
}

function cosineSimilarity(a: number[], b: number[]): number {
  const dot = a.reduce((sum, val, i) => sum + val * b[i], 0);
  const magA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
  const magB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
  return dot / (magA * magB);
}

Precompute centroids at startup from your labeled examples. Store them in memory or in a vector database if the type space is large (more than a few hundred types). Recompute centroids when you add labeled examples; it takes seconds.

A classification confidence below 0.75 is a signal to route to human review rather than proceed with extraction. Tune this threshold against your labeled holdout set.

---

Structured Extraction with LLMs

Classification tells you the document type; extraction pulls the structured fields from it. LLMs with JSON schema constraints are the most flexible extraction layer available today.

The key constraint is that you must treat LLM extraction output as untrusted external data. Parse it with a schema validator and capture every parse failure:

import { z } from "zod";

const InvoiceSchema = z.object({
  invoiceNumber: z.string(),
  vendorName: z.string(),
  vendorAddress: z.string().optional(),
  invoiceDate: z.string().regex(/^\d{4}-\d{2}-\d{2}$/),
  dueDate: z.string().regex(/^\d{4}-\d{2}-\d{2}$/).optional(),
  lineItems: z.array(
    z.object({
      description: z.string(),
      quantity: z.number(),
      unitPrice: z.number(),
      total: z.number(),
    })
  ),
  subtotal: z.number(),
  taxAmount: z.number().optional(),
  totalAmount: z.number(),
  currency: z.string().default("USD"),
});

type Invoice = z.infer<typeof InvoiceSchema>;

async function extractInvoice(ocrText: string): Promise<StructuredExtraction> {
  const response = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [
      {
        role: "system",
        content: `Extract invoice data from the provided OCR text.
Return a JSON object matching the schema exactly.
For dates, use ISO 8601 format (YYYY-MM-DD).
If a field is not present in the document, omit it rather than guessing.
Do not invent values that are not clearly stated in the text.`,
      },
      {
        role: "user",
        content: `Extract invoice data from this text:\n\n${ocrText}`,
      },
    ],
    response_format: { type: "json_object" },
  });

  const rawResponse = response.choices[0].message.content ?? "{}";
  const parsed = JSON.parse(rawResponse);
  const result = InvoiceSchema.safeParse(parsed);

  if (!result.success) {
    return {
      schema: "invoice/v1",
      data: parsed,
      rawLLMResponse: rawResponse,
      parseErrors: result.error.errors.map((e) => `${e.path.join(".")}: ${e.message}`),
    };
  }

  return {
    schema: "invoice/v1",
    data: result.data,
    rawLLMResponse: rawResponse,
    parseErrors: [],
  };
}

Never use the raw LLM JSON output directly downstream. Always parse against a typed schema. Parse errors are a signal for confidence scoring, not a crash condition.

For documents with complex tables (lab results, financial statements, legal schedules), send document images to a vision model rather than relying on OCR text alone. OCR degrades table structure; vision models read tables directly from the image.

---

Confidence Scoring

Confidence scoring is where pipelines either earn trust or silently generate garbage. A single overall confidence number is not enough. Score per field and track specific failure flags:

function scoreExtraction(
  extraction: StructuredExtraction,
  classification: DocumentClassification,
  ocrText: string
): ConfidenceScore {
  const flags: string[] = [];
  const byField: Record<string, number> = {};

  // Classification confidence feeds into overall score
  if (classification.confidence < 0.80) {
    flags.push("low_classification_confidence");
  }

  // Parse errors are hard failures
  if (extraction.parseErrors.length > 0) {
    flags.push("schema_parse_errors");
    extraction.parseErrors.forEach((err) => {
      const field = err.split(":")[0];
      byField[field] = 0;
    });
  }

  // OCR quality heuristic: long runs of garbled characters
  const garbledRatio = countGarbledCharacters(ocrText) / ocrText.length;
  if (garbledRatio > 0.05) {
    flags.push("low_ocr_quality");
  }

  // Domain-specific sanity checks
  if (extraction.schema.startsWith("invoice/")) {
    const data = extraction.data as Partial<Invoice>;
    if (data.totalAmount !== undefined && data.subtotal !== undefined) {
      const diff = Math.abs(data.totalAmount - (data.subtotal + (data.taxAmount ?? 0)));
      if (diff > 0.01) {
        flags.push("math_inconsistency");
        byField["totalAmount"] = 0.3;
      }
    }
  }

  const penaltyPerFlag: Record<string, number> = {
    low_classification_confidence: 0.15,
    schema_parse_errors: 0.25,
    low_ocr_quality: 0.20,
    math_inconsistency: 0.30,
  };

  const totalPenalty = flags.reduce(
    (sum, flag) => sum + (penaltyPerFlag[flag] ?? 0.10),
    0
  );

  const overall = Math.max(0, 1 - totalPenalty) * classification.confidence;

  return { overall, byField, flags };
}

The domain-specific sanity checks (math consistency for invoices, date range plausibility for contracts, ID number checksum for regulated documents) are the highest-value confidence signals. They catch extractions where the LLM returned plausible-looking but wrong values.

---

Human-in-the-Loop Review Queue

Every pipeline that processes regulated or business-critical documents needs a human review path. The question is not whether to build one, but what to route to it.

Route to human review when:

• Overall confidence is below your acceptance threshold (typically 0.70-0.80, tune per document type)
• Any required field is missing after extraction
• A domain sanity check flag fires
• Classification confidence is below 0.75 (you may have the wrong extraction schema entirely)
• The document type is new or rare (fewer than 50 labeled examples)

type ReviewAction = "approve" | "reject" | "correct";

interface ReviewQueueItem {
  documentId: string;
  reason: string;
  confidence: ConfidenceScore;
  extraction: StructuredExtraction;
  ocrText: string;
  sourceKey: string;
  createdAt: Date;
  assignedTo?: string;
  reviewedAt?: Date;
  reviewAction?: ReviewAction;
  correctedData?: Record<string, unknown>;
}

function shouldRouteToReview(
  confidence: ConfidenceScore,
  extraction: StructuredExtraction
): { review: boolean; reason: string } {
  if (confidence.overall < 0.75) {
    return { review: true, reason: `confidence_below_threshold: ${confidence.overall.toFixed(2)}` };
  }

  if (extraction.parseErrors.length > 0) {
    return { review: true, reason: `parse_errors: ${extraction.parseErrors.join("; ")}` };
  }

  const hardFlags = ["math_inconsistency", "low_classification_confidence"];
  const triggeredHardFlag = confidence.flags.find((f) => hardFlags.includes(f));
  if (triggeredHardFlag) {
    return { review: true, reason: `hard_flag: ${triggeredHardFlag}` };
  }

  return { review: false, reason: "" };
}

Human corrections are training data. Every reviewed and corrected document should feed back into your classification centroids and your prompt refinement process. Without a feedback loop, your accuracy plateaus at whatever the model gives you out of the box.

---

Pipeline Orchestration

At small volume (a few hundred documents per day), a simple queue-based approach with a job worker is sufficient. At higher volume, you want a proper workflow engine.

The core requirements are: idempotency (processing a document twice must not create duplicate records), retryability per stage (OCR can fail transiently without losing classification work already done), and observability (you need to know where in the pipeline documents are sitting and why they stalled).

A minimal queue worker pattern in TypeScript:

import { Queue, Worker } from "bullmq";

const ocrQueue = new Queue("ocr");
const classifyQueue = new Queue("classify");
const extractQueue = new Queue("extract");

// Each worker processes one stage and enqueues the next
const ocrWorker = new Worker(
  "ocr",
  async (job) => {
    const { documentId } = job.data;
    const doc = await getDocument(documentId);

    if (doc.stage !== "normalized") {
      // Idempotency: skip if already past this stage
      return;
    }

    const engine = await selectOCREngine(doc);
    const ocrText = await runOCR(doc.sourceKey, engine);

    await updateDocument(documentId, {
      ocrText,
      ocrEngine: engine,
      stage: "ocr_complete",
    });

    await classifyQueue.add("classify", { documentId }, {
      attempts: 3,
      backoff: { type: "exponential", delay: 2000 },
    });
  },
  {
    connection: redisConnection,
    concurrency: 10,
  }
);

Each stage worker checks the current document stage before processing. This makes the pipeline replay-safe: re-enqueue a document at any stage and it either skips work already done or redoes only what needs redoing.

For LegalTech, RegTech, HealthTech, and FinTech use cases, add a dead-letter queue and alert on any document that fails all retry attempts. A document silently stuck in “failed” state is a compliance risk.

---

Production Considerations

OCR quality gates before extraction. Measure OCR confidence scores where available (Google Vision returns per-word confidence). Discard extractions where median word confidence is below 0.7 and route to manual entry instead.

Schema and prompt versioning. Your extraction schemas will change as you add fields or fix types. Version every schema (invoice/v1, invoice/v2) and store the version alongside the extraction. Track which prompt version and model version was used for each extraction. When you deploy schema v2, reprocess historical documents in a background job. When model behavior changes between API versions, you need to know which documents were processed with which prompt before investigating regressions.

Cost observability per document type. Invoice extraction through GPT-4o costs more than contract header extraction. Track actual API cost per document, broken down by OCR engine and LLM call. Without this visibility, a surge in complex document types can spike your API bill invisibly.

Timeout and retry policies by stage. OCR can take 30-60 seconds for a dense multi-page PDF. LLM extraction for a 50-page contract can take 2-3 minutes. Set stage-specific timeouts, not a single pipeline timeout. A 5-second timeout fine for single-page invoices will fail silently on complex documents.

Audit trail for regulated industries. In HealthTech (HIPAA), FinTech (SOX, PCI), and LegalTech contexts, store a complete audit log: which model version extracted data, every human review action and who performed it. Append-only events, not mutable state on the document record.

---

Where This Breaks Down

The approach described here works well for document types you have labeled examples for. It struggles with:

Completely novel document types. When a document type arrives that shares no vocabulary with your labeled set, embedding similarity will route it to the nearest known type. Set a hard lower bound on classification confidence: anything below 0.50 goes straight to review as “unrecognized type” rather than being misclassified.

Multi-document PDFs. A single uploaded PDF may contain a cover letter, a contract, an attachment, and a signature page. Naive processing treats this as one document and produces garbled extractions. Split PDFs by logical boundaries (blank pages, header pattern changes) before classification. This requires a dedicated preprocessing step and is frequently underestimated.

Scans with mixed orientations. A batch of scanned documents from a physical folder will have pages at various rotations. Tesseract produces garbage on rotated pages. Cloud APIs handle this better; vision models best. Add orientation detection and correction before OCR for human-uploaded batches.

---

The pipeline architecture described here is not novel, but the specifics matter: tiered OCR routing, embedding-based classification with updateable centroids, schema-validated LLM extraction, per-field confidence scoring, and a feedback loop from human corrections back to model improvement. Each of these is a distinct engineering decision with real tradeoffs. A system that handles all of them reliably is worth significantly more than one that handles only the happy path.