Computer Vision in Production: Object Detection, Image Classification, and Video Analysis Pipelines for SaaS Applications

Notebook demos for computer vision are deceptively easy. Load a model, call model.predict(), display bounding boxes on a sample image, done. The gap between that and a system your SaaS product can depend on is large. GPU cold starts, batch throughput, model warm-up, video frame extraction rates, confidence calibration, human review queues, and per-image cost tracking are all invisible in a notebook. They are the actual engineering.

This article covers how to build a production computer vision pipeline that handles object detection, image classification, and video analysis, with realistic TypeScript orchestration code, a model selection framework, and the production concerns that turn a proof of concept into a shipped feature.

---

Picking the Right Model for Your Task

The model selection decision is the one you will be most tempted to revisit later, so make it carefully. The three models worth knowing for SaaS CV work are YOLO variants, CLIP, and Florence-2. They solve different problems.

YOLO (You Only Look Once): The standard for real-time object detection and bounding box prediction. YOLOv8 and YOLOv9 are the current production defaults. Fast on GPU (sub-10ms per image at full resolution on an A10G), well-supported by Ultralytics, and trivially fine-tunable on custom classes. Use YOLO when you need to locate and label objects with bounding boxes. Not ideal for open-vocabulary queries or semantic understanding.

CLIP (Contrastive Language-Image Pre-Training): Learns joint embeddings for images and text. Useful for zero-shot classification (does this image contain a safety helmet?) and similarity search (find images visually similar to this one). Slower than YOLO for pure detection but requires no labeled bounding box data. Use CLIP when your classification categories change frequently or you need to query by natural language description.

Florence-2: Microsoft’s vision foundation model. Handles object detection, grounding, captioning, and OCR in a single model with a prompt-based API. Better than CLIP at structured scene understanding, competitive with YOLO on detection, and much more flexible. The tradeoff is size (0.2B or 0.7B params) and inference cost. Use Florence-2 when you need multiple CV tasks from one model and can afford the latency.

The decision framework is straightforward. If you need bounding boxes and your classes are fixed, use YOLO. If you need zero-shot classification or image search, use CLIP. If you need bounding boxes plus captioning plus OCR without separate models, use Florence-2 and accept the higher per-image cost.

---

Inference Infrastructure: GPU Serving and Model Warm-Up

The single biggest operational mistake in CV deployments is treating GPU instances like on-demand Lambda functions. GPU cold start for a YOLO model is 8-25 seconds depending on instance type and model size. For a user-facing SaaS feature, that is unacceptable. You need at least one warm replica at all times.

The serving architecture that works at moderate scale (up to roughly 200 requests per second) is a pool of GPU workers behind a queue. Workers pull jobs from the queue, run inference, and write results to a results store. The API layer is CPU-only and just enqueues jobs and polls for results.

// inference-worker/worker.ts
import { InferenceSession } from "onnxruntime-node";
import * as redis from "redis";

interface CVJob {
  jobId: string;
  imageUrl: string;
  task: "detect" | "classify" | "caption";
  modelId: "yolov9s" | "clip-vit-b32" | "florence2-base";
  requestedAt: number;
}

interface CVResult {
  jobId: string;
  modelId: string;
  inferenceMs: number;
  detections?: Detection[];
  classifications?: Classification[];
  caption?: string;
  error?: string;
  completedAt: number;
}

interface Detection {
  label: string;
  confidence: number;
  bbox: [number, number, number, number]; // x1, y1, x2, y2 normalized
}

interface Classification {
  label: string;
  confidence: number;
}

async function runWorker() {
  const client = redis.createClient({ url: process.env.REDIS_URL });
  await client.connect();

  // Load model into GPU memory at startup, not per-request
  const session = await InferenceSession.create("/models/yolov9s.onnx", {
    executionProviders: ["cuda"],
    graphOptimizationLevel: "all",
  });

  // Warm-up: run one empty inference to initialize CUDA kernels
  const dummyInput = new Float32Array(1 * 3 * 640 * 640);
  await session.run({ images: dummyInput });
  console.log("Worker ready, model warm");

  while (true) {
    const raw = await client.brPop("cv:jobs", 30);
    if (!raw) continue;

    const job: CVJob = JSON.parse(raw.element);
    const start = Date.now();

    try {
      const result = await processJob(session, job);
      await client.setEx(
        `cv:results:${job.jobId}`,
        300,
        JSON.stringify(result)
      );
    } catch (err) {
      await client.setEx(
        `cv:results:${job.jobId}`,
        300,
        JSON.stringify({
          jobId: job.jobId,
          error: String(err),
          completedAt: Date.now(),
        })
      );
    }
  }
}

A few decisions in that code are worth naming explicitly. The model loads once at process startup, not per request. The warm-up inference matters: CUDA kernel compilation happens on the first call, so the first real request after a cold start still takes 500ms-2s without it. The results TTL of 300 seconds gives the API layer a safe polling window while preventing unbounded Redis growth.

For batching, the decision point is throughput vs. latency. If you have a batch-oriented workflow (process 10,000 images overnight), set batchSize to 16-32 and process images together. A single A10G with batch size 16 on YOLOv8n achieves roughly 3,000 images per second. For interactive use, keep batch size at 1 to minimize queuing latency.

---

Image Preprocessing and the Normalization Contract

Every CV model has a strict input contract: image dimensions, pixel normalization range, channel order (RGB vs. BGR). Getting this wrong produces silently incorrect results, not errors. YOLO expects RGB, normalized to [0, 1], resized to 640x640 with letterboxing to preserve aspect ratio. CLIP expects RGB, center-cropped and resized to 224x224, normalized with ImageNet means and standard deviations.

// preprocessing/normalize.ts
import sharp from "sharp";

interface PreprocessedImage {
  tensor: Float32Array;
  originalWidth: number;
  originalHeight: number;
  paddingTop: number;
  paddingLeft: number;
  scale: number;
}

async function preprocessForYOLO(
  imageBuffer: Buffer,
  targetSize = 640
): Promise<PreprocessedImage> {
  const meta = await sharp(imageBuffer).metadata();
  const { width = 640, height = 640 } = meta;

  const scale = Math.min(targetSize / width, targetSize / height);
  const scaledW = Math.round(width * scale);
  const scaledH = Math.round(height * scale);

  const paddingLeft = Math.floor((targetSize - scaledW) / 2);
  const paddingTop = Math.floor((targetSize - scaledH) / 2);

  const raw = await sharp(imageBuffer)
    .resize(scaledW, scaledH)
    .extend({
      top: paddingTop,
      bottom: targetSize - scaledH - paddingTop,
      left: paddingLeft,
      right: targetSize - scaledW - paddingLeft,
      background: { r: 114, g: 114, b: 114 }, // standard YOLO gray padding
    })
    .raw()
    .toBuffer();

  const tensor = new Float32Array(3 * targetSize * targetSize);
  for (let i = 0; i < targetSize * targetSize; i++) {
    tensor[i] = raw[i * 3] / 255.0;                     // R
    tensor[i + targetSize * targetSize] = raw[i * 3 + 1] / 255.0; // G
    tensor[i + 2 * targetSize * targetSize] = raw[i * 3 + 2] / 255.0; // B
  }

  return { tensor, originalWidth: width, originalHeight: height, paddingTop, paddingLeft, scale };
}

Tracking the letterbox padding and scale factor is load-bearing. When you map predicted bounding boxes back to original image coordinates, you need to undo the padding offset and scale. Skip this and your bounding boxes will be offset for any non-square input, which is almost every real image.

---

Video Frame Extraction and Analysis

Video analysis is image analysis applied at scale with one extra constraint: frame selection strategy. Processing every frame of a 30fps video is expensive and redundant. Most consecutive frames are nearly identical. The question is how aggressively to subsample without missing events.

Three strategies in order of cost:

Fixed interval subsampling. Extract one frame every N seconds. Simple, predictable cost. Misses fast events (a defect that appears for 0.5s at 2fps subsampling is invisible). Use for slow-changing scenes.

Scene change detection. Compute inter-frame difference (absolute pixel delta or perceptual hash). Only process frames where the scene changes significantly. More accurate but adds a preprocessing pass.

Motion-based extraction. Use optical flow or background subtraction to detect motion events and extract frames around those events. Best accuracy for action detection use cases.

// video/extractor.ts
import { createFFmpeg, fetchFile } from "@ffmpeg/ffmpeg";
import { createHash } from "crypto";

interface ExtractedFrame {
  frameIndex: number;
  timestampMs: number;
  imageBuffer: Buffer;
  phash: string;
}

async function extractKeyFrames(
  videoBuffer: Buffer,
  options: {
    maxFps: number;       // upper bound on frame rate to process
    sceneThreshold: number; // 0-1: how different must frames be to count as new scene
  }
): Promise<ExtractedFrame[]> {
  const ffmpeg = createFFmpeg({ log: false });
  await ffmpeg.load();

  ffmpeg.FS("writeFile", "input.mp4", await fetchFile(new Blob([videoBuffer])));

  // Extract frames at maxFps using ffmpeg scene change filter
  await ffmpeg.run(
    "-i", "input.mp4",
    "-vf", `select=gt(scene\\,${options.sceneThreshold}),fps=${options.maxFps}`,
    "-vsync", "vfr",
    "-frame_pts", "1",
    "frame-%06d.jpg"
  );

  const frames: ExtractedFrame[] = [];
  const files = ffmpeg.FS("readdir", "/");

  for (const file of files) {
    if (!file.startsWith("frame-")) continue;
    const data = ffmpeg.FS("readFile", file);
    const buf = Buffer.from(data);
    const phash = createHash("sha256").update(buf.slice(0, 1024)).digest("hex").slice(0, 16);

    const match = file.match(/frame-(\d+)/);
    const frameIndex = match ? parseInt(match[1]) : 0;

    frames.push({
      frameIndex,
      timestampMs: (frameIndex / options.maxFps) * 1000,
      imageBuffer: buf,
      phash,
    });

    ffmpeg.FS("unlink", file);
  }

  return frames;
}

The scene filter in ffmpeg computes a perceptual difference score between consecutive frames. A threshold of 0.3 is aggressive (catches most scene changes). A threshold of 0.1 is sensitive (catches subtle changes). For SaaS products where video analysis is a paid feature, err toward 0.1 initially and tune up once you have data on what your users’ videos actually look like.

---

Confidence Thresholds and Human Review Loops

Every CV model returns a confidence score. The question is what to do with predictions below a given threshold. The naive answer is discard them. The production answer depends on your error cost asymmetry.

For a defect detection system in a manufacturing SaaS, a missed defect (false negative) costs more than a false alarm (false positive). Set a low confidence threshold and route low-confidence detections to a human review queue rather than discarding them. For a content moderation system, a false positive (incorrectly flagging clean content) damages user trust, so set a higher threshold and accept more misses.

The routing structure that works:

// review/router.ts
interface PredictionWithConfidence {
  jobId: string;
  predictions: Array<{
    label: string;
    confidence: number;
    bbox?: [number, number, number, number];
  }>;
  imageUrl: string;
  context: Record<string, string>;
}

interface RoutingDecision {
  action: "auto-accept" | "auto-reject" | "human-review";
  reason: string;
  priority: "high" | "medium" | "low";
}

function routePrediction(
  prediction: PredictionWithConfidence,
  config: {
    autoAcceptThreshold: number;  // e.g. 0.90: accept without review
    autoRejectThreshold: number;  // e.g. 0.10: reject without review
    reviewPriorityThreshold: number; // e.g. 0.60: route to high-priority review
  }
): RoutingDecision {
  const maxConfidence = Math.max(...prediction.predictions.map((p) => p.confidence));

  if (maxConfidence >= config.autoAcceptThreshold) {
    return { action: "auto-accept", reason: `confidence ${maxConfidence.toFixed(3)} above auto-accept threshold`, priority: "low" };
  }

  if (maxConfidence <= config.autoRejectThreshold) {
    return { action: "auto-reject", reason: `confidence ${maxConfidence.toFixed(3)} below auto-reject threshold`, priority: "low" };
  }

  const priority = maxConfidence < config.reviewPriorityThreshold ? "high" : "medium";
  return {
    action: "human-review",
    reason: `confidence ${maxConfidence.toFixed(3)} in review band`,
    priority,
  };
}

One non-obvious detail: the review queue needs a latency SLA as much as an accuracy requirement. A human review queue that backs up creates ghost predictions sitting in limbo, blocking downstream workflows. Set a max queue depth and alert before it fills. If review capacity is exceeded, auto-accept or auto-reject based on which error is cheaper for your use case and inform the user.

Track your false positive and false negative rates per confidence band. This is how you tune thresholds over time and detect model drift (when the model’s confidence calibration degrades as your input distribution shifts).

---

Tradeoffs: Model and Infrastructure Choices

Dimension	YOLO	CLIP	Florence-2
Task fit	Bounding box detection	Zero-shot classification, similarity search	Detection + captioning + OCR
Inference latency (A10G)	5-15ms	20-50ms	80-200ms
Fine-tuning requirement	Required for custom classes	Not required for new classes	Not required (prompt-based)
Per-image cost at scale	Lowest	Low	Highest
Open-vocabulary queries	No	Yes	Yes
Video suitability	High (fast)	Medium	Low (too slow per frame)

Dimension	Dedicated GPU	Serverless GPU (Modal, RunPod)	Cloud Vision API
Cold start	None (warm)	5-30s	None
Cost at low volume	High (idle GPU)	Low (pay per second)	Medium (per-call pricing)
Cost at high volume	Low (amortized)	Medium	High
Model control	Full	Full	None
Operational overhead	High	Medium	Low

---

Cost Modeling for CV Workloads

GPU compute is billed per second of usage, not per image. The number that matters is throughput: images processed per GPU-second. At batch size 1 on an A10G ($1.30/hr), YOLOv9s processes roughly 100 images per second. That is $0.0000036 per image in GPU cost alone. At 10 million images per month, GPU cost is $36. At 100 million images per month, it is $360. Storage, networking, and preprocessing are the costs that grow unexpectedly.

The cost model that catches surprises:

// cost/model.ts
interface CVCostEstimate {
  imageCount: number;
  gpuCostUsd: number;
  storageCostUsd: number;    // results + thumbnails
  transferCostUsd: number;   // egress for image downloads
  reviewCostUsd: number;     // human reviewer cost at $0.03/review
  totalCostUsd: number;
  costPerImageUsd: number;
}

function estimateMonthlyCost(params: {
  imagesPerMonth: number;
  avgImageSizeKb: number;
  imagesPerGpuSecond: number;
  reviewRate: number;          // fraction routed to human review
  gpuHourlyRateUsd: number;
  storageGbMonthUsd: number;
  egressPerGbUsd: number;
  reviewerCostPerImageUsd: number;
}): CVCostEstimate {
  const gpuSeconds = params.imagesPerMonth / params.imagesPerGpuSecond;
  const gpuCostUsd = (gpuSeconds / 3600) * params.gpuHourlyRateUsd;

  const totalDataGb = (params.imagesPerMonth * params.avgImageSizeKb) / (1024 * 1024);
  const storageCostUsd = totalDataGb * params.storageGbMonthUsd;
  const transferCostUsd = totalDataGb * params.egressPerGbUsd;

  const reviewCount = params.imagesPerMonth * params.reviewRate;
  const reviewCostUsd = reviewCount * params.reviewerCostPerImageUsd;

  const totalCostUsd = gpuCostUsd + storageCostUsd + transferCostUsd + reviewCostUsd;

  return {
    imageCount: params.imagesPerMonth,
    gpuCostUsd,
    storageCostUsd,
    transferCostUsd,
    reviewCostUsd,
    totalCostUsd,
    costPerImageUsd: totalCostUsd / params.imagesPerMonth,
  };
}

Run this before you price your CV feature. If you are charging $0.001 per image and the all-in cost is $0.0004, your margin is fine. If human review at 15% review rate is costing $0.03 per reviewed image, and your review rate is 20%, your effective cost is $0.006 per image before GPU, storage, or transfer. Pricing decisions made with a Jupyter notebook running on a pre-warmed GPU will underestimate cost by 3-10x.

---

Production Considerations

Model versioning. Treat CV model files like code. Store model versions in object storage, reference them by content hash in your job queue schema, and never overwrite a deployed model in place. When you ship a new model version, run it in shadow mode (process images with both versions, compare outputs) before routing production traffic. This is how you catch regressions in precision or recall before users do.

Observability. The metrics that matter for CV pipelines: inference P50/P95/P99 latency, queue depth and drain rate, confidence score distribution per label (alert when it shifts), false positive rate (approximated from human review overrides), and GPU utilization. A model that degrades silently as input distribution drifts will look fine on latency metrics. Confidence distribution shifts are your early warning signal.

Idempotency. Image processing jobs should be idempotent. If a worker crashes mid-job, the job must be re-runnable without double-charging the user or creating duplicate results. Use job IDs as idempotency keys and check the results store before running inference. Workers that crash without acknowledging should have their jobs re-queued after a timeout, not silently dropped.

Input validation. Reject malformed inputs at the API boundary, not inside the worker. Validate MIME type, file size (reject anything over your max, e.g., 20MB), and image dimensions before enqueuing. A 200MB TIFF dropped into a worker that expects 640x640 JPEG will exhaust GPU memory and crash the process. GPU OOM events are not gracefully catchable in all ONNX Runtime configurations.

Preprocessing failures. Image corruption, truncated uploads, and unsupported color profiles (CMYK PDFs embedded as images) are common. Handle sharp decoding failures explicitly and return a structured error rather than an uncaught exception. Corrupted images should fail fast with a 422 before they reach a GPU worker.

Model warm-up in auto-scaling environments. If you use auto-scaling GPU workers, a scale-out event will add cold workers. The warm-up inference on startup absorbs the first-request latency spike. If your workers load-balance without health checks, requests will land on cold workers before warm-up completes. Add a readiness probe that waits for the warm-up inference to finish before the worker joins the pool.

---

The Gap That Matters

The distance between a notebook and a production CV system is mostly operational, not mathematical. The model itself is the easiest part. Everything around it, serving infrastructure, preprocessing contracts, video extraction strategy, confidence routing, human review capacity planning, cost modeling, observability, model versioning, input validation, is what determines whether the feature ships and stays shipped.

Start with the smallest model that meets your accuracy requirement on your actual data, not on a public benchmark. Run it on representative production images before committing to an architecture. The confidence distribution you see in the lab will not match what your users send you, and that gap is where most CV production incidents originate.