Designing an Image Processing Pipeline: Uploads, Thumbnails, and On-the-Fly Transformations at Scale

Images are deceptively simple to ship and surprisingly expensive to scale. A naive pipeline takes about an hour to build: accept an upload, save it to S3, return the URL. That works until your users start uploading 12-megapixel phone photos and your frontend is resizing them in CSS, or until you realize you are serving 8MB JPEGs to users on 3G, or until your S3 egress bill comes in.

A production image pipeline has five distinct concerns: ingestion (how images get into your system without touching your API servers), processing (resizing, format conversion, quality optimization), storage (where originals and derivatives live), delivery (CDN configuration and cache keys), and the pre-generation vs. lazy-generation tradeoff that determines your cost structure. Each of these has failure modes that are not obvious until you hit them.

Ingestion: Direct-to-Storage with Presigned URLs

The first mistake most teams make is routing image uploads through their API server. A single 8MB upload buffered through a Node.js server occupies memory for the entire transfer duration. At ten concurrent uploads you are holding 80MB in process memory before you have done any actual work.

The correct pattern uses a presigned URL to give the client a time-limited, scoped upload credential for direct-to-storage writes.

// Generate a presigned upload URL
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { createId } from "@paralleldrive/cuid2";

const s3 = new S3Client({ region: "us-east-1" });

interface PresignedUploadParams {
  userId: string;
  contentType: "image/jpeg" | "image/png" | "image/webp" | "image/gif";
  fileSizeBytes: number;
}

interface PresignedUploadResult {
  uploadUrl: string;
  objectKey: string;
  expiresAt: Date;
}

async function createPresignedUpload(
  params: PresignedUploadParams
): Promise<PresignedUploadResult> {
  const maxSizeBytes = 20 * 1024 * 1024; // 20MB limit
  if (params.fileSizeBytes > maxSizeBytes) {
    throw new Error(`File size ${params.fileSizeBytes} exceeds limit ${maxSizeBytes}`);
  }

  const objectKey = `originals/${params.userId}/${createId()}.${extensionFor(params.contentType)}`;

  const command = new PutObjectCommand({
    Bucket: process.env.UPLOAD_BUCKET,
    Key: objectKey,
    ContentType: params.contentType,
    ContentLength: params.fileSizeBytes,
    // Tag the object so S3 lifecycle rules can clean up unconfirmed uploads
    Tagging: "status=pending",
  });

  const uploadUrl = await getSignedUrl(s3, command, { expiresIn: 300 });
  const expiresAt = new Date(Date.now() + 300 * 1000);

  return { uploadUrl, objectKey, expiresAt };
}

function extensionFor(contentType: string): string {
  const map: Record<string, string> = {
    "image/jpeg": "jpg",
    "image/png": "png",
    "image/webp": "webp",
    "image/gif": "gif",
  };
  return map[contentType] ?? "bin";
}

Two details matter here. First, include ContentLength in the presigned URL parameters. Without it, a client can upload a 200MB file against a URL you intended for 2MB. Second, tag new objects as status=pending and set an S3 lifecycle rule to delete pending objects after 24 hours. Users abandon uploads constantly; without cleanup you accumulate gigabytes of orphaned originals.

After the upload completes, the client calls a confirmation endpoint. That endpoint validates the object exists, updates the tag to status=confirmed, and enqueues a processing job.

Async Thumbnail Generation

Do not generate thumbnails synchronously in the upload confirmation handler. Image processing is CPU-bound, can take several seconds for large originals, and your API server should not be doing it at all.

The confirmation handler enqueues a message. A separate worker processes it.

// Worker: processes thumbnail generation jobs
interface ThumbnailJob {
  objectKey: string;
  userId: string;
  imageId: string;
}

interface ThumbnailSpec {
  name: string;
  width: number;
  height: number;
  fit: "cover" | "contain" | "fill";
  quality: number;
}

const THUMBNAIL_SPECS: ThumbnailSpec[] = [
  { name: "thumb", width: 200, height: 200, fit: "cover", quality: 80 },
  { name: "card", width: 600, height: 400, fit: "cover", quality: 82 },
  { name: "hero", width: 1200, height: 630, fit: "cover", quality: 85 },
  { name: "og", width: 1200, height: 630, fit: "fill", quality: 90 },
];

async function processThumbnailJob(job: ThumbnailJob): Promise<void> {
  // Download original from S3
  const original = await downloadFromS3(process.env.UPLOAD_BUCKET!, job.objectKey);

  // Process all specs in parallel
  const results = await Promise.allSettled(
    THUMBNAIL_SPECS.map(async (spec) => {
      const processed = await sharp(original)
        .resize(spec.width, spec.height, { fit: spec.fit })
        .webp({ quality: spec.quality })
        .toBuffer();

      const derivativeKey = `derivatives/${job.imageId}/${spec.name}.webp`;

      await uploadToS3(process.env.ASSETS_BUCKET!, derivativeKey, processed, {
        contentType: "image/webp",
        cacheControl: "public, max-age=31536000, immutable",
      });

      return { spec: spec.name, key: derivativeKey, bytes: processed.length };
    })
  );

  // Log failures without failing the entire job
  const failures = results.filter((r) => r.status === "rejected");
  if (failures.length > 0) {
    console.error("Thumbnail generation partial failures", {
      imageId: job.imageId,
      failures: failures.map((f) => (f as PromiseRejectedResult).reason?.message),
    });
  }

  // Mark image as processed even if some variants failed
  // A retry will fill gaps; do not block the entire record
  await markImageProcessed(job.imageId, results);
}

Sharp is the right library for this. It wraps libvips, which is faster than ImageMagick and has much lower memory overhead. On a 2-vCPU worker, sharp can process a 10-megapixel JPEG into five WebP thumbnails in under 400ms.

One tricky detail: Promise.allSettled instead of Promise.all. If the 1200x630 hero thumbnail fails for some transient reason (out-of-memory spike, transient S3 error), you do not want to retry the entire job and reprocess every variant. Mark the image processed, log which variants failed, and let a separate reconciliation job backfill them.

On-the-Fly Transformations with Cloudflare Workers

Pre-generating a fixed set of thumbnails breaks down as soon as your product team asks for a new image variant. They want 400x400 square crops for a new grid layout. You either add a new spec and reprocess millions of originals, or you build on-the-fly transformation from the start.

On-the-fly transformation uses a URL schema like /images/{imageId}?w=400&h=400&fit=cover&f=webp and processes the image at request time. The key insight: you only need to process each unique parameter combination once, because you cache the result at the CDN edge.

Cloudflare Workers Image Resizing handles this directly if you are on a Pro plan or higher.

// Cloudflare Worker: image transformation proxy
export default {
  async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
    const url = new URL(request.url);

    // Parse and validate transform parameters
    const params = parseTransformParams(url.searchParams);
    if (!params.ok) {
      return new Response(params.error, { status: 400 });
    }

    // Check edge cache first
    const cacheKey = buildCacheKey(url);
    const cache = caches.default;
    const cached = await cache.match(cacheKey);
    if (cached) return cached;

    // Fetch from origin storage
    const imageId = url.pathname.split("/")[2];
    const originUrl = `${env.STORAGE_BASE_URL}/originals/${imageId}`;

    const transformed = await fetch(originUrl, {
      cf: {
        image: {
          width: params.value.width,
          height: params.value.height,
          fit: params.value.fit,
          format: params.value.format,
          quality: params.value.quality ?? 82,
        },
      },
    });

    if (!transformed.ok) {
      return new Response("Image not found", { status: 404 });
    }

    // Build cacheable response with immutable headers
    const response = new Response(transformed.body, {
      headers: {
        "Content-Type": transformed.headers.get("Content-Type") ?? "image/webp",
        "Cache-Control": "public, max-age=31536000, immutable",
        "Vary": "Accept", // Needed if you do format negotiation
        "X-Image-Id": imageId,
      },
    });

    // Store in edge cache without blocking the response
    ctx.waitUntil(cache.put(cacheKey, response.clone()));

    return response;
  },
};

function buildCacheKey(url: URL): Request {
  // Normalize parameter order so ?w=400&h=400 and ?h=400&w=400 hit the same key
  const params = new URLSearchParams();
  for (const key of ["w", "h", "fit", "f", "q"].sort()) {
    const value = url.searchParams.get(key);
    if (value) params.set(key, value);
  }
  const normalized = new URL(url.pathname, url.origin);
  normalized.search = params.toString();
  return new Request(normalized.toString());
}

type TransformParams = {
  width?: number;
  height?: number;
  fit: "cover" | "contain" | "fill" | "scale-down";
  format: "webp" | "avif" | "jpeg" | "png";
  quality?: number;
};

function parseTransformParams(
  params: URLSearchParams
): { ok: true; value: TransformParams } | { ok: false; error: string } {
  const width = params.get("w") ? Number(params.get("w")) : undefined;
  const height = params.get("h") ? Number(params.get("h")) : undefined;

  if (width !== undefined && (isNaN(width) || width < 1 || width > 4000)) {
    return { ok: false, error: "Invalid width" };
  }
  if (height !== undefined && (isNaN(height) || height < 1 || height > 4000)) {
    return { ok: false, error: "Invalid height" };
  }

  const fit = (params.get("fit") ?? "cover") as TransformParams["fit"];
  const format = (params.get("f") ?? "webp") as TransformParams["format"];

  return { ok: true, value: { width, height, fit, format } };
}

The cache key normalization in buildCacheKey is easy to miss and expensive to get wrong. If parameter order is not normalized, ?w=400&h=400 and ?h=400&w=400 create two cache entries for the same image. At scale, un-normalized cache keys cut your CDN hit ratio significantly.

Storage Strategy: S3 vs. R2

The storage decision comes down to egress costs.

S3 charges for egress. At $0.09/GB, a pipeline delivering 100TB/month of image data pays $9,000/month just in S3 egress, before CDN costs. For read-heavy image pipelines, this adds up fast.

Cloudflare R2 charges zero egress fees. You pay for storage ($0.015/GB/month) and operations (Class A writes at $4.50/million, Class B reads at $0.36/million), but not for data transferred out. For images served through Cloudflare CDN, where R2 and the CDN are in the same network, this is almost always cheaper than S3 at meaningful scale.

The migration path is straightforward if you design the storage layer correctly from the start:

interface StorageAdapter {
  put(key: string, data: Buffer, metadata: ObjectMetadata): Promise<void>;
  get(key: string): Promise<Buffer | null>;
  getUrl(key: string): string;
  delete(key: string): Promise<void>;
}

interface ObjectMetadata {
  contentType: string;
  cacheControl?: string;
  userMetadata?: Record<string, string>;
}

// Swap implementations without changing call sites
class R2StorageAdapter implements StorageAdapter {
  constructor(private readonly bucket: R2Bucket) {}

  async put(key: string, data: Buffer, metadata: ObjectMetadata): Promise<void> {
    await this.bucket.put(key, data, {
      httpMetadata: {
        contentType: metadata.contentType,
        cacheControl: metadata.cacheControl,
      },
      customMetadata: metadata.userMetadata,
    });
  }

  async get(key: string): Promise<Buffer | null> {
    const object = await this.bucket.get(key);
    if (!object) return null;
    return Buffer.from(await object.arrayBuffer());
  }

  getUrl(key: string): string {
    return `${process.env.R2_PUBLIC_URL}/${key}`;
  }

  async delete(key: string): Promise<void> {
    await this.bucket.delete(key);
  }
}

If you are starting fresh and deploying on Cloudflare’s network, R2 is the right default for derivative storage. Keep originals on S3 or wherever your existing data lives; derivatives are the high-egress objects that drive cost.

WebP vs. AVIF: Format Selection

WebP and AVIF are not interchangeable. They make different tradeoffs.

Dimension	WebP	AVIF
Compression ratio	Good (25-35% over JPEG)	Better (40-50% over JPEG)
Encode time	Fast (tens of milliseconds)	Slow (hundreds of milliseconds to seconds)
Browser support	97%+	90%+ as of 2025
Decode speed	Fast	Slower on mobile
Animation support	Yes	Yes (but limited tooling)
Lossless support	Yes	Yes

For most image pipelines, the right answer is: serve WebP by default, offer AVIF as an opt-in for hero images and thumbnails where encode time is acceptable.

AVIF encoding at quality 60 is roughly equivalent to WebP at quality 80 visually, but AVIF will take 5-10x longer to produce. If your thumbnail worker processes 200ms WebP and 1500ms AVIF per image, and you have a backlog of 2 million images to process, that is the difference between a 2-hour and 15-hour migration window.

Format negotiation via the Accept header is an alternative to explicit format parameters:

function selectFormat(acceptHeader: string | null): "avif" | "webp" | "jpeg" {
  if (!acceptHeader) return "jpeg";
  if (acceptHeader.includes("image/avif")) return "avif";
  if (acceptHeader.includes("image/webp")) return "webp";
  return "jpeg";
}

If you do format negotiation, you must add Vary: Accept to your cached responses and make sure your CDN is configured to treat Vary correctly. Cloudflare strips Vary headers by default and manages format variants internally when you use their image resizing. Check your CDN’s behavior explicitly before relying on Vary for format selection.

Pre-generation vs. Lazy Generation

This is the tradeoff that most determines your operational cost structure.

Pre-generation: When an image is uploaded, immediately produce every derivative variant and store them. Requests hit cached storage with no compute at request time.

Lazy generation: On the first request for a variant, generate and cache it. Subsequent requests hit the cache. Cold variants incur compute cost.

Dimension	Pre-generation	Lazy generation
Storage cost	High (N variants per image)	Low (only requested variants)
Compute cost	Predictable, front-loaded	Unpredictable, scales with unique requests
First-request latency	Fast (variant already exists)	Slow for first hit
Variant flexibility	Low (changing specs = reprocessing)	High (add params without reprocessing)
Cache warming	Automatic	Requires traffic
Useful for	Known, fixed variant sets	Dynamic or user-defined transforms

In practice, most production pipelines use both. Pre-generate a small set of canonical variants (thumb, card, hero, og) that are used everywhere and latency-sensitive. Let everything else resolve lazily through the on-the-fly transformation layer.

The risk with lazy generation is the cold start problem for new images. When a high-traffic image is uploaded and first shared, the burst of first requests all miss the edge cache and hit the transformation layer simultaneously. You need either request coalescing at the transformation layer (queue duplicate in-flight requests rather than spawning them) or a warm-up step that pre-requests canonical variants after upload completes.

Production Considerations

Observability. Track hit ratio per route, not aggregate. A 98% aggregate hit ratio can hide a frequently accessed but uncacheable route that is hammering your origin. Tag transformation metrics by image variant and source format so you can see which variants are expensive.

Idempotency in thumbnail workers. Thumbnail generation jobs are retried on failure. If your worker writes derivative keys deterministically (same input, same output key), you can safely retry without producing duplicate storage objects. The write is idempotent; the key will be overwritten with the same data.

Original immutability. Never modify original objects after upload. Treat originals as append-only. If a user replaces an image, create a new image record with a new ID and generate new derivatives. This keeps your derivative cache keys stable and avoids the stampede that comes from cache busting.

Deletion propagation. When a user deletes an image, your pipeline needs to delete: the original, all pre-generated derivatives, and invalidate any CDN-cached on-the-fly variants. Build a deletion job that walks all derivative paths for an image ID and deletes them, then issues a CDN purge for the image path prefix. Tag-based CDN purge (/images/{imageId}/*) is the right tool here.

Cost monitoring. Set up per-bucket cost alerts separately for originals and derivatives. Derivatives will typically be 5-10x the storage volume of originals. Watch for unbounded growth if users can request arbitrary transformation parameters: a bot requesting 10,000 unique size combinations can generate 10,000 cached derivatives for a single image.

Closing Insight

The image pipeline is one of the few places where storage and delivery costs dominate compute costs. Designing for the right abstraction boundaries (direct uploads, async processing, lazy-generated derivatives with CDN caching) keeps compute costs low, but the real cost lever is egress: where you store derivatives and how aggressively you cache them. Get the cache key normalization right, colocate derivative storage with your CDN, and let the edge serve the vast majority of requests. At that point the pipeline is mostly invisible and image delivery is just another CDN cache hit ratio problem.