Designing a File Upload Pipeline: Presigned URLs, Chunked Uploads, and Processing at Scale

The file upload implementation in most production systems starts as a 20-line Express handler: accept a multipart request, pipe the body to S3, return a URL. That works until you need to handle 500MB video files, or until your API server starts running out of memory because three concurrent uploads are buffering through it simultaneously, or until a user on a flaky mobile connection loses their 200MB upload at 98% and has to start over.

The real architecture is not complicated, but it has more moving parts than the naive version. This article walks through every layer: direct-to-storage uploads with presigned URLs, chunked and resumable transfers, validation and virus scanning, async processing queues, CDN delivery, and the failure modes you will hit in production.

Why You Should Not Buffer Uploads Through Your API Server

The instinctive approach routes upload traffic through your application:

// What most teams start with
app.post('/upload', upload.single('file'), async (req, res) => {
  const file = req.file; // multer has buffered this into memory
  await s3.putObject({
    Bucket: process.env.S3_BUCKET!,
    Key: `uploads/${file.originalname}`,
    Body: file.buffer,
  });
  res.json({ url: `https://cdn.example.com/uploads/${file.originalname}` });
});

This has three problems. First, every byte of every upload passes through your application process, consuming CPU for I/O work that your storage layer is better equipped to handle. Second, you cannot horizontally scale cleanly: uploads to one instance cannot be accessed by another for any post-processing. Third, streaming large files correctly requires careful Node.js stream handling that most teams skip, so you end up with files buffered entirely in memory.

The better path: generate a signed URL, send the client directly to storage, and have your application only handle the coordination.

Presigned URLs for Direct-to-Storage Uploads

A presigned URL is a temporary credential baked into a URL. The client uploads directly to S3 (or R2, or GCS) with no traffic touching your API server.

import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { randomUUID } from 'crypto';

const s3 = new S3Client({ region: process.env.AWS_REGION });

interface PresignRequest {
  filename: string;
  contentType: string;
  contentLength: number;
  userId: string;
}

interface PresignResponse {
  uploadId: string;
  uploadUrl: string;
  key: string;
  expiresAt: Date;
}

async function createPresignedUpload(req: PresignRequest): Promise<PresignResponse> {
  // Validate before generating the URL
  const allowedTypes = ['image/jpeg', 'image/png', 'image/webp', 'video/mp4', 'application/pdf'];
  if (!allowedTypes.includes(req.contentType)) {
    throw new Error(`Content type not allowed: ${req.contentType}`);
  }

  const maxBytes = 100 * 1024 * 1024; // 100MB for single-part
  if (req.contentLength > maxBytes) {
    throw new Error('File too large for single-part upload, use multipart');
  }

  const uploadId = randomUUID();
  const ext = req.filename.split('.').pop() ?? 'bin';
  // Use a structured key: user-scoped, not guessable from filename alone
  const key = `uploads/${req.userId}/${uploadId}.${ext}`;

  const command = new PutObjectCommand({
    Bucket: process.env.S3_BUCKET!,
    Key: key,
    ContentType: req.contentType,
    ContentLength: req.contentLength,
    Metadata: {
      'upload-id': uploadId,
      'user-id': req.userId,
      'original-filename': encodeURIComponent(req.filename),
    },
  });

  const uploadUrl = await getSignedUrl(s3, command, { expiresIn: 900 }); // 15 minutes

  return {
    uploadId,
    uploadUrl,
    key,
    expiresAt: new Date(Date.now() + 900_000),
  };
}

The client receives this URL and PUTs directly to S3:

// Client-side upload
async function uploadFile(file: File, presign: PresignResponse): Promise<void> {
  const response = await fetch(presign.uploadUrl, {
    method: 'PUT',
    headers: {
      'Content-Type': file.type,
      'Content-Length': String(file.size),
    },
    body: file,
  });

  if (!response.ok) {
    throw new Error(`Upload failed: ${response.status} ${response.statusText}`);
  }
}

After the PUT completes, the client notifies your API. This is where you record the upload in your database and kick off async processing.

// API handler: confirm upload completed
app.post('/uploads/:uploadId/confirm', async (req, res) => {
  const { uploadId } = req.params;

  // Verify the object actually landed in S3
  const headResult = await s3.headObject({
    Bucket: process.env.S3_BUCKET!,
    Key: req.body.key,
  });

  await db.insert(uploads).values({
    id: uploadId,
    userId: req.user.id,
    key: req.body.key,
    contentType: headResult.ContentType!,
    sizeBytes: headResult.ContentLength!,
    status: 'pending',
    createdAt: new Date(),
  });

  // Enqueue for processing
  await queue.send('file-processing', { uploadId, key: req.body.key });

  res.json({ uploadId, status: 'pending' });
});

Chunked and Resumable Uploads for Large Files

For files over 100MB, presigned single-part uploads break down. Network interruptions on large files are common, and retrying the entire transfer is a poor experience. S3 Multipart Upload solves this: you split the file into parts, upload each independently, and finalize with a completion call. If a part fails, you retry only that part.

import {
  CreateMultipartUploadCommand,
  UploadPartCommand,
  CompleteMultipartUploadCommand,
  AbortMultipartUploadCommand,
} from '@aws-sdk/client-s3';

const PART_SIZE = 10 * 1024 * 1024; // 10MB minimum enforced by S3

interface MultipartSession {
  uploadId: string;
  s3UploadId: string;
  key: string;
  totalParts: number;
}

async function initiateMultipartUpload(
  key: string,
  contentType: string,
  fileSizeBytes: number,
): Promise<MultipartSession> {
  const { UploadId } = await s3.send(
    new CreateMultipartUploadCommand({
      Bucket: process.env.S3_BUCKET!,
      Key: key,
      ContentType: contentType,
    }),
  );

  const totalParts = Math.ceil(fileSizeBytes / PART_SIZE);
  const uploadId = randomUUID();

  // Persist session so it survives page reloads
  await db.insert(multipartSessions).values({
    id: uploadId,
    s3UploadId: UploadId!,
    key,
    totalParts,
    completedParts: [],
    createdAt: new Date(),
  });

  return { uploadId, s3UploadId: UploadId!, key, totalParts };
}

The client uploads each part using a presigned URL generated per-part:

import { UploadPartCommand } from '@aws-sdk/client-s3';

async function getPartUploadUrl(
  key: string,
  s3UploadId: string,
  partNumber: number,
): Promise<string> {
  const command = new UploadPartCommand({
    Bucket: process.env.S3_BUCKET!,
    Key: key,
    UploadId: s3UploadId,
    PartNumber: partNumber,
  });

  return getSignedUrl(s3, command, { expiresIn: 3600 });
}

// Client-side: upload a single part with retry
async function uploadPart(
  url: string,
  chunk: Blob,
  onProgress: (bytes: number) => void,
): Promise<string> {
  const response = await fetch(url, {
    method: 'PUT',
    body: chunk,
  });

  if (!response.ok) throw new Error(`Part upload failed: ${response.status}`);

  // ETag is required for the completion call
  const etag = response.headers.get('ETag');
  if (!etag) throw new Error('Missing ETag in part response');

  onProgress(chunk.size);
  return etag;
}

Once all parts are uploaded, finalize:

import { CompleteMultipartUploadCommand } from '@aws-sdk/client-s3';

async function completeMultipartUpload(
  key: string,
  s3UploadId: string,
  parts: Array<{ PartNumber: number; ETag: string }>,
): Promise<void> {
  await s3.send(
    new CompleteMultipartUploadCommand({
      Bucket: process.env.S3_BUCKET!,
      Key: key,
      UploadId: s3UploadId,
      MultipartUpload: { Parts: parts },
    }),
  );
}

Resumability comes from persisting the completed parts list. When a user returns after an interruption, fetch the session from your database, generate presigned URLs only for the missing parts, and continue. S3 holds incomplete multipart uploads for up to 7 days by default. Add a lifecycle rule to abort them sooner or you will accumulate storage costs from abandoned uploads.

Validation and Virus Scanning

You cannot trust content-type headers. Clients send whatever they want. Validate based on actual file content using magic bytes:

async function detectMimeType(key: string): Promise<string> {
  // Read only the first 8KB: enough for magic byte detection
  const { Body } = await s3.getObject({
    Bucket: process.env.S3_BUCKET!,
    Key: key,
    Range: 'bytes=0-8191',
  });

  const buffer = Buffer.from(await Body!.transformToByteArray());

  // JPEG: FF D8 FF
  if (buffer[0] === 0xff && buffer[1] === 0xd8 && buffer[2] === 0xff) return 'image/jpeg';
  // PNG: 89 50 4E 47 0D 0A 1A 0A
  if (
    buffer[0] === 0x89 && buffer[1] === 0x50 &&
    buffer[2] === 0x4e && buffer[3] === 0x47
  ) return 'image/png';
  // PDF: 25 50 44 46
  if (buffer.slice(0, 4).toString() === '%PDF') return 'application/pdf';
  // MP4: check ftyp box
  if (buffer.slice(4, 8).toString() === 'ftyp') return 'video/mp4';

  return 'application/octet-stream';
}

For virus scanning, integrate ClamAV (self-hosted) or a cloud scanning API. The pattern is the same: download the file from your private bucket, scan, mark the record, and either quarantine or delete on detection.

interface ScanResult {
  clean: boolean;
  threat?: string;
}

async function scanFile(key: string): Promise<ScanResult> {
  const { Body } = await s3.getObject({
    Bucket: process.env.S3_BUCKET!,
    Key: key,
  });

  const buffer = Buffer.from(await Body!.transformToByteArray());

  // Example using a hypothetical scanning client
  // In practice: ClamAV via clamdjs, or an HTTP API like VirusTotal
  const result = await virusScanner.scan(buffer);

  if (!result.clean) {
    // Move to quarantine bucket, do not delete (for forensics)
    await s3.copyObject({
      CopySource: `${process.env.S3_BUCKET}/${key}`,
      Bucket: process.env.S3_QUARANTINE_BUCKET!,
      Key: key,
    });
    await s3.deleteObject({ Bucket: process.env.S3_BUCKET!, Key: key });
  }

  return result;
}

Keep your upload bucket private. Only the finalized, scanned content should ever be publicly accessible, and only through a CDN with a separate origin bucket.

Async Processing Pipeline

Uploads land in a private bucket. Processing (image resizing, video transcoding, metadata extraction) happens asynchronously. The architecture: a job queue, one or more worker types, and status polling.

interface ProcessingJob {
  uploadId: string;
  key: string;
  contentType: string;
  operations: ProcessingOperation[];
}

type ProcessingOperation =
  | { type: 'scan' }
  | { type: 'resize'; widths: number[] }
  | { type: 'transcode'; format: 'mp4' | 'webm'; maxBitrate: number }
  | { type: 'extract-metadata' };

// Worker: processes one job at a time
async function processUpload(job: ProcessingJob): Promise<void> {
  await db.update(uploads)
    .set({ status: 'processing' })
    .where(eq(uploads.id, job.uploadId));

  try {
    // Step 1: always scan first
    const scan = await scanFile(job.key);
    if (!scan.clean) {
      await db.update(uploads)
        .set({ status: 'rejected', rejectionReason: `Threat detected: ${scan.threat}` })
        .where(eq(uploads.id, job.uploadId));
      return;
    }

    // Step 2: validate actual mime type
    const detectedType = await detectMimeType(job.key);
    if (detectedType !== job.contentType) {
      await db.update(uploads)
        .set({ status: 'rejected', rejectionReason: 'Content type mismatch' })
        .where(eq(uploads.id, job.uploadId));
      return;
    }

    // Step 3: content-type-specific processing
    const outputs: Record<string, string> = {};

    for (const op of job.operations) {
      if (op.type === 'resize' && detectedType.startsWith('image/')) {
        const resized = await resizeImage(job.key, op.widths);
        outputs['thumbnails'] = JSON.stringify(resized);
      }
      if (op.type === 'extract-metadata') {
        const meta = await extractMetadata(job.key, detectedType);
        outputs['metadata'] = JSON.stringify(meta);
      }
    }

    // Move processed file to public bucket
    const publicKey = job.key.replace('uploads/', 'public/');
    await s3.copyObject({
      CopySource: `${process.env.S3_BUCKET}/${job.key}`,
      Bucket: process.env.S3_PUBLIC_BUCKET!,
      Key: publicKey,
    });

    await db.update(uploads).set({
      status: 'ready',
      publicKey,
      outputs,
      processedAt: new Date(),
    }).where(eq(uploads.id, job.uploadId));

  } catch (err) {
    await db.update(uploads)
      .set({ status: 'failed', lastError: String(err) })
      .where(eq(uploads.id, job.uploadId));
    throw err; // Let the queue handle retry
  }
}

Status polling from the client:

// Client polls until status is terminal
async function pollUploadStatus(uploadId: string): Promise<Upload> {
  const terminalStatuses = new Set(['ready', 'failed', 'rejected']);
  let backoff = 1000;

  while (true) {
    const upload = await api.get<Upload>(`/uploads/${uploadId}`);
    if (terminalStatuses.has(upload.status)) return upload;

    await new Promise(resolve => setTimeout(resolve, backoff));
    backoff = Math.min(backoff * 1.5, 10_000);
  }
}

For long-running jobs like video transcoding, consider WebSocket or SSE push instead of polling. Polling works fine for processing times under 30 seconds.

CDN Delivery

Serve public files through a CDN, not directly from S3. Direct S3 URLs expose your bucket name, bypass caching, and incur per-request costs. A CDN origin pointed at your public bucket gives you edge caching, signed URL support for gated content, and a stable URL structure that survives bucket migrations.

For gated content (files that require authentication to view), generate time-limited signed CDN URLs server-side:

import { createHmac } from 'crypto';

function signCdnUrl(key: string, expiresInSeconds: number): string {
  const expiresAt = Math.floor(Date.now() / 1000) + expiresInSeconds;
  const path = `/files/${key}`;
  const signature = createHmac('sha256', process.env.CDN_SECRET!)
    .update(`${path}:${expiresAt}`)
    .digest('hex');

  return `https://cdn.example.com${path}?expires=${expiresAt}&sig=${signature}`;
}

Tradeoffs

Approach	Throughput	Resumability	Complexity	Best For
Buffer through API	Low	No	Low	Prototypes, < 5MB files
Presigned PUT	High	No	Medium	Files < 100MB, simple flow
S3 Multipart	High	Yes	High	Files > 100MB, mobile clients
Tus protocol (self-hosted)	High	Yes	Very High	Full resumability standard

Multipart upload is the right default for anything above 100MB. Below that, a presigned PUT is simpler and covers most use cases.

Production Failure Modes

Incomplete multipart uploads accumulating cost. S3 charges for the storage consumed by in-progress multipart uploads. An S3 lifecycle rule like AbortIncompleteMultipartUploads after 1 day prevents this from becoming a surprise line item.

Client reporting upload success when S3 never received it. Always verify with HeadObject in your confirmation handler. Do not trust the client’s claim that the upload succeeded.

Processing jobs stuck in “processing” forever. Workers crash. Set a max processing time and a watchdog query that requeues any upload stuck in processing for longer than N minutes.

Presigned URLs used for unauthorized file types. The presigned URL grants a PUT with the exact ContentType and ContentLength you specified. A client can still upload different bytes. This is why magic-byte validation in the processing step is non-negotiable.

Race condition between confirm and processing. If your confirmation handler enqueues the job and the worker fires before the database transaction commits, the worker reads no record. Use transactional outbox: insert the job record in the same transaction as the upload record, and have a separate poller dispatch it.

CDN serving stale content after file replacement. If you allow re-uploads to the same key, invalidate the CDN cache explicitly. Better: use content-addressed keys (hash of the file contents), which makes cache invalidation irrelevant.

What This Looks Like in Practice

The end-to-end flow for a 50MB image upload:

1. Client requests a presigned URL from your API, sending filename, content type, and file size.
2. API validates the request, generates a key, persists an upload record with status initiated, and returns the presigned URL.
3. Client PUTs the file directly to S3.
4. Client calls your confirmation endpoint with the upload ID and key.
5. API runs HeadObject, confirms the file is there and matches expected size, updates status to pending, and enqueues a processing job.
6. Worker picks up the job, scans, validates mime type, resizes, copies to the public bucket, updates status to ready.
7. Client polls status until ready, then gets the CDN URL for display.

The API server handles only small JSON requests. Storage handles the bytes. Workers handle the computation. Each layer scales independently.

The naive POST-to-API handler is not wrong for a prototype. It is wrong for anything that needs to survive real upload volumes, large files, or users on mobile connections. The architecture above is not significantly harder to build the first time if you approach it with the right primitives. The cost of retrofitting it later, when you are debugging memory exhaustion on your API servers at 2 AM, is much higher.