Designing a Video Processing Pipeline: Upload, Transcoding, Adaptive Bitrate Streaming, and CDN Delivery at Scale

A video processing pipeline is one of the more humbling infrastructure problems in software engineering. The raw surface area is large: you need to accept files that are gigabytes in size from unreliable connections, transcode them into a matrix of quality levels and container formats, generate time-aligned segments and streaming manifests, warm a globally distributed CDN before the first viewer hits play, and do all of it at a cost that does not destroy your margin. Then add DRM, thumbnails, content fingerprinting, and per-tenant isolation, and you have a system that touches nearly every discipline in backend engineering.

This article walks through each layer in detail, with TypeScript for the upload and orchestration layers, a tradeoffs table for transcoding strategies, and production considerations that only surface once you are past the happy path.

The Upload Layer: Resumable Chunked Ingestion

The simplest possible upload interface (a single POST with a multipart body) fails in two ways for video. First, raw video files are large: a 30-minute uncompressed 1080p recording is easily 20-50 GB. Your API server is not a proxy for multi-gigabyte binary streams. Second, the connection from a client (particularly a mobile client, or a professional creator on a conference hotel Wi-Fi) is unreliable. If you lose 98% of an upload and require the user to restart, you have a product problem.

The production approach is a resumable chunked upload protocol. The client splits the file into fixed-size chunks, gets presigned storage URLs for each, uploads chunks in parallel, and reports completion to your API. On any failure, the client can query which chunks arrived and resume from there.

interface UploadSession {
  sessionId: string;
  tenantId: string;
  assetId: string;
  filename: string;
  contentType: string;
  totalBytes: number;
  chunkSize: number;   // typically 32MB-128MB per chunk
  totalChunks: number;
  status: "pending" | "uploading" | "assembling" | "queued" | "failed";
  receivedChunks: number[];
  storagePrefix: string;
  expiresAt: string;
  createdAt: string;
}

async function createUploadSession(params: {
  tenantId: string;
  filename: string;
  contentType: string;
  totalBytes: number;
}): Promise<{ session: UploadSession; presignedUrls: string[] }> {
  const CHUNK_SIZE = 64 * 1024 * 1024; // 64MB chunks
  const assetId = generateAssetId();
  const totalChunks = Math.ceil(params.totalBytes / CHUNK_SIZE);
  const storagePrefix = `raw/${params.tenantId}/${assetId}`;

  const session: UploadSession = {
    sessionId: crypto.randomUUID(),
    tenantId: params.tenantId,
    assetId,
    filename: params.filename,
    contentType: params.contentType,
    totalBytes: params.totalBytes,
    chunkSize: CHUNK_SIZE,
    totalChunks,
    status: "pending",
    receivedChunks: [],
    storagePrefix,
    expiresAt: new Date(Date.now() + 24 * 60 * 60 * 1000).toISOString(),
    createdAt: new Date().toISOString(),
  };

  await db.uploadSessions.insert(session);

  const presignedUrls = await Promise.all(
    Array.from({ length: totalChunks }, (_, i) =>
      storage.presignPut({
        key: `${storagePrefix}/chunk-${i.toString().padStart(6, "0")}`,
        expiresIn: 3600,
        contentLengthMax: CHUNK_SIZE * 1.01, // 1% slack for last chunk
      })
    )
  );

  return { session, presignedUrls };
}

The client uploads chunks directly to object storage, then calls a confirmation endpoint for each chunk. That endpoint updates the receivedChunks array atomically. When all chunks are confirmed, the finalization step triggers:

async function finalizeUpload(sessionId: string): Promise<void> {
  const session = await db.uploadSessions.findOne({ sessionId });

  if (session.receivedChunks.length !== session.totalChunks) {
    const missing = Array.from(
      { length: session.totalChunks },
      (_, i) => i
    ).filter((i) => !session.receivedChunks.includes(i));
    throw new UploadIncompleteError(missing);
  }

  // Compose chunks into a single object. S3 supports up to 10,000 part
  // multipart compose. For very large files assembled from many chunks,
  // you may need an intermediate concatenation step.
  await storage.compose({
    destination: `${session.storagePrefix}/source`,
    parts: Array.from({ length: session.totalChunks }, (_, i) =>
      `${session.storagePrefix}/chunk-${i.toString().padStart(6, "0")}`
    ),
  });

  await db.uploadSessions.update(sessionId, { status: "queued" });

  await transcodingQueue.enqueue({
    assetId: session.assetId,
    tenantId: session.tenantId,
    sourceKey: `${session.storagePrefix}/source`,
    filename: session.filename,
    priority: await resolvePriority(session.tenantId),
  });
}

One important note: do not clean up chunk objects until after the composition step succeeds. If composition fails midway, you need the chunks intact for retry. Schedule chunk cleanup as a separate async job that runs after the composed object is verified.

Job Orchestration: Transcoding Across Quality Presets

Once the source video is in object storage, you need to fan out transcoding jobs across multiple quality presets. A standard ladder for an adaptive bitrate delivery system looks like this:

Resolution	Bitrate (video)	Audio bitrate	Target device
240p	300 kbps	64 kbps	Mobile, low bandwidth
360p	700 kbps	96 kbps	Mobile, standard
480p	1,200 kbps	128 kbps	Tablet, desktop
720p	2,500 kbps	128 kbps	HD desktop, TV
1080p	5,000 kbps	192 kbps	Full HD
1080p HQ	8,000 kbps	192 kbps	High-fidelity masters
4K	18,000 kbps	256 kbps	4K displays

Each row is an independent transcoding job. They can run in parallel, which is where the orchestration layer matters.

interface TranscodingJob {
  jobId: string;
  assetId: string;
  tenantId: string;
  sourceKey: string;
  preset: QualityPreset;
  status: "queued" | "running" | "succeeded" | "failed" | "retrying";
  workerId: string | null;
  attempt: number;
  maxAttempts: number;
  startedAt: string | null;
  completedAt: string | null;
  outputKey: string | null;
  errorMessage: string | null;
}

type QualityPreset =
  | "240p"
  | "360p"
  | "480p"
  | "720p"
  | "1080p"
  | "1080p-hq"
  | "4k";

const PRESET_CONFIG: Record<QualityPreset, {
  width: number;
  height: number;
  videoBitrate: number;
  audioBitrate: number;
  codec: "h264" | "h265" | "vp9" | "av1";
}> = {
  "240p":    { width: 426,  height: 240,  videoBitrate: 300000,   audioBitrate: 64000,  codec: "h264" },
  "360p":    { width: 640,  height: 360,  videoBitrate: 700000,   audioBitrate: 96000,  codec: "h264" },
  "480p":    { width: 854,  height: 480,  videoBitrate: 1200000,  audioBitrate: 128000, codec: "h264" },
  "720p":    { width: 1280, height: 720,  videoBitrate: 2500000,  audioBitrate: 128000, codec: "h264" },
  "1080p":   { width: 1920, height: 1080, videoBitrate: 5000000,  audioBitrate: 192000, codec: "h264" },
  "1080p-hq":{ width: 1920, height: 1080, videoBitrate: 8000000,  audioBitrate: 192000, codec: "h265" },
  "4k":      { width: 3840, height: 2160, videoBitrate: 18000000, audioBitrate: 256000, codec: "h265" },
};

async function dispatchTranscodingJobs(params: {
  assetId: string;
  tenantId: string;
  sourceKey: string;
  presets: QualityPreset[];
}): Promise<TranscodingJob[]> {
  const jobs: TranscodingJob[] = params.presets.map((preset) => ({
    jobId: crypto.randomUUID(),
    assetId: params.assetId,
    tenantId: params.tenantId,
    sourceKey: params.sourceKey,
    preset,
    status: "queued",
    workerId: null,
    attempt: 0,
    maxAttempts: 3,
    startedAt: null,
    completedAt: null,
    outputKey: null,
    errorMessage: null,
  }));

  await db.transcodingJobs.insertMany(jobs);

  await Promise.all(
    jobs.map((job) =>
      transcodingQueue.enqueue(job, {
        priority: presetPriority(job.preset),
      })
    )
  );

  return jobs;
}

function presetPriority(preset: QualityPreset): number {
  // Lower presets get higher priority so that low-bandwidth viewers
  // can start watching while higher quality renditions are still processing.
  const order: QualityPreset[] = ["360p", "240p", "480p", "720p", "1080p", "1080p-hq", "4k"];
  return order.indexOf(preset);
}

The priority ordering here is non-obvious but important: 360p should finish before 4K so that viewers on slow connections have something to play while the high-quality renditions are still encoding. The player will upgrade automatically as higher renditions become available and the manifest is updated.

Workers pick up jobs using a SELECT FOR UPDATE SKIP LOCKED pattern if you are running this on Postgres, or a visibility-timeout model if you are using SQS or a similar managed queue. The critical production requirement: workers must be idempotent. If a worker crashes mid-encode and the job is retried on a different worker, it should restart cleanly without producing corrupt output.

HLS and DASH: Manifest Generation and Segment Packaging

After transcoding, each rendition exists as a continuous video file. For adaptive bitrate streaming, you need to segment these into short chunks (typically 2-6 seconds each) and generate manifests that tell the player where to find each segment.

HLS uses an M3U8 format. A master playlist points to per-rendition playlists:

#EXTM3U
#EXT-X-VERSION:6

#EXT-X-STREAM-INF:BANDWIDTH=700000,RESOLUTION=640x360,CODECS="avc1.64001F,mp4a.40.2"
360p/playlist.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=2500000,RESOLUTION=1280x720,CODECS="avc1.640028,mp4a.40.2"
720p/playlist.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080,CODECS="avc1.640028,mp4a.40.2"
1080p/playlist.m3u8

Each per-rendition playlist lists its segments with their durations. A key property: all renditions must be segmented at the same timestamps. If the 360p rendition has a cut at 4.002 seconds and the 1080p rendition has a cut at 4.008 seconds, the player’s quality switch mid-stream will produce a brief glitch. Most production pipelines use a two-pass approach: first pass establishes keyframe positions, second pass transcodes all renditions forced to those exact keyframe positions using FFmpeg’s -force_key_frames option.

DASH (Dynamic Adaptive Streaming over HTTP) uses an MPD (Media Presentation Description) XML manifest and is structurally equivalent but more flexible in terms of codec support, particularly for H.265 and AV1 at the high end.

The manifest generation step runs after all segment files are written to storage:

async function generateHLSManifest(params: {
  assetId: string;
  tenantId: string;
  completedRenditions: Array<{
    preset: QualityPreset;
    segments: Array<{ filename: string; duration: number }>;
    outputPrefix: string;
  }>;
}): Promise<string> {
  const lines: string[] = ["#EXTM3U", "#EXT-X-VERSION:6", ""];

  for (const rendition of params.completedRenditions) {
    const config = PRESET_CONFIG[rendition.preset];
    const bandwidth = config.videoBitrate + config.audioBitrate;
    const codecs = config.codec === "h264"
      ? "avc1.640028,mp4a.40.2"
      : "hvc1.1.6.L120.90,mp4a.40.2";

    lines.push(
      `#EXT-X-STREAM-INF:BANDWIDTH=${bandwidth},` +
      `RESOLUTION=${config.width}x${config.height},` +
      `CODECS="${codecs}"`
    );
    lines.push(`${rendition.preset}/playlist.m3u8`);
  }

  const masterPlaylist = lines.join("\n");
  const masterKey = `processed/${params.tenantId}/${params.assetId}/master.m3u8`;

  await storage.put({
    key: masterKey,
    body: masterPlaylist,
    contentType: "application/vnd.apple.mpegurl",
    cacheControl: "public, max-age=300", // 5-minute TTL, refreshed as renditions complete
  });

  return masterKey;
}

One production detail: set a short cache TTL on the master manifest during the period when renditions are still completing. A player that fetches the manifest while 4K is still transcoding should not cache an incomplete rendition list for an hour. Once all renditions are done and the manifest is final, update the cache control to a longer TTL.

CDN Delivery and Cache Warming

The default CDN behavior is pull-through: a viewer requests a segment, the CDN edge node checks its cache, misses on first request, fetches from origin, and caches it. For long-tail content that few viewers watch simultaneously, this is fine. For a popular video that 10,000 viewers will request in the first minute after publish, a cold cache means 10,000 origin fetches all happening at once.

Cache warming pushes content to CDN edge nodes proactively before viewers arrive. The mechanics vary by CDN provider, but the concept is the same: enumerate the segments your content delivery tier will need and issue prefetch requests to the CDN.

interface CacheWarmingJob {
  assetId: string;
  tenantId: string;
  priority: "urgent" | "standard" | "background";
  segments: string[];
  edgeRegions: string[];
  status: "pending" | "running" | "completed" | "failed";
}

async function scheduleWarmCache(params: {
  assetId: string;
  tenantId: string;
  masterManifestUrl: string;
  expectedViewers: number;
}): Promise<void> {
  // Collect all segment URLs from all rendition manifests.
  // In practice, parse the manifests to enumerate segments.
  const segments = await collectAllSegmentUrls(params.masterManifestUrl);

  // Warm only if high expected concurrency justifies the cost.
  if (params.expectedViewers < 500) return;

  const job: CacheWarmingJob = {
    assetId: params.assetId,
    tenantId: params.tenantId,
    priority: params.expectedViewers > 5000 ? "urgent" : "standard",
    segments,
    edgeRegions: await selectWarmingRegions(params.tenantId),
    status: "pending",
  };

  await cacheWarmingQueue.enqueue(job);
}

async function executeCacheWarming(job: CacheWarmingJob): Promise<void> {
  // Send warming requests in parallel, but cap concurrency to avoid
  // hammering origin with a thundering herd from the warming job itself.
  const CONCURRENCY = 20;

  for (let i = 0; i < job.segments.length; i += CONCURRENCY) {
    const batch = job.segments.slice(i, i + CONCURRENCY);
    await Promise.all(
      batch.map((url) =>
        cdn.prefetch({ url, regions: job.edgeRegions })
      )
    );
  }
}

A realistic warming strategy warms the first 30 seconds of each rendition (early segments are most latency-sensitive) plus the master manifest and all per-rendition playlists. Beyond that, let the CDN’s natural cache fill in as viewers watch.

Transcoding Strategy Tradeoffs

The biggest lever in pipeline design is where and how you run transcoding. The options span a spectrum from on-demand compute to always-on worker pools.

Strategy	Latency to first playback	Cost profile	Complexity	Best fit
On-demand serverless (Lambda/Cloud Run)	High: cold start adds 30-60s per job	Pay per second of CPU	Low	Low-volume, cost-sensitive
Spot instance pool with autoscaling	Medium: 2-5 min to scale up	Low cost, but interruption risk	Medium	High-volume batch workloads
Reserved instance pool (always-on workers)	Low: jobs start in seconds	High fixed cost	Low	Latency-critical, predictable volume
Managed transcoding service (MediaConvert, Zencoder)	Low to medium: varies by provider	Premium per-minute pricing	Very low	Teams without dedicated infra eng
Hybrid: reserved for 360p/720p, spot for 1080p/4K	Low for base quality, medium for high quality	Balanced	Medium-high	Production platforms with volume

The hybrid approach is what most mature platforms land on. Viewers can start watching in 360p or 720p within minutes, while higher-quality renditions finish on cheaper spot capacity. The tradeoff is orchestration complexity: you need the manifest update logic to add renditions incrementally as they complete.

Production Considerations

Thumbnail Generation

Thumbnails are not a postprocessing afterthought. They are the first signal a viewer uses to decide whether to watch. Run thumbnail extraction in parallel with transcoding, not after it, to avoid adding latency to your asset availability window.

A practical approach: extract a frame every N seconds during transcoding (a free operation when you are already decoding the source), filter candidates by perceptual quality metrics (avoid black frames, heavily blurred frames, frames dominated by a single solid color), and store the top 5-10 candidates. Let the content owner pick one, or run a heuristic selection if you need automation.

Content Fingerprinting

If your platform accepts user-uploaded content, you will encounter copyright violations. The standard tooling for audio fingerprinting is AcoustID and Chromaprint. For video, YouTube’s Content ID uses a frame-hash matching approach, but licensable alternatives include Audible Magic and Vobile.

The integration point is straightforward: extract a fingerprint during processing and match it against a reference database before making the asset available. Run this as a parallel job, not a blocking step, unless your policy requires pre-clearance.

async function runContentFingerprinting(params: {
  assetId: string;
  sourceKey: string;
}): Promise<FingerprintResult> {
  // Extract audio track for acoustic fingerprinting
  const audioTrack = await extractAudioTrack(params.sourceKey);
  const fingerprint = await acoustid.fingerprint(audioTrack);
  const matches = await fingerprintDatabase.query(fingerprint);

  return {
    assetId: params.assetId,
    matched: matches.length > 0,
    matches: matches.map((m) => ({
      referenceId: m.referenceId,
      owner: m.owner,
      confidence: m.confidence,
      policy: m.policy, // "block" | "monetize" | "track"
    })),
    fingerprintedAt: new Date().toISOString(),
  };
}

DRM Integration

DRM (Digital Rights Management) is required for premium content, particularly in the broadcast and OTT space. The two dominant DRM systems are Widevine (Google, Chrome/Android) and FairPlay (Apple, Safari/iOS). Most platforms implement both, plus PlayReady for Microsoft environments, through a common encryption standard (CENC).

In the processing pipeline, DRM encryption happens at packaging time, after transcoding. The packager encrypts each segment with a content encryption key (CEK), and a key server (KMS) stores the mapping from content ID to CEK. When a player requests a license to decrypt a stream, it authenticates with the license server and receives the CEK in the DRM-specific license format.

The key operational concern: key rotation. CEKs should be rotated periodically for live content, and you need a revocation mechanism for subscribers whose access you need to terminate.

Multi-Tenant Isolation

In a multi-tenant video platform (SaaS video hosting, white-label streaming), you need strong isolation between tenant assets, compute, and observability.

At the storage layer, use per-tenant prefixes with bucket policies or IAM conditions that enforce tenant boundaries. A misconfigured cross-tenant storage key is a serious data breach.

At the compute layer, you have two options: shared worker pools with tenant tagging, or per-tenant worker pools. Shared pools are more cost-efficient but create noisy-neighbor problems: a tenant uploading 1,000 videos simultaneously will starve other tenants’ jobs if you do not enforce per-tenant queue limits. Per-tenant pools eliminate noisy-neighbor effects but are expensive for tenants with low volume.

A practical hybrid: shared worker pool with per-tenant rate limiting and priority bands. High-tier tenants get a dedicated priority lane. Standard tenants share a pool with per-tenant concurrency caps.

async function resolveJobPriority(params: {
  tenantId: string;
  preset: QualityPreset;
}): Promise<number> {
  const tenant = await db.tenants.findOne(params.tenantId);

  const tierBase: Record<string, number> = {
    enterprise: 0,
    growth: 100,
    starter: 200,
  };

  const presetOffset = presetPriority(params.preset);
  const currentJobCount = await countPendingJobsForTenant(params.tenantId);
  const fairnessOffset = Math.min(currentJobCount * 2, 50); // soft rate limiting

  return (tierBase[tenant.tier] ?? 200) + presetOffset + fairnessOffset;
}

At the observability layer, tag every metric, log, and trace with tenant ID. Without this, debugging a transcoding failure report from a specific customer is painful.

Latency-Optimized vs Cost-Optimized Pipelines

The core architectural decision in a video processing pipeline is where you sit on the latency-cost spectrum.

A latency-optimized pipeline prioritizes time-to-first-play. It uses reserved compute, parallel job dispatch across all presets simultaneously, pre-warmed CDN caching on publish, and real-time manifest updates as each rendition completes. It is expensive: reserved compute is idle during off-peak hours, and cache warming costs money for content that few people end up watching.

A cost-optimized pipeline prioritizes cost per processed video minute. It uses spot or serverless compute, sequential preset processing (low resolution first to unblock viewers, high resolution later when spot is available), CDN pull-through caching without pre-warming, and batch manifest updates after all renditions complete. The tradeoff is that initial playback quality may be limited and the delay from upload to playback availability is measured in minutes rather than seconds.

For most platforms, the answer is contextual. Content scheduled for a live premiere or a high-traffic release window should go through the latency path. Bulk content migration or archival processing belongs on the cost path. Build both paths and route based on metadata, not hardcoded configuration.

Where This Gets Complicated

The parts of a video processing pipeline that cause the most production pain are rarely the ones you plan for. Corrupted source files that pass validation but cause FFmpeg to hang mid-encode, requiring a watchdog that kills and retries workers. Source videos with variable frame rates that cause segment alignment drift across renditions. Mobile uploads where the OS terminated the upload app mid-chunk and the client has no mechanism to detect which chunks were actually committed. CDN purges that do not fully propagate, leaving stale manifests at some edge nodes after content is updated or taken down.

Each of these requires a specific operational response: FFmpeg timeout enforcement at the worker level, VFR-to-CFR conversion as a preprocessing step, server-side chunk verification before presigned URL expiry, and CDN propagation monitoring with a canary check before considering a purge complete.

The pipeline itself is not particularly complex. The operational discipline required to keep it reliable at scale is.