Designing a Batch Processing System: Job Partitioning, Checkpoint Recovery, and Parallel Execution at Scale

Batch processing is one of those topics where the conceptual answer is easy (process records in chunks) and the production reality is not. The failure modes emerge at scale: a worker dies halfway through a 200 million row export, a partition skew causes one worker to hold 80% of the data, or a downstream queue floods because the batch is running faster than consumers can absorb.

This article covers how to design a batch processing system that handles these problems, with concrete partitioning strategies, checkpoint mechanics, execution models, and the tradeoffs you will face at each layer.

The Core Problem

A batch job has three properties that make it harder than it looks:

1. It operates on a bounded dataset that is large enough to make single-threaded processing impractical.
2. It needs to be restartable without reprocessing work already done.
3. It shares infrastructure with other workloads, so it cannot consume resources without bound.

The naive approach runs all records in a loop. It works until the dataset grows, the process crashes mid-run, or a downstream system slows down and nothing upstream notices.

Job Partitioning

The first design decision is how to divide the work. The choice of partition strategy determines how evenly work is distributed, whether you can restart partial failures cheaply, and how you handle data that arrives after the job starts.

Range-Based Partitioning

Split the keyspace by value ranges. If processing user records, partition by user_id ranges: worker 1 handles IDs 1-1000000, worker 2 handles 1000001-2000000, and so on.

interface RangePartition {
  workerId: number;
  minId: number;
  maxId: number;
  status: "pending" | "in_progress" | "complete" | "failed";
  checkpoint?: number; // last processed ID within this range
}

function buildRangePartitions(
  totalRecords: number,
  workerCount: number
): RangePartition[] {
  const rangeSize = Math.ceil(totalRecords / workerCount);
  return Array.from({ length: workerCount }, (_, i) => ({
    workerId: i,
    minId: i * rangeSize + 1,
    maxId: Math.min((i + 1) * rangeSize, totalRecords),
    status: "pending",
  }));
}

Range partitioning is simple to reason about and allows deterministic restart: if a worker fails, you know exactly which ID range it was processing. The problem is skew. If your IDs are not uniformly distributed (inactive users cluster in old ranges, active users are recent), some workers finish in minutes while others run for hours.

Hash-Based Partitioning

Apply a hash function to the record key and assign to a bucket. This distributes records more uniformly regardless of key distribution.

function hashPartition(key: string, bucketCount: number): number {
  let hash = 0;
  for (let i = 0; i < key.length; i++) {
    hash = (hash * 31 + key.charCodeAt(i)) >>> 0;
  }
  return hash % bucketCount;
}

async function processPartition(
  bucket: number,
  bucketCount: number,
  processFn: (record: Record<string, unknown>) => Promise<void>
): Promise<void> {
  const cursor = await db.query(
    `SELECT * FROM records WHERE hash_bucket(id, $1) = $2 ORDER BY id`,
    [bucketCount, bucket]
  );
  for await (const record of cursor) {
    await processFn(record);
  }
}

Hash partitioning gives you uniform distribution but loses locality. Range scans are cheap on a B-tree index. A hash scan requires a full table scan unless you precompute and store the bucket column. For large tables, that column adds write overhead and index maintenance cost.

Dynamic Partitioning

Instead of pre-assigning partitions, workers claim chunks from a work queue. A coordinator inserts pending chunks; workers poll for available work, process it, and mark it done.

interface WorkChunk {
  chunkId: string;
  filter: Record<string, unknown>;
  claimedAt?: Date;
  workerId?: string;
  completedAt?: Date;
}

class WorkQueue {
  async claimChunk(workerId: string): Promise<WorkChunk | null> {
    // Atomic claim using a CAS-style update
    const result = await db.query<WorkChunk>(
      `UPDATE work_chunks
       SET claimed_at = NOW(), worker_id = $1, status = 'in_progress'
       WHERE chunk_id = (
         SELECT chunk_id FROM work_chunks
         WHERE status = 'pending'
           OR (status = 'in_progress' AND claimed_at < NOW() - INTERVAL '10 minutes')
         ORDER BY chunk_id
         LIMIT 1
         FOR UPDATE SKIP LOCKED
       )
       RETURNING *`,
      [workerId]
    );
    return result.rows[0] ?? null;
  }

  async completeChunk(chunkId: string): Promise<void> {
    await db.query(
      `UPDATE work_chunks SET status = 'complete', completed_at = NOW() WHERE chunk_id = $1`,
      [chunkId]
    );
  }
}

The FOR UPDATE SKIP LOCKED pattern is important. Without it, multiple workers racing to claim the same chunk will block each other. With SKIP LOCKED, each worker skips already-locked rows and grabs the next available one. The stale claim reclaim (chunks in-progress for more than 10 minutes) handles worker crashes.

Dynamic partitioning handles skew naturally: slow workers process fewer chunks, fast workers pick up more. The tradeoff is coordination overhead and the need for a reliable work queue.

Checkpoint and Recovery

Checkpoints let a job resume from the middle of a partition rather than restarting from scratch. For a job processing 50 million records, restarting from record 0 after a crash at record 40 million is expensive and may violate SLAs.

What to Checkpoint

At minimum, checkpoint the last successfully processed record identifier and the count of records processed. For jobs with side effects (writes to external systems), you also need idempotency guarantees because the checkpoint may not align exactly with what was already written.

interface Checkpoint {
  jobId: string;
  partitionId: string;
  lastProcessedKey: string;
  processedCount: number;
  updatedAt: Date;
}

class CheckpointStore {
  private readonly flushIntervalMs = 1000;
  private pendingCount = 0;
  private lastFlushedKey = "";

  constructor(
    private readonly db: Database,
    private readonly jobId: string,
    private readonly partitionId: string
  ) {}

  async update(key: string): Promise<void> {
    this.pendingCount++;
    this.lastFlushedKey = key;

    // Flush on interval, not on every record
    if (this.pendingCount % 500 === 0) {
      await this.flush();
    }
  }

  async flush(): Promise<void> {
    await this.db.query(
      `INSERT INTO checkpoints (job_id, partition_id, last_processed_key, processed_count, updated_at)
       VALUES ($1, $2, $3, $4, NOW())
       ON CONFLICT (job_id, partition_id)
       DO UPDATE SET last_processed_key = $3, processed_count = $4, updated_at = NOW()`,
      [this.jobId, this.partitionId, this.lastFlushedKey, this.pendingCount]
    );
  }

  async load(): Promise<Checkpoint | null> {
    const result = await this.db.query<Checkpoint>(
      `SELECT * FROM checkpoints WHERE job_id = $1 AND partition_id = $2`,
      [this.jobId, this.partitionId]
    );
    return result.rows[0] ?? null;
  }
}

Checkpointing every record is too slow. Checkpointing too infrequently means more re-processing on restart. 500 records per flush is a reasonable starting point; tune based on record processing latency and acceptable re-processing cost.

Idempotent Side Effects

A checkpoint at record 5000 means records 4001-5000 may have been processed but the checkpoint not yet written, or the checkpoint written but some records failed mid-batch. Both cases result in records being processed again on restart.

Design side effects to be idempotent: upsert instead of insert, use job-scoped idempotency keys for external API calls, and prefer append-only writes where the reader can deduplicate by record ID.

async function processWithIdempotency(
  record: UserRecord,
  jobId: string
): Promise<void> {
  const idempotencyKey = `${jobId}:${record.id}`;

  await externalApi.post("/events", {
    type: "user_processed",
    userId: record.id,
    payload: record,
    idempotencyKey, // external systems use this to deduplicate
  });

  await db.query(
    `INSERT INTO processed_users (job_id, user_id, processed_at)
     VALUES ($1, $2, NOW())
     ON CONFLICT (job_id, user_id) DO NOTHING`,
    [jobId, record.id]
  );
}

Parallel Execution Models

Two dominant models: MapReduce-style and DAG-based.

MapReduce Style

Map phase: distribute records across workers. Reduce phase: aggregate results. The two phases are separate and synchronize at a barrier. Simple to reason about; the barrier is both its strength (clear phase boundary) and its weakness (the slowest mapper blocks the reduce phase).

async function mapReduce<T, M, R>(config: {
  source: AsyncIterable<T>;
  mapFn: (record: T) => Promise<M>;
  reduceFn: (accumulator: R, mapped: M) => R;
  initialValue: R;
  concurrency: number;
}): Promise<R> {
  const { source, mapFn, reduceFn, initialValue, concurrency } = config;
  const semaphore = new Semaphore(concurrency);
  const mapped: M[] = [];

  for await (const record of source) {
    await semaphore.acquire();
    mapFn(record)
      .then((result) => {
        mapped.push(result);
        semaphore.release();
      })
      .catch((err) => {
        semaphore.release();
        throw err;
      });
  }

  await semaphore.drain(); // wait for all in-flight maps to complete
  return mapped.reduce(reduceFn, initialValue);
}

DAG-Based Execution

Model the job as a directed acyclic graph of stages. Each stage consumes from one or more upstream stages and emits to one or more downstream stages. Stages run concurrently; dependencies enforce ordering without a global barrier.

interface Stage<TIn, TOut> {
  id: string;
  dependencies: string[];
  process: (input: TIn) => AsyncIterable<TOut>;
}

class DAGExecutor {
  private readonly stages = new Map<string, Stage<unknown, unknown>>();
  private readonly channels = new Map<string, AsyncQueue<unknown>>();

  register<TIn, TOut>(stage: Stage<TIn, TOut>): void {
    this.stages.set(stage.id, stage as Stage<unknown, unknown>);
    this.channels.set(stage.id, new AsyncQueue());
  }

  async execute(rootInput: unknown): Promise<void> {
    const rootStageIds = [...this.stages.values()]
      .filter((s) => s.dependencies.length === 0)
      .map((s) => s.id);

    for (const id of rootStageIds) {
      this.channels.get(id)!.push(rootInput);
    }

    const runStage = async (stageId: string): Promise<void> => {
      const stage = this.stages.get(stageId)!;
      const inputChannel = this.channels.get(stageId)!;

      for await (const input of inputChannel) {
        const outputChannel = this.getDownstreamChannel(stageId);
        for await (const output of stage.process(input)) {
          outputChannel?.push(output);
        }
      }
    };

    await Promise.all([...this.stages.keys()].map(runStage));
  }

  private getDownstreamChannel(stageId: string): AsyncQueue<unknown> | null {
    for (const [id, stage] of this.stages) {
      if (stage.dependencies.includes(stageId)) {
        return this.channels.get(id)!;
      }
    }
    return null;
  }
}

DAG execution is more complex to implement correctly but avoids the skew problem at the barrier. A slow stage builds up backlog in its input channel; the upstream stage either blocks or applies backpressure.

Backpressure and Resource Management

Without backpressure, a fast producer will outpace a slow consumer and exhaust memory. The fix is to couple producer throughput to consumer capacity.

The simplest form is a bounded queue: the producer blocks when the queue is full.

class BoundedQueue<T> {
  private readonly items: T[] = [];
  private readonly waiters: Array<() => void> = [];

  constructor(private readonly capacity: number) {}

  async push(item: T): Promise<void> {
    if (this.items.length >= this.capacity) {
      await new Promise<void>((resolve) => this.waiters.push(resolve));
    }
    this.items.push(item);
  }

  async pop(): Promise<T> {
    while (this.items.length === 0) {
      await new Promise<void>((resolve) =>
        setTimeout(resolve, 10)
      );
    }
    const item = this.items.shift()!;
    const waiter = this.waiters.shift();
    waiter?.();
    return item;
  }
}

For resource management beyond memory, track active worker slots and refuse to start new work items when the system is saturated. This is the admission control layer: it decides what enters the system, not just how fast things move through it.

class WorkerPool {
  private activeWorkers = 0;

  constructor(
    private readonly maxWorkers: number,
    private readonly onWorkerAvailable: () => void
  ) {}

  async acquire(): Promise<() => void> {
    while (this.activeWorkers >= this.maxWorkers) {
      await new Promise<void>((resolve) => setTimeout(resolve, 50));
    }
    this.activeWorkers++;
    return () => {
      this.activeWorkers--;
      this.onWorkerAvailable();
    };
  }
}

Partitioning Strategy Tradeoffs

Strategy	Distribution	Restart granularity	Coordination overhead	Skew risk
Range-based	Even if data is uniform	Exact range boundary	None	High if data is not uniform
Hash-based	Uniform regardless of key	Bucket boundary	Column storage cost	Low
Dynamic (work queue)	Adaptive	Chunk size	DB or queue required	Minimal
Fixed assignment	Depends on data	Partition boundary	None	Medium

For most production batch jobs, dynamic partitioning wins unless you need locality (for range scans) or have a pre-existing sharding scheme you can align with.

Production Considerations

Job observability. Track records processed, records failed, throughput (records/second), and estimated time to completion per partition. Store these in a metrics table or emit to a time-series backend. Without these, a job that is silently slow is indistinguishable from one that is making progress.

interface BatchMetrics {
  jobId: string;
  partitionId: string;
  processedCount: number;
  failedCount: number;
  throughputRps: number;
  estimatedRemainingMs: number;
  timestamp: Date;
}

Poison record handling. A record that consistently causes a worker to crash will block partition progress indefinitely. Add a per-record retry limit and a dead-letter table. After N retries, move the record to dead-letter, log why, and continue processing.

SLA tracking. For time-sensitive batch jobs (end-of-day reports, billing runs), track job start time against the SLA deadline and alert well before the deadline is in jeopardy. A job that started 3 hours ago and is 40% complete will miss its 4-hour SLA window; you need to know that at hour 2, not hour 4.

Cost optimization. Batch jobs are a good fit for spot/preemptible instances because they are restartable. The tradeoff: spot instances can be reclaimed mid-job, which requires your checkpoint mechanism to be reliable. Test this explicitly. A checkpoint that only flushes at job completion does not help with spot reclamation.

Parallel job isolation. Running multiple batch jobs concurrently against the same database will compete for I/O and connection pool capacity. Use job-level resource tagging (separate read replicas, query priority hints, or time-of-day scheduling) to prevent a batch job from degrading OLTP query latency.

Monitoring for stalls. A worker that has claimed a partition but stopped making progress is different from a slow worker. Set a heartbeat: workers emit a progress ping every N seconds. If the coordinator sees no ping from a partition within 2x the expected heartbeat interval, reassign it. This catches deadlocks, network partitions to downstream systems, and worker processes that are alive but stuck.

Execution Model Tradeoffs

Model	Complexity	Skew tolerance	Memory overhead	When to use
Single-threaded loop	None	N/A	Minimal	Small datasets, exploratory
MapReduce (barrier)	Low	Low (barrier blocks on slowest mapper)	Medium	Independent map steps, simple aggregation
DAG-based	Medium	High (stages run independently)	Higher (buffering between stages)	Multi-stage pipelines, heterogeneous stage costs
Streaming with backpressure	High	High	Controlled via queue depth	Continuous batch, near-real-time requirements

DAG-based execution is the right default for production batch systems because it composes naturally with checkpoint recovery (each stage has its own cursor) and handles heterogeneous stage costs without manual tuning.

Closing

Batch processing systems fail in predictable ways: partition skew, missing checkpoints, absent backpressure, and no visibility into progress. Solving these problems is not intellectually complex, but each one requires deliberate design decisions that are easy to skip when the initial dataset is small. The checkpoint store and the dynamic work queue are the two investments with the highest return: they turn a brittle one-shot job into a restartable, observable pipeline that can grow with the data.