Designing a Distributed Job Queue: Task Scheduling, Worker Pools, and Delivery Guarantees at Scale

Most applications need work to happen asynchronously: sending emails, generating reports, resizing images, syncing data to third-party APIs. A job queue is the coordination layer between the code that says “do this later” and the code that actually does it.

At small scale, a single Redis list and a few workers is enough. At larger scale, the design decisions compound: how do you guarantee a job runs exactly once? How do you detect a stuck worker? How do you prevent a surge of work from crushing your database? These are not abstract questions. They are the bugs that cause double-charges, missed notifications, and silent data loss.

This article walks through designing a distributed job queue from scratch. Not a tutorial on using BullMQ or SQS (those comparisons exist elsewhere), but the underlying architecture: what components you need, what guarantees you can offer, and where the tradeoffs live.

Core Components

A job queue has five logical components. Each can be simple or complex depending on your requirements.

Producers enqueue tasks. They serialize a job definition (task type, payload, scheduling metadata) and write it to the queue storage. Producers should be fast and non-blocking. They should not know or care which worker picks up the job.

Queue storage holds pending jobs. This is the heart of the system. It needs to support atomic enqueue and dequeue, persistence across restarts, and ideally some form of ordering or priority. Relational databases (Postgres), Redis, and message brokers (Kafka, SQS) are all viable depending on your consistency and throughput requirements.

Workers pull jobs from the queue, execute them, and report results. Each worker runs a loop: claim a job, process it, acknowledge completion or failure.

Scheduler handles delayed and recurring jobs. Jobs that should run “in 15 minutes” or “every hour” need to live somewhere until their execution time arrives. This is typically a separate data structure from the main queue.

Result store (optional) holds job output and status. Some systems need to answer “what happened to job X?” Others only care about side effects and discard results. If you need queryable job history, a result store is necessary.

interface Job {
  id: string;
  type: string;
  payload: Record<string, unknown>;
  priority: number;
  scheduledAt: Date | null;   // null = run immediately
  maxAttempts: number;
  attemptCount: number;
  createdAt: Date;
  visibleAt: Date;            // when this job becomes claimable
}

type JobStatus = "pending" | "running" | "completed" | "failed" | "dead";

interface JobRecord extends Job {
  status: JobStatus;
  workerId: string | null;
  lockedUntil: Date | null;
  completedAt: Date | null;
  lastError: string | null;
}

Queue Storage: Postgres as a Job Queue

Postgres is underused as queue storage. It is not the highest-throughput option, but for most systems handling fewer than a few thousand jobs per second, it is the right call: ACID transactions, rich querying, no extra infrastructure, and SELECT FOR UPDATE SKIP LOCKED for race-free job claiming.

// Claim a batch of jobs atomically
async function claimJobs(
  db: Pool,
  workerId: string,
  batchSize: number,
  lockDurationMs: number
): Promise<JobRecord[]> {
  const now = new Date();
  const lockUntil = new Date(now.getTime() + lockDurationMs);

  const result = await db.query<JobRecord>(
    `
    UPDATE jobs
    SET
      status = 'running',
      worker_id = $1,
      locked_until = $3,
      attempt_count = attempt_count + 1
    WHERE id IN (
      SELECT id FROM jobs
      WHERE status = 'pending'
        AND visible_at <= $2
      ORDER BY priority DESC, visible_at ASC
      LIMIT $4
      FOR UPDATE SKIP LOCKED
    )
    RETURNING *
    `,
    [workerId, now, lockUntil, batchSize]
  );

  return result.rows;
}

SKIP LOCKED is the key primitive here. It lets multiple workers claim different jobs concurrently without blocking on each other. Without it, you get lock contention and serialized throughput even with multiple workers.

The priority DESC, visible_at ASC ordering gives you priority queuing with FIFO tiebreaking within the same priority level.

Scheduling Strategies

FIFO is the default. Jobs execute in the order they were enqueued. Simple, fair, predictable. The visible_at ASC ordering in the query above gives you this for free.

Priority queuing assigns a numeric priority to each job. High-priority jobs (password reset emails) jump ahead of low-priority ones (analytics aggregation). Priority inversion is a real risk: if the queue is saturated with high-priority work, low-priority jobs starve indefinitely. You need starvation prevention, typically by bumping the effective priority of jobs that have been waiting beyond a threshold.

Delayed jobs are jobs that should not execute until some future time. They are stored with a visible_at timestamp in the future. A periodic scan (or a sorted set in Redis, or a Postgres index on visible_at) surfaces them when their time arrives.

async function enqueueDelayed(
  db: Pool,
  job: Omit<Job, "id" | "createdAt" | "visibleAt">,
  delayMs: number
): Promise<string> {
  const id = crypto.randomUUID();
  const now = new Date();
  const visibleAt = new Date(now.getTime() + delayMs);

  await db.query(
    `INSERT INTO jobs (id, type, payload, priority, scheduled_at, max_attempts, attempt_count, created_at, visible_at, status)
     VALUES ($1, $2, $3, $4, $5, $6, 0, $7, $8, 'pending')`,
    [id, job.type, JSON.stringify(job.payload), job.priority, job.scheduledAt, job.maxAttempts, now, visibleAt]
  );

  return id;
}

Scheduled (recurring) jobs are a separate concern from the queue itself. The standard pattern is a dedicated scheduler process that reads a cron-like schedule table, computes the next run time, and enqueues a new job record. The job queue then executes it like any other job. Never try to make the queue itself model recurrence.

Delivery Guarantees

This is where most job queue bugs originate. Let’s be precise about what each guarantee actually means.

At-most-once: The job runs zero or one times. If the worker crashes mid-execution, the job is lost. This is appropriate for metrics or cache warming where duplicates are harmful, but missing one is acceptable. It is rarely the right default.

At-least-once: The job runs one or more times. If the worker crashes before acknowledging, the job becomes visible again and another worker picks it up. This is the practical default for most systems. The consequence: your job handlers must be idempotent.

Exactly-once: The job runs exactly once, even across worker crashes, network partitions, and restarts. True exactly-once is not achievable without coordination between the queue and the downstream system. What you can achieve is effectively-once: at-least-once delivery with idempotent handlers that detect and suppress duplicate executions.

The visibility timeout is the mechanism that drives at-least-once delivery. When a worker claims a job, it acquires a lock for a bounded duration (the lock period). If it does not acknowledge completion before the lock expires, the job becomes visible again for another worker to claim.

async function acknowledgeJob(db: Pool, jobId: string, workerId: string): Promise<void> {
  const result = await db.query(
    `UPDATE jobs SET status = 'completed', completed_at = NOW(), worker_id = NULL, locked_until = NULL
     WHERE id = $1 AND worker_id = $2 AND status = 'running'`,
    [jobId, workerId]
  );

  if (result.rowCount === 0) {
    // The lock expired and another worker may have claimed this job.
    // Log and alert — this means the handler ran but the lock was too short.
    throw new Error(`Failed to acknowledge job ${jobId}: lock may have expired`);
  }
}

Idempotency Keys

For at-least-once delivery to be safe, handlers need idempotency. The standard pattern is to include an idempotency key in the job payload and record it in the side-effectful system before committing.

async function sendWelcomeEmail(job: Job): Promise<void> {
  const { userId, email, idempotencyKey } = job.payload as {
    userId: string;
    email: string;
    idempotencyKey: string;
  };

  // Check if we already sent this email
  const existing = await db.query(
    "SELECT id FROM sent_emails WHERE idempotency_key = $1",
    [idempotencyKey]
  );

  if (existing.rows.length > 0) {
    return; // Already sent, skip
  }

  await mailer.send({ to: email, template: "welcome", userId });

  await db.query(
    "INSERT INTO sent_emails (idempotency_key, user_id, sent_at) VALUES ($1, $2, NOW())",
    [idempotencyKey, userId]
  );
}

Use the job id as the idempotency key when you control job creation. For jobs triggered by external events (webhooks, API calls), derive the key from stable properties of the event.

Worker Management

Concurrency control is how many jobs a single worker process handles in parallel. Too low and you leave capacity unused during I/O-heavy work (network calls, database queries). Too high and you exhaust connection pools and memory.

The right model depends on the work: CPU-bound jobs want concurrency equal to core count. I/O-bound jobs can run with much higher concurrency (10-50x) because most of the time is spent waiting.

class WorkerPool {
  private active = 0;
  private queue: Array<() => void> = [];

  constructor(private concurrency: number) {}

  async run(handler: () => Promise<void>): Promise<void> {
    if (this.active >= this.concurrency) {
      await new Promise<void>((resolve) => this.queue.push(resolve));
    }

    this.active++;
    try {
      await handler();
    } finally {
      this.active--;
      const next = this.queue.shift();
      if (next) next();
    }
  }
}

Heartbeats solve the problem of distinguishing a long-running job from a dead worker. A worker processing a large file might hold a lock for 10 minutes legitimately. If the visibility timeout is 5 minutes, the job gets requeued while still running. Heartbeats allow workers to extend their lock before it expires.

async function heartbeat(
  db: Pool,
  jobId: string,
  workerId: string,
  extensionMs: number
): Promise<boolean> {
  const newLockUntil = new Date(Date.now() + extensionMs);
  const result = await db.query(
    `UPDATE jobs SET locked_until = $1
     WHERE id = $2 AND worker_id = $3 AND status = 'running' AND locked_until > NOW()`,
    [newLockUntil, jobId, workerId]
  );
  return (result.rowCount ?? 0) > 0;
}

// In the worker main loop
async function processWithHeartbeat(
  db: Pool,
  job: JobRecord,
  workerId: string,
  handler: (job: JobRecord) => Promise<void>
): Promise<void> {
  const interval = setInterval(async () => {
    const extended = await heartbeat(db, job.id, workerId, 30_000);
    if (!extended) {
      // Lock was lost, stop processing
      clearInterval(interval);
    }
  }, 10_000);

  try {
    await handler(job);
  } finally {
    clearInterval(interval);
  }
}

Graceful shutdown is often missing in naive implementations. When a worker process receives SIGTERM, it should stop accepting new jobs, wait for in-progress jobs to complete (up to a timeout), acknowledge them, and exit. Without this, deployments cause jobs to become visible again and trigger duplicate execution.

let shutdownRequested = false;

process.on("SIGTERM", () => {
  shutdownRequested = true;
});

async function workerLoop(db: Pool, workerId: string): Promise<void> {
  while (!shutdownRequested) {
    const jobs = await claimJobs(db, workerId, 5, 30_000);

    if (jobs.length === 0) {
      await sleep(1000);
      continue;
    }

    await Promise.allSettled(
      jobs.map((job) => processWithHeartbeat(db, job, workerId, dispatch(job)))
    );
  }
}

Failure Handling

When a job fails, you have three choices: retry immediately, retry with backoff, or give up. The retry strategy belongs in the queue, not in the handler. Handlers should throw errors; the queue should decide what to do next.

async function failJob(
  db: Pool,
  jobId: string,
  workerId: string,
  error: Error,
  baseDelayMs: number
): Promise<void> {
  const job = await db.query<JobRecord>(
    "SELECT * FROM jobs WHERE id = $1 FOR UPDATE",
    [jobId]
  );

  const record = job.rows[0];
  const isExhausted = record.attempt_count >= record.max_attempts;

  if (isExhausted) {
    // Move to dead letter queue
    await db.query(
      `UPDATE jobs SET status = 'dead', last_error = $1, worker_id = NULL, locked_until = NULL WHERE id = $2`,
      [error.message, jobId]
    );
    return;
  }

  // Exponential backoff with jitter
  const delay = baseDelayMs * Math.pow(2, record.attempt_count - 1);
  const jitter = Math.random() * delay * 0.1;
  const nextVisible = new Date(Date.now() + delay + jitter);

  await db.query(
    `UPDATE jobs SET status = 'pending', last_error = $1, worker_id = NULL, locked_until = NULL, visible_at = $2
     WHERE id = $3`,
    [error.message, nextVisible, jobId]
  );
}

Dead letter queues (DLQ) hold jobs that have exhausted retries. They serve two purposes: preventing poison pills from circling the main queue indefinitely, and giving operators a place to inspect and replay failed work. A DLQ is just a different status flag (or a separate table) with tooling to re-enqueue jobs after fixing the underlying cause.

Poison pills are jobs that always crash their worker process rather than throwing a recoverable error (memory exhaustion, infinite loops, unhandled native exceptions). The detection pattern is tracking consecutive worker crashes on the same job ID and quarantining the job after a threshold.

Scaling Patterns

Partitioned queues distribute load across multiple independent queues, each owned by a subset of workers. Partition by tenant, by job type, or by shard key. Each partition is a full queue: its own storage, its own worker pool, its own DLQ. Partitioning gives you isolation (a runaway tenant cannot starve others), horizontal scalability, and independent scaling per partition.

The tradeoff: global ordering and global priority no longer exist. If a high-priority job lands on a heavily loaded partition while an idle partition sits empty, the priority signal is lost. Only use global ordering when you actually need it.

Consumer groups (Kafka’s model) allow multiple logical consumers to each receive a copy of every message on a topic. This is the broadcast pattern, useful when multiple systems need to react to the same event. It is distinct from the competing-consumers pattern (multiple workers share a single workload). Do not conflate them; they solve different problems.

Backpressure is what happens when workers cannot keep up with the enqueue rate. Without backpressure, the queue grows unboundedly, memory and disk fill up, and eventually the whole system falls over. Backpressure mechanisms include:

• Rate limiting producers: reject or slow enqueue requests when queue depth exceeds a threshold.
• Adaptive polling: workers back off their polling interval when the queue is empty; they poll aggressively when there is work to do.
• Capacity signaling: workers publish their current load; producers check before enqueueing (push-based backpressure).

async function enqueueWithBackpressure(
  db: Pool,
  job: Omit<Job, "id" | "createdAt" | "visibleAt">,
  maxDepth: number
): Promise<string | null> {
  const depth = await db.query<{ count: string }>(
    "SELECT COUNT(*) as count FROM jobs WHERE status = 'pending'"
  );

  if (parseInt(depth.rows[0].count, 10) >= maxDepth) {
    return null; // Caller must handle the rejection
  }

  return enqueueDelayed(db, job, 0);
}

Architecture Tradeoffs

Decision	Option A	Option B	When to pick A	When to pick B
Queue storage	Postgres with SKIP LOCKED	Redis sorted sets	You need ACID, queryability, no extra infra	You need sub-millisecond enqueue/dequeue, >10K jobs/sec
Delivery guarantee	At-least-once + idempotent handlers	Exactly-once via distributed TX	Most systems; simpler to implement	Financial transactions, metered billing; requires 2PC or outbox pattern
Worker model	Competing consumers (shared queue)	Consumer groups (each worker gets all jobs)	Work distribution, throughput scaling	Fan-out to multiple systems from same event
Scheduling	Delayed visibility in main queue	Separate scheduler process	Simple delayed jobs, single-digit second precision	High-frequency recurring jobs, cron-style scheduling with missed-job semantics
Partitioning	Single queue, all workers share	Partitioned by tenant or type	Simple topology, global ordering matters	Tenant isolation, independent scaling per job type
Backpressure	Queue depth threshold (reject)	Adaptive producer rate limiting	Bursty producers, hard capacity limits	Sustained overload, need smooth throttling

Production Considerations

Monitor queue depth per job type, not just total depth. A healthy overall depth can hide a specific job type that is accumulating because its handler is broken.

Set lock durations based on p95 execution time, not average. If your median job takes 2 seconds but the p99 takes 90 seconds, a 5-second lock will cause constant requeuing for slow jobs.

Log job lifecycle events (enqueued, claimed, completed, failed, dead) with the job ID, worker ID, attempt count, and duration. Without this, debugging production failures is blind guessing.

Test your failure paths explicitly: kill a worker mid-job and confirm the job gets reclaimed. Saturate the queue and confirm backpressure triggers correctly. Run a job that always fails and confirm it lands in the DLQ after the expected number of retries.

Index (status, visible_at, priority) on the jobs table. The claim query runs constantly; without this index it will scan the full table as the queue grows.

CREATE INDEX idx_jobs_claimable
  ON jobs (priority DESC, visible_at ASC)
  WHERE status = 'pending';

The partial index (WHERE status = 'pending') keeps the index small and fast. Completed and dead jobs do not need to participate in the claim scan.

The Architecture in One Sentence

A distributed job queue is a coordination contract: producers write to a durable store, workers claim work atomically with bounded leases, and the queue recovers from failures by returning unclaimed work to the pool after the lease expires.

Everything else (priority, scheduling, partitioning, backpressure) is an optimization layered on top of that contract. Start with the simplest version that gives you at-least-once delivery and idempotent handlers. Add complexity only when you have a specific problem that simpler approaches cannot solve.

The visibility timeout, the idempotency key, and the dead letter queue are not optional features. They are the minimum viable set for a job queue that you can trust in production.