Designing a Distributed Task Queue: Priority Scheduling, Worker Pools, and Exactly-Once Processing

Most teams reach for a task queue when they hit the same wall: a slow operation in the request path is making the API unresponsive. Email sending, PDF generation, third-party webhook calls, report aggregation. The first instinct is to offload it. The second instinct, after the first queue implementation bites them, is to understand what they actually built.

A task queue is not just a list of jobs. It is a distributed system with its own consistency model, failure modes, and capacity constraints. Getting the design right upfront saves you from the class of production incidents that only appear at scale: duplicate charges, lost jobs, worker starvation, and cascading overload.

This article covers the core design decisions you will face when building or evaluating a task queue: scheduling, worker management, delivery guarantees, failure handling, and the tradeoffs between building on Redis, Postgres, or a dedicated system.

The Core Problem

A task queue sits between a producer (your API or application code) that enqueues work and a pool of consumers (workers) that execute it. The contract sounds simple: producers push work, workers pull it, work gets done. The complexity is in the edge cases.

What happens when a worker crashes mid-execution? What if two workers pick up the same job? What if the queue fills up faster than workers can drain it? What if some jobs are more time-sensitive than others? Each of these questions drives a specific architectural decision.

Priority Scheduling

The naive implementation uses a single FIFO queue. This works until you have mixed workloads where a batch report generation job is blocking a time-sensitive transactional email. Priority scheduling solves this by assigning numeric priority to jobs and processing higher-priority jobs first.

The two common approaches are weighted priority queues and multiple queue tiers.

Multiple queue tiers are simpler to reason about and easier to implement correctly. You maintain separate queues for each priority level, and workers poll from higher-priority queues first:

type Priority = "critical" | "high" | "normal" | "low";

const QUEUE_POLL_ORDER: Priority[] = ["critical", "high", "normal", "low"];

interface Job {
  id: string;
  type: string;
  payload: unknown;
  priority: Priority;
  idempotencyKey: string;
  enqueuedAt: Date;
  attemptCount: number;
  maxAttempts: number;
}

async function pollNextJob(redis: Redis): Promise<Job | null> {
  for (const priority of QUEUE_POLL_ORDER) {
    const raw = await redis.lpop(`queue:${priority}`);
    if (raw) {
      return JSON.parse(raw) as Job;
    }
  }
  return null;
}

This works, but it has a starvation problem: if critical and high queues are always populated, low-priority jobs never run. You need to add a fairness mechanism. The simplest is a weighted poll: poll critical 8 times per cycle, high 4 times, normal 2 times, low 1 time.

const POLL_WEIGHTS: Record<Priority, number> = {
  critical: 8,
  high: 4,
  normal: 2,
  low: 1,
};

function buildPollSchedule(): Priority[] {
  const schedule: Priority[] = [];
  for (const [priority, weight] of Object.entries(POLL_WEIGHTS) as [Priority, number][]) {
    for (let i = 0; i < weight; i++) {
      schedule.push(priority);
    }
  }
  return schedule;
}

// Rotate through this schedule across worker iterations
const POLL_SCHEDULE = buildPollSchedule();

Weighted priority queues (a single sorted set with numeric scores) are what systems like BullMQ use. They are more flexible for fine-grained priority but require atomic score-based dequeue operations, which have subtle race conditions under concurrent workers without proper locking.

Worker Pool Management

A worker pool manages concurrency. Too few workers and the queue backs up. Too many and you overwhelm downstream services or run into database connection limits.

The key parameters are:

• Concurrency: how many jobs run simultaneously per worker process
• Worker count: how many worker processes run in total
• Polling interval: how frequently idle workers check for work

interface WorkerPoolConfig {
  concurrency: number;
  minWorkers: number;
  maxWorkers: number;
  pollingIntervalMs: number;
  gracefulShutdownTimeoutMs: number;
}

class WorkerPool {
  private activeJobs = new Map<string, Promise<void>>();
  private running = false;

  constructor(
    private config: WorkerPoolConfig,
    private executor: (job: Job) => Promise<void>,
    private queue: Queue
  ) {}

  async start(): Promise<void> {
    this.running = true;
    await this.loop();
  }

  private async loop(): Promise<void> {
    while (this.running) {
      if (this.activeJobs.size >= this.config.concurrency) {
        await Promise.race(this.activeJobs.values());
        continue;
      }

      const job = await this.queue.dequeue();
      if (!job) {
        await sleep(this.config.pollingIntervalMs);
        continue;
      }

      const execution = this.execute(job);
      this.activeJobs.set(job.id, execution);
      execution.finally(() => this.activeJobs.delete(job.id));
    }
  }

  private async execute(job: Job): Promise<void> {
    try {
      await this.executor(job);
      await this.queue.ack(job.id);
    } catch (err) {
      await this.queue.nack(job.id, err as Error);
    }
  }

  async shutdown(): Promise<void> {
    this.running = false;
    await Promise.race([
      Promise.all(this.activeJobs.values()),
      sleep(this.config.gracefulShutdownTimeoutMs),
    ]);
  }
}

The shutdown method is important. Workers get SIGTERM on deploy. If you kill them mid-job, you either lose work or create inconsistent state. Graceful shutdown waits for active jobs to finish up to a timeout, then exits. Anything still running gets re-queued when the next worker picks it up.

For horizontal scaling, the right signal depends on your workload. Queue depth works for batch workloads: when the queue depth exceeds a threshold, spin up more workers. For latency-sensitive workloads, p95 processing time is a better signal than raw depth. Worker CPU and memory utilization are trailing indicators that respond too slowly for bursty workloads.

Exactly-Once Processing

Distributed systems give you at-least-once delivery or at-most-once delivery. Exactly-once is a guarantee at the application layer, not the infrastructure layer, and it requires idempotency.

At-least-once means a job may run more than once. This happens when a worker crashes after executing the job but before acknowledging it to the queue. The queue assumes failure and re-delivers the job.

At-most-once means a job is acknowledged before execution. If the worker crashes during execution, the job is lost.

Idempotency converts at-least-once delivery into exactly-once semantics. You design your job handler so that running it multiple times with the same inputs produces the same result with no side effects on duplicate runs.

interface IdempotencyRecord {
  key: string;
  status: "processing" | "completed" | "failed";
  result?: unknown;
  completedAt?: Date;
  expiresAt: Date;
}

class IdempotentJobExecutor {
  constructor(private db: Database, private ttlMs = 86_400_000) {}

  async execute<T>(
    idempotencyKey: string,
    handler: () => Promise<T>
  ): Promise<T> {
    const existing = await this.db.idempotency.findUnique({
      where: { key: idempotencyKey },
    });

    if (existing?.status === "completed") {
      return existing.result as T;
    }

    if (existing?.status === "processing") {
      // Another worker is handling this. Wait and retry, or throw.
      throw new Error(`Job ${idempotencyKey} is already being processed`);
    }

    await this.db.idempotency.upsert({
      where: { key: idempotencyKey },
      create: {
        key: idempotencyKey,
        status: "processing",
        expiresAt: new Date(Date.now() + this.ttlMs),
      },
      update: { status: "processing" },
    });

    try {
      const result = await handler();

      await this.db.idempotency.update({
        where: { key: idempotencyKey },
        data: { status: "completed", result, completedAt: new Date() },
      });

      return result;
    } catch (err) {
      await this.db.idempotency.update({
        where: { key: idempotencyKey },
        data: { status: "failed" },
      });
      throw err;
    }
  }
}

The idempotency key should be derived from the job’s semantic identity, not its queue ID. A payment charge job should use charge:${paymentIntentId}, not job:${uuid}. The UUID changes if you re-enqueue; the payment intent ID does not. This ensures that re-enqueueing after a failure still deduplicates correctly.

Dead Letter Queues

Not all failures are transient. Some jobs fail because of bad input, upstream bugs, or data inconsistencies that retrying will not fix. After N retry attempts, those jobs should move to a dead letter queue (DLQ) instead of poisoning the main queue with infinite retries.

async function handleJobFailure(
  job: Job,
  error: Error,
  queue: Queue,
  dlq: Queue
): Promise<void> {
  const nextAttempt = job.attemptCount + 1;

  if (nextAttempt >= job.maxAttempts) {
    await dlq.enqueue({
      ...job,
      failureReason: error.message,
      failedAt: new Date(),
    });
    return;
  }

  // Exponential backoff: 2^attempt * 1000ms, capped at 1 hour
  const delayMs = Math.min(Math.pow(2, nextAttempt) * 1_000, 3_600_000);

  await queue.enqueueDelayed(
    { ...job, attemptCount: nextAttempt },
    delayMs
  );
}

The DLQ is not a graveyard. It is an ops tool. You need:

1. A way to inspect DLQ contents with failure reasons
2. A way to replay individual jobs after you fix the underlying bug
3. Alerting when DLQ depth crosses a threshold (any growth in DLQ usually means a systemic bug, not random failures)

Replay should re-enqueue the original job with the original idempotency key, not create a new one. Otherwise your idempotency records will not protect against the duplicate run.

Backpressure

Backpressure is what you apply when producers enqueue faster than workers can drain. Without it, the queue grows unbounded, memory pressure mounts, and eventual processing of jobs that were relevant hours ago wastes resources.

Two mechanisms matter here:

Queue depth limits: Reject enqueue attempts when the queue exceeds a maximum depth. The producer gets an error and decides what to do: drop, retry later, or surface an error to the user.

Rate limiting at the producer: Apply a token bucket or leaky bucket at the enqueue path. This smooths bursts and prevents a single producer from overwhelming the queue.

class TokenBucketRateLimiter {
  private tokens: number;
  private lastRefill: number;

  constructor(
    private capacity: number,
    private refillRatePerSecond: number
  ) {
    this.tokens = capacity;
    this.lastRefill = Date.now();
  }

  tryAcquire(count = 1): boolean {
    this.refill();
    if (this.tokens < count) {
      return false;
    }
    this.tokens -= count;
    return true;
  }

  private refill(): void {
    const now = Date.now();
    const elapsed = (now - this.lastRefill) / 1_000;
    this.tokens = Math.min(
      this.capacity,
      this.tokens + elapsed * this.refillRatePerSecond
    );
    this.lastRefill = now;
  }
}

In distributed systems, per-process rate limiting is not sufficient. You need a shared rate limiter backed by Redis (using the token bucket algorithm with Lua scripts for atomicity) to enforce limits across multiple producer instances.

Choosing Your Storage Backend

Approach	Throughput	Ordering	Persistence	Operational cost	Best for
Redis (BullMQ)	Very high	FIFO per queue	AOF/RDB (can lose tail)	Low	High-volume, latency-sensitive, loss-tolerant workloads
Postgres (polling)	Moderate	FIFO with FOR UPDATE SKIP LOCKED	Full ACID	Low if already on PG	Transactional correctness, low-to-moderate volume
AWS SQS	High	Best-effort (FIFO queues available)	Managed	Near-zero	Cloud-native, teams that don’t want to operate infra
Kafka	Very high	Partitioned order	Durable, configurable	High	High-volume event streaming, fan-out to many consumers

Redis is fast and simple to operate if you already run it. BullMQ handles delayed jobs, priorities, repeatable jobs, and flow dependencies. The risk is data loss on crash if persistence is not configured correctly. Redis AOF persistence with appendfsync everysec gives you at most one second of data loss. For jobs that process payments or send emails, that may not be acceptable.

Postgres is underrated for task queues at moderate volume. The FOR UPDATE SKIP LOCKED pattern gives you a correct, concurrent-safe dequeue without external locking:

WITH next_job AS (
  SELECT id FROM jobs
  WHERE status = 'pending'
    AND scheduled_at <= NOW()
  ORDER BY priority DESC, created_at ASC
  LIMIT 1
  FOR UPDATE SKIP LOCKED
)
UPDATE jobs
SET status = 'processing', started_at = NOW(), worker_id = $1
FROM next_job
WHERE jobs.id = next_job.id
RETURNING *;

This is ACID. The job is either claimed or it is not, with no race conditions. You get priority ordering, delayed scheduling, and the full power of SQL for DLQ inspection and replay. The ceiling is roughly 1,000-5,000 jobs/second on a reasonably sized Postgres instance before queue polling becomes a meaningful load driver. For most applications, that is more than enough.

SQS removes operational burden but adds network latency (50-200ms per operation) and pricing complexity at high volume. SQS FIFO queues give you exactly-once delivery within a five-minute deduplication window, which handles the common case of worker crash and re-delivery. The deduplication window is not configurable, which is a real constraint for some workloads.

Production Considerations

Monitoring. Track queue depth per priority tier, job processing latency (time from enqueue to completion), DLQ depth, worker concurrency utilization (active/max), and retry rate. A sudden spike in retry rate usually indicates a downstream dependency is failing. A queue depth growing faster than it drains is a capacity alert.

Job timeouts. Every job should have a maximum execution time. A job that hangs indefinitely holds a worker slot and causes the queue to back up. Implement a per-job timeout by wrapping the handler in a Promise.race against a timeout promise, then marking the job as failed and releasing the slot.

Visibility timeout. For systems like SQS, the visibility timeout is the window during which a dequeued message is hidden from other consumers. Set it to 1.5-2x your expected maximum job duration. If a worker crashes before the timeout expires, SQS re-delivers the message. If your timeout is too short, you get duplicate deliveries on long-running jobs.

Schema versioning. Job payloads are serialized and stored. When you change a job’s payload shape, old jobs in the queue have the old shape. Always version your job payloads and handle unknown versions in your handler. A simple version field and a migration path in the handler avoids broken deserialization when you deploy.

Poison pills. A job that consistently crashes workers (out-of-memory, infinite loop) before it can be acked or nacked is a poison pill. The system will keep re-delivering it. Set a max-attempt count before moving to DLQ, and ensure your timeout mechanism catches runaway jobs before they exhaust worker memory.

The Layer Map

A complete distributed task queue has these layers, each independently testable:

1. Enqueue API: validates payload, assigns priority and idempotency key, applies rate limiting, enforces queue depth limits
2. Storage backend: Redis, Postgres, or SQS with the dequeue/ack/nack primitives
3. Worker pool: concurrency management, polling loop, graceful shutdown
4. Job executor: handler dispatch, idempotency check, timeout enforcement
5. Retry and DLQ logic: exponential backoff, max attempts, DLQ enqueue
6. Observability: metrics per layer, DLQ alerting, replay tooling

The complexity in a task queue is not in any single component. It is in the interactions between them when things go wrong: a worker crashes at step three of a five-step job, a downstream API returns 429 under load, a payload schema change lands in production while old jobs are still in the queue. Design each layer to handle failure from the layers around it, and you end up with a system that is boring to operate in the best possible sense.

Pick the storage backend that matches your consistency requirements, not your throughput ambitions. Most applications need Postgres-level correctness more than Redis-level speed. Optimize for the failure modes you will actually hit before optimizing for the throughput ceiling you will probably never reach.