Designing a Job Scheduling System: Priority Queues, Fair Scheduling, and Failure Recovery at Scale

Most systems start with cron. A cron job fires every minute, scans a database table for pending work, and processes what it finds. This works at small scale. It stops working when you need priorities, fairness across tenants, reliable failure handling, and more than one worker.

This article walks through how production job scheduling systems are built: from the failure modes of naive polling to the algorithms and coordination primitives that make distributed schedulers reliable.

Why Naive Approaches Break Down

The classic pattern looks like this:

// Worker polls every 30 seconds
setInterval(async () => {
  const jobs = await db.query(
    `SELECT * FROM jobs WHERE status = 'pending' ORDER BY created_at LIMIT 10 FOR UPDATE SKIP LOCKED`
  );
  for (const job of jobs) {
    await processJob(job);
  }
}, 30_000);

FOR UPDATE SKIP LOCKED is a genuine improvement over naive polling. It prevents two workers from grabbing the same job. But the problems compound quickly:

No priority model. Every job competes on created_at. A low-priority batch export blocks a user-facing report that arrived later.

Polling latency. A 30-second interval means average latency of 15 seconds before a job starts. Shrink the interval and you hammer the database with empty scans.

No fairness. If one tenant submits 10,000 jobs, they consume all worker capacity. Other tenants starve.

Recovery is ad-hoc. Failed jobs sit in an error state until someone writes a script to retry them, usually inconsistently.

Scaling is coarse. You can add more workers, but they all poll the same table. Lock contention grows. The database becomes the bottleneck.

These aren’t edge cases. They show up as soon as you have multiple job types, multiple tenants, or meaningful traffic.

Priority Queue Implementation

A priority queue gives you a way to express that some work matters more than other work. The simplest implementation uses a sorted set, which Redis natively supports.

import { createClient } from "redis";

const redis = createClient();

type JobPriority = "critical" | "high" | "normal" | "low";

const PRIORITY_SCORES: Record<JobPriority, number> = {
  critical: 0,
  high: 25,
  normal: 50,
  low: 75,
};

async function enqueue(
  queueName: string,
  jobId: string,
  priority: JobPriority
): Promise<void> {
  // Score combines priority band with timestamp for FIFO within a band
  const timestamp = Date.now() / 1e13; // normalize to small fraction
  const score = PRIORITY_SCORES[priority] + timestamp;
  await redis.zAdd(queueName, { score, value: jobId });
}

async function dequeue(queueName: string): Promise<string | null> {
  // Atomically pop the lowest-score (highest-priority) job
  const result = await redis.zPopMin(queueName);
  return result?.value ?? null;
}

The score encodes both priority band and insertion time. Within the same priority band, jobs are FIFO. A critical job always sorts ahead of a normal job, regardless of when it arrived.

This pattern is what BullMQ uses under the hood. Jobs move between sorted sets as they transition through states: waiting, active, completed, failed. The worker atomically moves a job from waiting to active using a Lua script to avoid race conditions.

Fair Scheduling with Weighted Fair Queuing

Priority queues solve the “what runs first” problem. They do not solve the “is anyone starved” problem. In a multi-tenant system, a high-priority flood from one tenant will still block all other tenants.

Weighted Fair Queuing (WFQ) assigns each tenant a weight and guarantees that over any window, each tenant gets throughput proportional to their weight.

A practical implementation uses virtual time. Each tenant tracks a virtual finish time for their last dequeued job:

interface TenantState {
  weight: number;
  virtualFinishTime: number;
  queue: string[]; // pending job IDs
}

class WeightedFairScheduler {
  private tenants = new Map<string, TenantState>();

  addTenant(tenantId: string, weight: number): void {
    this.tenants.set(tenantId, {
      weight,
      virtualFinishTime: 0,
      queue: [],
    });
  }

  enqueue(tenantId: string, jobId: string): void {
    const tenant = this.tenants.get(tenantId);
    if (!tenant) throw new Error(`Unknown tenant: ${tenantId}`);
    tenant.queue.push(jobId);
  }

  dequeue(): { tenantId: string; jobId: string } | null {
    let selected: string | null = null;
    let minVirtualFinish = Infinity;

    for (const [tenantId, tenant] of this.tenants) {
      if (tenant.queue.length === 0) continue;
      if (tenant.virtualFinishTime < minVirtualFinish) {
        minVirtualFinish = tenant.virtualFinishTime;
        selected = tenantId;
      }
    }

    if (!selected) return null;

    const tenant = this.tenants.get(selected)!;
    const jobId = tenant.queue.shift()!;
    // Advance virtual time by the cost of one job unit divided by weight
    tenant.virtualFinishTime += 1 / tenant.weight;

    return { tenantId: selected, jobId };
  }
}

A tenant with weight 2 will, over time, get twice the throughput of a tenant with weight 1. No tenant can starve another indefinitely.

In practice you store this state in Redis (with atomic Lua scripts) rather than in-process, so multiple worker instances share a consistent view. The virtual finish time becomes the score in a sorted set of “eligible tenants”, and you pop the minimum.

Job Lifecycle as a State Machine

Every job moves through states. Modeling this as an explicit state machine prevents invalid transitions and makes recovery deterministic.

type JobStatus =
  | "pending"
  | "scheduled"   // scheduled for future execution
  | "queued"      // in the priority queue, ready for pickup
  | "running"
  | "completed"
  | "failed"
  | "retrying"
  | "dead";       // exhausted retries, in dead letter queue

interface JobTransition {
  from: JobStatus;
  to: JobStatus;
  trigger: string;
}

const VALID_TRANSITIONS: JobTransition[] = [
  { from: "pending",   to: "scheduled", trigger: "schedule" },
  { from: "pending",   to: "queued",    trigger: "enqueue" },
  { from: "scheduled", to: "queued",    trigger: "timer_fired" },
  { from: "queued",    to: "running",   trigger: "worker_claimed" },
  { from: "running",   to: "completed", trigger: "success" },
  { from: "running",   to: "failed",    trigger: "error" },
  { from: "failed",    to: "retrying",  trigger: "retry_scheduled" },
  { from: "retrying",  to: "queued",    trigger: "retry_ready" },
  { from: "failed",    to: "dead",      trigger: "max_retries_exceeded" },
  { from: "retrying",  to: "dead",      trigger: "max_retries_exceeded" },
];

function transition(
  current: JobStatus,
  trigger: string
): JobStatus {
  const match = VALID_TRANSITIONS.find(
    (t) => t.from === current && t.trigger === trigger
  );
  if (!match) {
    throw new Error(`Invalid transition: ${current} + ${trigger}`);
  }
  return match.to;
}

Every status change goes through this function. You never update a completed job to running. You never mark a dead job as retrying. The state machine is the contract.

Persist each transition as an event (not just the current state). This gives you a complete audit trail and makes debugging production failures much easier.

Failure Recovery: Retry, Backoff, and Dead Letter Queues

The naive retry strategy: catch the error, sleep, try again. This causes thundering herds when a downstream service recovers and every queued job hammers it simultaneously.

Exponential backoff with jitter is the correct default:

interface RetryConfig {
  maxAttempts: number;
  baseDelayMs: number;
  maxDelayMs: number;
  jitterFactor: number; // 0.0 to 1.0
}

function computeRetryDelay(attempt: number, config: RetryConfig): number {
  const exponential = config.baseDelayMs * Math.pow(2, attempt - 1);
  const capped = Math.min(exponential, config.maxDelayMs);
  const jitter = capped * config.jitterFactor * Math.random();
  return Math.floor(capped - jitter);
}

async function handleJobFailure(
  job: Job,
  error: Error,
  config: RetryConfig
): Promise<void> {
  const nextAttempt = job.attemptCount + 1;

  if (nextAttempt > config.maxAttempts) {
    await moveToDeadLetterQueue(job, error);
    return;
  }

  const delayMs = computeRetryDelay(nextAttempt, config);
  const retryAt = new Date(Date.now() + delayMs);

  await db.transaction(async (tx) => {
    await tx.jobs.update({
      where: { id: job.id },
      data: {
        status: "retrying",
        attemptCount: nextAttempt,
        lastError: error.message,
        retryAt,
      },
    });
    await tx.jobEvents.create({
      data: { jobId: job.id, event: "retry_scheduled", metadata: { delayMs } },
    });
  });
}

The dead letter queue (DLQ) is not a trash can. It is a queue of jobs that need human review or special handling. Keep the original payload, all error messages, and every retry attempt. You want to be able to replay DLQ jobs after fixing the underlying bug.

For circuit breaking at the job handler level, track error rates per job type and temporarily disable processing when a handler is clearly broken:

interface CircuitState {
  status: "closed" | "open" | "half-open";
  failureCount: number;
  lastFailureAt: Date | null;
  openedAt: Date | null;
}

const CIRCUIT_THRESHOLD = 10;      // failures before opening
const CIRCUIT_TIMEOUT_MS = 60_000; // how long to stay open

function shouldSkipJob(circuit: CircuitState): boolean {
  if (circuit.status === "closed") return false;
  if (circuit.status === "open") {
    const elapsed = Date.now() - circuit.openedAt!.getTime();
    if (elapsed > CIRCUIT_TIMEOUT_MS) {
      // Transition to half-open: allow one probe
      circuit.status = "half-open";
      return false;
    }
    return true;
  }
  // half-open: allow one through
  return false;
}

Celery implements circuit breaking at the worker pool level. BullMQ leaves it to the application. Temporal takes a different approach: workflows are durable by default, so failure recovery is part of the execution model rather than a layer you add on top.

Distributed Coordination: Leases and Fencing Tokens

When multiple workers compete for jobs, you need exactly-once execution semantics. “At-most-once” drops jobs on worker crash. “At-least-once” duplicates them. Getting to exactly-once requires coordination.

The standard approach is a lease: the worker claims a job for a bounded time window, and must renew the lease to keep processing it. If the worker dies, the lease expires and another worker can pick up the job.

interface JobLease {
  jobId: string;
  workerId: string;
  fencingToken: number; // monotonically increasing version
  expiresAt: Date;
}

async function acquireLease(
  jobId: string,
  workerId: string,
  ttlMs: number
): Promise<JobLease | null> {
  const expiresAt = new Date(Date.now() + ttlMs);

  // Atomic compare-and-swap: only succeed if no active lease exists
  const result = await redis.eval(
    `
    local current = redis.call('GET', KEYS[1])
    if current ~= false then return nil end
    local token = redis.call('INCR', KEYS[2])
    redis.call('SET', KEYS[1], ARGV[1], 'PX', ARGV[2])
    return token
    `,
    2,
    `lease:${jobId}`,
    `fencing:${jobId}`,
    workerId,
    ttlMs.toString()
  ) as number | null;

  if (result === null) return null;

  return {
    jobId,
    workerId,
    fencingToken: result,
    expiresAt,
  };
}

The fencing token is the critical piece. When the worker writes the job result, it includes the token. If the lease expired and another worker took over (with a higher token), the original worker’s write is rejected. This prevents the split-brain scenario where two workers both believe they own a job.

async function completeJob(
  jobId: string,
  result: unknown,
  lease: JobLease
): Promise<boolean> {
  // Only write if our fencing token is still current
  const updated = await db.$executeRaw`
    UPDATE jobs
    SET status = 'completed', result = ${JSON.stringify(result)}, completed_at = NOW()
    WHERE id = ${jobId}
      AND fencing_token = ${lease.fencingToken}
      AND status = 'running'
  `;
  return updated > 0;
}

Temporal abstracts this entirely. Workflows are persisted event sequences. A worker that crashes simply re-executes the workflow from the last checkpoint. Fencing tokens and lease renewal become the platform’s problem.

Scaling Patterns

Horizontal scaling is straightforward: add more workers, all reading from the same queue backend. The queue becomes the bottleneck at some point, so choose your backend carefully. Redis with cluster mode handles millions of operations per second. A Postgres queue works for moderate throughput but tops out faster.

Queue partitioning is the next step. Instead of one global queue, you shard by tenant, job type, or region. Workers can specialize: high-memory jobs go to high-memory workers, GPU jobs to GPU workers. BullMQ supports named queues natively. Celery uses queue routing with worker affinity.

Backpressure prevents the queue from growing without bound. Accept rate limiting at the API layer (before enqueue), not at the worker layer. Refusing a job before it’s enqueued is cleaner than having 500,000 jobs sit in the queue for six hours.

async function enqueueWithBackpressure(
  job: JobPayload,
  queueName: string,
  maxQueueDepth: number
): Promise<{ accepted: boolean; queueDepth: number }> {
  const depth = await redis.zCard(queueName);
  if (depth >= maxQueueDepth) {
    return { accepted: false, queueDepth: depth };
  }
  await enqueue(queueName, job.id, job.priority);
  return { accepted: true, queueDepth: depth + 1 };
}

Delayed jobs are a distinct concern. A job scheduled to run in 4 hours should not sit in the active queue for 4 hours consuming a slot. Use a separate sorted set scored by execution timestamp, and a dedicated scheduler process that moves jobs to the active queue when their time arrives. Both BullMQ and Sidekiq use this two-queue model.

Comparing BullMQ, Celery, and Temporal

	BullMQ	Celery	Temporal
Backend	Redis	Redis / RabbitMQ / SQS	Temporal Server (Cassandra / Postgres)
Language	TypeScript / Node	Python (workers in any language)	Go SDK, TypeScript SDK, Java SDK
Durability	Redis persistence	Broker-dependent	Strong: workflows survive worker restarts
Fair scheduling	Manual (multiple queues)	Manual (queue routing)	Not built-in, workflow-level
Distributed coordination	Lua scripts, atomic ops	Celery beat + locks	Native (event sourcing model)
DLQ	Built-in failed queue	Manual or broker feature	Workflow history
Ideal for	Node services, moderate scale	Python-heavy backends	Long-running, complex workflows

BullMQ is the right starting point for most TypeScript services. It handles priority queues, delayed jobs, concurrency limits, and retries out of the box with Redis as the only dependency.

Celery is mature and battle-tested, but the Python coupling is real. If your workers are Python, it’s a natural choice. Otherwise the serialization boundary becomes friction.

Temporal is a different abstraction entirely. You write workflows as code, and Temporal handles durability, retries, and distributed coordination transparently. The tradeoff is operational overhead: you’re running a stateful cluster instead of stateless workers pointing at Redis. For workflows that span hours or days, or that require human approval steps, it’s worth it.

Production Checklist

Before shipping a job scheduling system:

• Idempotent job handlers. Workers will execute some jobs more than once. Write handlers that produce the same result if called twice with the same input.
• Observability per job type. Track enqueue rate, queue depth, processing latency, failure rate, and DLQ growth as separate time series.
• Lease TTL calibrated to actual job duration. A 30-second TTL on a job that takes 25 seconds will cause unnecessary re-execution. A 10-minute TTL on a job that takes 5 seconds delays failure detection.
• DLQ alerting. A DLQ that fills silently is a support ticket waiting to happen. Alert when DLQ depth crosses a threshold.
• Graceful shutdown. Workers must finish in-flight jobs (or release their leases cleanly) before stopping. A SIGTERM handler that calls worker.close() and waits is not optional.
• Schema versioning for job payloads. You will deploy new code while old jobs are in the queue. Version your payloads and handle both old and new formats in the worker.

The patterns here are not novel. They appear, in some form, in every production scheduler. The discipline is in applying them consistently and not skipping steps because things “mostly work” at current load.