Distributed Locks with Redis: Preventing Double Processing in Job Workers

Background job systems break in surprisingly expensive ways. The most common one is duplicate execution: two workers process the same task, both attempt side effects, and your system ends up with duplicate emails, duplicate payouts, corrupted inventory, or race-condition bugs that are painful to debug.

If you are building with multiple workers, autoscaling, retries, and at-least-once delivery, duplicate processing is not an edge case. It is expected behavior unless you design against it.

This guide covers a production approach for Redis-backed distributed locks that actually holds up under real failure conditions.

Why Duplicate Processing Happens

In distributed systems, work assignment and work execution are separated by network calls, queues, and worker crashes. That creates windows where two consumers can believe they own the same task.

Typical causes:

• Queue visibility timeout expires before a slow worker finishes.
• Worker crashes after side effect but before ack.
• Retry policy re-enqueues a task still being processed.
• Two schedulers trigger the same logical job.
• Deployment restarts create handoff races.

A lock can reduce overlap, but lock design is the difference between a safe system and a false sense of safety.

What a Lock Must Guarantee

A useful distributed lock for jobs should provide:

1. Mutual exclusion: one worker performs critical section at a time.
2. Bounded ownership: lock expires if owner dies.
3. Safe release: only owner can release.
4. Renewal support: long jobs can extend ownership.
5. Stale owner protection: if ownership changes, old owner cannot continue writing.

If you skip #5, you can still get corruption during network partitions and long GC pauses.

Minimal Correct Redis Lock

The basic acquire operation is:

SET lock:job:{jobId} {ownerId} NX PX 30000

• NX: create only if absent.
• PX 30000: expire in 30 seconds.
• ownerId: unique token for this worker attempt.

Safe release requires checking ownership atomically with Lua:

if redis.call('get', KEYS[1]) == ARGV[1] then
  return redis.call('del', KEYS[1])
else
  return 0
end

This prevents worker B from deleting a lock currently owned by worker C.

That is the minimum. It is still not enough for robust production workloads.

Failure Modes You Must Handle

1) Worker pauses longer than TTL

A worker can freeze (CPU starvation, GC, I/O stalls). TTL expires, another worker acquires the lock, then the old worker resumes and continues side effects.

2) Network partition

Worker cannot renew lock due to transient network issue, ownership moves, then partition heals and stale worker resumes writes.

3) Long-running jobs

Fixed TTL is either too short (false lock loss) or too long (slow recovery after crash).

4) Clock assumptions

Do not rely on local machine clocks for correctness decisions. Use Redis state and monotonic owner tokens.

Production Pattern: Lock + Heartbeat + Fencing Token

Use three mechanisms together:

1. Lock key with TTL for ownership.
2. Heartbeat renewal while processing.
3. Fencing token (monotonic number) attached to writes.

Fencing token is critical. Each new lock acquisition gets a higher token. Downstream writes reject stale tokens.

Example policy:

• Worker A acquires token 41.
• Lock expires, Worker B acquires token 42.
• Any write from A with token 41 is rejected.

Now stale workers cannot corrupt state even if they continue running.

Data Model in Redis

Use these keys:

• lock:job:{jobId} -> {ownerId} with TTL
• lock:job:{jobId}:token -> integer counter for fencing token

Acquire flow:

1. SET NX PX lock key.
2. If success, INCR token key.
3. Return {ownerId, token, ttlMs}.

Renew flow:

• Lua script checks GET lockKey == ownerId; if true, extends TTL via PEXPIRE.

Release flow:

• Lua compare-and-delete.

TypeScript Implementation

import { randomUUID } from "node:crypto";
import { Redis } from "ioredis";

type LockHandle = {
  key: string;
  ownerId: string;
  fencingToken: number;
  ttlMs: number;
};

export class RedisJobLock {
  constructor(private redis: Redis) {}

  async acquire(jobId: string, ttlMs = 30_000): Promise<LockHandle | null> {
    const key = `lock:job:${jobId}`;
    const ownerId = randomUUID();

    const acquired = await this.redis.set(key, ownerId, "PX", ttlMs, "NX");
    if (acquired !== "OK") return null;

    const tokenKey = `${key}:token`;
    const fencingToken = await this.redis.incr(tokenKey);

    return { key, ownerId, fencingToken, ttlMs };
  }

  async renew(handle: LockHandle): Promise<boolean> {
    const script = `
      if redis.call('get', KEYS[1]) == ARGV[1] then
        return redis.call('pexpire', KEYS[1], ARGV[2])
      else
        return 0
      end
    `;

    const result = await this.redis.eval(
      script,
      1,
      handle.key,
      handle.ownerId,
      String(handle.ttlMs)
    );

    return Number(result) === 1;
  }

  async release(handle: LockHandle): Promise<boolean> {
    const script = `
      if redis.call('get', KEYS[1]) == ARGV[1] then
        return redis.call('del', KEYS[1])
      else
        return 0
      end
    `;

    const result = await this.redis.eval(script, 1, handle.key, handle.ownerId);
    return Number(result) === 1;
  }
}

Worker Execution Pattern

Use a watchdog timer that renews at 1/3 of TTL, and abort if renewal fails.

type JobPayload = {
  jobId: string;
  userId: string;
  amountCents: number;
};

export async function processJob(payload: JobPayload, lockClient: RedisJobLock) {
  const lock = await lockClient.acquire(payload.jobId, 30_000);
  if (!lock) return { status: "skipped", reason: "lock-not-acquired" };

  let alive = true;
  const heartbeat = setInterval(async () => {
    try {
      const ok = await lockClient.renew(lock);
      if (!ok) alive = false;
    } catch {
      alive = false;
    }
  }, 10_000);

  try {
    // Guard every critical side effect with alive check
    if (!alive) throw new Error("Lost lock before execution");

    await writePaymentLedger({
      jobId: payload.jobId,
      userId: payload.userId,
      amountCents: payload.amountCents,
      fencingToken: lock.fencingToken,
    });

    if (!alive) throw new Error("Lost lock before completion");
    return { status: "done" };
  } finally {
    clearInterval(heartbeat);
    await lockClient.release(lock);
  }
}

Enforce Fencing at the Database Layer

Fencing tokens are only useful if downstream state changes validate them.

For SQL tables, keep last_fencing_token and reject smaller values:

UPDATE payouts
SET status = 'processed',
    processed_at = NOW(),
    last_fencing_token = $new_token
WHERE id = $payout_id
  AND $new_token > COALESCE(last_fencing_token, 0);

If rows_affected = 0, this worker is stale and must stop.

For event streams, include token in event metadata and enforce monotonic tokens per aggregate.

Locking vs Idempotency: Use Both

A lock reduces concurrent execution overlap. Idempotency protects side effects against retries and replays.

Do not choose one.

• Lock: concurrency control.
• Idempotency key: duplicate side effect protection.

For payment jobs, store idempotency key as payment:{jobId} in your payment provider and internal ledger.

Operational Tuning Guidelines

TTL selection

Start with:

• ttlMs = p99(job_duration) * 2
• renewal interval = ttlMs / 3

Then tune after observing renewal failures and worker pause behavior.

Backoff on lock miss

If lock not acquired:

• short jittered retry for low-latency flows
• requeue for batch systems
• dead-letter after max attempts with clear reason

Metrics to track

Minimum lock dashboard:

• lock acquire success rate
• lock acquire latency
• renewal failure count
• stale write rejection count (fencing)
• duplicate side effect incidents

Alert on spikes in stale write rejection and renewal failures. That often indicates cluster pressure or network instability.

Tradeoffs

Approach	Strengths	Weaknesses	Best Use
Redis lock only	Simple, fast	Stale owner writes still possible	Low-risk tasks without hard side effects
Redis lock + renewal	Handles long jobs	Still vulnerable to stale writes	Moderate-risk pipelines
Redis lock + renewal + fencing	Strong correctness under failure	More implementation effort	Payments, billing, inventory, user-visible critical jobs
Queue-level dedup only	Easy with managed queues	Time-window based, not full mutual exclusion	Non-critical async notifications

Common Mistakes

1. Releasing locks without owner check.
2. Using one static owner ID per process instead of per attempt.
3. No heartbeat for long jobs.
4. Assuming lock implies exactly-once processing.
5. Forgetting fencing checks in downstream data store.

Practical Rollout Plan

1. Add lock library with safe acquire/renew/release.
2. Add fencing token field to critical tables.
3. Gate critical writes on monotonic token.
4. Add lock and stale-write metrics.
5. Enable by queue or job type with feature flags.
6. Run shadow mode first: log stale token rejections without blocking for 24 hours.
7. Switch to hard enforcement.

Final Takeaway

Distributed locks with Redis are useful, but only when designed for real failures.

If you implement only SET NX PX, you solved the easy part and left the dangerous part open. For production systems with real money or core user state, use lock renewal and fencing tokens together, then enforce token monotonicity where writes happen.

That combination gives you a system that fails safely instead of failing silently.