Designing a Change Data Capture Pipeline: Log-Based CDC, Debezium, and Real-Time Data Synchronization

Your analytics database is 20 minutes behind production. Your search index has stale records. Your data warehouse misses updates because the polling job failed silently overnight. You can timestamp-filter your way through this for a while, but eventually polling cannot keep pace with write volume, and the gaps become business problems.

This is where change data capture (CDC) earns its complexity budget.

CDC captures every committed change from a database and streams it downstream. Not via a cron job that re-scans rows, but by reading the database’s internal write log. The result is low-latency, ordered, complete propagation of changes with no application-code changes required.

This article covers how log-based CDC works, how Debezium fits into the architecture, how to write a production TypeScript consumer, and when to use Debezium versus direct WAL consumption versus a managed service.

Why Polling-Based Sync Fails

The polling approach seems reasonable at first: every N seconds, query for rows where updated_at > last_checked. Problems emerge at scale.

Deletions are invisible. A deleted row has no updated_at to query. You need a separate soft-delete convention, enforced everywhere, with no exceptions. In practice, some team always inserts a direct SQL delete.

Updates that reset timestamps are invisible. Bulk update scripts frequently skip ORM hooks and do not touch updated_at.

Clock skew creates gaps. If your app servers have even a 50ms clock difference, rows written during that window can fall outside your query range.

High-frequency writes create load spikes. A large-batch write followed by a poller scan hits the source database with a full-table read at exactly the moment the database is under the most pressure.

You need to poll all tables separately. Each entity becomes its own polling job, each with its own timing and failure handling. At 50 tables, you have 50 independent latency sources.

Log-based CDC avoids all of these. It reads the database’s write-ahead log (WAL in PostgreSQL, binlog in MySQL), which captures every committed insert, update, and delete with the exact values before and after the change.

How Log-Based CDC Works

PostgreSQL WAL

PostgreSQL’s WAL exists for crash recovery and replication. By default, it uses a compact format that records physical page changes. For CDC, you need logical replication, which outputs row-level change events in a consumable format.

Enable logical replication by setting wal_level = logical in postgresql.conf. Then create a replication slot:

-- Create a logical replication slot using pgoutput (built-in since PG10)
SELECT pg_create_logical_replication_slot('cdc_slot', 'pgoutput');

-- Create a publication covering the tables you want to stream
CREATE PUBLICATION cdc_publication
  FOR TABLE orders, order_items, customers, inventory;

The replication slot acts as a cursor into the WAL. It retains WAL segments until your consumer acknowledges them. This is the core guarantee: you will not miss changes as long as you advance the slot position after processing each batch.

A consumer connects to the slot using the streaming replication protocol and receives BEGIN, RELATION, INSERT, UPDATE, DELETE, and COMMIT messages in commit order.

MySQL Binlog

MySQL’s binlog has served as the replication mechanism for decades. For CDC, set binlog_format = ROW to get full row images instead of SQL statements, and binlog_row_image = FULL to capture both old and new values on updates.

Debezium connects as a replica, authenticates with a replication user, and requests the binlog stream starting from a saved offset (file name + position or GTID).

The key difference from PostgreSQL: MySQL has no concept of slot retention. You must ensure your consumer is alive and progressing, or the binlog position can be flushed by the server’s binlog rotation policy. A Debezium failure of more than a few days against a busy MySQL server can leave you needing a full snapshot to resync.

Debezium Architecture

Debezium is a set of Kafka Connect source connectors. You run it as a Kafka Connect worker, and it manages the connection to your source database, the offset tracking, and the serialization of change events into Kafka topics.

The deployment model looks like this:

Source DB (Postgres/MySQL)
    |
    | replication protocol
    v
Debezium (Kafka Connect worker)
    |
    | produces to topics per table
    v
Kafka
    |
    | consumed by
    v
Downstream systems (search index, analytics DB, cache invalidation)

Each table gets its own topic, named by convention: <connector>.<schema>.<table>. An orders table in the public schema with connector prefix myapp becomes myapp.public.orders.

Each message contains:

{
  "before": { "id": 42, "status": "pending", "total": 5000 },
  "after":  { "id": 42, "status": "paid",    "total": 5000 },
  "source": {
    "ts_ms": 1711871400000,
    "db": "myapp",
    "schema": "public",
    "table": "orders",
    "lsn": 24532992,
    "txId": 89123
  },
  "op": "u"
}

op is c (create), u (update), d (delete), or r (read, during initial snapshot). The before field is null for inserts.

Initial Snapshot

When you start a new Debezium connector, it performs a consistent snapshot of existing rows before switching to streaming. For PostgreSQL, it acquires a table-level lock briefly to establish the consistent point, exports the data, then streams WAL events from that exact LSN. The snapshot is serialized through the same topic, so consumers see a complete ordered view from the start.

For large tables (tens of millions of rows), the snapshot phase can take hours. Plan accordingly. Debezium 2.x supports incremental snapshots via watermarking, which avoids the full lock and allows pausing and resuming.

Building a TypeScript Consumer

A CDC consumer reads change events from Kafka and applies them downstream. The critical properties are idempotency, ordering, and schema awareness.

Event Types

type CdcOperation = "c" | "u" | "d" | "r";

interface CdcSource {
  ts_ms: number;
  db: string;
  schema: string;
  table: string;
  txId?: number;
  lsn?: number;
}

interface CdcEvent<T = Record<string, unknown>> {
  before: T | null;
  after: T | null;
  source: CdcSource;
  op: CdcOperation;
}

interface OrderRecord {
  id: number;
  customer_id: number;
  status: string;
  total_cents: number;
  created_at: string;
  updated_at: string;
}

Consumer Loop with Idempotency

import { Kafka } from "kafkajs";

const kafka = new Kafka({ brokers: ["kafka:9092"], clientId: "cdc-consumer" });
const consumer = kafka.consumer({ groupId: "search-index-sync" });

await consumer.subscribe({ topic: "myapp.public.orders", fromBeginning: false });

await consumer.run({
  eachMessage: async ({ message, topic, partition }) => {
    if (!message.value) {
      // Tombstone message: Debezium delete with key retention
      return;
    }

    const event = JSON.parse(message.value.toString()) as CdcEvent<OrderRecord>;

    await applyChangeEvent(event, {
      offset: message.offset,
      partition,
      topic,
    });
  },
});

async function applyChangeEvent(
  event: CdcEvent<OrderRecord>,
  position: { offset: string; partition: number; topic: string }
) {
  const { op, after, before, source } = event;

  // Idempotency check: skip if we have already processed this LSN
  const alreadyProcessed = await checkProcessedOffset(
    source.table,
    source.lsn ?? 0
  );
  if (alreadyProcessed) return;

  switch (op) {
    case "c":
    case "r":
      if (after) await searchIndex.upsert(after.id, toSearchDoc(after));
      break;
    case "u":
      if (after) await searchIndex.upsert(after.id, toSearchDoc(after));
      break;
    case "d":
      if (before) await searchIndex.delete(before.id);
      break;
  }

  await markOffsetProcessed(source.table, source.lsn ?? 0);
}

The checkProcessedOffset / markOffsetProcessed pair should be backed by your target database, not an in-memory map. Kafka consumer group offsets handle at-least-once delivery at the transport layer, but your downstream writes need their own idempotency layer because Kafka offsets and your application’s processed-event tracking are not atomic.

Ordering Guarantees

Kafka provides ordering within a partition. Debezium routes all events for a given primary key to the same partition by default, so all changes to orders:42 arrive in order.

Cross-table ordering is weaker. If your consumer joins across tables, a transaction that writes to both orders and order_items will appear as separate events on separate topics. They will be close in time and share a txId in the source field, but you cannot assume they arrive in commit order across topics without coordination.

For use cases that require cross-table consistency, consider the transactional outbox pattern: write events to an outbox table inside the application transaction, and CDC that table instead of the domain tables directly. You get single-table ordering with full payload control.

Schema Evolution

This is where most CDC pipelines eventually break.

The Problem

When a developer adds a nullable column to orders, existing Debezium messages do not include it. A consumer deserializing with a strict TypeScript type will either fail or silently drop the field. When a column is renamed, before and after disagree with each other mid-stream.

Debezium Schema Registry Integration

Debezium integrates with Confluent Schema Registry (or compatible registries). Each message includes an Avro or JSON Schema identifier, and the consumer resolves the schema at read time. This allows producers and consumers to evolve independently as long as changes are backward-compatible (adding nullable fields, default values).

import { SchemaRegistry } from "@kafkajs/confluent-schema-registry";

const registry = new SchemaRegistry({ host: "http://schema-registry:8081" });

async function decodeMessage(message: Buffer): Promise<CdcEvent<OrderRecord>> {
  const decoded = await registry.decode(message);
  return decoded as CdcEvent<OrderRecord>;
}

For teams not using Avro, the simpler approach is to version your consumer type defensively:

function toSearchDoc(row: Partial<OrderRecord>): SearchDoc {
  return {
    id: row.id!,
    customerId: row.customer_id!,
    status: row.status ?? "unknown",
    totalCents: row.total_cents ?? 0,
    // New field added 2026-03: nullable, default safely
    shippingAddress: row.shipping_address ?? null,
  };
}

Treat CDC event payloads the same way you treat external API responses: parse defensively, never cast.

Column Renames and Drops

These are breaking changes. A column rename appears as a schema change in the Debezium connector config but also produces a brief window where before has the old name and after has the new name if your migration is not atomic. For destructive migrations, use the expand/contract pattern: add the new column, backfill it, migrate consumers, then drop the old column. CDC streams both columns during the overlap.

CDC vs Direct WAL Consumption vs Managed Services

Dimension	Debezium (Kafka Connect)	Direct WAL (pg_logical)	AWS DMS / GCP Datastream
Operational complexity	Medium (Kafka required)	Low (library in your app)	Low (managed)
Latency	Sub-second	Sub-second	30s-5min typical
Schema registry	Built-in integration	DIY	Limited
Ordering guarantee	Per-key, within topic	Per-slot	Weak cross-table
Multi-target fanout	Kafka handles this	Consumer-side	Single target
Snapshot support	Full and incremental	DIY	Yes
Cost	Infrastructure	Infrastructure	Per-GB processed
PostgreSQL support	Excellent	Native	Yes
MySQL support	Excellent	N/A	Yes
Best for	High-volume, multi-consumer	Simple single-target sync	Cloud-native, low ops

Direct WAL consumption makes sense when you have one consumer, one database, and want to avoid Kafka. Libraries like pg-logical-replication (Node.js) or wal2json give you a replication slot consumer without Debezium’s operational weight.

import { LogicalReplicationService, PgoutputPlugin } from "pg-logical-replication";

const service = new LogicalReplicationService({
  connectionString: process.env.DATABASE_URL,
});

const plugin = new PgoutputPlugin({
  protoVersion: 1,
  publicationNames: ["cdc_publication"],
});

service.on("data", async (lsn: string, log: unknown) => {
  await processWalMessage(log);
  await service.acknowledge(lsn);
});

await service.subscribe(plugin, "cdc_slot");

The acknowledge call advances the replication slot. If your consumer crashes before acknowledging, it will re-receive messages from the last acknowledged LSN. Build your downstream writes to handle this.

AWS DMS is worth using when you are doing a one-time migration or have strong requirements to avoid self-managed infrastructure. The latency is higher and the schema handling is less flexible, but for cloud teams that cannot justify Kafka, it covers most synchronization use cases.

Debezium is the right choice when you have multiple downstream consumers (search index, analytics, cache invalidation, data warehouse), high write volume, and need the full ordered event stream with schema history. Kafka becomes the integration bus that absorbs the complexity of fan-out.

The Outbox Pattern with CDC

The outbox pattern and CDC are frequently combined. Instead of CDC-ing every domain table, you write structured domain events to an outbox_events table inside your application transactions, then CDC that single table.

This gives you:

• Application-controlled payload: the event contains only what you want to expose, not the full row image.
• Single-table ordering: all downstream events flow through one topic, preserving transaction-level ordering.
• Schema stability: the outbox table schema changes rarely; domain table migrations are invisible to consumers.
• Explicit retry semantics: you control what is an event, not the database engine.

The cost is indirection. Every application write path must be aware of the outbox. For greenfield services this is straightforward. For systems with existing writes scattered across ORMs, raw queries, and migration scripts, CDC on domain tables is often the more complete choice because it captures everything, including writes the application does not know about.

Production Considerations

Replication slot lag. If your consumer falls behind, the PostgreSQL replication slot retains WAL segments, causing disk growth on the source database. Monitor pg_replication_slots for confirmed_flush_lsn lag. Alert before the disk fills.

SELECT slot_name,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS lag
FROM pg_replication_slots
WHERE active = true;

Consumer group rebalancing. When a Kafka consumer group rebalances, in-flight messages can be reprocessed. Your idempotency layer must handle this. Partitions reassigned mid-batch will replay from the last committed Kafka offset, not from your application’s last processed LSN.

Table bloat from REPLICA IDENTITY. For Debezium to capture the before image on updates and deletes, PostgreSQL needs REPLICA IDENTITY FULL on each table (or at minimum the primary key, which is the default). FULL causes every update to write the full old row to WAL. On wide tables with frequent updates, this increases WAL volume significantly. Use REPLICA IDENTITY DEFAULT (primary key only) unless you specifically need the before-image for your consumers.

Large transactions. A single database transaction that updates millions of rows produces millions of CDC events. Your consumer must handle this without OOMing. Use bounded batch sizes and back-pressure signals from your downstream system to throttle processing rate.

Schema registry availability. If you use Avro with Schema Registry and the registry becomes unavailable, consumer decoding fails hard. Include schema registry availability in your CDC pipeline’s SLO.

Connector restart semantics. Debezium stores offsets in Kafka’s internal connect-offsets topic. If you restart the connector, it resumes from the stored offset. If you delete and recreate the connector with the same name, it inherits the stored offset. Use distinct connector names for distinct pipelines to avoid offset collisions.

A CDC pipeline that handles these concerns reliably is one of the highest-leverage infrastructure investments in a data-intensive system. You stop writing polling jobs, stop reasoning about updated_at gaps, and stop explaining to stakeholders why the analytics dashboard is behind. The complexity lives in the pipeline, not scattered across every service that wants to sync data.