Change Data Capture with PostgreSQL and Debezium: Building Reliable Event Pipelines

Every application that updates a database and then does something else — pushes to a search index, invalidates a cache, sends a notification, writes to a data warehouse — faces the same fundamental problem. You’re doing dual-writes: writing to two or more systems, hoping they all succeed, hoping you don’t crash between steps. Eventually, one of those hopes fails you.

Change Data Capture (CDC) flips the model. Instead of writing to multiple places, you write to your database like you normally would. A CDC tool watches the database’s transaction log and streams every committed change to downstream consumers. Your database becomes the source of truth, and everything else derives from it.

Let’s walk through how CDC works with PostgreSQL logical replication, set up a working pipeline with Debezium, and cover the patterns that make CDC reliable in production.

The Dual-Write Problem

Consider a typical e-commerce flow: when an order is placed, you insert a row in PostgreSQL and publish an event to Kafka so the inventory and shipping services can react.

# The dual-write anti-pattern
def place_order(order):
    db.execute(
        "INSERT INTO orders (id, customer_id, total) VALUES (?, ?, ?)",
        (order.id, order.customer_id, order.total)
    )
    # If the process crashes here, the order exists in the DB
    # but no event was ever published. Ghost order.
    kafka_producer.send("orders", key=order.id, value=order.to_dict())

If the process crashes between the database commit and the Kafka publish, you have a committed order that no downstream service knows about. If you publish first and the database insert fails, you have phantom events for orders that don’t exist. There is no correct ordering of these two operations that eliminates the race.

You could wrap both in a distributed transaction, but that couples your message broker to your database with all the performance and availability penalties that implies. CDC sidesteps the problem entirely by reading changes from the database’s own transaction log after they’re committed.

How PostgreSQL Logical Replication Works

PostgreSQL maintains a Write-Ahead Log (WAL) — an append-only sequence of every change made to the database. The WAL exists primarily for crash recovery: if the database goes down, it replays the WAL to restore consistency. But PostgreSQL also supports logical replication, which decodes WAL entries into row-level change events that external consumers can read.

To enable logical replication, three settings must be configured in postgresql.conf:

wal_level = logical
max_wal_senders = 10
max_replication_slots = 10

The logical WAL level adds enough information to reconstruct row-level changes from the binary WAL stream. Replication slots ensure that the database retains WAL segments until every consumer has acknowledged them — preventing data loss if a consumer falls behind.

Setting Up Debezium with PostgreSQL

Debezium is an open-source CDC platform (over 12,800 GitHub stars, Apache 2.0 licensed) that wraps Kafka Connect around database-specific connectors. For PostgreSQL, it uses the pgoutput logical decoding plugin — the same plugin PostgreSQL ships natively for logical replication — so no additional extensions are required beyond creating a replication-enabled role.

First, create a replication slot and grant the necessary privileges:

-- Create a role with replication privileges
CREATE ROLE debezium_user REPLICATION LOGIN PASSWORD 'secure_password';
GRANT USAGE ON SCHEMA public TO debezium_user;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO debezium_user;

-- Debezium needs to identify rows uniquely for incremental snapshots
-- (required for tables without a primary key)
ALTER TABLE orders REPLICA IDENTITY FULL;

Then deploy the connector with a JSON configuration. Here’s a minimal setup that captures changes from an orders table:

{
  "name": "orders-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres.production",
    "database.port": "5432",
    "database.user": "debezium_user",
    "database.password": "secure_password",
    "database.dbname": "shop",
    "topic.prefix": "pg-production",
    "table.include.list": "public.orders",
    "plugin.name": "pgoutput",
    "slot.name": "debezium_orders",
    "publication.name": "dbz_publication",
    "snapshot.mode": "initial"
  }
}

The snapshot.mode setting controls what happens on first startup. initial takes a consistent snapshot of existing data before streaming live changes — so downstream consumers get the full current state plus ongoing updates. Use no_data if you only want new changes going forward.

Understanding Change Events

Every row-level change produces an event with before and after states, operation type, and source metadata. Here’s what a UPDATE on the orders table looks like:

{
  "before": {
    "id": 1001,
    "customer_id": 42,
    "total": "99.95",
    "status": "pending"
  },
  "after": {
    "id": 1001,
    "customer_id": 42,
    "total": "99.95",
    "status": "shipped"
  },
  "source": {
    "version": "3.6.0.Final",
    "connector": "postgresql",
    "name": "pg-production",
    "db": "shop",
    "schema": "public",
    "table": "orders",
    "ts_ms": 1752259200000,
    "lsn": 23456789,
    "txId": 5432
  },
  "op": "u",
  "ts_ms": 1752259200123
}

The op field tells you the operation: c for create, u for update, d for delete, and r for read (snapshot events). The source block contains the WAL Log Sequence Number (LSN) and transaction ID — critical for exactly-once processing and offset management.

Consuming Events in Python

Downstream services consume CDC events from Kafka like any other topic. The key design decision is how to handle idempotency: since consumers can restart and reprocess events, every handler must be safe to execute more than once.

from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    "pg-production.public.orders",
    bootstrap_servers=["kafka:9092"],
    group_id="shipping-service",
    auto_offset_reset="earliest",
    enable_auto_commit=False,
    value_deserializer=lambda m: json.loads(m.decode("utf-8")),
)

for message in consumer:
    event = message.value
    op = event["op"]
    after = event.get("after")

    if op in ("c", "u") and after:
        sync_to_search_index(after)
    elif op == "d":
        before = event["before"]
        delete_from_search_index(before["id"])

    # Manual commit ensures we only advance the offset
    # after successful processing
    consumer.commit()

The Outbox Pattern: Reliable Event Publishing

One of the most common CDC use cases is the Transactional Outbox pattern. Instead of writing to the database and publishing to Kafka (the dual-write problem), you write both the business data and an event record in the same database transaction. Debezium then captures the event row and routes it to Kafka.

-- Both operations succeed or fail together
BEGIN;

INSERT INTO orders (id, customer_id, total, status)
VALUES (1001, 42, 99.95, 'confirmed');

INSERT INTO outbox (id, aggregatetype, aggregateid, type, payload)
VALUES (
  gen_random_uuid(),
  'Order',
  '1001',
  'OrderConfirmed',
  '{"orderId": 1001, "customerId": 42, "total": 99.95}'
);

COMMIT;

Debezium’s Outbox Event Router Single Message Transform (SMT) unwraps the outbox row into a clean event structure — extracting the event type as the Kafka message key and the JSON payload as the message value, while filtering out the outbox table’s mechanical columns. The transform handles the routing logic so downstream consumers receive well-structured domain events instead of raw database rows.

Production Considerations

WAL retention and replication slots. If a consumer falls behind, PostgreSQL retains WAL segments until the replication slot acknowledges them. A stuck consumer can cause WAL to accumulate and fill the disk. Monitor pg_replication_slots and set up alerts for slot lag. If a consumer is permanently offline, drop the slot to let PostgreSQL reclaim space.

-- Check replication slot health
SELECT slot_name, active, restart_lsn,
       pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS lag_bytes
FROM pg_replication_slots;

Schema evolution. When you add a column to a source table, downstream consumers must handle the new field. Debezium publishes schema information with every event (using Avro with a schema registry is the standard approach), so consumers can detect schema changes. The schema.history.internal.kafka.topic setting tracks DDL changes to support connector restarts.

Snapshot consistency. Debezium’s incremental snapshot feature (using watermarks) allows re-snapshotting specific tables without locking them or stopping the connector. This is essential for backfilling new consumers or recovering from data inconsistencies without downtime.

What’s New in Debezium 3.6

The latest Debezium 3.6.0 release (July 2026) introduces Debezium Platform — a management layer with a monitoring dashboard, REST API for pipeline management, and built-in JDBC sink support. A new JBang-based CLI lets you manage sources, destinations, and transformations from the command line. The Quarkus-based runtime extensions now support filtering on column-level changes, allowing fine-grained control over which changes trigger events — useful for high-throughput tables where most updates are irrelevant to downstream consumers.

When to Use CDC (and When Not To)

CDC shines for cache invalidation, search index synchronization, data warehouse loading, audit trails, and microservice integration. It provides eventual consistency between systems with minimal coupling.

It’s not the right tool when you need synchronous notification (use a regular API call), when the source database can’t support logical replication (some managed databases restrict it), or when your change volume is so low that polling is simpler and cheaper. CDC adds infrastructure — Kafka, Kafka Connect, schema registry — that must be operated and monitored.

For systems where data consistency across services matters — and that’s most systems — CDC with PostgreSQL and Debezium eliminates the dual-write problem at its root. The database transaction log becomes your event backbone, and every downstream system derives its state from the same committed source of truth.