Every production database tells a continuous story — orders placed, statuses updated, accounts created. But most applications treat their database as a black box, reading its current state without any awareness of the stream of changes that produced it. Change Data Capture (CDC) flips that assumption. Instead of periodically polling tables or building fragile dual-write pipelines, CDC turns your database’s transaction log into a reliable event stream that downstream services can consume in real time.
Debezium is the most widely used open-source CDC platform in the JVM ecosystem. It’s built on top of Kafka Connect and reads database transaction logs — PostgreSQL’s WAL, MySQL’s binlog, MongoDB’s oplog — capturing row-level changes as structured events without imposing any query load on your source database. The project recently shipped version 3.6.0, which introduced the Debezium Platform management layer with a monitoring dashboard and a JBang-based CLI that simplifies pipeline management considerably.
Let’s walk through setting up Debezium with PostgreSQL, understanding the event format, and building a consumer that reacts to changes in real time.
How CDC Actually Works
Traditional data synchronization falls into two camps. Polling periodically queries the source database for changes — it’s simple but creates load spikes, misses deletes, and struggles with timing gaps. Dual writes have the application code write to both the database and a message queue — but this creates a distributed consistency problem where one write can succeed while the other fails, leaving your systems out of sync.
CDC solves this by reading the database’s own transaction log, the authoritative record of every committed change. PostgreSQL uses a Write-Ahead Log (WAL) to ensure durability — every modification is written to the WAL before it’s applied to the data files. Debezium creates a logical replication slot that reads this WAL stream and transforms each row-level change into a structured event published to a Kafka topic.
The key advantage: the database is the single source of truth. Your application writes to one place. CDC handles the rest asynchronously, with exactly-once delivery semantics backed by Kafka’s offset tracking.
Configuring PostgreSQL for Logical Replication
PostgreSQL needs a few configuration changes to support logical replication. In postgresql.conf:
wal_level = logical
max_wal_senders = 10
max_replication_slots = 10Debezium uses the pgoutput plugin, which is built into PostgreSQL — no additional extensions required. You’ll also need a replication role:
CREATE ROLE debezium_user REPLICATION LOGIN PASSWORD 'secret';
GRANT USAGE ON SCHEMA public TO debezium_user;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO debezium_user;For tables without a primary key, set REPLICA IDENTITY FULL so that Debezium can capture the full row image for updates and deletes during incremental snapshots:
ALTER TABLE orders REPLICA IDENTITY FULL;Registering the PostgreSQL Connector
Debezium connectors are deployed as Kafka Connect plugins. Once Kafka Connect is running, you register the PostgreSQL connector by posting a JSON configuration to the Connect REST API:
curl -X POST http://localhost:8083/connectors \
-H "Content-Type: application/json" \
-d '{
"name": "postgres-orders-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"topic.prefix": "orders_pg",
"database.hostname": "localhost",
"database.port": "5432",
"database.user": "debezium_user",
"database.password": "secret",
"database.dbname": "shop",
"plugin.name": "pgoutput",
"slot.name": "debezium_slot",
"publication.name": "dbz_pub",
"table.include.list": "public.orders",
"snapshot.mode": "initial"
}
}'A few properties deserve attention:
topic.prefix— Sets the Kafka topic prefix. Each captured table gets a topic named{prefix}.{schema}.{table}, so theorderstable produces events onorders_pg.public.orders.snapshot.mode—initialtakes a full snapshot of existing data before streaming live changes. Useno_datato skip the snapshot and stream only new changes.table.include.list— Filters which tables to capture, inschema.tableformat.
Understanding Change Events
Every row change produces a structured event. Here’s what an UPDATE on the orders table looks like when using a JSON converter with schemas.enable=false:
{
"before": {"id": 1001, "status": "pending", "total": 42.50},
"after": {"id": 1001, "status": "shipped", "total": 42.50},
"source": {
"version": "3.6.0.Final",
"connector": "postgresql",
"db": "shop",
"schema": "public",
"table": "orders",
"ts_ms": 1752259200123,
"lsn": 5123456,
"txId": 98765
},
"op": "u",
"ts_ms": 1752259200123
}The op field tells you what happened: c for create, u for update, d for delete, and r for read (emitted during the initial snapshot). The before and after blocks contain the full row state before and after the change. The source block includes the Log Sequence Number (lsn) and transaction ID (txId), which are essential for exactly-once processing and offset management.
Consuming Change Events
When consuming events with a JSON converter, the fields op, before, after, source, and ts_ms are top-level keys — there is no payload wrapper. Here’s a Python consumer that processes order status changes and reacts to shipped orders:
import json
from kafka import KafkaConsumer
consumer = KafkaConsumer(
"orders_pg.public.orders",
bootstrap_servers=["localhost:9092"],
auto_offset_reset="earliest",
group_id="order-processor",
value_deserializer=lambda m: json.loads(m.decode("utf-8")),
)
for message in consumer:
event = message.value
operation = event["op"]
after = event.get("after")
if operation in ("c", "u"):
if after:
order_id = after["id"]
status = after["status"]
if status == "shipped":
print(f"Order {order_id} shipped — triggering fulfillment")
# Update search index, send notification, sync warehouse system
elif operation == "d":
before = event.get("before")
if before:
print(f"Order {before['id']} deleted — cleaning up")
elif operation == "r":
if after:
print(f"Snapshot record: order {after['id']}")For Go services, the pattern is the same — deserialize the JSON, read op and after directly:
type ChangeEvent struct {
Before map[string]interface{} `json:"before"`
After map[string]interface{} `json:"after"`
Source SourceMetadata `json:"source"`
Op string `json:"op"`
TsMs int64 `json:"ts_ms"`
}
type SourceMetadata struct {
Version string `json:"version"`
DB string `json:"db"`
Schema string `json:"schema"`
Table string `json:"table"`
LSN int64 `json:"lsn"`
TxID int64 `json:"txId"`
}
func handleEvent(event ChangeEvent) {
switch event.Op {
case "c", "u":
if event.After["status"] == "shipped" {
log.Printf("Order %v shipped", event.After["id"])
}
case "d":
if event.Before != nil {
log.Printf("Order %v deleted", event.Before["id"])
}
}
}The Outbox Pattern: Clean Domain Events
CDC captures every column change, but downstream consumers often don’t care about field-level updates — they want domain events like “OrderShipped” or “PaymentProcessed”. The Outbox Event Router Single Message Transform solves this by reading from a dedicated outbox table within the same transaction as your business write, then routing those rows as clean Kafka messages.
Your application writes to both the business table and the outbox table in a single database transaction:
BEGIN;
UPDATE orders SET status = 'shipped' WHERE id = 1001;
INSERT INTO outbox (event_id, aggregatetype, aggregateid, type, payload)
VALUES (
gen_random_uuid(),
'Order',
'1001',
'OrderShipped',
'{"order_id": 1001, "shipped_at": "2026-08-01T12:00:00Z"}'
);
COMMIT;Debezium’s Outbox Event Router SMT then unwraps the outbox row into a clean Kafka message: the type column becomes the message key, and the payload column becomes the message value. All the mechanical outbox columns (event_id, aggregatetype) are stripped out. Downstream consumers receive a clean OrderShipped event instead of a raw column-level change.
Monitoring Replication Slot Health
One of the most common operational issues with CDC is WAL accumulation. When a Kafka consumer falls behind or goes offline, the PostgreSQL replication slot retains WAL segments until the consumer catches up. If left unchecked, this can fill up the database disk and take down the entire database. Monitor slot health regularly:
SELECT slot_name,
active,
restart_lsn,
pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS lag_bytes
FROM pg_replication_slots;If a consumer is permanently offline, drop the replication slot to release the accumulated WAL:
SELECT pg_drop_replication_slot('debezium_slot');Wrapping Up
Change Data Capture fundamentally changes how you think about data movement. Instead of building fragile polling jobs or accepting dual-write inconsistency, you get an asynchronous, reliable event stream derived from your database’s own transaction log. Debezium makes this practical at scale, with connector support for PostgreSQL, MySQL, MongoDB, and more.
The patterns worth remembering: configure logical replication properly, understand the flat event structure when consuming with JSON converters, use the Outbox Event Router for clean domain events, and always monitor replication slot health. With those foundations, CDC becomes a reliable backbone for cache invalidation, search index synchronization, audit logging, and real-time analytics pipelines — all without adding query load to your source database.