The Transactional Outbox Pattern: Reliable Events Without Distributed Transactions

Posted on

Sending a message to a broker after committing a database write sounds trivial: insert the order, publish the “OrderPlaced” event, done. Except the two operations are not atomic. If the process crashes between the commit and the publish, downstream consumers never learn the order exists. If you publish first and the transaction rolls back, consumers react to an order that was never created. This is the dual-write problem, and every system that writes to a database and publishes to a message broker simultaneously has it, whether the team knows it or not.

The usual proposed fixes — distributed transactions like two-phase commit, or “just retry until it works” — each carry serious costs. Two-phase commit couples your database to your broker and destroys availability under load. Naive retries can’t distinguish “the publish failed” from “the commit failed.” The transactional outbox pattern solves the problem with primitives you already have: a database transaction and one extra table. This post walks through the mechanics, a working Go implementation using PostgreSQL’s SKIP LOCKED, and the delivery-semantics gotchas that bite in production.

The Core Idea: Make the Event Part of the Transaction

Instead of publishing to the broker from application code, you insert the event into an outbox table inside the same transaction as your business data. The atomicity guarantee you already trust for your orders now covers event production too: either both the order and the event exist, or neither does. There is no window for the dual-write failure because there is only one commit.

package orders

import (
	"context"
	"database/sql"
	"encoding/json"
	"time"
)

type Order struct {
	ID    string
	Total int
}

func PlaceOrder(ctx context.Context, db *sql.DB, order Order) error {
	tx, err := db.BeginTx(ctx, nil)
	if err != nil {
		return err
	}
	defer tx.Rollback()

	_, err = tx.ExecContext(ctx,
		`INSERT INTO orders (id, total, status) VALUES ($1, $2, 'placed')`,
		order.ID, order.Total)
	if err != nil {
		return err
	}

	payload, err := json.Marshal(map[string]any{
		"order_id": order.ID,
		"total":    order.Total,
	})
	if err != nil {
		return err
	}

	_, err = tx.ExecContext(ctx,
		`INSERT INTO outbox (aggregate_id, type, payload, created_at)
		 VALUES ($1, $2, $3, $4)`,
		order.ID, "OrderPlaced", payload, time.Now().UTC())
	if err != nil {
		return err
	}

	return tx.Commit()
}

The matching table schema is deliberately boring — an identity column for ordering, the aggregate it belongs to, an event type, the payload, and a nullable published_at timestamp that doubles as the delivery marker:

CREATE TABLE outbox (
    id           BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    aggregate_id TEXT        NOT NULL,
    type         TEXT        NOT NULL,
    payload      JSONB       NOT NULL,
    created_at   TIMESTAMPTZ NOT NULL,
    published_at TIMESTAMPTZ
);

CREATE INDEX idx_outbox_unpublished
    ON outbox (id) WHERE published_at IS NULL;

The Relay: Moving Events From Table to Broker

Something still has to deliver those rows to the broker. Two mainstream options exist: a polling relay (a separate process or goroutine that reads unpublished rows and publishes them) and change-data-capture, where a tool like Debezium tails the database’s write-ahead log and emits the outbox inserts as Kafka records with no polling loop at all.

The polling relay is easier to reason about and has no infrastructure beyond your existing database. The trick to running multiple relay instances safely is FOR UPDATE SKIP LOCKED: each worker locks only the rows it is processing, and concurrent workers skip past locked rows instead of blocking. Two relays can run side by side without stepping on each other or publishing the same row twice concurrently.

package orders

import (
	"context"
	"database/sql"
)

type Publisher interface {
	Publish(ctx context.Context, key string, payload []byte) error
}

type event struct {
	id          int64
	aggregateID string
	payload     []byte
}

func RunRelay(ctx context.Context, db *sql.DB, pub Publisher) error {
	tx, err := db.BeginTx(ctx, nil)
	if err != nil {
		return err
	}
	defer tx.Rollback()

	rows, err := tx.QueryContext(ctx, `
		SELECT id, aggregate_id, payload
		FROM outbox
		WHERE published_at IS NULL
		ORDER BY id
		LIMIT 100
		FOR UPDATE SKIP LOCKED`)
	if err != nil {
		return err
	}

	var batch []event
	for rows.Next() {
		var e event
		if err := rows.Scan(&e.id, &e.aggregateID, &e.payload); err != nil {
			return err
		}
		batch = append(batch, e)
	}
	if err := rows.Err(); err != nil {
		return err
	}
	rows.Close()

	for _, e := range batch {
		if err := pub.Publish(ctx, e.aggregateID, e.payload); err != nil {
			return err
		}
		if _, err := tx.ExecContext(ctx,
			`UPDATE outbox SET published_at = now() WHERE id = $1`, e.id); err != nil {
			return err
		}
	}

	return tx.Commit()
}

Note what this loop guarantees and what it does not. The relay publishes each message and marks the row in the same transaction. If the process dies after Publish succeeds but before the commit rolls in, the row stays unpublished and a later attempt publishes it again. The outbox gives you at-least-once delivery to the broker — never at-most-once. That is the correct trade: a duplicate event is cheap to handle, a lost event is usually not.

Polling Versus Change Data Capture

Polling’s main weakness is latency and load coupling: the relay polls on an interval, and each poll scans the table. With the partial index above, the scan is cheap, and sub-second delivery is achievable with a tight interval — but you are paying query overhead even when the queue is empty. CDC flips that equation. Debezium reads the WAL once, streams each outbox insert as a message, and adds near-zero load at idle. The outbox event router even promotes payload fields into Kafka message keys and headers, so partitioning by aggregate ID falls out of the configuration.

The cost of CDC is operational: you run and monitor Kafka Connect (or an equivalent WAL reader), and you inherit its failure modes. A reasonable default for most teams is to start with a polling relay — it is a few dozen lines and one deployment — and move to CDC when latency requirements or outbox volume justify running the extra infrastructure. Because the outbox table is the contract, swapping the relay for CDC later changes nothing about your write path.

Idempotent Consumers Are Part of the Pattern

Since at-least-once is the contract, consumers must tolerate duplicates. The standard mechanism is a consumer-side deduplication table keyed on a message ID, checked in the same transaction as the consumer’s state change. If the insert of the message ID conflicts, the event was already processed and the consumer acknowledges without doing work again.

  • Give every outbox row a globally unique event ID (a UUID column works) and propagate it as the broker message key or header.
  • Make consumers transactional with respect to dedup: mark processed and apply effects in one commit.
  • Design event payloads to be replayable. A consumer that can safely process the same OrderPlaced twice is one that can be redeployed, rewound, or recovered without ceremony.

The Gotchas That Bite in Production

Cleanup. Rows with published_at set are dead weight forever unless you delete them. A scheduled job that removes rows older than a retention window keeps the table and its indexes small. Keep the window long enough to support replay and debugging — days, not minutes.

Ordering. The outbox guarantees a total order per table (by id), but your broker almost certainly does not. Partition by aggregate ID so all events for one order traverse the same Kafka partition, and accept that events for different aggregates interleave freely. If you need strict global ordering, the outbox will not give it to you at any scale worth having.

Schema evolution. Outbox payloads are serialized the moment the business transaction commits, but they may be read months later. Version your payload format from day one — even a simple version field in the JSON — and prefer additive changes. Consumers locked on an old schema will still be replaying last quarter’s events while you deploy the new one.

Hot aggregates. A single busy aggregate becomes a single busy partition. Partitioning by aggregate ID gives you ordering per entity at the cost of throughput per entity; if one entity genuinely exceeds a partition’s throughput, the fix is domain modeling (split the aggregate), not infrastructure.

When Not to Use It

The outbox pattern earns its complexity when the event must eventually reach consumers and the write must not be lost — order lifecycles, payments, inventory. If the notification is advisory (“a dashboard counter can miss one”), publishing after commit with an alert on failure is simpler and fine. And if you do not have a broker at all — say, a monolith with an in-process event bus — the dual-write problem may not exist yet, because both sides already share the database transaction. Introduce the outbox when the second system of record appears, not before.

The transactional outbox is one of those patterns that looks like ceremony until the first lost event, after which it looks like the minimum viable design. One table, one relay loop, idempotent consumers — all built from guarantees your database already provides. Start with the polling implementation, instrument delivery lag, and let real volume tell you when to graduate to log-based capture.

Leave a Reply

Your email address will not be published. Required fields are marked *