A message arrives that your consumer cannot parse. Maybe the producer shipped a new schema without telling anyone, maybe the payload is genuinely corrupt, maybe it hits a downstream API that rejects it with a 422. What happens next decides whether you have an outage or an incident ticket. If your consumer re-raises the exception, the offset never advances, and every subsequent message behind the poison pill queues up behind it. If your consumer swallows the error, you silently lose data. Both failure modes are common in production, and both are avoidable with one pattern: the dead letter queue.
A dead letter queue (DLQ) is a side topic where messages go when processing fails permanently. Instead of blocking the partition or dropping the message, the consumer routes the failure somewhere it can be inspected, alerted on, and replayed later. The idea sounds trivial — catch the error, publish elsewhere, commit the offset — but the production-grade version has sharp edges: infinite retry loops, ordering loss on replay, and DLQ topics that fill up unnoticed until someone needs the data. This post walks through a Go consumer that gets the basics right, then covers the failure modes that break DLQ implementations in real systems.
Why Kafka Has No Built-In DLQ
Unlike some messaging systems, Kafka does not ship a native dead letter mechanism for regular consumers. The broker delivers messages and tracks offsets; what your consumer does with a malformed payload is entirely your code’s problem. The one exception is Kafka Connect, which supports dead letter queues through connector configuration:
errors.tolerance=all
errors.deadletterqueue.topic.name=orders.dlq
errors.deadletterqueue.context.headers.enable=true
errors.deadletterqueue.topic.replication.factor=3For everything else — plain consumers, Kafka Streams topologies (which do offer DeserializationExceptionHandler support), and application-level processing failures — you implement the pattern yourself. That is not a deficiency. A DLQ policy encodes business decisions: which errors are retryable, how long to wait, when to give up. A generic broker feature could not make those calls for you.
Classify Errors Before You Route Anything
The single most important design decision is distinguishing transient failures from permanent ones. A network timeout deserves a retry. A payload that fails schema validation will fail identically every time — retrying it is pure waste. Route them differently:
- Transient — downstream timeouts, connection resets, temporary unavailability. Retry with backoff first; DLQ only after the budget is exhausted.
- Permanent — deserialization failures, schema violations, business-rule rejections (negative quantity on an order). Go straight to the DLQ. Retrying changes nothing.
The usual way to encode this in Go is a sentinel error or a custom error type that the handler returns and the consumer loop inspects:
package processing
import "errors"
// ErrPermanent marks failures that retrying will never fix:
// bad payloads, schema violations, rejected business rules.
var ErrPermanent = errors.New("permanent failure")
func HandleOrder(payload []byte) error {
order, err := parseOrder(payload)
if err != nil {
// A payload that cannot be parsed will never parse.
return errors.Join(ErrPermanent, err)
}
if order.Quantity <= 0 {
return errors.Join(ErrPermanent, errors.New("non-positive quantity"))
}
return chargeAndFulfill(order)
}With classification in place, the consumer loop becomes a router: permanent errors go to the DLQ immediately, transient errors go through a bounded retry before they do.
A Bounded Retry Consumer in Go
The consumer below processes records from a topic, retries transient failures up to a limit, and publishes anything that still fails to orders.dlq. Enrichment headers preserve the original topic, partition, offset, and error message so the DLQ record is self-describing:
package consumer
import (
"context"
"errors"
"log/slog"
"strconv"
"time"
"github.com/segmentio/kafka-go"
"yourapp/processing"
)
const (
mainTopic = "orders"
dlqTopic = "orders.dlq"
maxRetry = 3
)
type Consumer struct {
reader *kafka.Reader
writer *kafka.Writer
log *slog.Logger
}
func (c *Consumer) Run(ctx context.Context) error {
for {
m, err := c.reader.FetchMessage(ctx)
if err != nil {
return err
}
c.process(ctx, m)
// Commit regardless of outcome: the DLQ, not the offset,
// is now responsible for this message.
if err := c.reader.CommitMessages(ctx, m); err != nil {
return err
}
}
}
func (c *Consumer) process(ctx context.Context, m kafka.Message) {
var err error
for attempt := 0; attempt < maxRetry; attempt++ {
err = processing.HandleOrder(m.Value)
if err == nil {
return
}
if errors.Is(err, processing.ErrPermanent) {
break // retrying cannot help
}
time.Sleep(backoff(attempt))
}
if dlqErr := c.toDLQ(ctx, m, err); dlqErr != nil {
// Publishing to the DLQ failed. Do not commit; the
// message will be redelivered and we try again.
c.log.Error("dlq publish failed", "offset", m.Offset, "err", dlqErr)
return
}
c.log.Warn("sent to dlq",
"topic", m.Topic, "partition", m.Partition,
"offset", m.Offset, "err", err)
}
func (c *Consumer) toDLQ(ctx context.Context, m kafka.Message, cause error) error {
dlqMsg := kafka.Message{
Topic: dlqTopic,
Key: m.Key,
Value: m.Value,
Headers: append(kafka.Header{}, m.Headers...,
kafka.Header{Key: "dlq.original.topic", Value: []byte(m.Topic)},
kafka.Header{Key: "dlq.original.partition", Value: []byte(strconv.Itoa(m.Partition))},
kafka.Header{Key: "dlq.original.offset", Value: []byte(strconv.FormatInt(m.Offset, 10))},
kafka.Header{Key: "dlq.error", Value: []byte(cause.Error())},
kafka.Header{Key: "dlq.timestamp", Value: []byte(time.Now().UTC().Format(time.RFC3339))},
),
}
return c.writer.WriteMessages(ctx, dlqMsg)
}
func backoff(attempt int) time.Duration {
// 1s, 2s, 4s... capped to keep a stuck consumer responsive.
d := time.Second << attempt
if d > 30*time.Second {
d = 30 * time.Second
}
return d
}Three details in this loop matter more than they look. First, the offset is committed even for DLQ’d messages — the dead letter topic is the durable record now, and blocking the partition on a poison pill is exactly what the pattern exists to prevent. Second, if the DLQ publish itself fails, the offset is not committed and the message gets redelivered; the worst outcome is a message that is neither processed nor parked. Third, the enrichment headers carry everything a future replay needs: where the message came from and why it failed. Without them, a DLQ full of opaque payloads is an archive of mysteries.
Do Not Build an Infinite Loop by Accident
The classic DLQ failure is a retry topology that feeds itself. A consumer on orders fails a message and routes it to orders.retry. A second consumer on orders.retry processes it, fails again, and routes it back to orders.retry — or worse, back to the main topic. Each hop is individually sensible; together they form a loop that churns forever, burning broker traffic and hiding a genuinely broken message behind a wall of retries.
Two defenses prevent this. First, carry a retry-count header and enforce a hard maximum: once the count hits the cap, the next hop is the DLQ, never another retry topic. Second, keep the direction one-way — retry topics may forward toward the DLQ, never back toward the main topic. Replay from the DLQ is a deliberate, human-initiated action with its own guardrails, not an automated hop.
Delays, Ordering, and the Cost of Parking Messages
Kafka has no built-in delayed delivery. If you want “retry after five minutes,” you either sleep in the consumer (blocking the partition — fine for seconds, terrible for minutes) or use a tiered topic scheme where each tier’s consumer delays before forwarding: orders.retry-5m, orders.retry-30m, then the DLQ. Tiered schemes work, but be aware of the trade: rerouting a message to a different topic breaks key ordering. If your downstream expects per-key sequencing — order updates for the same customer, for instance — a message sitting in a retry tier while its successors process on the main topic can arrive out of order after replay. Design downstream handlers to tolerate that, or serialize per-key state through the same topology.
Operating the DLQ: Alerts, Capacity, Replay
An unmonitored DLQ is a data graveyard. Three operational habits keep it useful:
- Alert on lag and arrival rate. A sudden burst into the DLQ usually means a producer deploy went out with a schema change. The DLQ growth rate is often your fastest signal that an upstream contract broke.
- Set retention deliberately. DLQ topics inherit the default retention policy unless you override it. Park messages for repair, not forever — pick a retention window your team can realistically act within and make replay tooling part of the runbook.
- Replay with intent. When replaying, always run the current handler code and deduplicate on idempotency keys — the original failure may have been fixed days ago, and side effects from the first attempt may or may not have landed. Blind replay of a full DLQ is how double charges happen.
The DLQ pairs naturally with the transactional outbox pattern on the producing side: the outbox guarantees messages get published reliably, and the DLQ guarantees failures get handled honestly once they arrive. Together they cover both halves of reliable event flow.
Wrapping Up
A dead letter queue is a small amount of code that buys a large property: no single bad message can stop your pipeline, and no failure goes silently missing. The implementation checklist is short — classify errors as transient or permanent, bound your retries, enrich DLQ records with origin and cause headers, keep retry flow one-directional, and alert on DLQ traffic. If your current consumer just logs the error and moves on, a poison pill is already a data-loss event waiting for its moment. The Kafka protocol gives you the primitives; the failure policy is yours to write.