Networks are unreliable. Clients retry requests. Messages get delivered twice. In a distributed system, the question isn’t whether a duplicate will arrive — it’s whether your system handles it gracefully. That’s idempotency: the property that processing the same request multiple times produces the same result as processing it once.
Most REST APIs get this wrong. A user clicks “Place Order,” the request times out, they click again, and now they have two orders. A payment webhook arrives, your server processes it, acknowledges it, but the payment provider never received the ACK and retries — double charge. These aren’t edge cases. They’re routine failures in any production system.
This post walks through how to implement idempotency correctly in Go, from the fundamental pattern to a production-ready middleware that handles the common pitfalls most developers miss.
The Idempotency-Key Pattern
The standard approach is the Idempotency-Key pattern, The client generates a unique key for each logical operation and sends it as a request header. The server checks whether it has already processed a request with that key — if so, it returns the cached response without re-executing the operation.
This sounds simple, but the implementation details matter. A naive version stores a key-to-response mapping and calls it done. A correct version also handles concurrent requests with the same key, key expiry, partial failures, and response mismatch detection.
Why Not Just Use HTTP Method Semantics?
GET and PUT are naturally idempotent by HTTP spec. DELETE mostly is too. The problem is POST — the method used for most mutations. POST is not idempotent by definition, and most business operations (creating orders, processing payments, sending notifications) map to POST. You can’t avoid POST by redefining your API around PUT; that stretches the semantics beyond what HTTP intended and confuses consumers.
The idempotency-key header works with POST (and any other method). It’s explicit, client-controlled, and doesn’t depend on HTTP method definitions.
A Correct Implementation in Go
Here’s the core data structure and the handler logic. The key insight is that we need to track three states: “in progress,” “completed,” and “expired.”
package idempotency
import (
"bytes"
"context"
"crypto/sha256"
"errors"
"fmt"
"io"
"net/http"
"time"
)
var (
ErrNoKey = errors.New("idempotency key required")
ErrConflict = errors.New("request already in progress")
ErrMismatch = errors.New("parameters do not match original request")
)
// RequestKey is the composite key: client-generated idempotency key + request fingerprint.
type RequestKey struct {
Key string `json:"key"`
Fingerprint string `json:"fingerprint"`
Method string `json:"method"`
Path string `json:"path"`
Status string `json:"status"` // "processing", "completed"
Response []byte `json:"response,omitempty"`
StatusCode int `json:"status_code,omitempty"`
CreatedAt time.Time `json:"created_at"`
ExpiresAt time.Time `json:"expires_at"`
}
// Store defines the persistence interface for idempotency records.
// Implementations can use Redis, a database, or in-memory storage.
type Store interface {
// Get retrieves a record. Returns nil, nil if not found.
Get(ctx context.Context, key string) (*RequestKey, error)
// Set atomically creates a record. Returns ErrConflict if key exists.
Set(ctx context.Context, key string, record *RequestKey) error
// Update replaces the record (used when processing completes).
Update(ctx context.Context, key string, record *RequestKey) error
}
// fingerprint reads the request body and returns a SHA-256 hash.
// This catches the case where a client reuses the same idempotency key
// with different request parameters.
func fingerprint(r *http.Request) (string, error) {
body, err := io.ReadAll(io.LimitReader(r.Body, 1<<20)) // 1MB limit
if err != nil {
return "", fmt.Errorf("reading body: %w", err)
}
// Replay the body so downstream handlers can read it.
r.Body = io.NopCloser(io.MultiReader(bytes.NewReader(body), r.Body))
if len(body) == 0 {
return "empty", nil
}
h := sha256.Sum256(body)
return fmt.Sprintf("%x", h)[:16], nil
}
// Middleware returns an HTTP middleware that enforces idempotency.
func Middleware(store Store, ttl time.Duration) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
idempotencyKey := r.Header.Get("Idempotency-Key")
if idempotencyKey == "" {
next.ServeHTTP(w, r)
return
}
ctx := r.Context()
fp, err := fingerprint(r)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
record, err := store.Get(ctx, idempotencyKey)
if err != nil {
// If the store is down, fall through to normal processing
// rather than rejecting all requests.
next.ServeHTTP(w, r)
return
}
if record != nil {
if record.Status == "processing" {
// Another request with this key is in flight.
// Return 409 so the client knows to retry later.
http.Error(w, ErrConflict.Error(), http.StatusConflict)
return
}
if record.Fingerprint != fp || record.Method != r.Method || record.Path != r.URL.Path {
// Same key, different request parameters.
// This is a client error — the key should be unique per operation.
http.Error(w, ErrMismatch.Error(), http.StatusBadRequest)
return
}
if time.Now().After(record.ExpiresAt) {
// Expired — treat as a new request.
record = nil
} else {
// Return the cached response.
w.Header().Set("X-Idempotency-Replayed", "true")
w.WriteHeader(record.StatusCode)
w.Write(record.Response)
return
}
}
// Mark this key as in-progress.
newRecord := &RequestKey{
Key: idempotencyKey,
Fingerprint: fp,
Method: r.Method,
Path: r.URL.Path,
Status: "processing",
CreatedAt: time.Now(),
ExpiresAt: time.Now().Add(ttl),
}
if err := store.Set(ctx, idempotencyKey, newRecord); err != nil {
if errors.Is(err, ErrConflict) {
http.Error(w, ErrConflict.Error(), http.StatusConflict)
return
}
// Store error — fall through to normal processing.
next.ServeHTTP(w, r)
return
}
// Execute the actual handler and capture the response.
rec := &responseRecorder{ResponseWriter: w, statusCode: http.StatusOK}
next.ServeHTTP(rec, r)
// Cache the result.
newRecord.Status = "completed"
newRecord.Response = rec.body.Bytes()
newRecord.StatusCode = rec.statusCode
_ = store.Update(ctx, idempotencyKey, newRecord)
})
}
}
// responseRecorder captures the status code and body for caching.
type responseRecorder struct {
http.ResponseWriter
body bytes.Buffer
statusCode int
}
func (r *responseRecorder) WriteHeader(code int) {
r.statusCode = code
r.ResponseWriter.WriteHeader(code)
}
func (r *responseRecorder) Write(b []byte) (int, error) {
r.body.Write(b)
return r.ResponseWriter.Write(b)
}The Redis-backed Store
For production use, Redis is the natural choice. It's fast, atomic with SET NX, and supports TTL natively. Here's a concrete implementation:
package idempotency
import (
"context"
"encoding/json"
"errors"
"time"
"github.com/redis/go-redis/v9"
)
type RedisStore struct {
client *redis.Client
ttl time.Duration
}
func NewRedisStore(client *redis.Client, ttl time.Duration) *RedisStore {
return &RedisStore{client: client, ttl: ttl}
}
func (s *RedisStore) Get(ctx context.Context, key string) (*RequestKey, error) {
data, err := s.client.Get(ctx, "idem:"+key).Bytes()
if err != nil {
if errors.Is(err, redis.Nil) {
return nil, nil
}
return nil, err
}
var record RequestKey
if err := json.Unmarshal(data, &record); err != nil {
return nil, err
}
return &record, nil
}
func (s *RedisStore) Set(ctx context.Context, key string, record *RequestKey) error {
data, err := json.Marshal(record)
if err != nil {
return err
}
ok, err := s.client.SetNX(ctx, "idem:"+key, data, s.ttl).Result()
if err != nil {
return err
}
if !ok {
return ErrConflict
}
return nil
}
func (s *RedisStore) Update(ctx context.Context, key string, record *RequestKey) error {
data, err := json.Marshal(record)
if err != nil {
return err
}
return s.client.Set(ctx, "idem:"+key, data, s.ttl).Err()
}The go-redis SetNX call is the critical part — it's atomic, so two concurrent requests with the same key can't both create a record. One wins and proceeds; the other gets a 409 Conflict and knows to retry.
Wiring It Up
package main
import (
"log"
"net/http"
"time"
"github.com/redis/go-redis/v9"
"yourapp/internal/idempotency"
)
func main() {
rdb := redis.NewClient(&redis.Options{
Addr: "localhost:6379",
})
store := idempotency.NewRedisStore(rdb, 24*time.Hour)
mux := http.NewServeMux()
mux.HandleFunc("POST /orders", createOrder)
// Wrap only the endpoints that need idempotency.
handler := idempotency.Middleware(store, 24*time.Hour)(mux)
log.Println("listening on :8080")
log.Fatal(http.ListenAndServe(":8080", handler))
}What Most Implementations Get Wrong
The skeleton above handles the three most common failure modes, but there are subtle issues worth calling out:
Request fingerprinting. If you only store the key without any request context, a client could accidentally send two different operations with the same key. The SHA-256 fingerprint of the request body catches this. It doesn't need to be perfect — it needs to catch the common case of "same key, different payload."
TTL selection. Too short and a slow client retries after the record expires, causing a duplicate. Too long and you accumulate stale records. For payment operations, 24–48 hours is typical. For fast idempotent writes like "update user profile," 1 hour is usually sufficient.
The "processing" state race. Without it, two concurrent identical requests both pass the store check, both execute the handler, and you get a double write. The SETNX-based locking ensures exactly one request proceeds. The other gets a 409 and retries — which, on retry, finds the cached response from the first request.
Store failures as fallback. If Redis is down, the middleware falls through to normal processing. This is intentional — it's better to process a request without idempotency protection than to reject all requests because the store is unavailable. You lose safety during the outage, but you gain availability. For critical financial operations, you might want the opposite tradeoff (fail closed), but that's a business decision.
Large response bodies. The middleware buffers the entire response to cache it. For endpoints returning large payloads (file uploads, long reports), this is wasteful. Either exclude those endpoints from idempotency middleware, or cap the cached response size and fall through when exceeded.
Idempotency Beyond HTTP
The same pattern applies to message queues and event-driven systems. When consuming from Kafka or RabbitMQ, include the message ID (or a deduplication header) as the idempotency key. The processing logic checks the store before executing the handler. The difference is that the key source changes — instead of a client-generated header, it's typically the message offset, the producer's message ID, or a content-based hash.
For webhook processing, the idempotency key is often the event ID from the provider. Stripe includes a unique event id in the payload body, GitHub sends an X-GitHub-Delivery header, and others follow similar conventions. The pattern is identical: check, set, process, cache.
When to Skip Idempotency
Not every endpoint needs it. Read endpoints (GET) are naturally idempotent. High-frequency writes where duplicates are cheap to process (analytics events, log entries) don't benefit from the overhead. And operations that genuinely need to produce different results on each call (generating a nonce, creating a timestamp) can't be made idempotent — the correct approach is to make them deterministic based on the key instead.
The sweet spot is any operation where a duplicate execution causes a real problem: financial transactions, order placement, notification delivery, and resource provisioning. These are exactly the operations where network retries are most likely and where the cost of a double-write is highest. Add idempotency keys there first.