Idempotency Keys: Making API Retries Safe by Design

Posted on

The network drops the connection two seconds after your client sends the payment request. Did the charge go through? Your client has no way to know, so it does the only sensible thing: it retries. If the first attempt actually landed, you have just charged the customer twice. The retry is not a bug — retrying after an ambiguous failure is exactly what a well-behaved client should do. The bug is a server that cannot tell a repeated request from a new one. Idempotency keys fix that, and implementing them well touches every layer of your API: headers, storage, concurrency control, and error semantics.

The core contract is simple. The client generates a unique key for the logical operation — usually a UUID — and sends it on every attempt of that operation, typically via an Idempotency-Key header. The server records the key with the outcome of the first request it sees. If the same key shows up again, the server does not re-execute the operation; it returns the stored result. This is the design Stripe’s API popularized in the payments world, and it has become the de facto convention for any mutating endpoint where a duplicate would be expensive: charges, transfers, bookings, order creation.

Where the Uniqueness Lives

The key alone is not enough — you need a scope. If two different clients coincidentally generate the same UUID (vanishingly rare, but the failure mode is silent double-execution), or if a client library reuses keys across endpoints, your dedup logic misfires. Store keys scoped to something the client already has: the API key or tenant ID, plus the endpoint and HTTP method. The logical identity of a request is then the tuple, not the bare key. One subtle consequence: if a client sends the same key twice with different bodies, that is a client bug, and the API should reject it with an error rather than returning the cached response. Silently returning the old result for a request that was actually different hides data corruption.

How long should keys live? Long enough to outlast every realistic retry window. Retries happen on connection timeouts, load balancer resets, and deploy blips — minutes, not months. A retention window on the order of a day comfortably covers automated retry behavior while keeping the keys table small, which is why the services that run this pattern at scale purge keys after roughly 24 hours. After expiry, a replayed key simply creates a new operation, which is the correct behavior once no client will legitimately retry that old a request.

The Hard Part: The Race Window

Storing “key → result” in a table and checking it before execution is the naive version, and it fails under the exact conditions idempotency exists for. Two attempts of the same request often arrive simultaneously — the client retried because it thought the first attempt timed out, but the first attempt is still executing. If both requests check the table, find nothing, and proceed, both execute. You need the check-and-insert to be atomic. The tool for that is a unique constraint, and the graceful way to use it is to insert first and let the database say no:

CREATE TABLE idempotency_keys (
    key          TEXT        NOT NULL,
    scope        TEXT        NOT NULL,      -- tenant/API key + endpoint + method
    request_hash TEXT        NOT NULL,      -- hash of the request body
    status       TEXT        NOT NULL,      -- 'in_progress' | 'completed' | 'failed'
    response     JSONB,                     -- stored response for replays
    created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (scope, key)
);

The insert acts as a lock. One request wins and proceeds; every concurrent duplicate gets a constraint violation and is told the original is still in flight:

package idempotency

import (
	"context"
	"errors"
	"fmt"
	"net/http"
	"time"

	"github.com/jackc/pgx/v5"
	"github.com/jackc/pgx/v5/pgconn"
)

var ErrConflictInProgress = errors.New("request already in progress")

type Store struct {
	db *pgx.Conn
}

// Begin claims the key by inserting an in_progress row.
// The PRIMARY KEY constraint makes this atomic: exactly one
// concurrent request can win the insert.
func (s *Store) Begin(ctx context.Context, scope, key, requestHash string) error {
	tag, err := s.db.Exec(ctx, `
		INSERT INTO idempotency_keys (scope, key, request_hash, status)
		VALUES ($1, $2, $3, 'in_progress')`,
		scope, key, requestHash)
	if err != nil {
		var pgErr *pgconn.PgError
		if errors.As(err, &pgErr) && pgErr.Code == "23505" {
			return ErrConflictInProgress // duplicate insert lost the race
		}
		return err
	}
	if tag.RowsAffected() == 0 {
		return ErrConflictInProgress
	}
	return nil
}

// Result returns the stored response for a completed request,
// or ErrConflictInProgress if the original attempt is still running.
func (s *Store) Result(ctx context.Context, scope, key, requestHash string) ([]byte, int, error) {
	var status string
	var response []byte
	err := s.db.QueryRow(ctx, `
		SELECT status, response FROM idempotency_keys
		WHERE scope = $1 AND key = $2`,
		scope, key).Scan(&status, &response)
	if errors.Is(err, pgx.ErrNoRows) {
		return nil, 0, fmt.Errorf("key %q not found after conflict", key)
	}
	if err != nil {
		return nil, 0, err
	}
	if status == "in_progress" {
		return nil, http.StatusConflict, ErrConflictInProgress
	}
	return response, http.StatusOK, nil
}

// Finish persists the response and releases the claim.
func (s *Store) Finish(ctx context.Context, scope, key string, response []byte) error {
	_, err := s.db.Exec(ctx, `
		UPDATE idempotency_keys
		SET status = 'completed', response = $3
		WHERE scope = $1 AND key = $2`,
		scope, key, response)
	return err
}

// Fail releases the claim so the client's next retry can proceed.
func (s *Store) Fail(ctx context.Context, scope, key string) error {
	_, err := s.db.Exec(ctx, `
		DELETE FROM idempotency_keys WHERE scope = $1 AND key = $2`,
		scope, key)
	return err
}

const keyTTL = 24 * time.Hour

When the losing request gets ErrConflictInProgress, the right response is 409 Conflict with a short body telling the client to retry after a delay. That turns a correctness race into a well-defined protocol step.

The Consistency Trap: Keys Must Share the Transaction

Here is the mistake that quietly reintroduces double charges. The idempotency row and the business data must be written in the same database transaction. If you mark the key completed in one commit and write the payment row in another, a crash between them leaves a completed key with no charge — the client’s retry gets the cached “success” response for a payment that never happened. In PostgreSQL, the fix is to run the business write and the key-state update inside one transaction. This is also the point where the idempotency pattern connects to the transactional outbox: both exist because “execute side effect” and “record that we did it” are two writes, and two writes need one transaction or a protocol for reconciling them.

Failure semantics need just as much care. When the handler errors before producing a response, release the key (the Fail path above) so the client’s retry gets a genuine second attempt — an error is not a result worth replaying. Only successful responses get cached. If your handler performs the operation but crashes before responding, the in_progress row lingers; add a reaper that treats claims older than a few minutes as abandoned and deletes them, letting the retry proceed.

Fingerprinting: Same Key, Different Request

Store a hash of the request body alongside the key, and verify it on replay. If a client reuses a key with a different payload, return 422 Unprocessable Entity with an error explaining the key reuse — never the cached response. Without the fingerprint check, a buggy client that resets its key counter can silently alias two different operations into one, and the second operation’s “success” will be a lie borrowed from the first.

A Middleware That Ties It Together

Wrapped as HTTP middleware, the pattern becomes reusable across every mutating endpoint:

package middleware

import (
	"bytes"
	"crypto/sha256"
	"encoding/hex"
	"errors"
	"io"
	"net/http"

	"yourapp/idempotency"
)

type responseRecorder struct {
	http.ResponseWriter
	status int
	body   bytes.Buffer
}

func (r *responseRecorder) WriteHeader(code int) {
	r.status = code
	r.ResponseWriter.WriteHeader(code)
}

func (r *responseRecorder) Write(b []byte) (int, error) {
	r.body.Write(b)
	return r.ResponseWriter.Write(b)
}

func Idempotency(store *idempotency.Store, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		key := r.Header.Get("Idempotency-Key")
		if key == "" {
			http.Error(w, "missing Idempotency-Key header", http.StatusBadRequest)
			return
		}
		scope := r.Header.Get("Authorization") + "|" + r.Method + "|" + r.URL.Path

		body, err := io.ReadAll(io.LimitReader(r.Body, 1<<20))
		if err != nil {
			http.Error(w, "cannot read body", http.StatusBadRequest)
			return
		}
		r.Body = io.NopCloser(bytes.NewReader(body))
		sum := sha256.Sum256(body)
		requestHash := hex.EncodeToString(sum[:])

		err = store.Begin(r.Context(), scope, key, requestHash)
		if errors.Is(err, idempotency.ErrConflictInProgress) {
			stored, status, resultErr := store.Result(r.Context(), scope, key, requestHash)
			if resultErr != nil {
				// Original request is still executing.
				w.Header().Set("Retry-After", "1")
				http.Error(w, "request already in progress", http.StatusConflict)
				return
			}
			w.Header().Set("Idempotent-Replayed", "true")
			w.WriteHeader(status)
			w.Write(stored)
			return
		}
		if err != nil {
			http.Error(w, "storage error", http.StatusInternalServerError)
			return
		}

		rec := &responseRecorder{ResponseWriter: w, status: http.StatusOK}
		next.ServeHTTP(rec, r)

		if rec.status >= 500 {
			// Server-side failure: release the key so the
			// client's retry gets a real attempt.
			store.Fail(r.Context(), scope, key)
			return
		}
		if err := store.Finish(r.Context(), scope, key, rec.body.Bytes()); err != nil {
			// The operation succeeded but the key write failed;
			// the reaper will eventually clear the stale claim.
			http.Error(w, "storage error", http.StatusInternalServerError)
			return
		}
	})
}

One scope caveat visible in this code: using the raw Authorization header in the scope works when every request carries the same credential format, but if your tokens rotate per request, scope on a stable client or tenant identifier instead. The scope must be stable across retries of the same logical operation, or dedup breaks.

Practical Checklist

  • Require idempotency keys on every mutating endpoint where duplicates are expensive; return 400 when the header is missing rather than silently accepting unkeyed writes.
  • Scope keys by tenant plus endpoint plus method; reject same-key-different-body with 422.
  • Claim keys with a unique-constraint insert, answer concurrent duplicates with 409 and Retry-After.
  • Write the key outcome in the same transaction as the business data.
  • Cache only successful responses; release keys on failure so retries are genuine.
  • Expire keys after roughly a day, and reap abandoned in-progress claims.
  • Return the replayed response with an Idempotent-Replayed header so clients and logs can distinguish replays from first executions.

None of this is exotic — one table, one unique constraint, one middleware — but together it converts your API’s most dangerous ambiguity into an explicit protocol. Clients will retry whether you plan for it or not; the only question is whether the retry is safe.

Leave a Reply

Your email address will not be published. Required fields are marked *