Idempotency Keys Done Right: Building Retry-Safe APIs That Never Double-Charge

Posted on

You hit “Place Order,” the spinner hangs for ten seconds, and you tap the button again. Now the fear sets in: did the first request land? Will the customer get charged twice? This moment — the double-click, the mobile network handoff, the retry after a gateway timeout — is where well-built APIs quietly prove their design and poorly built ones double-charge customers. The fix has a name: idempotency. A protocol-level contract between client and server that says replaying this request is safe.

The idea sounds simple. The implementation is where teams get burned, because the naive approaches all fail in ways that only show up under real traffic. This post covers why retries are unavoidable, why the common shortcuts break under load, and how to build an idempotency key implementation that survives concurrent requests, partial failures, and long-running operations.

Why Retries Are Unavoidable

Start with the failure model. Between a client and your handler sit a mobile network, possibly a CDN, a load balancer, and your service’s own connection pool. Any hop can drop a connection after the request was sent but before the response came back. From the client’s perspective, the request didn’t complete — so it retries. From your server’s perspective, the first request may have fully succeeded and mutated your database.

Timeouts make this worse than it sounds. A client that times out after 10 seconds will retry a payment your server is still processing — not one that failed. The retry isn’t an error condition; it’s the correct behavior. Exponential backoff with jitter controls when retries arrive; it does nothing about what happens when they do. That’s the job of idempotency.

HTTP gives you partial help. GET, PUT, and DELETE are defined as idempotent methods — repeating them has the same effect as doing it once. POST is explicitly not, and most dangerous operations (create order, submit payment, book reservation) are naturally POST. The protocol punts the problem to your application layer.

The Idempotency Key Contract

The pattern that’s become the de facto standard — used by Stripe, PayPal, and Azure — is a client-generated key sent with each mutating request:

curl -X POST https://api.example.com/v1/orders \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Idempotency-Key: 8f3c1a2e-9b4d-4e6f-a1b2-c3d4e5f60718" \
  -d '{"sku": "widget-42", "qty": 3}'

The rules of the contract:

  • The client generates the key — a UUID or a hash of the request’s business identity. The server can’t generate it, because the server has no way to know that two requests are “the same operation” from the client’s point of view.
  • The key is scoped per endpoint and per API key. The same UUID sent to two different endpoints is two different operations.
  • On the first request, the server executes the operation and stores the response alongside the key.
  • On a replay (same key, same payload), the server skips execution and returns the stored response — ideally with the same status code and headers.
  • A replay with the same key but a different payload is a client bug. Return 409 Conflict or 422 Unprocessable Entity; do not silently execute it.

That last rule matters more than teams expect. Without a payload check, a client that builds its key once at app startup and reuses it for every request will get stale responses forever, and debugging turns into archaeology.

Naive Approaches That Break in Production

Before showing the full design, here’s why the shortcuts fail.

“Just check if the record exists”

The most common first attempt: before inserting an order, SELECT for one with the same key, and skip if found. Under concurrent load this is a race — two requests can both run the check, both see nothing, both insert. You’ve moved the duplicate from “always” to “occasionally,” which is strictly worse because it’s now a heisenbug that appears during traffic spikes or retry storms.

“Use a UNIQUE constraint and catch the error”

A unique index on the idempotency key column does close the race: the second insert fails, and the transaction rolls back. This is genuinely the right building block — but on its own it doesn’t solve the response problem. The retried request now gets a 500 from the unique violation, not the original 201. The client retries again (it’s a 5xx, so of course it retries), gets another conflict, and eventually shows the user an error — for an order that was actually created. Deduplication without response replay is only half the pattern.

“Store the key in Redis with a TTL”

A Redis SET NX gate is fast and tempting, but now you have two systems that can disagree. If your service marks the key as “seen” in Redis and then crashes before committing the database transaction, the key exists but the operation never happened — legitimate retries get rejected forever. If Redis evicts the key early, replays leak through as duplicates. A cache can be a fast path in front of the source of truth, but the database — where the operation’s outcome is committed atomically — must be the authority.

The Full Design: Store the Outcome, Not Just the Key

The complete pattern keeps the idempotency record in the same database as the operation itself, commits them atomically, and stores the serialized response. Schema first:

CREATE TABLE idempotency_keys (
    key             TEXT        NOT NULL,
    endpoint        TEXT        NOT NULL,
    request_hash    TEXT        NOT NULL,
    response_status INT         NULL,
    response_body   JSONB       NULL,
    status          TEXT        NOT NULL DEFAULT 'processing',
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at      TIMESTAMPTZ NOT NULL,
    PRIMARY KEY (key, endpoint)
);

CREATE INDEX idx_idempotency_expiry ON idempotency_keys (expires_at)
    WHERE status = 'completed';

Key design decisions in this schema:

  • Composite primary key on (key, endpoint) enforces scoping — the same UUID against different endpoints doesn’t collide.
  • request_hash is a hash of the request body. On replay you recompute and compare; a mismatch means key reuse with different data, and you reject with 422.
  • status tracks the lifecycle: processing → completed (or failed) — this is what makes concurrent replays safe.
  • Storing the response body turns deduplication into true replay: the retried client gets the original 201 with the original order payload.
  • expires_at bounds storage. Keys past expiry are deleted by a background job; a replay after expiry executes as a new request — the standard tradeoff, since platforms like Stripe retain keys roughly 24 hours.

The handler flow, in order:

  1. Extract the key from the header. Missing key on an endpoint that requires one: either reject with 400, or generate a server-side key (which means you’ve accepted that retries are not protected — document this).
  2. INSERT a row with status processing and the request hash. If the insert conflicts on the primary key, someone else got there first — go to step 4.
  3. On first insert: execute the operation and update the status to completed (with the response body) in one transaction. This atomic commit is the foundation of the pattern — the key is marked done exactly when the operation is done, no window in between.
  4. On conflict: SELECT ... FOR UPDATE the existing row and check it:
    • status completed → verify request hash, return stored response.
    • status processing → another request holds the key. Wait briefly and re-check, or return 409 Conflict with a Retry-After header.
    • status failed → the previous attempt errored out. Delete the stale row and re-execute — a client retrying after a failure deserves a fresh attempt, not a permanent tombstone.
    • request hash mismatch → 422, always.

The Go Implementation

The core logic as a reusable store:

package orders

import (
	"context"
	"crypto/sha256"
	"encoding/hex"
	"encoding/json"
	"errors"
	"fmt"
	"net/http"

	"github.com/jackc/pgx/v5"
)

var ErrKeyReuse = errors.New("idempotency key reused with different payload")

type KeyStore struct {
	db Pool
}

func hashRequest(body []byte) string {
	sum := sha256.Sum256(body)
	return hex.EncodeToString(sum[:])
}

// Process implements the idempotency contract for one request.
// execute is the real operation; it runs at most once per key.
func (s *KeyStore) Process(
	ctx context.Context,
	key, endpoint string,
	reqBody []byte,
	execute func(ctx context.Context) (int, any, error),
) (int, any, error) {
	reqHash := hashRequest(reqBody)

	// Fast path: completed key, return the stored response.
	row, err := s.findByKey(ctx, key, endpoint)
	if errors.Is(err, pgx.ErrNoRows) {
		// First attempt: claim the key, then execute.
		if err := s.claim(ctx, key, endpoint, reqHash); err != nil {
			return 0, nil, err
		}
		return s.runAndStore(ctx, key, endpoint, execute)
	}
	if err != nil {
		return 0, nil, err
	}

	if row.RequestHash != reqHash {
		return http.StatusUnprocessableEntity, nil, ErrKeyReuse
	}

	switch row.Status {
	case "completed":
		var body any
		if err := json.Unmarshal(row.RespBody, &body); err != nil {
			return 0, nil, fmt.Errorf("stored response corrupt: %w", err)
		}
		return row.RespStatus, body, nil

	case "failed":
		// Previous attempt errored: allow a fresh attempt.
		if err := s.reset(ctx, key, endpoint, reqHash); err != nil {
			return 0, nil, err
		}
		return s.runAndStore(ctx, key, endpoint, execute)

	default: // "processing": a concurrent request holds the key.
		return http.StatusConflict, map[string]string{
			"error": "request in progress, retry shortly",
		}, nil
	}
}

func (s *KeyStore) runAndStore(
	ctx context.Context,
	key, endpoint string,
	execute func(ctx context.Context) (int, any, error),
) (int, any, error) {
	status, respBody, execErr := execute(ctx)

	// Whether the operation succeeded or failed, record the outcome.
	newStatus := "completed"
	storeBody := respBody
	if execErr != nil {
		newStatus = "failed"
		storeBody = nil
	}
	if err := s.finalize(ctx, key, endpoint, newStatus, status, storeBody); err != nil {
		return 0, nil, err
	}
	return status, respBody, execErr
}

The helper methods wrap single SQL statements; the two that matter are claim and finalize:

func (s *KeyStore) claim(ctx context.Context, key, endpoint, reqHash string) error {
	tag, err := s.db.Exec(ctx, `
		INSERT INTO idempotency_keys (key, endpoint, request_hash, status, expires_at)
		VALUES ($1, $2, $3, 'processing', now() + interval '24 hours')
		ON CONFLICT (key, endpoint) DO NOTHING`,
		key, endpoint, reqHash)
	if err != nil {
		return err
	}
	if tag.RowsAffected() == 0 {
		// Lost a race with a concurrent first attempt; treat as replay.
		return errConcurrentRequest
	}
	return nil
}

func (s *KeyStore) finalize(ctx context.Context, key, endpoint, status string, code int, body []byte) error {
	_, err := s.db.Exec(ctx, `
		UPDATE idempotency_keys
		SET status = $3, response_status = $4, response_body = $5
		WHERE key = $1 AND endpoint = $2`,
		key, endpoint, status, code, body)
	return err
}

Handling Long-Running Operations

Payment captures, report generation, and provisioning jobs often don’t finish within one request. Forcing them into a 30-second request/response cycle breaks both the client and your server, and the failure mode is ugly: the client times out, retries, and your handler is still running the original.

The status machine in the schema handles this cleanly. For slow operations, the endpoint returns 202 Accepted immediately with a status URL, marks the key processing, and a worker completes it asynchronously. A client that replays the same key during processing gets the same 202 and the same status URL — no duplicate jobs enqueued. The worker flips the row to completed when the job finishes, and any replay after that returns the final result. You get exactly-once enqueueing (the hard part) while accepting at-least-once execution inside the worker, which is the correct place to absorb that cost.

One subtlety: the worker must finalize the key row in the same transaction that commits the operation’s side effects. If your worker writes the result and updates the key in separate transactions, a crash between them leaves a completed operation behind a processing key — and the next replay re-enqueues the job. This is the same atomicity principle as the synchronous path, one layer down.

Operational Details That Decide Whether It Works

TTL choice. The retention window must exceed your clients’ maximum retry horizon. Mobile clients with aggressive backoff can retry for hours; a 24-hour window covers essentially every legitimate replay, and the expiry index keeps the cleanup job cheap. Deleting expired keys is mandatory hygiene — the table grows with every mutating request, and an unbounded idempotency table eventually becomes your slowest query.

Missing header policy. Rejecting keyless requests outright is the safest contract, but it’s a breaking change for existing clients. A pragmatic rollout: generate a random server-side key (retries unprotected), require the key on new API versions, and make the policy uniform — per-endpoint inconsistency here is how double charges slip through the one endpoint everyone forgot.

Response fidelity. Replays should return the original status code, content type, and business payload. Headers that describe the current request (rate-limit counters, trace IDs) should be regenerated; headers that describe the original operation (a Location pointing at the created resource) should be replayed.

Putting It Together

  • Client-generated key on every mutating request, scoped per endpoint.
  • Key and outcome stored in the operation’s database, committed atomically.
  • Request hash verification on every replay; hard 422 on mismatch.
  • A status machine (processing → completed/failed) making concurrent replays safe and failures retryable.
  • Response replay with original status and body; TTL cleanup with an indexed expiry.
  • For long operations: 202 + status URL, worker finalizing the key inside its commit.

Start with the synchronous path — it covers most APIs, and the schema above supports both modes from day one. The pattern costs one extra table, one index, and two queries per mutating request; against the cost of a single duplicate-charge incident, it’s among the highest-leverage design decisions an API that moves money or inventory can make.

Leave a Reply

Your email address will not be published. Required fields are marked *