Every web developer has been there: a user clicks “Submit Payment” and nothing happens. The spinner keeps spinning. They click again. And again. When the network recovers, three identical charges hit their card.
This isn’t a hypothetical. Double- and triple-submits are one of the most common causes of data corruption in web applications, and they happen for reasons entirely outside your control: network timeouts, mobile connectivity drops, retry logic in HTTP clients, load balancer failovers, and users who simply don’t trust that spinner.
Idempotency — the property that a request can be repeated without producing additional side effects — is the answer. Let’s walk through how to implement it correctly, from the HTTP semantics down to the storage layer.
What Idempotency Actually Means
An operation is idempotent if executing it once has the same effect as executing it multiple times. f(f(x)) = f(x). The HTTP specification (RFC 9110) defines certain methods as inherently idempotent: GET, PUT, and DELETE are supposed to be safe to retry by definition. POST and PATCH are not.
In practice, even DELETE can violate idempotency if it returns different responses on each call (404 on the first retry, 200 on the original). But the real problem is POST — the method we use for creating resources, processing payments, and triggering side effects. Every non-idempotent POST endpoint is a potential double-charge waiting to happen.
The Idempotency Key Pattern
The solution is straightforward in concept: the client generates a unique key for each logical operation and sends it with the request. The server stores the result of the first request. On subsequent requests with the same key, it returns the stored result instead of re-executing the operation.
The Idempotency-Key header specification formalizes this as an HTTP header: Idempotency-Key: <client-generated-uuid>. The client generates a UUID (or any sufficiently unique token) before the first attempt and includes it in every retry of that same logical request. The server uses it to detect duplicates and respond consistently.
The Three States
Every idempotency key moves through exactly three states:
- New: The server has never seen this key. Execute the operation, store the result, return it.
- In-Flight: The server is currently processing a request with this key. Return a 409 Conflict or retry instruction.
- Completed: The operation finished. Return the stored result.
The critical insight is that in-flight state must be tracked. Without it, two concurrent requests with the same key can both pass the “have I seen this?” check and execute in parallel — defeating the entire purpose.
Implementing It in Go
Let’s build a practical implementation using Redis for the key store. Redis is ideal here because it provides atomic operations and built-in TTL — we can set an expiration on stored keys so the store doesn’t grow forever.
The Store
package idempotency
import (
"context"
"encoding/json"
"errors"
"time"
"github.com/redis/go-redis/v9"
)
var (
ErrKeyNotFound = errors.New("idempotency key not found")
ErrKeyInFlight = errors.New("idempotency key is in-flight")
ErrKeyMismatch = errors.New("idempotency key belongs to different payload")
)
const (
statusInFlight = "in-flight"
statusDone = "done"
)
type StoredResponse struct {
Status int `json:"status"`
Body json.RawMessage `json:"body"`
}
type Store struct {
rdb *redis.Client
ttl time.Duration
}
func NewStore(rdb *redis.Client, ttl time.Duration) *Store {
return &Store{rdb: rdb, ttl: ttl}
}
// Acquire attempts to claim the key for processing.
// Returns ErrKeyInFlight if already being processed,
// ErrKeyMismatch if the key exists with a different payload.
func (s *Store) Acquire(ctx context.Context, key, payloadHash string) error {
inFlightKey := "idem:if:" + key
set, err := s.rdb.SetNX(ctx, inFlightKey, payloadHash, s.ttl).Result()
if err != nil {
return err
}
if !set {
// Key exists — check if same payload
stored, _ := s.rdb.Get(ctx, inFlightKey).Result()
if stored != payloadHash {
return ErrKeyMismatch
}
return ErrKeyInFlight
}
return nil
}
// StoreResult saves the completed response and removes the in-flight marker.
func (s *Store) StoreResult(ctx context.Context, key string, resp StoredResponse) error {
data, err := json.Marshal(resp)
if err != nil {
return err
}
doneKey := "idem:done:" + key
pipe := s.rdb.TxPipeline()
pipe.Set(ctx, doneKey, data, s.ttl)
pipe.Del(ctx, "idem:if:"+key)
_, err = pipe.Exec(ctx)
return err
}
// GetResult retrieves a stored response for a completed key.
func (s *Store) GetResult(ctx context.Context, key string) (*StoredResponse, error) {
doneKey := "idem:done:" + key
data, err := s.rdb.Get(ctx, doneKey).Result()
if errors.Is(err, redis.Nil) {
return nil, ErrKeyNotFound
}
if err != nil {
return nil, err
}
var resp StoredResponse
if err := json.Unmarshal([]byte(data), &resp); err != nil {
return nil, err
}
return &resp, nil
}The SetNX command (set-if-not-exists) is the backbone of the acquire step. It’s atomic at the Redis level — two concurrent requests cannot both succeed. The TTL ensures that even if the server crashes mid-processing, the in-flight marker eventually expires and the client can retry. The implementation above uses go-redis v9, the standard Redis client for Go.
The HTTP Middleware
package idempotency
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"errors"
"io"
"net/http"
)
// Middleware wraps an http.Handler with idempotency protection.
func Middleware(store *Store) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Only protect non-idempotent methods
if r.Method != http.MethodPost && r.Method != http.MethodPatch {
next.ServeHTTP(w, r)
return
}
key := r.Header.Get("Idempotency-Key")
if key == "" {
next.ServeHTTP(w, r)
return
}
// Read and hash the request body for payload matching
bodyBytes, err := io.ReadAll(r.Body)
if err != nil {
http.Error(w, "failed to read body", http.StatusBadRequest)
return
}
r.Body.Close()
r.Body = io.NopCloser(bytes.NewReader(bodyBytes))
hash := sha256.Sum256(bodyBytes)
payloadHash := hex.EncodeToString(hash[:])
ctx := r.Context()
// Try to acquire the key
err = store.Acquire(ctx, key, payloadHash)
if errors.Is(err, ErrKeyInFlight) {
http.Error(w, "request already in progress", http.StatusConflict)
return
}
if errors.Is(err, ErrKeyMismatch) {
http.Error(w, "idempotency key reused with different payload",
http.StatusUnprocessableEntity)
return
}
if err != nil {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
// Check for a previously stored result
stored, err := store.GetResult(ctx, key)
if err == nil && stored != nil {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(stored.Status)
w.Write(stored.Body)
return
}
// Capture the response
rec := &responseRecorder{
ResponseWriter: w,
status: http.StatusOK,
body: &bytes.Buffer{},
}
next.ServeHTTP(rec, r)
// Store the result for future retries
store.StoreResult(ctx, key, StoredResponse{
Status: rec.status,
Body: rec.body.Bytes(),
})
})
}
}
type responseRecorder struct {
http.ResponseWriter
status int
body *bytes.Buffer
}
func (r *responseRecorder) WriteHeader(code int) {
r.status = code
r.ResponseWriter.WriteHeader(code)
}
func (r *responseRecorder) Write(b []byte) (int, error) {
r.body.Write(b)
return r.ResponseWriter.Write(b)
}A few design decisions worth noting in this implementation:
- Payload hashing: The middleware hashes the request body and includes it in the acquire step. If a client reuses the same idempotency key with a different payload, the server rejects it with 422. This prevents subtle bugs where a retry accidentally sends a different request body.
- Response recording: The
responseRecordercaptures the status code and body as they’re written. On a replay, we serve the exact same response — same status, same body, same headers. - Non-blocking on GET/PUT/DELETE: These methods are already idempotent by HTTP contract, so the middleware skips them.
Client-Side Implementation
The server side is only half the battle. The client must generate the key before the first attempt and reuse it on every retry. Here’s what that looks like in TypeScript:
async function createOrder(orderData: Order): Promise {
const idempotencyKey = crypto.randomUUID()
let lastError: Error | null = null
for (let attempt = 0; attempt < 3; attempt++) {
try {
const resp = await fetch("/api/orders", {
method: "POST",
headers: {
"Content-Type": "application/json",
"Idempotency-Key": idempotencyKey,
},
body: JSON.stringify(orderData),
})
if (resp.status === 409) {
// Request in-flight on server — wait and retry
await new Promise(r => setTimeout(r, 1000 * (attempt + 1)))
continue
}
if (!resp.ok) throw new Error(`HTTP ${resp.status}`)
return await resp.json()
} catch (err) {
lastError = err as Error
await new Promise(r => setTimeout(r, 1000 * (attempt + 1)))
}
}
throw lastError ?? new Error("max retries exceeded")
} The key is generated once and reused across all retry attempts. The UUID is stored on the client side — if the user navigates away and comes back to submit again, a new key is generated (which is correct: it’s a new logical operation). The retry logic handles 409 Conflict by backing off and trying again, expecting the original request to have completed by then.
Common Pitfalls
Storing Results Too Briefly
If your TTL is too short, a slow client retry after a network blip might arrive after the stored result has expired, causing a legitimate double-execution. For payment endpoints, 24–48 hours is reasonable. For general-purpose endpoints, 1–2 hours usually suffices. The TTL should be longer than your longest plausible client-side retry window.
Not Handling Concurrent Requests
Without the in-flight marker, two requests with the same key arriving within milliseconds of each other can both pass the existence check and both execute. The SetNX pattern prevents this by atomically claiming the key, but only if you check the result before proceeding.
Hashing Inconsistent Payloads
If your client serializes JSON differently on each attempt (different key ordering, whitespace, etc.), the payload hash will differ and the mismatch check will reject the retry. Normalize your serialization before hashing, or skip payload hashing entirely if you trust the client to send consistent bodies.
Returning Cached Errors Indefinitely
If a request fails with a transient error (say, a 503), caching that response and returning it on retry is counterproductive. Consider only caching successful responses (2xx) and client errors (4xx that indicate a genuine validation failure). Transient server errors (5xx) should not be cached — the client should be able to retry and get a fresh attempt.
When You Need Idempotency
Not every endpoint needs idempotency. Read operations, status checks, and other side-effect-free calls are safe to retry by nature. But any endpoint that creates resources, moves money, sends messages, or triggers external side effects should implement it. The cost is minimal — one Redis call, one extra header — and the protection is absolute.
The pattern scales naturally. In a distributed system with multiple service instances behind a load balancer, all instances share the same Redis store, so idempotency works regardless of which instance handles the request. If you’re already using Redis for caching or sessions, adding idempotency is a small incremental investment.
Idempotency keys are one of those patterns that feels like overkill right up until the moment a production incident traces back to a double-submit. Adding them proactively — especially on payment and order endpoints — is one of the highest-leverage reliability improvements you can make to an API.