Every production incident report with a root cause of “the downstream service was slow” is really a story about a missing timeout, a naive retry loop, or a client that kept hammering a dying service. Resilience at the HTTP client level is not one feature — it is a stack of three mechanisms that fail in a specific order: timeouts bound how long you wait, retries handle transient failures, and circuit breakers stop you from paying for an outage you cannot fix. This post walks through building that stack in Go, with the trade-offs each layer introduces.
Timeouts: The Default Is Infinity
Go’s default HTTP client is a landmine for the unwary. A zero-valued http.Client has no timeout at all:
package main
import (
"fmt"
"net/http"
"time"
)
func main() {
// No timeout set anywhere: this client will wait forever.
client := &http.Client{}
start := time.Now()
resp, err := client.Get("https://slow-backend.internal/report")
if err != nil {
fmt.Println("error:", err)
return
}
defer resp.Body.Close()
fmt.Println("status:", resp.Status, "after", time.Since(start))
}
If the remote server accepts the connection and then stops reading, this request hangs indefinitely. Each hung request pins a goroutine and, more importantly, a connection in the client’s idle pool. Under load, hung requests compound until the whole service runs out of file descriptors. This is such a common failure mode that some teams run go vet extensions specifically to flag clients constructed without timeouts.
There are two layers to bound. http.Client.Timeout covers the entire exchange — connection, redirects, and reading the body — and is the right blunt instrument for simple calls. Per-request deadlines via context are finer-grained: they propagate cancellation to everything downstream of the request, including database queries triggered by a middleware chain, and they let different callers impose different budgets on the same client.
package main
import (
"context"
"io"
"net/http"
"time"
)
func fetch(url string) ([]byte, error) {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
return io.ReadAll(resp.Body)
}
One subtlety: Client.Timeout and a context deadline are not redundant. The context deadline is what you control per call site; the client timeout is the safety net that catches callers who forgot. Setting both is normal. What matters is that neither is zero.
Retries: Only What Deserves a Second Attempt
A retry turns one failure into multiple attempts, which means it amplifies load precisely when the system can least afford it. The rules that keep retries safe:
- Retry network errors and
5xxresponses. Do not retry4xx— the server understood the request and rejected it; repeating it will fail identically. The exception is429 Too Many Requests, which explicitly asks for a later attempt. - Never retry an operation whose effect is not idempotent unless you have an idempotency key. Retrying a payment capture can double-charge a customer.
- Always cap the total budget — a deadline on the context, not just a retry count.
- Back off exponentially and add jitter so a thousand clients do not retry in lockstep after an outage window closes.
package main
import (
"context"
"errors"
"fmt"
"io"
"math/rand"
"net/http"
"time"
)
// retryable reports whether a failure is worth another attempt.
// A canceled context is not: the caller already gave up.
func retryable(err error, status int) bool {
if err != nil {
return !errors.Is(err, context.Canceled)
}
return status == http.StatusTooManyRequests || status >= 500
}
func getWithRetry(ctx context.Context, url string, maxAttempts int) (*http.Response, error) {
var lastErr error
for attempt := 1; attempt <= maxAttempts; attempt++ {
if attempt > 1 {
backoff := time.Duration(1<<uint(attempt-2)) * 100 * time.Millisecond
jitter := time.Duration(rand.Int63n(int64(backoff / 2)))
select {
case <-time.After(backoff + jitter):
case <-ctx.Done():
return nil, ctx.Err()
}
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, err
}
resp, err := http.DefaultClient.Do(req)
if err == nil && !retryable(nil, resp.StatusCode) {
return resp, nil // final answer, including non-retryable 4xx
}
if resp != nil {
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
}
status := 0
if resp != nil {
status = resp.StatusCode
}
if !retryable(err, status) {
return nil, err
}
lastErr = err
}
if lastErr == nil {
lastErr = fmt.Errorf("GET %s: still failing after %d attempts", url, maxAttempts)
}
return nil, lastErr
}
Three details in this loop are easy to get wrong. First, the response body is drained and closed before retrying — leaking a body per attempt exhausts connections just as effectively as no timeout. Second, the sleep honors ctx.Done(), so a caller canceling mid-backoff gets its error immediately instead of waiting out the jitter window. Third, a canceled context is classified as non-retryable: the deadline expiring on attempt two is information, not a transient blip.
Respecting the server’s hints matters too. A 429 response with a Retry-After header is telling you exactly when to come back, and honoring it beats computing your own backoff. If a service is rate-limiting you, aggressive retries turn a soft limit into a hard block.
Circuit Breakers: Stop Paying for an Outage
Retries assume the failure is transient. When a dependency is down for five minutes, retrying is just self-inflicted load: every caller burns its full retry budget against a service that cannot respond, tripling the request volume and slowing your own recovery. A circuit breaker tracks recent failures and, past a threshold, refuses to send requests at all — failing fast instead of failing slow.
The pattern has three states. Closed (normal traffic, failures counted), open (requests rejected immediately), and half-open (a small number of probe requests to test whether the dependency has recovered). The gobreaker library from Sony implements this as a generic wrapper, and its v2 API adds a rolling-window failure counter and error exclusion:
package main
import (
"context"
"errors"
"net/http"
"time"
"github.com/sony/gobreaker/v2"
)
func newPaymentBreaker() *gobreaker.CircuitBreaker[*http.Response] {
return gobreaker.NewCircuitBreaker[*http.Response](gobreaker.Settings{
Name: "payment-service",
MaxRequests: 3, // concurrent probes allowed in half-open
Interval: time.Minute, // clear failure counts each minute while closed
Timeout: 30 * time.Second, // how long the breaker stays open
ReadyToTrip: func(c gobreaker.Counts) bool {
if c.Requests < 20 {
return false
}
return float64(c.TotalFailures)/float64(c.Requests) >= 0.5
},
IsExcluded: func(err error) bool {
// A caller canceling its own request says nothing about the
// payment service's health.
return errors.Is(err, context.Canceled)
},
})
}
The interesting knob is ReadyToTrip. The library’s default trips after five consecutive failures, which is fine for quiet services but noisy for high-traffic ones — five failures in a million requests is a blip, not an outage. A failure ratio over a minimum sample size, as above, scales the sensitivity to your traffic. Deadline expirations should count as failures here (a dependency that times out is unhealthy by definition), while caller-initiated cancellation should not, which is what IsExcluded expresses.
Using the breaker is a single call to Execute. The ordering question — does the breaker wrap the retry loop, or the other way around — has a clear answer: the breaker goes outside. One logical operation should count as one breaker observation, whether it took one HTTP attempt or three. A breaker inside the retry loop would trip three times faster than intended and count each retry as separate evidence of failure:
package main
import (
"context"
"errors"
"github.com/sony/gobreaker/v2"
)
var errCircuitOpen = errors.New("payment service unavailable, queued for retry")
// charge runs one logical payment: the breaker counts the whole call as a
// single observation, while the retry loop inside may make several HTTP
// attempts. chargeWithRetry is the retrying helper from the previous
// section, adapted for a POST with an idempotency key.
func charge(ctx context.Context, cb *gobreaker.CircuitBreaker[string]) (string, error) {
receipt, err := cb.Execute(func() (string, error) {
return chargeWithRetry(ctx, "https://payments.internal/charge")
})
if errors.Is(err, gobreaker.ErrOpenState) {
// Fail fast and let an async worker replay the queued charge later.
return "", errCircuitOpen
}
return receipt, err
}
The open-state rejection is not an error condition to hide — it is a decision point. A good client treats ErrOpenState as its cue to switch to a fallback: queue the work, serve a cached response, or shed load with a clear 503 and a Retry-After header of its own.
Putting the Layers Together
The full stack, from the inside out: a shared http.Client with a nonzero timeout as the safety net, a per-request context deadline set by the caller, a retry loop that classifies errors before repeating anything, and a circuit breaker wrapping the whole logical operation. Two configuration pitfalls bite teams that assemble this naively. If your retry timeout budget is longer than the breaker’s observation window, the breaker never sees enough completed calls to trip — size Interval to comfortably exceed your worst-case single-call duration. And if you put the breaker inside the retry loop, as noted above, your trip threshold silently shrinks by the retry factor.
Finally, wire the breaker’s state changes into your metrics. A transition to open is one of the most actionable signals a service can emit — it tells you a dependency just crossed from degraded to down, roughly a minute before your dashboards make it obvious. The OnStateChange hook exists for exactly this; a single counter increment there gives you alerting on dependency health for free.
None of this requires exotic tooling: the standard library covers timeouts and context propagation, a retry loop is thirty lines, and gobreaker handles the state machine you should not hand-roll. The hard part is not the code — it is deciding, per dependency, what deserves a retry, what deserves a fallback, and what deserves to fail fast.