Every distributed system has a breaking point. The downstream service you depend on will fail — not “might” fail, will fail. A network partition, a slow query, a deploy gone wrong, or a traffic spike will push a dependency past its limits. When that happens, the clients hammering that service don’t just get errors; they amplify the failure. Every request that would normally return in 50ms now hangs for 30 seconds waiting for a timeout. Thread pools exhaust, connection queues fill, and the failure cascades through your entire architecture.
This is the cascading failure problem, and the circuit breaker pattern is the primary defense. Unlike a retry, which tries harder, a circuit breaker stops trying. It wraps each external call in a state machine that monitors success and failure rates, opens to protect the system when things go wrong, and automatically recovers when health returns. Let’s look at how circuit breakers work in practice, when to use them, and how to implement them cleanly in Go.
The Three States Every Circuit Breaker Knows
A circuit breaker operates as a finite state machine with three states:
- Closed — Normal operation. All requests pass through. The breaker tracks success and failure counts. When failures exceed a threshold, it trips to Open.
- Open — Requests are rejected immediately. No call is made to the downstream service. This is the protection phase. After a configurable timeout, the breaker moves to Half-Open.
- Half-Open — A limited number of test requests are allowed through. If they succeed, the breaker closes and normal traffic resumes. If they fail, it re-opens. This is the recovery phase.
The genius of this design is that recovery is automatic. No human intervention, no restart, no deploy. The breaker continuously probes the downstream service and restores traffic the moment it’s healthy.
Why Retries Alone Are Not Enough
Retries and circuit breakers solve different problems. A retry handles transient failures — a momentary network blip, a 503 from an overloaded server. A circuit breaker handles sustained failures — the downstream service is down and will stay down for a while.
The danger of combining them naively is retry storms. If every client retries failed requests with exponential backoff, and the downstream service is already struggling, the retry traffic can double or triple the load. With a circuit breaker in front, retries only happen when the breaker is Closed or Half-Open. When it’s Open, requests fail fast before the retry layer even sees them.
Implementing a Circuit Breaker in Go
While you can build a circuit breaker from scratch, the gobreaker library provides a well-tested, generics-based implementation. The v2 API uses type parameters so your breaker is statically typed — no interface{} casting required.
package main
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"time"
"github.com/sony/gobreaker/v2"
)
type User struct {
ID int `json:"id"`
Email string `json:"email"`
}
var userCB *gobreaker.CircuitBreaker[*User]
func init() {
userCB = gobreaker.NewCircuitBreaker[*User](gobreaker.Settings{
Name: "user-service",
MaxRequests: 3,
Interval: 60 * time.Second,
Timeout: 30 * time.Second,
ReadyToTrip: func(c gobreaker.Counts) bool {
// Trip when failure ratio exceeds 60%
failureRatio := float64(c.TotalFailures) / float64(c.Requests)
return c.Requests > 10 && failureRatio > 0.6
},
OnStateChange: func(name string, from, to gobreaker.State) {
fmt.Printf("Circuit %s: %s -> %s\n", name, from, to)
},
})
}
func GetUser(ctx context.Context, userID int) (*User, error) {
return userCB.Execute(func() (*User, error) {
req, err := http.NewRequestWithContext(ctx, "GET",
fmt.Sprintf("https://api.example.com/users/%d", userID), nil)
if err != nil {
return nil, err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode >= 500 {
return nil, fmt.Errorf("user service error: %d", resp.StatusCode)
}
var user User
if err := json.NewDecoder(resp.Body).Decode(&user); err != nil {
return nil, err
}
return &user, nil
})
}The key decisions are in the Settings struct. MaxRequests controls how many test calls are allowed in Half-Open — keeping this low (3 is reasonable) prevents overwhelming a recovering service. ReadyToTrip is where you define what “broken” means for your service. The default trips after 5 consecutive failures, but a failure-rate threshold is more resilient to brief blips.
Classifying Errors: Not Every Error Is a Failure
One of the most overlooked aspects of circuit breaking is error classification. A 404 response from the user service is not a service failure — it’s a valid business response. You don’t want client errors (4xx) tripping your circuit breaker, only server errors (5xx) and network failures.
type transientError struct{ err error }
func (e *transientError) Error() string { return e.err.Error() }
func (e *transientError) Unwrap() error { return e.err }
userCB = gobreaker.NewCircuitBreaker[*User](gobreaker.Settings{
Name: "user-service",
// ... other settings ...
// IsSuccessful: treat 4xx errors as success (they're valid responses)
IsSuccessful: func(err error) bool {
var httpErr *transientError
if errors.As(err, &httpErr) {
return true // Client errors are "successful" for circuit purposes
}
return err == nil
},
// IsExcluded: don't count context cancellations at all
IsExcluded: func(err error) bool {
return errors.Is(err, context.Canceled) ||
errors.Is(err, context.DeadlineExceeded)
},
})The IsSuccessful callback lets you define which errors are “real” failures. Client-side errors like validation failures shouldn’t count. The IsExcluded callback goes further — it removes certain errors from the count entirely. Context cancellations (from request timeouts or client disconnects) should always be excluded, since they reflect caller behavior, not downstream health.
Using Rolling Windows Instead of Fixed Counts
By default, a circuit breaker resets its counts on each state change. This means a burst of failures just after a reset might not trip the breaker because the count starts from zero. The BucketPeriod setting solves this by implementing a rolling window — counts are maintained across multiple time buckets and aged out gradually.
userCB = gobreaker.NewCircuitBreaker[*User](gobreaker.Settings{
Name: "user-service",
Interval: 60 * time.Second,
BucketPeriod: 10 * time.Second, // 6 buckets of 10s
ReadyToTrip: func(c gobreaker.Counts) bool {
failureRatio := float64(c.TotalFailures) / float64(c.Requests)
return c.Requests > 10 && failureRatio > 0.6
},
// ... other settings ...
})With this configuration, the breaker maintains 6 ten-second buckets. Old failures age out instead of disappearing all at once. The Interval is automatically adjusted to be a multiple of BucketPeriod.
Fallback Strategies: What Happens When the Circuit Is Open
When the circuit is open, callers get an error immediately. That’s better than waiting 30 seconds for a timeout, but it’s still an error. A good circuit breaker setup includes a fallback strategy:
- Cached response: Return the last known good value from a local cache or Redis.
- Default value: Return a safe default (empty result set, degraded feature flag).
- Alternative service: Route to a secondary endpoint or read replica.
func GetUserWithFallback(ctx context.Context, userID int) (*User, error) {
user, err := GetUser(ctx, userID)
if errors.Is(err, gobreaker.ErrOpenState) {
// Circuit is open — try cache
if cached, ok := userCache.Get(userID); ok {
return cached, nil
}
return &User{}, nil // safe default
}
return user, err
}Observability: Making Breaker State Visible
A circuit breaker that silently opens is almost as dangerous as no breaker at all. When traffic drops, you need to know why. The OnStateChange callback is your hook for alerting. In production, wire it to your metrics system:
OnStateChange: func(name string, from, to gobreaker.State) {
// Emit a metric for dashboarding
breakerState.WithLabelValues(name).Set(float64(to))
// Log for audit trail
logger.Info("circuit breaker state change",
"breaker", name,
"from", from.String(),
"to", to.String(),
)
// Alert on open state
if to == gobreaker.StateOpen {
alerter.Notify(name + " circuit opened")
}
}The key metric to expose is the current state of each breaker. Track it as a gauge (0=closed, 1=open, 2=half-open) so dashboards can show breaker health at a glance. Also track the counts — Requests, ConsecutiveFailures, and TotalSuccesses — so you can understand why a breaker tripped.
Common Pitfalls to Avoid
- Setting thresholds too aggressively. A threshold of 3 consecutive failures in a system with 1% error rate will trip constantly. Size your threshold to your traffic volume and normal error rate.
- Forgetting to exclude caller-side errors. Context cancellations from client disconnects should never trip a breaker — they tell you nothing about downstream health.
- No fallback. If the only thing a caller can do with an open circuit is panic, your circuit breaker hasn’t solved the problem — it’s just made it fail faster.
- One breaker per service, not per endpoint. A monolithic breaker for “the payment service” will trip if the refund endpoint fails, blocking all payments. Create separate breakers for separate failure domains.
Wrapping Up
Circuit breakers are one of those patterns that seems simple in theory but has enough edge cases to matter in practice. The core idea — stop calling a failing service — is sound. But the details of error classification, threshold tuning, rolling windows, and fallback strategies determine whether your breaker is a safety net or a source of false alarms. Start with conservative thresholds, make breaker state observable, and tune based on real traffic patterns. Your downstream services (and your on-call engineers) will thank you.
If you’re building Go services, gobreaker gives you a clean, generics-based implementation that’s production-proven across thousands of projects. For a broader resilience toolkit, also look at failsafe-go which combines circuit breaking with retries, timeouts, and bulkheads in a unified API.