Here is a failure mode that shows up in every system with more than two backends: round robin distributes requests perfectly evenly, and yet one instance is melting while its siblings idle. No unfair algorithm did this. The algorithm did exactly what it was designed to do — it just measured the wrong thing. Round robin counts requests; your users experience milliseconds. When request durations vary — and they always vary — equal request counts are unequal load.
This post walks through the load balancing algorithms that actually matter in production, why duration-aware and hash-based strategies exist, and how to wire the right one into a Go service. The examples use Envoy and a small Go client, but the reasoning transfers to any proxy or service mesh, including Istio, which builds on Envoy’s load balancing primitives.
Why round robin lies to you
Round robin assumes homogeneity: every request costs the same, every backend is equally fast. Under that assumption it is optimal — perfectly fair, zero state, O(1) per dispatch. Production violates the assumption in two ways:
- Request duration variance. A mix of 5ms health checks and 5s report queries means the backend that happened to catch the heavy requests is drowning while its neighbor has handled ten trivial calls in the same window.
- Backend heterogeneity. A canary instance with half the CPU of its stable siblings receives the same share of traffic as the full-size ones.
The classic result from queueing theory makes the damage precise: utilization climbs toward saturation, and queueing delay grows non-linearly with it. A backend at 90% utilization has far worse tail latency than one at 50% — the relationship is hyperbolic, not linear. Two backends receiving “equal” request counts can sit at wildly different utilizations purely because of request-mix luck, and your p99 lives in the unlucky one. This is why duration-aware balancing tends to outperform round robin on tail latency in real workloads even though average throughput looks identical on a dashboard.
Least outstanding requests: measure work, not counts
The fix is to balance on in-flight work instead of dispatch counts. Least-outstanding-request (LOR) balancing sends each new request to the backend with the fewest requests currently being processed. A backend that drew three slow queries shows three outstanding requests and stops receiving new work until it drains; the idle backend gets everything in the meantime. No request-duration model is needed — the in-flight count is a self-updating proxy for load.
LOR is not exotic. AWS Application Load Balancers default to it, and Envoy’s documentation of its supported load balancers describes the least-request policy with an active-health-check-based fast rejection path: before picking the low-count host, Envoy can compare two candidate hosts’ health and skip the choice if one is failing. Istio goes further in its guidance — its DestinationRule reference notes that LEAST_REQUEST generally outperforms ROUND_ROBIN and recommends it as a drop-in replacement.
The cost is trivial in practice: maintain a per-backend in-flight counter, increment on dispatch, decrement on completion, pick the minimum. In Go, an atomic counter per upstream is enough:
type backend struct {
addr string
inFlight atomic.Int64
healthy atomic.Bool
}
func pickLeastOutstanding(backends []*backend) *backend {
var best *backend
var bestCount int64 = math.MaxInt64
for _, b := range backends {
if !b.healthy.Load() {
continue
}
if n := b.inFlight.Load(); n < bestCount {
best, bestCount = b, n
}
}
if best == nil {
return nil // all backends unhealthy
}
best.inFlight.Add(1)
return best
}
// The caller must run: defer chosen.inFlight.Add(-1)
Note what this counter is not: it is not a concurrency limiter and not a health signal by itself. It only decides where the next request goes. Circuit breaking and health checking remain separate concerns — and conflating them is a common design mistake.
Consistent hashing: when affinity beats fairness
Least-outstanding is the right default for stateless request/response workloads. But some workloads are not really stateless in the way HTTP semantics pretend. Consider:
- Per-instance caches. If each backend memoizes expensive computations, sending the same key to different instances on every request destroys the hit rate.
- Session state held locally. Sticky routing keeps session data local instead of paying for a shared session store.
- Sharded work. Anything where “which shard owns this key” is a routing decision.
Naive hashing — hash(key) % N — solves affinity but breaks catastrophically on topology changes: removing one backend remaps nearly every key, instantly flushing the caches you were trying to protect. Consistent hashing fixes this by placing backends on a hash ring; each key routes to the next ring position clockwise. When a backend leaves, only the keys that hashed to it move — roughly 1/N, not nearly all.
Envoy exposes this as the ring-hash load balancer, and its ring hash policy is the standard tool for the canonical use case: routing all requests for a given entity to the same upstream so caches stay warm. A minimal hash ring in Go shows why the structure works:
type HashRing struct {
nodes sortedKeys // sorted hash values
backends map[uint64]*backend
}
func NewHashRing(backends []*backend, replicas int) *HashRing {
hr := &HashRing{backends: make(map[uint64]*backend)}
for _, b := range backends {
for i := 0; i < replicas; i++ {
h := hash64(fmt.Sprintf("%s#%d", b.addr, i))
hr.backends[h] = b
hr.nodes = append(hr.nodes, h)
}
}
sort.Slice(hr.nodes, func(i, j int) bool { return hr.nodes[i] < hr.nodes[j] })
return hr
}
func (hr *HashRing) Get(key string) *backend {
if len(hr.nodes) == 0 {
return nil
}
h := hash64(key)
idx := sort.Search(len(hr.nodes), func(i int) bool { return hr.nodes[i] >= h })
if idx == len(hr.nodes) {
idx = 0 // wrap the ring
}
return hr.backends[hr.nodes[idx]]
}
The replicas parameter (virtual nodes) matters: with one position per backend, hash-space coverage is uneven and some instance inherits a disproportionately large key range. A hundred or so virtual nodes per backend flattens the distribution. One trade-off to internalize: consistent hashing is load-oblivious. It gives you affinity at the price of duration-awareness — a hot key lands wherever it lands, regardless of that backend’s queue depth. Ring-hash plus a per-key concurrency cap is the usual mitigation when one key can get hot.
Weighted variants and canary deployments
Both strategies compose with weights. Weighted round robin handles heterogeneous hardware — give the 2x instance a 2x weight — and is the mechanism behind most canary rollouts: 95% of traffic to stable, 5% to the new build, adjusted over time. Envoy’s newer client-side weighted round robin policy pushes this further: instead of static weights configured by operators, the proxy computes weights dynamically from load reports the backends themselves emit, so a struggling instance’s weight drops automatically. That closes the loop that static weights leave open — hardware heterogeneity handled once at config time is fine; runtime variability (compaction storms, GC pauses, cache flushes) needs feedback.
A word of caution on weights and LOR together: weighting least-outstanding is subtler than multiplying counters. Get it wrong and you have built a priority inversion where the “less loaded” backend by weight is the busier one by queue depth. If you need both, prefer the proxy’s native implementation over a hand-rolled composite.
Outlier detection: balancing needs ejection
No balancing algorithm survives contact with partially-failing backends. Least-outstanding will happily send traffic to an instance that accepts connections but returns 503s — it has, after all, very few in-flight requests. This is why load balancing and outlier detection ship together in every serious proxy. Envoy’s outlier detection continuously monitors upstream responses and ejects hosts that breach thresholds: consecutive 5xx responses eject a host for a configured interval, after which it is probed again and, if healthy, gradually returned to rotation.
The passively-detected pattern is worth restating in Go terms, because it is small:
func (b *backend) recordResult(ok bool) {
if ok {
b.consecutiveErrors.Store(0)
return
}
if n := b.consecutiveErrors.Add(1); n == 5 {
b.healthy.Store(false)
time.AfterFunc(30*time.Second, func() {
// probe before restoring full traffic in production
b.healthy.Store(true)
b.consecutiveErrors.Store(0)
})
}
}
In production you would gate re-entry behind an active health probe rather than an unconditional timer, and add a cap on the fraction of the pool that can be ejected simultaneously — otherwise one bad deployment can eject your entire fleet and leave the pool empty.
Choosing: a decision table
- Stateless request/response, heterogeneous request cost → least outstanding requests. This is the majority of microservice traffic.
- Per-backend caches or local session state → consistent hashing (ring-hash) on a stable key.
- Known hardware asymmetry or canary routing → weighted round robin, preferably with runtime weight feedback.
- Everything → outlier detection with bounded ejection, plus health checks. Balancing chooses among healthy backends; health checking decides who is healthy.
The deeper principle under all of this: a load balancer is a feedback system, and every algorithm above is just a choice of signal — dispatch count, in-flight count, key identity, or reported load. Round robin fails not because it is naive but because dispatch count is the signal most weakly correlated with what you actually care about: tail latency. Pick the signal closest to your SLO, add ejection for the backends that lie about their health, and the “which instance got the slow query” pager incidents quietly stop happening.