Your service was up 99.95% of the time last month. Great number — until you realize that users filed complaints all month, the on-call phone never stopped buzzing, and half the “uptime” was spent serving requests so slowly that clients timed out anyway. Raw availability tells you the server responded. It says nothing about whether the service was actually usable. That gap is why SLOs — service level objectives — and their companion concept, the error budget, have become the standard way serious teams talk about reliability.
This post walks through building SLOs that hold up in production: choosing indicators that reflect user experience, setting targets you can actually sustain, deriving an error budget your team can spend deliberately, and wiring it into alerting so that pages fire only when the budget is genuinely burning — not on every transient blip. The examples use Prometheus alerting rules and Kubernetes-native deployment, but the concepts apply to any stack.
SLIs: Measure What Users Experience
A service level indicator (SLI) is the measurement itself — a carefully chosen ratio of “good events” to “total events.” The most common indicator is availability: the fraction of requests answered successfully. But the word “successful” is where most SLO programs quietly go wrong. Is a 500 an unsuccessful request? Obviously. Is a request that took nine seconds and was abandoned by the client successful? The server returned 200, so by naive counting it was — yet from the user’s chair, it failed.
The fix is to define success from the consumer’s point of view: a request is good if it returned a useful response within an acceptable time. Concretely, that means counting both status codes and latency in the same indicator. The Google SRE Workbook’s chapter on implementing SLOs recommends exactly this pattern — a single ratio of good events to valid events, where “good” encodes everything the user cares about.
For most request-serving systems, two indicators cover the ground:
- Availability SLI — the fraction of requests answered with a correct response, where a correct response excludes both server errors and client errors caused by your own infrastructure (an overloaded gateway returning 429 is your failure, not the client’s).
- Latency SLI — the fraction of requests answered faster than a threshold your users would call responsive, typically somewhere between 100ms and 1s depending on the operation.
Resist the urge to measure everything. An SLI set with a dozen indicators fragments attention and makes budgets impossible to reason about. If you can only afford one, start with availability plus a latency threshold — that combination captures most of what users perceive as “the service is broken.”
Choosing Targets You Can Sustain
An SLO is the target you set on an SLI — 99.9% of requests served in under 300ms, for example. The number should come from three inputs: what users actually need, what your infrastructure can deliver, and what the business can afford. Skipping any of the three produces a target that looks good in a slide deck and fails in production.
The most common mistake is defaulting to “four nines” everywhere. Each additional nine costs dramatically more than the last, and the difference between 99.9% and 99.99% is invisible to users if your latency threshold is wrong. Four minutes of downtime per month versus fifty-two minutes matters only if those minutes land during traffic peaks. Meanwhile, nailing four nines usually means gold-plated redundancy, aggressive over-provisioning, and an on-call rotation that never sleeps — for a benefit most products can’t monetize.
A practical calibration approach:
- Pick an internal target slightly tighter than what you’ll publish externally, so normal operational noise doesn’t breach the public promise.
- Set the latency threshold by looking at real traffic distributions, not round numbers. If p99 today is 240ms, a 100ms target is a re-architecture, not an objective.
- Give every dependency its own realistic target. Your API can’t sustain a tighter SLO than the slowest database it depends on.
The Error Budget: Reliability as a Spendable Resource
The error budget is simply the complement of your SLO: at 99.9% availability, you’re budgeted 0.1% of failed requests over the window. On a service handling 100 million requests per day, that’s 100,000 failed or too-slow requests per day that you are allowed to spend. This reframing changes everything about how engineering teams operate, because reliability stops being an absolute demand and becomes a finite resource with a balance.
That balance is the arbiter in the perennial conflict between feature velocity and stability. The deal is explicit: while budget remains, ship features, take measured risks, deploy on Fridays. When the budget is exhausted, feature work pauses and the team pays down reliability debt — hardening the failing component, fixing the flaky deploy pipeline, adding the load test that would have caught the regression. Neither side wins permanently; the number decides, and both sides agreed to the number in advance.
Budgets also protect you from your own alerts. If you currently page on every error spike, you’re spending human attention — your scarcest reliability resource — on failures the budget can absorb. A single 503 from a rolling restart shouldn’t wake anyone if it doesn’t meaningfully dent the budget. The budget tells you which failures matter.
Multi-Window Burn-Rate Alerting
The standard pattern — popularized by the Google SRE Workbook’s alerting-on-SLOs chapter and now built into tooling like the sloth SLO generator and kube-prometheus-stack — is burn-rate alerting. Instead of asking “did we breach the SLO,” it asks “are we consuming budget fast enough that we will breach it.” Burn rate is the ratio of actual consumption to sustainable consumption: a burn rate of 1 means you’re on pace to exactly exhaust the budget over the window; a burn rate of 14 means you’ll blow through a 30-day budget in about two days.
Why multi-window? Because a single short window is noisy and a single long window is slow. A pure 1-hour check pages on a five-minute blip that recovers on its own. A pure 3-day check lets an outage smolder for hours before anyone looks. The fix is to require both a fast window and a slow window to be burning before alerting: the fast window provides responsiveness, the slow window confirms the problem is real and ongoing rather than a spike that already ended.
The canonical configuration pairs two severity levels:
- Page when the burn rate is 14.4x over 1 hour AND 14.4x over 5 minutes. At that pace, a 30-day budget (2% headroom for a 98% SLO) burns out in roughly an hour of sustained failure — this is a genuine emergency.
- Page (lower urgency) or ticket at 6x over 6 hours AND 6x over 30 minutes — budget exhaustion in about three days if nothing changes. Serious, but not a drop-everything fire.
With Prometheus recording rules, the implementation is compact. Assume a http_requests_total counter labeled by code, and an SLO of 99.9% availability:
groups:
- name: slo:availability
interval: 30s
rules:
# Error ratio over the fast windows
- record: slo:error_ratio_rate5m
expr: |
sum(rate(http_requests_total{code=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))
- record: slo:error_ratio_rate1h
expr: |
sum(rate(http_requests_total{code=~"5.."}[1h]))
/ sum(rate(http_requests_total[1h]))
# Error ratio over the slow confirmation windows
- record: slo:error_ratio_rate30m
expr: |
sum(rate(http_requests_total{code=~"5.."}[30m]))
/ sum(rate(http_requests_total[30m]))
- record: slo:error_ratio_rate6h
expr: |
sum(rate(http_requests_total{code=~"5.."}[6h]))
/ sum(rate(http_requests_total[6h]))Then the alerts compare burn against the budget. The math: with a 99.9% SLO, the tolerated error fraction is 0.001, so burning at rate R means the error ratio is R × 0.001.
groups:
- name: slo:alerts
rules:
- alert: AvailabilityBudgetBurningFast
expr: |
(slo:error_ratio_rate5m > (14.4 * 0.001))
and
(slo:error_ratio_rate1h > (14.4 * 0.001))
for: 2m
labels:
severity: critical
annotations:
summary: "Availability error budget burning fast"
description: "Burn rate > 14.4x for 5m and 1h. At this pace
the 30-day budget is gone in ~1h of sustained failure."
- alert: AvailabilityBudgetBurningSlow
expr: |
(slo:error_ratio_rate30m > (6 * 0.001))
and
(slo:error_ratio_rate6h > (6 * 0.001))
for: 15m
labels:
severity: warning
annotations:
summary: "Availability error budget burning steadily"
description: "Burn rate > 6x for 30m and 6h. Budget gone in
~3 days if the current rate continues."The and operator is the heart of the design: it requires both windows to agree before firing. A momentary 15x spike for two minutes fails the 1-hour condition and stays silent. A steady 1.2x drip — annoying but budget-sustainable — never crosses either threshold. Pages now correspond to genuine emergencies, which is precisely the property that lets on-call engineers trust the pager.
One subtlety: alert on the fast-burn condition with a short for duration (or none), but keep the slow-burn alert as a ticket rather than a page if your volume is low. At low request volumes, percentage-based indicators swing wildly on a handful of failures, and a 6x burn may represent four failed requests. Scaling SLOs to low-traffic services is a real problem — if “total events” per hour is in the hundreds, consider counting failures in absolute terms or extending the window instead.
Enforcing the Budget in Kubernetes
Postgres can’t run your SLO policy, but your Kubernetes resource configuration quietly decides whether you can meet it. Reliability under load depends on the scheduler treating your latency-critical pods differently from batch work, and Kubernetes has first-class machinery for that.
First, resource requests and limits. A pod with requests equal to limits is in the Guaranteed QoS class, which means the kubelet treats it as the last thing to evict under node pressure and it gets the most stable CPU allocation. Latency-sensitive services that form your SLO’s critical path benefit measurably from Guaranteed class — CPU throttling from CFS quota on a Burstable pod is a classic hidden source of p99 latency spikes that burn budget without any error in your logs.
resources:
requests:
cpu: "2"
memory: 2Gi
limits:
cpu: "2"
memory: 2GiSecond, PriorityClasses and preemption. Give your SLO-critical workloads a high priority so that when a node is oversubscribed, the scheduler evicts best-effort batch jobs — never your request path. Combined with a PodDisruptionBudget to cap voluntary disruptions during node drains, this converts “we hope the important pods survive” into a scheduling guarantee.
Third, decide what to do when the budget is exhausted, and encode it. Teams at this maturity stage typically have two tiers: SLO-critical deployments get a change freeze when the budget is gone, while a “batch” tier of low-priority workloads is pre-configured to degrade — feature flags dark, autoscaling floor raised, or non-critical consumers paused. Whether the enforcement is a manual freeze declared in the incident channel or an automated policy that scales down non-critical consumers, the important part is that the response was agreed before the budget ran out, not negotiated during an outage.
Getting Started Without Overengineering
SLO programs fail most often from ambition: fourteen indicators, targets invented in a conference room, alerts wired to every threshold from day one. A workable first iteration fits in a sprint:
- Pick your single most important service and define one availability SLI and one latency SLI with thresholds grounded in current traffic.
- Set 99.9% as the starting target, measure for a month, and adjust based on what the data says rather than what the slide said.
- Add the two burn-rate alert rules above. Delete the cruder “error rate > 1% for 5m” alerts they replace.
- Review the budget in the weekly ops meeting. Spend it deliberately; freeze when it’s gone.
The payoff compounds: once paging is tied to budget burn, alert noise drops, on-call trust rises, and the feature-versus-stability argument gets settled by arithmetic instead of seniority. Reliability was never a feeling — it’s a budget, and now you can see the balance.