Every team says they care about reliability, but very few can answer the follow-up question: how reliable, exactly, and how do you know when you have stopped being reliable enough? Without a shared definition of “reliable,” every outage becomes a debate about opinions, every alert is either noise or a fire drill, and the tension between shipping features and keeping the system stable is settled by whoever argues loudest. Service level objectives, or SLOs, are the mechanism that turns that debate into arithmetic.
The core idea is simple to state and surprisingly hard to do well. You pick a small number of user-facing indicators — things like “the API responds within 300ms” or “the checkout endpoint returns success” — you set a target for how often those must hold over a rolling window, and then you manage the system against the resulting error budget: the small slice of failures you are allowed to spend. This post walks through how to define indicators that are worth measuring, how to compute budgets without fooling yourself, and how to wire them into alerting so that pages correspond to actual budget burn rather than raw metric spikes.
SLI, SLO, and the error budget in plain terms
Three terms do all the work here. A service level indicator (SLI) is a carefully specified measurement of a service’s behavior from the user’s perspective — the ratio of good events to total events. A service level objective (SLO) is a target value for that ratio over a rolling time window. The error budget is whatever you did not promise: 100% minus the SLO target.
Concretely, if your availability SLO says “99.9% of requests to the checkout service succeed over 30 days,” then your SLI is the success ratio itself, and your error budget is 0.1% of requests over 30 days — roughly 43 minutes of full-downtime equivalent in a month, or, more usefully, a pool of failed requests you can allocate across partial degradations. The critical mental shift is that the budget is a resource, like CPU or money. When you still have budget, you can take risks: ship the migration, roll out the aggressive caching change, tolerate a risky deploy. When the budget is gone, the arithmetic says stop and stabilize. That’s what makes an SLO a decision-making tool rather than a vanity metric — the Google SRE books frame this as the mechanism that aligns product and engineering incentives around a shared, quantified risk appetite.
One nuance worth internalizing early: the number of nines is not a score to maximize. 100% availability is not only unattainable, it is actively harmful as a target — it forces you to slow feature velocity to a crawl in exchange for reliability users cannot perceive. Users cannot tell 99.95% from 99.99%; they can absolutely tell when your release cadence died because the team spends every sprint chasing the last 0.04%. Pick the target that matches what users actually notice and what the business can afford, and treat everything up to that line as spendable.
Choosing SLIs that track user experience
The most common mistake in SLO programs is measuring what is easy instead of what matters. CPU utilization, memory pressure, and queue depths are useful health signals, but they are poor SLIs because they are indirect: a machine can be maxed out while users are fine, and users can be miserable while every dashboard is green. A good SLI measures the event the user actually experiences.
In practice, four indicator families cover most services:
- Availability — the fraction of requests that return a successful response. Define “success” precisely: for an HTTP API that usually means 5xx are bad while 4xx are good (the client made a mistake, but the service did its job), and you will likely want to explicitly exclude certain routes — health checks, bot traffic — from the denominator.
- Latency — the fraction of requests served faster than a threshold. Note the shape of this definition: it is a “good event ratio,” not an average. The mean hides the tail, and the tail is where humans live — the 99th-percentile user is often your most engaged user hitting the most data.
- Correctness — the fraction of responses that are not just fast and 200-OK, but right. Often requires product instrumentation: a background job that “succeeds” but silently wrote the wrong field is not a good event.
- Freshness / durability — for data systems, how stale reads are allowed to be, or the fraction of writes that survive any single-machine failure. These map naturally onto replication lag and write-acknowledgment semantics.
Keep the count small. A service with a dozen SLOs has no SLOs — nobody can reason about which budget matters when they conflict. Two to four per service, covering the interactions users actually care about, is the practical sweet spot described in the SRE guidance on objectives in practice. When two indicators fight (say, an aggressive retry policy makes availability look better while latency craters), that conflict is itself a finding worth surfacing to the team.
Computing the budget: aggregation windows and the math that misleads
Once you have indicators, the mechanical part is computing good-event ratios and comparing them against targets. But three implementation details decide whether your numbers mean anything.
First, define the denominator ruthlessly. Requests from synthetic monitors, crawlers, and broken internal tooling can dominate a low-traffic service’s error budget. Most teams scope the SLI to the user-facing endpoints that carry real traffic, and drop maintenance-window periods from the calculation — but document those choices in the SLO itself, because every exclusion is a place where the number can quietly drift from reality.
Second, prefer time-based availability for user-visible surfaces. “The service responded to at least some requests during 99.9% of one-minute windows” is often a better proxy for user experience than raw request ratios for low-QPS endpoints, because it captures the difference between “a blip” and “down for 45 minutes.” Request-based ratios are better for high-volume, per-interaction surfaces. The choice changes what your budget means, so pick it deliberately per SLI.
Third, beware the tyranny of the long window. A 30-day availability budget that is 100% spent tells you the month was bad, but it does not page anyone at 3 a.m. when things start going wrong. Long windows are for governance and quarterly reviews; fast alerting needs a shorter lens. That is where burn rates come in.
Alerting on burn rate, not on raw thresholds
The insight that separates SLO-based alerting from threshold alerting is this: what should page you is not “error rate exceeded X” but “you are consuming budget fast enough that the budget will be gone before anyone reasonable would notice.” That rate of consumption is the burn rate — how many times faster than sustainable pace you are spending the budget.
The math: if you have a 30-day budget, a burn rate of 1.0 means you will exhaust it in exactly 30 days. A burn rate of 36 means you exhaust it in about 20 hours. The widely used multiwindow scheme from the SRE workbook chapter on alerting pairs a fast window with a long window so that a brief spike does not page you but a sustained high burn does:
- Page: burn rate > 14.4 over 1 hour and > 14.4 over 5 minutes — exhausts a 30-day budget in 2 days; catches real outages within minutes.
- Page (slower): burn rate > 6 over 6 hours and > 6 over 30 minutes — catches slower burns within a few hours without firing on every blip.
- Ticket, don’t page: burn rate > 1 over 3 days — something is quietly eating the budget; look at it during working hours.
Here is what that looks like as a Prometheus alerting rule for a request-based availability SLI (5xx against a user-facing route):
groups:
- name: checkout-availability-slo
rules:
# Budget is 0.1% over 28 days. Sustained burn > 14.4x for 1h
# (confirmed on a 5m window) exhausts the budget in ~2 days.
- alert: CheckoutAvailabilityBudgetBurnFast
expr: |
(
sum(rate(http_requests_total{service="checkout",code=~"5.."}[1h]))
/
sum(rate(http_requests_total{service="checkout"}[1h]))
) > (14.4 * 0.001)
and
(
sum(rate(http_requests_total{service="checkout",code=~"5.."}[5m]))
/
sum(rate(http_requests_total{service="checkout"}[5m]))
) > (14.4 * 0.001)
for: 2m
labels:
severity: page
annotations:
summary: "Checkout availability burning error budget fast (14.4x)"
# Slower burn: 6x over 6h confirmed on 30m. Catches gradual decay.
- alert: CheckoutAvailabilityBudgetBurnSlow
expr: |
(
sum(rate(http_requests_total{service="checkout",code=~"5.."}[6h]))
/
sum(rate(http_requests_total{service="checkout"}[6h]))
) > (6 * 0.001)
and
(
sum(rate(http_requests_total{service="checkout",code=~"5.."}[30m]))
/
sum(rate(http_requests_total{service="checkout"}[30m]))
) > (6 * 0.001)
for: 15m
labels:
severity: page
annotations:
summary: "Checkout availability burning error budget (6x, slow)"
# Budget exhaustion tracking: what fraction of the 28d budget is left?
- record: checkout:slo_error_budget_remaining
expr: |
1 - (
(
sum(increase(http_requests_total{service="checkout",code=~"5.."}[28d]))
/
sum(increase(http_requests_total{service="checkout"}[28d]))
) / 0.001
)A few things are worth calling out in that rule set. The and operator is doing the multiwindow confirmation — both rate windows must agree the burn is happening, which kills single-tick false positives. The recording rule for remaining budget is deliberately a separate series, so your dashboards and reviews can query “how much budget is left” without recomputing long-window ratios on every panel load. And the thresholds encode the 14.4x/6x scheme, not arbitrary error percentages — if you change the SLO window or target, the burn thresholds stay meaningful because they are expressed relative to budget pace. The general query mechanics behind expressions like these are covered in the Prometheus query documentation, and rule-file syntax in the alerting rules reference.
One practical trap: burn-rate alerting on percentage-based ratios behaves badly for very low-traffic services, because one or two failed requests can trip a 14x burn in a tiny sample. For low-QPS endpoints, switch to time-based SLIs (was the endpoint up, per minute) or aggregate across replicas before computing the ratio.
Making the budget govern: policy beats dashboards
An SLO that nobody acts on is a dashboard ornament. The difference between teams that get value from budgets and teams that don’t is rarely the math — it is whether an explicit policy exists for what happens at budget thresholds. A workable policy is boring and short: above 50% budget consumed in the window, note it in the weekly review; above 75%, reliability work displaces feature work on the next sprint; at 100%, feature launches freeze until the budget recovers or the SLO is renegotiated. The point is not the specific numbers — it is that everyone, including product management, agreed to them before the outage, when nobody was defending a launch date. The error budget policy chapter in the SRE workbook sketches this negotiation pattern, including how to handle the inevitable “but this launch is special” conversations.
Implementation-wise, the plumbing to make budgets real is modest: export request counters with enough labels to compute your SLIs, keep the SLO definitions (targets, windows, exclusions) in version control next to the alerting rules, and render budget burn on a dedicated dashboard rather than burying it in service graphs. Tools like OpenTelemetry give you the metric primitives — counters, histograms — that SLIs are computed from, with the metrics data model specification defining the instruments you will aggregate; you do not need a commercial “SLO platform” to get 80% of the value. What you do need is the discipline to change the SLO when reality proves it wrong — targets that are never missed and never relaxed both indicate a program that has stopped doing its job.
Getting started without boiling the ocean
If you are starting from zero, resist the urge to define SLOs for every service at once. Pick the single service whose degradation your users would notice first. Draft one availability and one latency SLI with explicit good-event definitions, set provisional targets from a month of measured data (not from aspiration), and wire up one fast-burn and one slow-burn alert. Run it for a month. You will learn more from that one lived-in SLO than from any amount of upfront design — including whether your alerts fire when they should, whether your exclusions were right, and whether product actually respects the budget freeze when it triggers. Then expand to the next service, carrying the corrections with you.
The deeper payoff is cultural as much as technical. Once “how much budget is left” becomes the shared language for reliability arguments, the loudest-voice problem disappears: the data decides when to ship and when to hold, and reliability stops being a value statement and becomes a managed resource. That is the whole point of the exercise — and it starts with one honest indicator on one service that matters.