Most SLO programs die the same way. A team writes “99.9% availability” on a slide, everyone nods, and six months later the number has influenced exactly zero decisions. Nothing alerts on it, nothing freezes when it’s violated. The number itself was never the hard part. The hard parts are choosing a signal that reflects what users actually experience, converting the target into a budget someone can spend, and wiring that budget into alerting that fires early enough to matter but rarely enough to be ignored.
This post walks through that wiring: an SLI that isn’t theater, the arithmetic that turns an SLO into a budget, why the two most obvious alerting strategies both fail, and the multi-window burn-rate alerts that mature reliability programs have converged on — with a working Prometheus implementation.
Start With a Signal Users Can Feel
An SLI is the measured quantity; the SLO is the target you set for it. The most common mistake is choosing an SLI because it’s easy to export, not because users feel it. CPU utilization, pod restarts, and replica counts are internal plumbing — a service can be at 20% CPU and totally broken, or pegged at 95% CPU and perfectly healthy. The SRE workbook’s chapter on implementing SLOs makes a simple recommendation: pick SLIs that measure the user’s journey, expressed as the ratio of good events to valid events.
For an HTTP API, the workhorse is a request-based availability SLI: the fraction of requests that succeed, excluding server-side 5xx errors — client errors usually indicate a bad request, not a broken service. Latency deserves its own SLI: the proportion of requests completing faster than a threshold your product can defend, say 300 milliseconds — more honest than an average, which can stay flat while tail latency quietly doubles. One SLI per user-facing dimension is enough; teams that define fifteen end up defending none.
The Error Budget Is the Point
Here’s the arithmetic that makes an SLO useful. The error budget is 100% minus the SLO, over the compliance window. With a 99.9% availability SLO over 30 days, a service handling 3 million requests has a budget of 3,000 failed requests. That’s not a rounding error — it’s a quantity. One bad deploy that fails 1,500 requests spends half the month’s budget in an afternoon, and now you have a number you can put in a release decision.
This reframing is what separates working SLOs from dashboard decoration. A target percentage asks a yes/no question at the end of the month, when it’s too late to act. A budget is spendable now: every failed request draws it down, as do planned downtime, risky experiments, and fast-moving launches — all spend the same currency. When the budget is rich, you move fast. When it’s thin, you slow down. That trade-off, not the decimal, is the product of the whole exercise.
Why Percentage-of-Time Alerts Misfire
The naive implementation is to alert whenever the error rate exceeds the SLO threshold: “page me when the 5xx ratio goes above 0.1%.” This fails in two opposite directions at once. Over a short window, ordinary background noise crosses the threshold constantly, and you page on problems that would never meaningfully dent a 30-day budget. Over the full compliance window, the average crosses the threshold only after the budget is long gone — an obituary, not an alarm. You want to know one thing: how fast is the budget being consumed right now, and is that pace sustainable for the rest of the window?
Burn Rate: Pacing, Not Position
The burn rate is how fast you’re consuming the error budget relative to the pace that would exactly exhaust it by the end of the window. A burn rate of 1.0 means that if the current error rate holds, you’ll finish a 30-day window with exactly zero budget left — for a 99.9% SLO, a steady 0.1% error rate. Burn rates convert directly into budget-consumption deadlines, which is what makes them useful as alert thresholds. For a 30-day (720-hour) window:
- A 14.4× burn rate sustained over 1 hour consumes 2% of the monthly budget (14.4 × 1/720). Detectable in minutes, worth waking someone up.
- A 6× burn rate sustained over 6 hours consumes 5% of the budget. Slower bleed, still pager-worthy.
- A 1× burn rate sustained over 3 days consumes 10% of the budget. Not an incident — a trend worth a ticket.
The SRE workbook’s alerting chapter recommends exactly these three tiers for a 99.9% SLO, and they generalize well: the thresholds encode “how much of the month’s safety margin did this problem just eat?” rather than “did a rate cross an arbitrary line?”
Pairing Windows to Kill False Positives
There’s one more failure mode to close. A 1-hour window at 14.4× burn keeps firing long after the incident ends, because the elevated rate stays in the window average for up to an hour. The fix is to require a second, shorter window — the workbook suggests roughly 1/12 the length of the long one — to cross the same threshold. Both must breach for the alert to fire.
The long window provides sensitivity — it only accumulates enough burn to trip on sustained problems, filtering out blips. The short window provides recency — it clears within minutes of the errors stopping, so alerts self-resolve. A sustained 15% error rate trips the 14.4× pair within minutes, while a two-minute burst never accumulates enough long-window burn to trigger anything.
A Prometheus Implementation
Recording rules compute the error ratios once; alert rules compare them against burn-rate thresholds. First the SLI recordings:
groups:
- name: api-availability-sli
interval: 30s
rules:
- record: api_http_error_ratio:5m
expr: |
sum(rate(http_requests_total{job="api", code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api"}[5m]))
- record: api_http_error_ratio:30m
expr: |
sum(rate(http_requests_total{job="api", code=~"5.."}[30m]))
/
sum(rate(http_requests_total{job="api"}[30m]))
- record: api_http_error_ratio:1h
expr: |
sum(rate(http_requests_total{job="api", code=~"5.."}[1h]))
/
sum(rate(http_requests_total{job="api"}[1h]))
- record: api_http_error_ratio:6h
expr: |
sum(rate(http_requests_total{job="api", code=~"5.."}[6h]))
/
sum(rate(http_requests_total{job="api"}[6h]))
- record: api_http_error_ratio:3d
expr: |
sum(rate(http_requests_total{job="api", code=~"5.."}[3d]))
/
sum(rate(http_requests_total{job="api"}[3d]))Then the multi-window alerts. The 0.001 factor is the 0.1% error rate that a 99.9% SLO permits at burn rate 1, so each expression reads naturally: “is the ratio above 14.4 times the budgeted error rate, in both windows?”
groups:
- name: api-availability-burn-alerts
rules:
- alert: ApiAvailabilityFastBurn
expr: |
api_http_error_ratio:1h > 14.4 * 0.001
and
api_http_error_ratio:5m > 14.4 * 0.001
labels:
severity: page
annotations:
summary: "Fast burn: 2 percent of monthly budget spent in 1h"
- alert: ApiAvailabilitySlowBurn
expr: |
api_http_error_ratio:6h > 6 * 0.001
and
api_http_error_ratio:30m > 6 * 0.001
labels:
severity: page
annotations:
summary: "Slow burn: 5 percent of monthly budget spent in 6h"
- alert: ApiAvailabilityBudgetDrain
expr: |
api_http_error_ratio:3d > 1 * 0.001
and
api_http_error_ratio:6h > 1 * 0.001
labels:
severity: ticket
annotations:
summary: "Budget drain: 10 percent of monthly budget spent in 3d"The two page rules should route to the on-call rotation; the ticket rule goes to the backlog. Dashboards showing remaining budget and per-release budget attribution layer on top of the same ratios. The Prometheus alerting rules documentation covers the expression syntax if you need to adapt the recording rules to your own metric names.
When the Budget Runs Dry
An alert policy without a spending policy is half a program. Decide in advance what happens when the budget is exhausted: reliability work outranks features until the budget recovers, risky rollouts pause, and exceptions go through the SLO’s owner. The specific policy matters less than agreeing on one before the incident — mid-incident is the worst time to negotiate whether reliability or roadmap wins.
Budgets also refill on a rolling window, not a calendar. With a 30-day compliance window, a bad week isn’t forgiven on the first of the month — it keeps constraining you until it ages out. Some teams shorten the window to 7 days to make recovery faster; the math above works identically, just with different window arithmetic.
Pitfalls Worth Avoiding
- SLOs on plumbing instead of journeys. If users can’t feel a metric’s failure, it’s a health check, not an SLI.
- Chasing perfection. A 100% target means zero budget — no deploys, no maintenance, no experiments. The budget is the feature, not the flaw.
- Low-traffic blindness. A service taking 1,000 requests a day has a budget of one failure per day at 99.9%, and ratio-based alerts get noisy. Widen the windows or aggregate across replicas so the statistics mean something.
- Too many SLOs. One or two per critical user journey. Every additional target is another thing to alert on and eventually ignore.
- Alerting on the SLO directly. If your pages come from the burn-rate rules, delete the raw “error rate exceeded the SLO” alert — it will either spam you or fire too late.
Wrapping Up
The sequence that makes SLOs real is short: measure a signal users feel, convert the target into a spendable budget, and alert on the pace of spending with paired windows so pages are both early and trustworthy. Everything downstream — release policies, on-call sanity, the argument about shipping on Friday — gets a number it can point at. Start with one journey, one availability SLI, and the three-tier burn-rate configuration above; add latency targets once the first one has survived a real incident.