SLOs You Can Actually Alert On: Error Budgets and Burn Rates in Prometheus

Every on-call rotation has a story about the alert that fires at 3 a.m. for something that isn’t wrong. CPU is at 87%, but the service is fine. Latency ticked above 400ms for one scrape, then settled. Memory is “high,” whatever that means for this process. After a few months of this, the team develops a reflex: read the alert name, decide it’s noise, go back to sleep. That reflex is how real incidents get missed.

The problem isn’t the paging system; it’s the question the alerts are asking. Resource metrics ask “is this number unusual?” — a question about the infrastructure. Users don’t experience CPU utilization. They experience whether the checkout request succeeded and how long it took. Service level objectives flip the question: pick the metrics that describe user experience, decide how much imperfection you can afford, and alert on the rate at which you’re spending that allowance. Everything else is dashboard material.

This post builds the full chain in Prometheus: define an SLI, turn it into an objective with a budget, write multiwindow burn-rate alerts, and put a reliability policy behind the numbers. The SRE workbook’s chapter on implementing SLOs covers the org-side theory; what follows is the engineering side.

SLI: A Ratio, Not a Feeling

A service level indicator is a ratio of good events to total events over a window. That’s the whole trick, and it’s what makes SLOs computable. “Availability” stops being a vibe and becomes: of all HTTP requests the service received, how many returned a 5xx or a 429? Both sides of the ratio come straight off the RED metrics your instrumentation library is already emitting:

sum(rate(http_requests_total{job="checkout", code=~"5..|429"}[5m]))
/
sum(rate(http_requests_total{job="checkout"}[5m]))

The rate() function converts the counter into per-second events over the window, and the division produces an error fraction that behaves the same at 10 requests per second and 10,000. Picking what counts as “bad” is a product decision worth a conversation: a 404 on a mistyped URL isn’t the service’s fault, but a 429 from your own rate limiter absolutely is the user’s experience of the service, so it belongs in the numerator.

SLO and the Error Budget

Fix a window — 28 days is the common choice, long enough to smooth deploys and weekends, short enough to feel current — and pick an objective. A 99.9% availability objective over 28 days says: at most 0.1% of requests may fail. That 0.1% is the error budget: just over 40 minutes of acceptable total downtime, or — for a request-based SLI — roughly 1 failed request in every 1,000.

The budget reframes reliability as a resource to spend. Deploys that cause a blip spend budget. A memory leak spends budget. This is what makes the reliability policy enforceable: when the budget is intact, ship aggressively; when it’s nearly exhausted, freeze feature work and pay down the reliability debt. The argument stops being “I feel we should slow down” and becomes arithmetic — and it cuts both ways, which is the point. A team burning only 10% of budget is being too conservative and should be taking more deploy risk.

Making It Fast: Recording Rules

Alerting rules evaluate frequently, and a ratio of two rate() aggregations is wasteful to recompute every evaluation cycle. Recording rules precompute the expression into a new time series on a fixed interval:

groups:
  - name: checkout-slo
    interval: 30s
    rules:
      - record: job:slo_errors_per_request:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{job="checkout", code=~"5..|429"}[5m]))
          /
          sum(rate(http_requests_total{job="checkout"}[5m]))

The naming convention looks bureaucratic — level:metric:operations — and it earns its keep the moment someone else opens your rule files six months later. The burn-rate alerts below consume these precomputed series, which keeps alert evaluation cheap even on large setups.

Burn-Rate Alerts: The Multiwindow Trick

You could alert when the 28-day error ratio exceeds 0.1%, but you’d page four weeks after the damage. You could alert on a 5-minute ratio, but a single slow minute would page you. Production alerting uses burn rates — how many times faster than sustainable you’re consuming budget right now — and each burn-rate alert is an AND of two windows: a short window that reacts fast and a long window that confirms the pattern is sustained. For a 99.9% objective with a 28-day window, the standard pairings are:

Page, 14.4× burn: 1h window AND 5m window — a hard outage; at this rate the 28-day budget is gone in about 2 days, and 2% of it is spent in the first hour. This is the 3 a.m. page.
Page, 6× burn: 6h window AND 30m window — a serious degradation that still exhausts the budget inside a week.
Ticket, 3× burn: 1d window AND 2h window — a slow bleed, budget gone in about 9 days. Route to a queue, not a phone.
Ticket, 1× burn: 3d window AND 6h window — you’re consuming budget at exactly the sustainable rate; investigate before the next window closes.

Why those multipliers? They encode “page me early, ticket me before things drift”: at 14.4× burn, 2% of a 28-day budget is spent within one hour and the whole budget lasts about two days; at 6× burn, 5% is spent in six hours; at 3×, 10% per day; at 1×, 10% every three days. The thresholds and windows move together so faster burns page faster, and the derivation is laid out in the workbook’s alerting chapter.

# Fast burn - page. 2% of budget in 1h, confirmed by the 5m view.
(
  job:slo_errors_per_request:ratio_rate1h > (14.4 * 0.001)
  and
  job:slo_errors_per_request:ratio_rate5m > (14.4 * 0.001)
)
or
(
  job:slo_errors_per_request:ratio_rate6h > (6 * 0.001)
  and
  job:slo_errors_per_request:ratio_rate30m > (6 * 0.001)
)

# Slow burn - ticket. Budget exhausted in ~9 days (3x) or ~28 days (1x).
(
  job:slo_errors_per_request:ratio_rate1d > (3 * 0.001)
  and
  job:slo_errors_per_request:ratio_rate2h > (3 * 0.001)
)
or
(
  job:slo_errors_per_request:ratio_rate3d > (1 * 0.001)
  and
  job:slo_errors_per_request:ratio_rate6h > (1 * 0.001)
)

Wrapped in an alerting rule, the fast-burn expression becomes the paging alert with a for: 2m hold, and the slow-burn expression becomes a ticket-severity alert. The AND-of-two-windows structure is what kills the flappy single-scrape page: a five-minute blip never satisfies the 1h condition, and a genuine outage satisfies both within minutes.

Generating SLOs Instead of Hand-Writing Them

Once you have more than two services, hand-copying recording and alerting rules gets old. Generators take a service spec and emit the full rule set. Sloth is the widely used option: you write a small YAML describing the SLI queries and the objective, and it produces the Prometheus rules including the multiwindow burn alerts:

version: "prometheus/v1"
service: "checkout"
labels:
  owner: "payments"
slos:
  - name: "requests-availability"
    objective: 99.9
    description: "Availability of checkout HTTP responses."
    sli:
      events:
        error_query: sum(rate(http_requests_total{job="checkout", code=~"5..|429"}[{{.window}}]))
        total_query: sum(rate(http_requests_total{job="checkout"}[{{.window}}]))
    alerting:
      name: CheckoutHighErrorRate
      page_alert:
        labels:
          severity: page
      ticket_alert:
        labels:
          severity: ticket

The {{.window}} templating is the point: Sloth substitutes every window the burn-rate alerts need, so you can’t get the 1h/5m pairing wrong. Pyrra takes a different angle — same SLO definitions, but it adds a UI that shows remaining budget over time and pairs each objective with generated, slightly more aggressive alerts, which you can adopt wholesale or tune down.

The Rest of the Chain

Two practices make the system durable. First, publish the numbers: a dashboard showing the SLI, the objective line, and remaining budget, refreshed daily. The original treatment of SLOs in the SRE book is worth reading for how the error budget becomes the basis for launch reviews and reliability standups. Second, write the policy down: what happens at 50% budget consumed, what happens at 100%. A policy that only exists in one engineer’s head will not survive that engineer’s vacation.

Wrapping Up

The chain is short enough to build in an afternoon: choose a request-based SLI from metrics you already emit, set a 99.9% objective over 28 days, precompute ratios with recording rules, and page on the 14.4×/1h AND 5m pattern instead of on CPU percentages. The payoff is alerts that are rare because they’re real, and a shared vocabulary for the speed-versus-reliability argument that every team keeps having.

If you’re starting from a noisy rotation, pick the single service that generates the most pages and build its SLO first. One objective, two alerts, one budget dashboard. The first time a fast-burn page arrives and it’s an actual outage — and the dozen CPU alerts you deleted never wake anyone again — you’ll wonder what took so long.