Every team that adopts SLOs eventually hits the same wall: the service is “up 99.95% of the time,” nobody knows whether that’s good, and the on-call rotation is drowning in alerts that fire on every three-minute blip. Threshold alerting — “page me when the error rate exceeds 1%” — doesn’t work for reliability, because reliability problems are rarely about the instantaneous error rate. They’re about how fast you are consuming the finite budget of errors you promised stakeholders you wouldn’t exceed.
Error-budget alerting fixes this, but the naive version has a fatal flaw: a single-window alert either fires too fast and screams at you over noise, or fires too slow and only notices an outage hours in. The fix used across the industry — and formalized in the Google SRE Workbook’s chapter on alerting on SLOs — is the multi-window, multi-burn-rate alert. This post explains the math behind it, shows how to build it with Prometheus recording rules, and covers the operational details that determine whether the paging works or just adds noise.
Burn rate: the error budget expressed as a speed
Start with an SLO: 99.9% of requests succeed over a rolling 28-day window. That means you’re allowed 0.1% failures — the error budget. Over 28 days (672 hours), the budget is 0.672 hours of full outage, or its equivalent in partial failures.
The burn rate is how fast you’re spending that budget relative to the pace that would exactly exhaust it at the end of the window. Formally:
burn_rate = error_ratio / (1 - SLO)Three cases make the intuition concrete:
- Burn rate 1.0 — you fail at exactly the budgeted rate. If this pace holds for all 28 days, you end with zero budget left, which is precisely on target.
- Burn rate 14.4 — you spend 14.4× the budget. At this pace the entire 28-day budget is gone in about 47 hours (672 / 14.4). That is an emergency.
- Burn rate 0.5 — you’re consuming half your allocation. Keep this up and you end the month with budget to spare.
Why 14.4 and not some round number? Because the recommended alerting pairs are derived from fixed budget-consumption goals. If you want to page when 2% of the budget is consumed within one hour on a 28-day SLO, the required burn rate is: budget consumed × period / window = 0.02 × 672 / 1 = 13.44, which rounds to the canonical 14.4 (the 2%-in-1h figure is computed over a 30-day period). Similarly, 5% consumed in 6 hours gives 0.05 × 720 / 6 = 6, and 10% in 3 days gives 0.10 × 720 / 72 = 1. The SRE Workbook’s recommended table for a 99.9% SLO boils down to two alert classes: page on 14.4× over 1h, or 6× over 6h; create a ticket on 3× over 1 day, or 1× over 3 days.
Why a single window fails
The obvious implementation is one alert per burn rate: burn_rate(1h) > 14.4. This fails in both directions, depending on how you configure it.
Suppose you use a one-hour window with the 14.4 threshold. A short burst of errors — say a five-minute blip where errors spike to 30% — produces a one-hour average burn rate well below 14.4, so the page never fires even though you just torched a meaningful chunk of budget. To catch fast bursts you need a short window, but a short window alone is miserable: a five-minute evaluation window means transient spikes, deploys with momentary error tails, and retry storms all page someone. The signal-to-noise ratio collapses and on-call engineers learn to ignore the alert — the classic path to a missed real outage.
The other failure mode is reset time. Once an alert fires and the incident is fixed, how long until the condition clears? A burn-rate alert computed over a one-hour window can stay firing for up to an hour after the errors stop, because the damaged hour keeps contributing to the average. Long windows have proportionally long reset times, which means responders sit in a firefighting session that technically never ends.
The multi-window design solves both problems by requiring two windows to agree for every alert: a long window that determines sensitivity to the overall burn, and a short window that confirms the problem is still happening right now. The long window does the math correctly; the short window suppresses alerts for problems that already ended.
The standard configuration
For a 28-day, 99.9% SLO, the recommended setup uses four recording-rule windows paired two-by-two:
- Page (high urgency): burn rate over 1h > 14.4 AND burn rate over 5m > 14.4
- Page (medium urgency): burn rate over 6h > 6 AND burn rate over 30m > 6
- Ticket (low urgency): burn rate over 3d > 1 AND burn rate over 6h > 1
Each pair uses the same threshold on both windows; the short window is typically 1/12 of the long one. The AND condition is what kills noise: an error burst lasting four minutes can trip the 5m window but won’t sustain the 1h window above 14.4, so no page. A slow leak burning budget at 2× for hours on end won’t trip the paging pair, but it will eventually trip the 3-day ticket pair — it’s real budget consumption, just not an emergency.
The highest-severity pair catches incidents that exhaust the 28-day budget in roughly two days; the second pair catches problems that would exhaust it in under five days. Note what’s not here: no alert on raw error rate, no alert on percent uptime this hour. Everything keys off one number — the budget — which is exactly the number your SLO commit can be measured against.
Building it with Prometheus recording rules
Assume an application exposing a standard request counter with a code label (any instrumentation following Prometheus naming practice works the same way). First, define a recording rule that computes the per-window error ratio once, so the alerts stay readable:
groups:
- name: slo-baseline
interval: 30s
rules:
# Total error ratio per window, for a 99.9% / 28-day SLO.
# PromQL cannot use a 3d range directly, so 72h is used.
- record: slo:error_ratio_rate5m
expr: |
sum(rate(http_requests_total{job="api"}[5m]))
-
sum(rate(http_requests_total{job="api",code=~"2..|3.."}[5m]))
- record: slo:error_ratio_rate30m
expr: |
sum(rate(http_requests_total{job="api"}[30m]))
-
sum(rate(http_requests_total{job="api",code=~"2..|3.."}[30m]))
- record: slo:error_ratio_rate1h
expr: |
sum(rate(http_requests_total{job="api"}[1h]))
-
sum(rate(http_requests_total{job="api",code=~"2..|3.."}[1h]))
- record: slo:error_ratio_rate6h
expr: |
sum(rate(http_requests_total{job="api"}[6h]))
-
sum(rate(http_requests_total{job="api",code=~"2..|3.."}[6h]))
- record: slo:error_ratio_rate72h
expr: |
sum(rate(http_requests_total{job="api"}[72h]))
-
sum(rate(http_requests_total{job="api",code=~"2..|3.."}[72h]))Next, convert error ratios to burn rates (divide by the budget, 0.001 for a 99.9% SLO) and write the alert rules. Each alert ANDs a long window with its short companion:
groups:
- name: slo-alerts
rules:
# Fast burn: page immediately
- alert: SLOFastBurn
expr: |
(slo:error_ratio_rate1h / 0.001 > 14.4)
and
(slo:error_ratio_rate5m / 0.001 > 14.4)
labels:
severity: critical
annotations:
summary: "Fast error-budget burn (page)"
description: >-
1h burn rate {{ $value }} exceeds 14.4 on a 99.9% SLO.
At this pace the 28-day budget is exhausted in ~2 days.
# Slow but sustained burn: page with lower urgency
- alert: SLOSustainedBurn
expr: |
(slo:error_ratio_rate6h / 0.001 > 6)
and
(slo:error_ratio_rate30m / 0.001 > 6)
labels:
severity: warning
annotations:
summary: "Sustained error-budget burn"
description: >-
6h burn rate {{ $value }} exceeds 6 on a 99.9% SLO.
At this pace the budget is exhausted in ~5 days.
# Slow leak: file a ticket, do not page
- alert: SLOBudgetLeak
expr: |
(slo:error_ratio_rate72h / 0.001 > 1)
and
(slo:error_ratio_rate6h / 0.001 > 1)
labels:
severity: ticket
annotations:
summary: "Error budget leaking over days"
description: >-
3-day burn rate exceeds 1.0 on a 99.9% SLO — spending
budget faster than planned without an acute incident.The and operator in PromQL performs vector matching on the label sets of both sides. Because the recording rules here aggregate everything away with sum(), both sides are single-element vectors and the match is trivial. If you keep per-route or per-region labels instead, the AND will correctly match each label combination independently — an alert per burning route rather than a global average that hides the one route on fire. For those per-slice variants, consider adding for: 2m to the alert rules to ride out sub-window jitter on low-traffic slices.
Four operational details that decide whether this works
1. Counting errors correctly
The whole system is only as good as your definition of “bad.” Status-code-based counting is the default, but be explicit: 404s on URLs nobody should call are not SLO failures; 500s are. A common pattern is to count 5xx plus timeouts as errors and treat 4xx as client faults — but if your server can return 429 under load, that’s a real reliability failure for the callers being rejected. Whatever you decide, encode it in the recording rule so the definition is versioned and reviewable, not tribal knowledge.
2. Low-traffic services need event-based math
A service doing 100 requests an hour can have a 5-minute window with three failures and a burn rate that’s either enormous or zero depending on the minute. For low-QPS services, rate() over short windows produces garbage. Two mitigations: widen the short windows (the SRE Workbook suggests that short windows be no shorter than what your traffic supports), or compute burn over request counts using increase() over the window instead of rate(). If the service does so few requests that even that is noisy, an SLO based on request count may be the wrong contract — consider a freshness or latency SLO on a scheduled job instead.
3. Alert suppression between pairs
When a full outage hits, all three alerts will eventually fire — the fast pair first, then the sustained pair, then the leak. That’s three pages for one incident. Configure inhibition in Alertmanager so that a firing SLOFastBurn suppresses SLOSustainedBurn and SLOBudgetLeak for the same service. One incident, one page.
4. SLOs are per-user journey, not per-service
The mechanical work above is worthless if the SLO measures something users don’t experience. A checkout flow that depends on five services should have an SLO on checkout success rate, not five separate service SLOs that each look green while the journey fails. Keep the number of SLOs small — a handful per user journey — and apply the same recording-rule template to each. Teams that generate hundreds of SLOs end up alerting on everything and defending nothing.
Tuning the numbers without breaking the contract
The 14.4/6/1 table is a starting point, not scripture. The invariants you should preserve when tuning:
- Short window = long window / 12 keeps the confirmation semantics consistent. If you shrink the short window too far, you reintroduce noise; too far the other way, and the alert becomes sluggish.
- Budget-consumption math must hold: burn rate threshold × long window should correspond to a budget fraction you’re genuinely willing to page on. If you want to page at 1% burn in an hour, the threshold on a 28-day SLO is 0.01 × 672 / 1 ≈ 6.72.
- Keep one ticket-level pair. Paging pairs catch incidents; the ticket pair is what drives the reliability backlog. Teams that drop it lose the mechanism that converts chronic 1.5× burns into scheduled engineering work.
Adjust after a month of operation: if pages fired that nobody would act on, raise the short-window threshold or lengthen the short window; if incidents landed before any alert fired, your fast-burn threshold is too high for your traffic shape. The feedback loop — page quality review at every incident retrospective — is what makes SLO alerting improve over time.
Wrapping up
Multi-window, multi-burn-rate alerting converts an abstract promise — “99.9% of requests succeed over 28 days” — into a small set of precise, low-noise alerts. The recording rules do the arithmetic once; the paired windows separate real incidents from blips; the ticket tier turns slow budget leaks into scheduled work instead of 3 a.m. pages. Start with the canonical thresholds, wire up Alertmanager inhibition, and let a month of incident reviews tune the constants for your traffic. The payoff is an on-call that pages only when the budget — the thing you actually promised — is genuinely at risk.