Burn-Rate Alerts
Once the error budget exists, the alerting question changes shape. Chapter 9 asked "is the error rate above 2% right now?" — a threshold someone picked in a code review. The question worth paging on is different: how fast is the budget being spent, and will it survive the window? Burn rate is that speed. A burn rate of 1× spends exactly the whole budget over the whole window; 14.4× torches a 28-day budget in about two days.
Alerting on burn rate, over multiple windows at once, is the Google SRE Workbook pattern — and it is the piece that lets Harborline retire the guessed thresholds from Chapter 9. The pager goes quiet except when the checkout budget is actually at risk, which was the promise the whole chapter has been building toward.
Burn Rate, Defined
Burn rate is the observed bad-event ratio divided by the ratio the budget allows. Against the 99.9% checkout SLO the allowance is 0.1%, so a 0.1% error rate burns at 1× — the designed rate, budget lasting exactly the 28 days. A 1.44% error rate is 14.4×, which exhausts the budget in 28 ÷ 14.4 days, about 47 hours. Total failure — 100% errors — is 1000×, gone in 40 minutes. One dimensionless number, and the SLO is already inside it.
Thresholds Derived, Not Guessed
The Chapter 9 page fired at "error rate above 2% for 5 minutes" because 2% felt right to whoever wrote the rule. A burn-rate threshold is derived from the SLO itself: the same 14.4× template means 1.44% errors on a 99.9% service and 14.4% on a 99% service — correctly different sensitivities from identical rule code. Change the SLO next quarter and every alert retunes itself, because the allowance is a variable in the expression, not a constant baked into five rule files.
The Multi-Window Ladder
The Workbook runs three rungs in parallel: page at 14.4× over 1 hour, page at 6× over 6 hours, ticket at 1× over 3 days. The fast rung catches a budget-threatening spike within minutes of the hourly average crossing. The medium rung catches sustained degradation the fast one ignores — a steady 6× empties the window in under five days. The slow rung catches the leak: 1× held for three days is the designed rate, but sustained means something changed, and a ticket during business hours is the right-sized response.
The numbers 14.4, 6, and 1 look arbitrary until you see the derivation: they are the rates that spend 2%, 5%, and 10% of a 30-day budget in 1 hour, 6 hours, and 3 days respectively — 14.4 is just 0.02 × 30 × 24. The Workbook derived them for 30-day windows; on Harborline's 28 days the percentage of budget consumed per window nudges up a little (2% becomes ~2.1%, 10% becomes ~10.7%), but the burn-rate thresholds themselves never reference the period at all — a burn rate is just observed ratio over allowed ratio — so the ladder transfers unchanged.
The Short Window
Each rung ANDs in a second window at one-twelfth the size: 5 minutes alongside the hour, 30 minutes alongside the 6 hours, 6 hours alongside the 3 days. The long window establishes that enough budget burned to matter; the short window establishes that it is still burning now. Without it, a 10-minute spike holds the 1-hour average above threshold for most of the remaining hour, and the page keeps ringing about an incident the responder can watch being over. With it, the alert resets within minutes of the bleeding stopping.
Harborline's Rules, Concretely
Mara writes two burn-rate pairs per checkout SLI — availability and latency — against the Topic 50 recording rules evaluated at each ladder window, never against raw ad-hoc ratios, so every window reads the same numerator and denominator. The fast availability page looks like this:
# page: ~2% of the 28-day budget burned within the hour
- alert: CheckoutAvailabilityBurn14x1h
expr: |
(1 - sli:checkout_availability:ratio_rate1h) > 14.4 * 0.001
and
(1 - sli:checkout_availability:ratio_rate5m) > 14.4 * 0.001
labels:
severity: page
The rule reads as prose: the bad-event ratio over the last hour exceeds 14.4 times the 0.1% allowance, and the last five minutes agree. Firing means checkout is failing at 1.44% or worse. The name carries the burn rate and the window, so when this exact rule pages Mara at 02:40 in Chapter 12, the title alone tells her the budget impact and the urgency before a single dashboard loads.
- Paging on a 1× burn rate — 1× is the designed spend, a service performing exactly to target; after the third night of being woken for it, the on-call engineer mutes the pager, and the real pages go with it.
- Running only the fast 1-hour window — a steady 2× burn (0.2% errors) never approaches the 14.4× threshold yet halves the budget's lifetime to 14 days; without the 6-hour and 3-day rungs, the leak is discovered when the budget is already gone.
- Skipping the 5-minute short window — the 1-hour average stays elevated for the better part of an hour after a 10-minute spike ends, so the page keeps firing through a recovery the responder can see is complete.
- Computing burn rates from raw queries instead of the recording rules — four windows × two conditions × two SLIs as ad-hoc ratio expressions is where a subtle denominator mismatch hides, and one wrong
sum bymakes the page lie. - Keeping the old Chapter 9 threshold pages running "just in case" — every incident now pages twice with conflicting severities, and the responder's first task at 3 a.m. is reverse-engineering which alert to trust.
- Adopt the Workbook ladder as the starting point — 14.4× over 1 h paging, 6× over 6 h paging, 1× over 3 days ticketing, each with a 1/12-size short window — and tune only with evidence from real pages.
- Generate the rules with Sloth or Pyrra from a one-page SLO spec instead of hand-writing eight alert expressions per SLI — generated rules are consistent, reviewed as config, and regenerate when the SLO changes.
- Route the speeds differently in the Chapter 9 Alertmanager tree: burn-rate pages to the pager, the 1×-over-3-days rung to the ticket queue — the ladder is only humane if the slow rung cannot wake anyone.
- Name every alert with its burn rate and window,
CheckoutAvailabilityBurn14x1hstyle, so the page title alone communicates budget impact and urgency.
Knowledge Check
Checkout runs at a 0.1% error rate against its 99.9% SLO — a burn rate of exactly 1×. Why is paging on that wrong?
- A 1× burn exhausts the budget within about two days
- 1× is the designed spend rate; the service is performing exactly to target
- Burn rates below 6× cannot be measured reliably at a 15-second scrape interval
- It is not wrong — any budget spend at all deserves a page
At a sustained 14.4× burn rate, how long does the 28-day checkout budget survive?
- About 40 minutes
- About two days
- About five days
- About two weeks
A steady 0.2% checkout error rate — a 2× burn — never trips the 14.4×-over-1-hour page. Which rung of the ladder exists to catch it?
- The 6×-over-6-hours page
- The 5-minute short window
- The 1×-over-3-days ticket rung
- None — a 2× burn is within design and safely ignorable
What does ANDing the 5-minute window onto the 1-hour burn-rate rule accomplish?
- It makes the page fire sooner on a fresh spike
- It lets the alert reset promptly once the burning actually stops
- It reduces Prometheus query load by evaluating the cheaper five-minute window first
- It filters out false positives caused by missed scrapes
You got correct