Topic 51

Error Budgets

An error budget is the SLO turned inside out. If checkout must succeed 99.9% of the time over 28 days, then 0.1% of the time it is allowed to fail — about 40 minutes of full downtime, or proportionally more at partial failure. Nothing new gets measured; the same SLI is simply read from the other side.

That inversion converts reliability from a virtue into a resource: something that can be measured, spent, saved, and argued over. It is this chapter's central idea, and arguably the book's, because it is the mechanism that turns "is it safe to ship?" from a shouting match between product and engineering into arithmetic both sides can read.

The Arithmetic

One hundred percent minus the SLO is the budget. For checkout, 0.1% of a 28-day window is 40.3 minutes of total unavailability: 28 × 24 × 60 × 0.001. At 99.99% the same window allows 4 minutes; at 99% it allows 6.7 hours. Writing the target as allowed downtime is what makes stakeholders feel the difference between nines — "99.9 versus 99.99" reads like rounding, "40 minutes versus 4" reads like a staffing decision.

Partial Failure Spends Proportionally

The budget is measured in bad events, not wall-clock outage. An hour at a 1% checkout error rate produces the same number of bad events as 36 seconds fully down — both cost about 0.6 bad-minutes, roughly 1.5% of the 40.3-minute budget. This is why most budget disappears in slow leaks rather than headline outages: a degradation too mild to page anyone under Chapter 9's threshold alerts still drains the account, minute after minute.

The Budget as Negotiation Currency

Budget remaining is the one number product and engineering both read. Plenty left means ship the risky migration, run the load test in prod, deploy on Friday afternoon. Nearly spent means the next sprint buys reliability work instead of features. The budget converts a philosophical argument about caution into a balance check — the same request gets a different answer in week one and week four, and both answers are right.

Spent Budget Is Data, Not Blame

The budget exists to be spent. A quarter that ends with 100% remaining means the team over-invested in caution — deploys were slower and features later than the users' actual tolerance required, and Topic 53 turns that observation into policy. What matters is what spent it: the monthly report attributes spend to incidents, deploys, and dependency failures, which is exactly the pre-prioritized input a reliability roadmap needs.

Harborline's First Budget Review

Three weeks into the first window, 80% of the checkout budget is gone. The attribution is short: the Chapter 8 Redis connection-pool ceiling ate 22 minutes across four Saturday spikes, and one bad deploy ate 9. The bookings team defers a planned feature to finish the hardening the Chapter 8 fix began. The decision takes ten minutes, because the number is on the wall and nobody has to argue about whether Saturday "felt bad."

That meeting is the payoff of the whole chapter. Before the SLO, the pool fix competed against features on anecdote — after it, reliability walks into the roadmap conversation carrying a balance sheet, and wins for the first time on data.

The SLO, inverted into a spendable budget

99.9% / 28 days

The SLO: checkout must succeed 99.9% of the time over a rolling 28-day window — with obs-01’s Prometheus retention raised from the 15-day default so the whole window stays queryable.

40.3-min budget

The other 0.1% is allowed to fail: 28 × 24 × 60 × 0.001 = 40.3 minutes of total downtime — counted in bad events, so partial failure spends proportionally.

What spends it

Incidents, bad deploys, dependency failures, slow leaks. Harborline’s first review: the Redis pool ate 22 minutes, one deploy 9 — 80% gone in three weeks.

Common Mistakes

Setting the window to 7 days because it feels more responsive — one 30-minute incident consumes 300% of a 7-day 99.9% budget (about 10 minutes) and the number pins at zero for a week, then resets as if nothing happened; 28 days smooths incidents into a workable planning horizon.
Treating a spent budget as a failure to punish — the first time budget spend appears in a performance review, engineers start disputing the SLI definition instead of fixing the service, and the budget dies as an honest signal.
Computing the budget from uptime pings instead of the request-ratio SLI — a 1% error rate sustained for three days never trips an up/down check yet produces 43.2 bad-minutes (3 × 1440 × 0.01), more than the entire 40.3-minute budget; slow leaks are invisible to pings.
Maintaining a budget nobody can query — if "how much is left?" takes an engineer twenty minutes of PromQL, the number is absent from every shipping decision, which is the same as not having one.
Resetting the budget on calendar months instead of a rolling window — an incident on the 30th carries one day of consequences while an identical incident on the 1st carries thirty; a rolling 28-day window prices every incident the same.

Best Practices

Publish budget-remaining as a first-class Grafana stat panel on the checkout dashboard — one number, green/amber/red, computed from the same recording rules as the SLI.
Adopt a 28-day rolling window for user-facing SLOs — long enough that one incident doesn't zero it, short enough that last quarter's outage eventually stops dominating decisions.
Attribute every noticeable budget spend to a cause in the monthly report — deploy, dependency, capacity, unknown — because the attribution list is the reliability roadmap, pre-prioritized.
Agree the budget policy before the budget is spent — what happens at 50%, what happens at zero — because negotiating consequences during an incident produces whatever the loudest person in the room wants.

Comparable toolsSaaS Nobl9 — budget tracking and attribution as the core productBuilt-in Datadog SLOs / Grafana SLO — budget-remaining rendered nativelyGenerators Sloth / Pyrra — recording rules that make the budget a queryable seriesOrigin Google SRE books — the concept itself has no vendor

Knowledge Check

A service has a 99.9% availability SLO over a 28-day window. How much total downtime does the error budget allow?

About 4 minutes
About 40 minutes
About 6.7 hours
About 86 seconds

Why does a 7-day window break down for a 99.9% SLO in practice?

Prometheus cannot evaluate range queries longer than a few days reliably on a single node
A 7-day window reacts too slowly to fast-burning incidents
One ordinary incident overspends the whole budget, which then resets a week later
Seven days of history creates too much series cardinality to store

How does one hour at a 1% checkout error rate compare with 36 seconds of full downtime, budget-wise?

The hour costs 100 times more, because it lasted 100 times longer
Only the full outage counts; partial degradation never spends budget
They differ because only the full outage triggers a page
They spend the same amount — the budget counts bad events, not outage minutes

A quarter ends with 100% of the error budget unspent. What does the SRE Workbook framing say about that?

The SLO target should immediately be raised to 99.99% to match reality
It is also a finding: the team over-invested in caution
The surplus rolls over, doubling next quarter's budget
Nothing — an unspent budget is simply a good quarter

You got correct