SLOs for One Service

"Checkout should be fast" is a wish. "99.9 percent of POST /orders requests complete under 800 milliseconds, measured over 30 days" is a service-level objective, and the difference is everything that follows from it: it defines the alert, it sets the error budget that decides whether this week is for features or for the pool, and it is stated on a number the histogram of Topic 69 can answer exactly. Stagedoor has three SLOs, one per path a buyer or an organizer would notice failing, and each one was agreed with the person who owns that path before it was written into an alert rule.

This topic is how the three were chosen, how each is measured, and what the team does with them: the indicator first, then the objective, then the budget the objective implies, then the alert that fires on the rate the budget is being spent rather than on any single bad minute. It closes with the discipline that keeps the list at three, because a service with thirty SLOs has none.

The SLI First

The service-level indicator is a ratio the service can measure: good requests over total requests, where good has a definition that a buyer would agree with. For checkout, good is a status below 500 and a duration under 800 milliseconds; a 402 for a declined card is good, because the service answered correctly and quickly, and a 201 that took 3 seconds is not. Both halves come from the two instruments of Topic 69, the counter for the status and the histogram for the duration, and 800 milliseconds is a bucket boundary precisely so that this ratio is a count and not an estimate. The demonstrated query is the ratio as the collector computes it.

# good: finished under 800 ms and was not a 5xx; the 0.8 bucket counts every request under 800 ms,
# including the rare 5xx, so the 5xx count is subtracted, which errs in the strict direction
(
  sum(rate(http_request_duration_seconds_bucket{route="/orders", method="POST", le="0.8"}[30d]))
  - sum(rate(http_requests_total{route="/orders", method="POST", status=~"5.."}[30d]))
)
/
sum(rate(http_requests_total{route="/orders", method="POST"}[30d]))

# August: 0.99912 — 4,320,000 checkouts, 3,802 of them bad, 518 of the budget left

The numerator is the count of checkouts that finished inside the 800-millisecond bucket, less the count that returned a 5xx, and the denominator is every checkout; the ratio over 30 days is the SLI, and in August it was 99.912 percent. Two honesties about where it is measured. The number comes from the metrics ring inside the service, so it counts from the moment the process saw the request to the moment it wrote the response, and the buyer's experience includes the 40 milliseconds of network in front of it that Chapter 1 showed the service cannot see, plus whatever her carrier does. The indicator is therefore stated at the edge where possible, and Stagedoor checks its own number against the load balancer's latency once a month to know how optimistic it is. And the target is not 100 percent, because the network the service does not control fails at some rate whatever the code does, and a target that ignores that is a target the service cannot meet.

The Objective

The objective is the number the ratio must stay above over the window, and each of Stagedoor's three is a decision about how much failure is acceptable on that path, made with the people who own the product, and written down where the alert rules can read it. Checkout: 99.9 percent of POST /orders good over 30 days, because a failed checkout is a lost sale and a support ticket. The seat map: 99.95 percent of GET /events/{id}/seats under 200 milliseconds, because it is served from Redis 2,600 times a second and it should be, and because a slow seat map is what makes a buyer reload and reload. Organizer reports: 99 percent under 3.2 seconds, because they run on the replica, they can wait, and an organizer who sees a spinner for 3 seconds once in a hundred loads has not been harmed.

The numbers are not the same because the paths are not the same, and a single "the service is 99.9 percent available" would either be too loose for the seat map or too strict for the reports. Each one has an owner who is not Marek: the head of sales owns checkout's number, because she is the one who decides whether 4,320 lost sales a month is acceptable, and she decided it was, at the price of not demanding 99.99 and the on-call rota that would come with it. The number that an engineer chose alone and nobody else ever saw is not an objective. It is a preference, and it gets ignored the first time it costs a feature.

The Error Budget

An objective of 99.9 percent is a permission to fail 0.1 percent of the time, and that permission has a size. At 100 checkouts a minute, a 30-day month is 4,320,000 checkouts, and 0.1 percent of them is 4,320: that is the error budget, the number of bad checkouts the month may contain before the objective is missed. A Tuesday deploy that returned a 500 on every checkout for 20 minutes burned 2,000 of them, 46 percent of the month, in one evening. That was the signal to stop deploying features for the rest of the month and spend the time on the pool, which is the decision the budget exists to make.

The budget is what turns "reliability versus features" from an argument into arithmetic. When the month's budget is mostly unspent, a risky deploy on a Friday is affordable and the answer is yes. When 46 percent went on Tuesday, the answer is no until the cause is fixed, and nobody has to win a meeting to say so. The budget is also the honest name for what a load test does: the run of Chapter 12 against staging is the budget spent on purpose, in an environment where it costs nothing, to find the pool's limit at 800 requests a second before production spends the real budget finding it on the night.

Alerting on Burn Rate

The wrong alert is "error rate above 1 percent for 5 minutes." It pages at 3 a.m. for a 30-second blip that spent 50 checkouts of the budget, 1 percent of the month, and it sleeps through a slow regression at 0.3 percent that spends the whole budget by the 10th. The right alert is on the rate the budget is being spent, the burn rate: the error ratio divided by the ratio the objective allows. At a burn rate of 1 the budget lasts exactly the 30 days. At 14.4 it is gone in 50 hours, about 2 days. The approach comes from the Google SRE workbook, and it pairs a long window, which makes the alert quiet during a blip, with a short window, which makes it stop as soon as the problem does. The demonstrated rules are the fast one and the slow one.

# checkout:error_ratio: is 1 - SLI over that window, recorded every minute for 5m, 1h, 6h and 3d

- alert: CheckoutFastBurn
  expr: checkout:error_ratio:1h  > 14.4 * 0.001      # 14.4x: 2% of the month's budget spent in the last hour
    and checkout:error_ratio:5m  > 14.4 * 0.001      # and still burning now: stops paging once it stops
  labels: { severity: page }
  annotations: { summary: "Checkouts are failing or slow: at this rate the month's budget is gone in 2 days" }

- alert: CheckoutSlowBurn
  expr: checkout:error_ratio:3d  > 1 * 0.001         # 1x over 3 days: 10% of the budget spent quietly
    and checkout:error_ratio:6h  > 1 * 0.001
  labels: { severity: ticket }
  annotations: { summary: "Checkout is spending its budget on schedule: find the regression this week" }

The first rule pages when the last hour's error ratio is above 14.4 times the 0.1 percent the objective allows, that is, above 1.44 percent, and the last 5 minutes agree. Two percent of the month's budget in an hour is the workbook's threshold for a page, and a regression that fails 3 checkouts in 200 crosses it. The second rule opens a ticket when the last 3 days averaged above the allowed rate and the last 6 hours did too, which is 10 percent of the budget spent without any single bad hour: a slow leak, a query that got a little worse, a dependency a little slower, found within the week instead of at the end of the month. Between the two, the workbook adds a middle rule, 6 times over 6 hours, which Stagedoor runs as a second page.

The behaviour during a blip is the reason to prefer this shape. Thirty seconds of total failure inside an otherwise healthy hour is 0.8 percent of that hour's requests, under the 1.44 percent line, and the fast-burn rule stays silent, correctly, because 50 checkouts is a bad half-minute and not an incident. Twenty minutes of total failure is 33 percent of the hour, 23 times the line, and the page arrives within 5 minutes of the deploy. A regression to 0.5 percent errors, which a threshold alert would never see, trips the slow-burn rule within a day. Quiet during a blip, loud during a real regression, and never a page for something that spent nothing.

What Feeds the SLO

Every mechanism in this book is a way the checkout budget gets spent, and the postmortem for a burned budget names which. The pool of Chapter 6: 20 connections per loop, and at 800 requests a second the acquire wait alone pushes a checkout past 800 milliseconds. The breaker of Chapter 7: while it is open every checkout is a 503, which is 100 checkouts of budget a minute, so a breaker that flaps for an hour is the month. The stampede of Chapter 9: a seat-map rebuild that lands on the primary at the wrong moment holds the connections checkout needs. The render's CPU from Chapter 11: a container throttled to 2 cores for 4 processes doubles every latency at once. Payrail's own slowness at 3 seconds: a timeout inside the budget, but the buyer still waited 3 seconds, and 3 seconds is a bad checkout by the SLI's definition, whatever the status.

The rows are the reason the SLO is stated on the endpoint the buyer calls and not on any component. A pool SLO, a breaker SLO and a cache SLO would each be green while checkout was red, because each is a cause and the buyer experiences the sum. The gauges of Topic 69 are what a burned budget is explained by, after the fact, and the load test of Chapter 12 is where each row above was seen for the first time, at a controlled rate, with the budget it spent being staging's and not the month's.

SLOs Are Few

Three, for Stagedoor. The rule is one per path a user would notice failing, stated on the endpoint that user calls, and no more. A service with thirty SLOs, one per route and one per dependency, has an alert rule nobody can hold in their head, a dashboard nobody reads, and a page that fires for the reports endpoint at 3 a.m. while checkout is fine. When the thirty were tried, the on-call engineer learned within a week which twenty-seven to ignore, and one of the three that mattered was among them. Fewer objectives, each one watched, is the whole discipline.

The three are reviewed quarterly against what happened: whether the budget was spent, on what, whether 99.9 was too loose or too strict for what the business now expects, and whether a new path has appeared that a user would notice, such as the scanner's GET /tickets/{code} on the night of the door, which became the fourth SLO in the autumn. The review is 30 minutes with the product owners in the room, and its output is a change to a number in a rules file or no change at all. An objective that has never been revised is one that nobody is using.