The Metrics That Matter

A metric is a number the service exposes about itself, scraped every 15 seconds, and cheap enough to keep for a year: one series is a name, a handful of labels and a value every 15 seconds, and a million of them fit on one collector. The three that matter for every endpoint are Rate, Errors and Duration, RED, and duration is a histogram rather than an average, because the second on-sale night's average for checkout was 120 milliseconds while its P99 was 4 seconds, and the average told nobody. Behind the endpoints sit the gauges that explain them: pool in use and pool wait, queue age, cache hit ratio, loop lag, breaker state. Each is a number a previous chapter promised to this one.

This topic is the list, the shapes, and the labels that make the numbers useful without making them expensive. It ends at /metrics: the service exposes the endpoint, the platform scrapes it, and the collector, the storage and the dashboards are Observability Deep Dive's subject. What that course draws is only as good as what this topic exposes.

RED per Endpoint

Two instruments, declared once in the metrics ring of Chapter 4 and updated for every request whether or not its log line was sampled away. A counter, http_requests_total, labelled by route, method and status, carries the rate and the errors: its increase over a minute is the request rate, and the increase of the series with a 5xx status is the error rate. A histogram, http_request_duration_seconds, labelled by route and method, carries the duration. The demonstrated exposition is those two for POST /holds, as the scraper read them at 21:04 on the second on-sale night.

# HELP http_requests_total Requests by route template, method and status
# TYPE http_requests_total counter
http_requests_total{route="/holds",method="POST",status="201"}   184203.0
http_requests_total{route="/holds",method="POST",status="409"}   3117.0     # seat already held: the race, decided
http_requests_total{route="/holds",method="POST",status="503"}   42.0       # the pool's acquire timeout
# HELP http_request_duration_seconds Request duration by route template and method
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{route="/holds",method="POST",le="0.1"}    21540.0
http_request_duration_seconds_bucket{route="/holds",method="POST",le="0.2"}    168911.0
http_request_duration_seconds_bucket{route="/holds",method="POST",le="0.4"}    181022.0
http_request_duration_seconds_bucket{route="/holds",method="POST",le="0.8"}    186900.0
http_request_duration_seconds_bucket{route="/holds",method="POST",le="+Inf"}   187362.0
http_request_duration_seconds_count{route="/holds",method="POST"}   187362.0
http_request_duration_seconds_sum{route="/holds",method="POST"}     30412.7

Read it as the scraper does. The counter has three series for this route, one per status the endpoint has returned since the process started: 184,203 holds created, 3,117 refused because the seat was already held, 42 failed because the pool's acquire timed out, and every number only ever goes up, so the collector computes rates by subtracting scrapes. The histogram is a set of cumulative buckets: 21,540 requests finished within 100 milliseconds, 168,911 within 200, 186,900 within 800, and 187,362 in total, so 462 took longer than 800 milliseconds, which is the number the checkout SLO of Topic 71 is stated on. The sum, 30,412 seconds over 187,362 requests, is the average, 162 milliseconds, and it is the least useful line on the page. Some buckets are omitted above; the real endpoint has 12 plus the last one.

The route label is the template, /holds or /events/{id}/seats, never the concrete path. The framework knows which template matched, and the metrics ring reads it from the match rather than from the URL, because a label whose values are the paths is a series per event and per order, and the collector dies of it. The section on cardinality is about exactly that mistake; it is the one this ring makes most easily.

Histograms, Not Averages

An average is one number that hides a distribution, and latency distributions are the wrong shape for it: most requests are fast, a few are very slow, and the average sits where no request actually is. On the second on-sale night the checkout average was 120 milliseconds at 21:04 while the slowest 1 percent of buyers waited 4 seconds, because 1 percent of requests waiting 4 seconds moves an average by 40 milliseconds. A histogram keeps the counts per bucket, and from the buckets the collector computes any percentile over any window: P50, P95 and P99 for the last minute, the last hour or the whole night, from the same series.

The buckets are chosen once and they are a decision, because a percentile is interpolated inside the bucket it lands in and is only as precise as the bucket is narrow. Stagedoor's are 5, 10, 25, 50, 100, 200, 400 and 800 milliseconds, then 1.6, 3.2, 6.4 and 10 seconds: doubling above 100 milliseconds, which keeps the error under a factor of two anywhere, with 800 milliseconds as a boundary on purpose, so that "under 800 milliseconds" is an exact count rather than an interpolation. The SLO of Topic 71 is stated on a percentile and a threshold the histogram can answer exactly, and never on the average, which the first version of the dashboard showed in large type and which was green all night.

The Gauges That Explain

RED says that something is wrong and on which endpoint. The gauges say why, and every one of them was promised to this topic by the chapter that discovered the need for it. From Chapter 6: stagedoor_pool_in_use against the pool size of 24, and stagedoor_pool_acquire_wait_seconds, a histogram, which is the number that says the pool is the bottleneck before the latency does; and the replica's lag in seconds, sampled by the worker every 10 seconds. From Chapter 7: stagedoor_breaker_state per dependency, 0 closed, 1 open, 2 half-open, the one line on the dashboard that names a cause rather than a symptom, and the age of the oldest unpublished outbox row. From Chapter 8: jobs_depth, jobs_oldest_age_seconds and jobs_completed_total per kind, and the length of the dead-letter stream. From Chapter 9: stagedoor_cache_requests_total by family and result, hit or miss, for the seat map's hit ratio. From Chapter 1 and Chapter 11: stagedoor_loop_lag_seconds, resident memory and open descriptors from the process collector. From Chapter 10: reconcile_last_success_timestamp and the disagreements counter.

The naming is a rule rather than an accident: the process collector's process_ family comes with the client, the HTTP ring's family is http_, the worker's are jobs_ and reconcile_, and every gauge that is Stagedoor's own idea is prefixed stagedoor_, so that a dashboard's autocomplete lists the service's instruments in one place. The environment name from Chapter 4 reaches every series as the env label: the service exposes it once, on stagedoor_build_info with the version beside it, and the scraper stamps it on everything it collects from that target. Each gauge has an alert on it, and every alert is written on the symptom a buyer or an organizer would report, in the unit they would use: not "pool wait histogram P99 above 2", but "checkouts are queueing for the database"; not "replay timestamp lag", but "organizer reports are 30 seconds stale".

The rows are the alerts, read from the gauge side. Pool acquire wait above 1 second for 2 minutes. Loop lag above 50 milliseconds for 2 minutes. Breaker state at 1 for longer than 5 minutes, because a short opening is the breaker working and a long one is Payrail down. Cache hit ratio for the seat-map family below 90 percent for 5 minutes. Oldest job older than 60 seconds for 1 minute, per kind, as Chapter 8 wrote it. Outbox oldest unpublished row older than 30 seconds, which is a relay that has died. Replica lag above 30 seconds, as Chapter 6 promised. Reconciliation last success older than 26 hours. Each one names a mechanism the reader has already built, which is what makes the page from the alert to the fix short.

Labels and Cardinality

A series is one combination of a metric name and label values, and the collector's cost is the number of series, not the number of scrapes. A label per route, method and status on the counter is 25 route-and-method pairs times perhaps 8 statuses each, 200 series. The histogram at 13 buckets plus a count and a sum is 25 times 15, 375. Breaker state per dependency is 3. Job metrics per kind are 4 kinds times 4 instruments. The whole service is under a thousand series per process, and the collector keeps a year of them without noticing.

A label per user, per order or per request id is a different thing entirely. Stagedoor places 4.3 million orders a month; http_requests_total{order_id="..."} is 4.3 million new series a month, each with its own index entry and its own 15-second sample stream, and a collector sized for a thousand series is out of memory by the second day. A label's set of values has to be enumerable in advance: routes, methods, statuses, dependencies, job kinds, cache families, outcomes. Anything with an id in it, anything a buyer typed, anything that grows with the business, is a field on a log line and a search in the viewer, never a label. The concrete path is the version of this mistake the metrics ring makes by default, which is why the ring reads the route template from the match and refuses to label anything else.

Business Metrics Are Metrics Too

The on-sale dashboard that the organizers and Marek both watch shows none of the instruments above. It shows orders placed per minute, charges by outcome, seats held per minute, and tickets scanned at the door, and those come from the same client and the same endpoint. The domain layer increments orders_placed_total when the order commits, charges_total with an outcome label of charged, declined or failed when the Payrail client returns, and seats_held_total when a hold is created, from the code that already knows the event happened. The metrics ring cannot produce these, because it sees a 201 and not what the 201 meant.

On a Tuesday in August a deploy broke the checkout handler, and for 20 minutes every POST /orders returned a 500. The RED error rate said so, and so did 2,000 buyers. The line that got the deploy rolled back in 3 minutes was orders per minute dropping from 100 to zero at 19:04, because that is the line the product owner had on her screen and the one nobody needs an engineer to interpret. A business counter is an alert that everyone in the company can read, and "orders per minute is zero" is the alert that matters to all of them. The 2,000 failed checkouts are also 46 percent of the error budget of Topic 71, spent in one evening, and the counter is how the postmortem knew the number.

Exposition

The service exposes one endpoint, /metrics, rendered from the client library's registry in the Prometheus text format above, and whatever the platform runs scrapes it every 15 seconds. Nothing is pushed. There is no agent inside the process, no batching, no buffer that can fill, and no second thing that can fail: a scrape that does not arrive is the collector's problem, and the process keeps counting. The demonstrated code is Stagedoor's exposition, end to end.

from prometheus_client import Counter, Histogram, make_asgi_app

BUCKETS = (.005, .01, .025, .05, .1, .2, .4, .8, 1.6, 3.2, 6.4, 10)   # 0.8 is a boundary on purpose

REQUESTS = Counter("http_requests_total", "Requests by route template, method and status",
                   ["route", "method", "status"])
DURATION = Histogram("http_request_duration_seconds", "Request duration by route template and method",
                     ["route", "method"], buckets=BUCKETS)

app.mount("/metrics", make_asgi_app())       # scraped every 15 s; never pushed

# the metrics ring of Chapter 4, after the response exists
route = request.scope["route"].path            # the template: /events/{id}/seats, never the path
REQUESTS.labels(route, request.method, response.status_code).inc()
DURATION.labels(route, request.method).observe(elapsed)

Four things happen. The buckets are declared once, with 800 milliseconds among them. The counter and the histogram are declared once, at import, with their label names, and the client library registers them in its default registry. The registry is mounted as the /metrics endpoint, which renders every registered instrument in the text format whenever it is asked. And the ring, after the response exists so that it knows the final status, reads the matched route's template from the framework and updates both instruments. One caution belongs here: the registry is per process, so on api-01's four uvicorn processes a scrape reaches one of the four and sees a quarter of the truth. The client's multiprocess mode, pointed at a shared directory, merges them, and running one process per container as Topic 62 does makes the problem disappear, which is one of the reasons it does that.

The endpoint is where this book stops. Which collector scrapes it, where 15-second samples are stored for a year, how the histogram becomes a P99 on a graph and how the alerts above reach a phone, is Observability Deep Dive. The service's contribution is that every number on those graphs was chosen by the person who built the mechanism it measures.