Topic 10

Percentiles over Averages

Latency

Take 100 checkout requests: 99 finish in 100 ms, one takes 5 seconds. The mean is 149 ms — a number that describes not a single one of those requests and hides the one customer who waited 5 seconds. That customer is the Saturday complaint.

Percentiles are how latency is honestly reported, and they come with one mathematical inconvenience that shapes everything downstream: they cannot be averaged. That single fact dictates, all the way down to the client library, how latency must be recorded.

The Mean as a Hiding Place

The arithmetic is the whole indictment: 99 requests at 0.1 s contribute 9.9 s, the slow one adds 5 s, and 14.9 s divided by 100 is 149 ms. Now let the slowest 1% of requests triple, from 300 ms to 900 ms: the mean moves by 1% of the 600 ms delta — 6 ms. A tail can triple while the mean twitches, which makes a mean-latency panel the place where slow requests go to be forgotten.

Percentiles Defined

The p99 is the value that 99% of requests finish under; p50 is the median. The checkout SLO — 95% of requests under 500 ms — is a p95 statement with a target attached: it holds exactly when the p95 is at or under 500 ms. Harborline's reliability promise is denominated in percentiles whether or not anyone has ever graphed one.

The same 100 requests, three statistics — 99 at 100 ms, one at 5 s

Mean149 ms

14.9 s spread across 100 requests. Describes none of them, and the 5-second outlier moves it by only 1% of its weight — the slow customer vanishes.

p50 · median100 ms

Half of requests finish under this. It can stay flat forever while the slowest 5% quietly cross every SLO threshold.

p99the 5 s tail

The value 99% of requests finish under — the tail where the 5-second customer, and the Saturday complaint, actually live.

The Tail Is Where Users Live

Fan-out multiplies exposure to the tail. A page that issues 40 backend calls hits the p99 of at least one of them on 1 − 0.99⁴⁰ ≈ 33% of loads — a "1-in-100" latency becomes a one-in-three page experience. And the sampling is not uniform across customers: your heaviest users make the most requests, so they draw from the tail most often. The tail punishes precisely the customers you least want to lose.

Tails also grow under load, because queues grow under load — every saturated pool and busy worker adds wait time to the unlucky requests first. Which is why the complaint is specifically a Saturday morning complaint: the tail stretches exactly when Harborline's traffic peaks, while any average of the week looks fine.

The Averaging Trap

Quantiles do not compose. avg(p99 of app-01, p99 of app-02) is not the fleet p99, not an approximation of it, and not related to it in any fixed way. If app-01 serves 10 times app-02's traffic, the true fleet p99 is set almost entirely by app-01's distribution — while the average weighs both instances equally, answering a question nobody asked.

The output is worse than no number, because it looks like one: a smooth, credible line that dashboards render without complaint and mathematics does not stand behind. Every decision made from it inherits the meaninglessness, invisibly.

The Recording Consequence

Since finished percentiles cannot be merged, the distribution itself must survive to query time. That is exactly what histogram buckets are for: buckets are counters, so sum by (le) merges app-01's and app-02's buckets into one fleet-wide distribution, and histogram_quantile computes the percentile from the merged result. The canonical order of operations is histogram_quantile(0.95, sum by (le) (rate(harborline_checkout_duration_seconds_bucket[5m]))) — aggregate first, quantile second, never the reverse.

This is the statistics behind topic 07's verdict on summaries: a summary ships finished per-instance quantiles, which is precisely the un-mergeable form. "Record latency as a histogram" is not a style preference. It is the only recording strategy under which fleet percentiles exist at all.

The Complaint, Restated

Mara can now say exactly what the Saturday complaint means: is the p95 of harborline_checkout_duration_seconds above 500 ms on Saturday mornings, and for what fraction of requests? That is a question with a yes/no answer — one that no average, and no amount of ssh, htop, and docker logs on app-01 and app-02, can produce. Nothing is installed yet, so the question still has no answer. Chapter 3 starts building the machinery that produces one.

Common Mistakes

Averaging per-instance percentiles on a dashboard — the canonical footgun: avg() over two p99 series renders a smooth, credible, meaningless line, and every threshold, capacity plan, and incident call made from it inherits the meaninglessness.
Alerting on mean latency — the mean stays nearly flat while the p99 triples, so the first report of Saturday slowness arrives as a customer email instead of a page. Symptoms live in the tail, and the alert must live there too.
Running histogram_quantile over too few buckets — with boundaries only at 0.1 s and 10 s, a reported "p95 = 3.2 s" is pure linear interpolation inside one giant bucket, off by potentially an order of magnitude while looking precise to two decimals.
Recording latency as a summary and expecting fleet percentiles at query time — the per-instance quantiles are already cooked and the distribution behind them is gone; the only fix is re-instrumenting, and history starts over from the deploy.
Watching only the median — p50 is the best-hidden place for a tail problem, since half the requests can stay fast forever while the slowest 5% quietly cross every threshold the SLO cares about.

Best Practices

Record every latency as a histogram with an explicit bucket boundary at each threshold you will ever alert or report on — 0.5 s for the checkout SLO — so the SLO ratio is an exact bucket count, not an interpolation.
Aggregate buckets first and compute quantiles second — sum by (le) inside, histogram_quantile outside — as the fixed order of operations for every fleet percentile.
Alert and set SLOs on percentiles; keep the mean (_sum ÷ _count) for what it is legitimately good at — throughput and capacity arithmetic.
Graph p50, p95, and p99 together on every latency panel — divergence between them is the shape of the problem, and a tail pulling away from a flat median is the signature of saturation.

Comparable toolsHDR Histogram the JVM ecosystem's answer to recording full latency distributions cheaplyt-digest the mergeable quantile sketch behind Elasticsearch's percentile aggregationsDDSketch Datadog distributions — the same cannot-average-percentiles problem, solved server-sideNative histograms Prometheus and OTel exponential histograms — merge-then-quantile with automatic boundaries

Knowledge Check

99 requests finish in 100 ms and one takes 5 seconds. What does the 149 ms mean actually tell you?

Roughly how long a typical request took
A value that describes none of the requests and hides the slow one
The same thing the median would, within rounding
An upper bound on what the slowest customer actually experienced end to end

A dashboard shows avg() over the p99 series of app-01 and app-02. What is wrong?

Nothing at all, as long as the average is properly weighted by each instance's traffic
It is a close approximation that drifts a few percent under load
The number is meaningless — merge the buckets first, then compute the quantile
The query errors, because avg() rejects quantile series

A page issues 40 backend calls. How often does at least one call land at or beyond its p99?

About 1% of page loads, by definition of p99
About 33% of page loads — 1 − 0.99⁴⁰
About 40% of page loads — 40 × 1%
Still 1% — fan-out does not change per-request probabilities

Why must sum by (le) sit inside histogram_quantile, and not the other way around?

Distributions merge; finished quantiles do not — so aggregate first, quantile second
The reversed order is a PromQL syntax error
The inner sum is merely a performance optimization that reduces overall query latency on wide ranges
Either order gives the same result on counters

You got correct