histogram_quantile

"p99 checkout latency" cannot be stored as a number and averaged later — quantiles do not compose. Two instances each reporting a p99 give you two p99s and no correct way to combine them into the fleet's. Prometheus's classic answer is to store a histogram: a family of cumulative counters, one per bucket boundary, each tagged with the le (less-or-equal) label.

histogram_quantile() reconstructs an estimated quantile from those bucket rates at query time. The estimate is exactly as accurate as the bucket layout and not one millisecond better — a sentence this topic earns twice, once at query time here and once when Chapter 5 chooses the buckets.

Buckets Are Counters

harborline_checkout_duration_seconds_bucket{le="0.5"} counts every observation that took 500 ms or less since process start. Buckets are cumulative: the le="1" bucket contains everything in le="0.5" plus the observations between them, and the le="+Inf" bucket equals _count, the total. Because each bucket is an ordinary counter, everything from the rate() topic applies unchanged — rate the buckets before doing anything else with them.

The bookings service does not emit this metric yet; Chapter 5 instruments it. It previews here because histograms are where PromQL earns its keep. The Chapter 3 blackbox probe stores latency as a gauge, and quantile_over_time(0.99, probe_duration_seconds[1h]) can rank one stream's samples — but that trick works on a single series at a time and does not scale to a population of requests spread across instances, which is exactly what the checkout SLO is about.

The Canonical Shape

One line produces every latency percentile in the rest of the book, and it is worth reading inside-out once.

# fleet-wide p99 checkout latency, classic histogram
histogram_quantile(0.99,
  sum by (le) (rate(harborline_checkout_duration_seconds_bucket[5m])))

Rate first, because buckets are counters and reset detection works per raw series. Sum second, to merge the instances into one bucket family — the by (le) is load-bearing, because summing away the le label destroys the structure the quantile is computed from. Quantile last, over the merged bucket rates. Every histogram query you will write is a variation of this one line.

Interpolation, Honestly

The function finds the bucket the target rank falls in and linearly interpolates between that bucket's boundaries. With boundaries at 250 ms and 500 ms, a reported p99 of 487 ms means "somewhere between 250 and 500, probably nearer 500" — the three digits are interpolated fiction. The honest resolution of a classic-histogram quantile is the width of the bucket it lands in, which is why bucket layout is a design decision, not a default to inherit.

Edge Behavior Worth Memorizing

If the quantile lands in the +Inf bucket, the result is pinned to the highest finite boundary: a p99 flatlined at exactly 10 means your latencies escaped the bucket layout, not that they went stable. If it lands in the lowest bucket, the lower bound is assumed to be 0. Fewer than two buckets, or a bucket family missing +Inf, returns NaN.

Why Buckets Beat Recorded Percentiles

Bucket counters sum cleanly across instances, so one query yields the true fleet-wide p99 whether bookings runs on two hosts or twenty. A system that pre-computes per-instance percentiles — Prometheus's summary type, or a pipeline that stores "p99" as a plain number — can never aggregate them correctly afterwards, because the underlying distribution is already gone. That composability is the entire reason to pay the histogram's price of 10-plus series per label combination.

Native Histograms, Briefly

The successor format stores exponentially-scaled buckets inside a single series, giving far finer resolution at a fraction of the series cost, and histogram_quantile() works on them with no le label involved. Native histograms went stable in Prometheus 3.8, but a target still only exposes them when you turn on scrape_native_histograms in the scrape config, so the classic explicit-bucket form is still what most fleets run in production today; Chapter 5 covers instrumenting both.