Metric Types
Prometheus client libraries offer four instrument types — counter, gauge, histogram, summary — and the choice is not stylistic. It fixes, at instrumentation time, which questions the data can answer for the rest of its life.
Pick a gauge where a counter belonged and rates are lost. Pick a summary where a histogram belonged and fleet-wide percentiles become impossible without re-instrumenting, redeploying, and starting history from zero. Ten minutes of type discipline now saves a redeploy during an incident later.
Counter: the Monotonic Accumulator
A counter only ever increases, and resets to 0 when the process restarts. harborline_bookings_total is one: every confirmed booking adds 1, forever. The raw value is nearly useless on a graph — 8 million bookings since the last deploy answers no question anyone has. The question is always the per-second rate(), computed at query time, and rate() detects the restart reset and absorbs it instead of reporting a phantom collapse.
Gauge: the Current Value
A gauge goes up and down and is read directly: in-flight requests in bookings, RabbitMQ queue depth on mq-01, Redis connections in use on cache-01. No rate, no transformation — the number on the graph is the number in the system. The inverse discipline applies too: rate() over a gauge is meaningless, because rate assumes monotonic counter semantics and misreads every ordinary decrease as a counter reset. Chapter 4 makes that failure concrete in PromQL.
Histogram: Counted Buckets
A histogram sorts every observation into cumulative buckets, and exposes each bucket as its own counter with an le ("less than or equal") label, plus a _sum and a _count. Cumulative means the bucket le="0.5" counts every checkout at or under 500 ms — including everything already counted under le="0.25". No percentile is computed in the application; the buckets travel to the server raw, and histogram_quantile computes any percentile you want at query time. Here is what that looks like on the wire, in the text exposition format a scrape actually fetches.
# a counter is one line; a histogram is a family of counters
harborline_bookings_total{status="confirmed"} 8231
harborline_checkout_duration_seconds_bucket{le="0.25"} 9411
harborline_checkout_duration_seconds_bucket{le="0.5"} 11007
harborline_checkout_duration_seconds_bucket{le="+Inf"} 11213
harborline_checkout_duration_seconds_sum 2841.7
harborline_checkout_duration_seconds_count 11213
Read it as a story: 11,213 checkouts observed so far, 2,841.7 seconds spent across all of them, 11,007 finished within 500 ms. The +Inf bucket always equals _count, because every observation is at or under infinity. That "11,007 of 11,213 under 500 ms" line is worth staring at — it is the checkout SLO, sitting in two counters, waiting for Chapter 10.
Summary: Client-Side Quantiles
A summary computes quantiles inside the process, over a sliding window, and exposes them pre-cooked: {quantile="0.99"} plus the same _sum and _count. For a single process the numbers are precise — and they are a dead end beyond it, for reasons the next section makes exact. There is also a blunt practical strike against summaries at Harborline: the official Python client does not implement summary quantiles at all, exposing only _sum and _count, and every Harborline backend service is Python.
The Aggregation Difference
Histogram buckets are counters, and counters add. The le="0.5" buckets from bookings on app-01 and app-02 sum into one fleet-wide bucket, the whole bucket family sums into one fleet-wide distribution, and the quantile is computed from that — after the merge, never before.
Summary quantiles cannot be combined at all. There is no mathematics that turns two per-instance p95s into the fleet p95 — not averaging, not weighting, nothing. The checkout SLO — 95% of requests under 500 ms — is a statement about the fleet, not about whichever instance a request happened to hit. The SLO itself dictates the histogram.
Native Histograms, Briefly
Current Prometheus also offers native histograms: exponential bucket boundaries chosen automatically, higher resolution at lower series cost, no hand-picked le list. They are still opt-in, and classic explicit-bucket histograms remain the default, the bulk of what you will meet in production, and what this book teaches. Know the successor exists; learn the incumbent first.
Histogram — counts observations into fixed buckets on the client and computes quantiles at query time. Aggregatable across instances and one PromQL function away from any percentile, at the cost of bucket-boundary precision.
Summary — computes exact quantiles inside one process over a sliding window. No aggregation across instances, no recomputing at a different quantile later. Instrument with histograms unless you can prove the service will only ever run as one instance — and even then, the histogram costs you nothing.
- Calling
rate()on a gauge — rate assumes monotonic counter semantics and treats every decrease as a counter reset, so it returns confidently wrong numbers rather than an error, and the dashboard built on them looks perfectly healthy. - Graphing a raw counter — the panel shows an ever-climbing line with a cliff at every restart and answers no question anyone actually has; the question is always the rate.
- Diffing raw counter values across a restart — the counter dropped to zero, the naive subtraction goes negative, and the dashboard reports a phantom traffic collapse.
rate()andincrease()exist precisely to absorb resets. - Instrumenting checkout latency as a summary, then needing the fleet-wide p95 for the SLO — per-instance quantiles cannot be merged, so the fix costs a code change, a rollout, and every sample of history collected so far.
- Keeping library-default buckets without checking them against your thresholds — the defaults happen to include 0.5 s, but a 300 ms threshold would fall between boundaries and turn the SLO ratio into an interpolated guess instead of an exact bucket count.
- Instrument every latency as a histogram with an explicit bucket boundary at each threshold you will alert or report on — for Harborline, exactly 0.5 for the 500 ms SLO.
- Use a counter for anything that only accumulates — requests, errors, bytes — and query it exclusively through
rate()orincrease(). - Reserve gauges for values that legitimately decrease — queue depth, pool usage, temperature — and read them directly, never through
rate(). - Keep bucket counts deliberate — 10 to 15 boundaries, concentrated where decisions are made — because every bucket is a full series per label combination, and buckets multiply cardinality like any other label.
Knowledge Check
Requests served, RabbitMQ queue depth, checkout duration — which instrument types fit, in order?
- Counter, counter, gauge
- Counter, gauge, histogram
- Gauge, gauge, histogram
- Counter, histogram, summary
Why can histogram buckets from app-01 and app-02 produce a fleet-wide p95 while summary quantiles cannot?
- Prometheus scrapes histograms before summaries in each cycle
- Summaries are too imprecise for fleet-level statistics
- Buckets are counters that sum across instances; a computed quantile cannot be merged
- histogram_quantile accepts summary data if you attach the right label to reconcile each instance
A panel applies rate() to the Redis connections-in-use gauge. What does it show?
- An error, because rate() rejects non-counter series
- Garbage — every decrease is misread as a counter reset
- A smoothed moving average of connection usage over the scrape window
- The per-second rate at which connections are acquired
Why must the checkout SLO threshold of 500 ms be an explicit histogram bucket boundary?
- Prometheus refuses to evaluate quantiles at values that fall between two bucket boundaries
- Boundary-aligned buckets are cheaper to scrape
- Alertmanager can only fire on bucket boundaries
- With a boundary at 0.5, the SLO ratio is an exact count instead of an interpolation
You got correct