Exemplars
Sometime in Chapter 6, a Grafana panel is going to show checkout p95 spiking at 09:14 on a Saturday, and the next question in the room is always the same: show me one of those slow requests. The histogram cannot answer. Aggregation destroyed the individual requests on purpose — a histogram keeps bucket counts, not requests, and the previous topic spent a page defending exactly that trade.
An exemplar is the escape hatch: one raw observation, carrying a trace_id, attached to a histogram bucket as it goes by. The count stays an aggregate; the exemplar dangles off it as a specimen. When the pieces line up, the dashboard spike links straight from the p95 panel to the exact slow trace in Tempo.
One Sampled Observation, Riding Along
In the exposition, an exemplar is a value-plus-labels annotation on a bucket line. OpenMetrics reserves a # section after the bucket's count for it: the exemplar's labels, the raw observed value, and a timestamp.
# an OpenMetrics bucket line carrying an exemplar
harborline_checkout_duration_seconds_bucket{le="2"} 1743 # {trace_id="4bf92f3577b34da6a3ce929d0e0e4736"} 1.9 1721460831.123
The bucket count, 1,743, aggregates exactly as before — Prometheus still sums it across app-01 and app-02. Everything after the # is the specimen: a trace_id label, the observed value of 1.9 seconds that put this request in the 2-second bucket, and when it happened. One request, from one worker, preserved whole while its 1,742 siblings dissolve into the count.
Attaching It in Code
from opentelemetry import trace
ctx = trace.get_current_span().get_span_context()
if ctx.trace_flags.sampled:
CHECKOUT_LATENCY.observe(
elapsed_seconds,
exemplar={"trace_id": format(ctx.trace_id, "032x")})
The checkout handler reads the active OpenTelemetry span context and passes an exemplar dict to observe(). The sampled check is not optional: attach the id of an unsampled span and the dashboard renders a dot whose click lands on "trace not found," because the trace it names was never stored. Chapter 8 covers how sampling decides; the rule applies from the first line of exemplar code.
The second rule is a size limit. OpenMetrics caps the combined length of exemplar label names and values at 128 UTF-8 characters, and prometheus_client enforces it with a ValueError raised inside observe() — which is to say, inside the checkout request path. Keep the label set to trace_id alone and the limit never comes close.
OpenMetrics or Nothing
The classic Prometheus text format has no exemplar syntax at all. Exemplars travel only in the OpenMetrics exposition, which make_asgi_app() serves automatically when the scraper's Accept header asks for it. Topic 22's advice against hand-rolling the endpoint turns out to have been load-bearing: a hardcoded classic-format response strips every exemplar the code attaches, silently.
Storage on the Prometheus Side
Prometheus discards scraped exemplars unless started with --enable-feature=exemplar-storage. Storage is an in-memory circular buffer — 100,000 exemplars by default, tunable — queried through a dedicated API rather than written into TSDB blocks. Exemplars are recent specimens, not history, and that is acceptable: the trace each one points at lives in Tempo under its own retention, and an old spike's specimen matters far less than this morning's.
The Payoff, and the Multiprocess Hole
With the feature flag on and the exemplar toggle enabled on Grafana's Prometheus data source, the checkout latency panel grows dots — one per exemplar, plotted at its observed value. A click opens the Tempo trace behind the dot. "The p95 spiked at 09:14" becomes "here is a request that did it, span by span," in two clicks, which is the single largest quality-of-life upgrade this chapter ships.
The honest caveat: prometheus_client multiprocess mode does not support exemplars, and bookings runs multi-worker gunicorn. The two features do not compose, and pretending otherwise wastes a sprint. Harborline's ways out are one worker per container with scaling by replicas (the container-native shape anyway), or generating exemplar-bearing span metrics in the collector — a bridge Chapter 8 builds from the tracing side.
- Instrumenting exemplars in code while Prometheus runs without
--enable-feature=exemplar-storage— nothing errors anywhere; the exemplars are scraped and silently discarded, and the team concludes the feature "doesn't work." - Hand-rolling
/metricswith a hardcoded classic-format response — content negotiation never happens, OpenMetrics is never served, and no exemplar leaves the process regardless of what the code attaches. - Attaching the trace id of an unsampled span — the dashboard dot renders, the click lands on "trace not found," and after three dead links nobody clicks again.
- Stuffing extra context like route or user hints into exemplar labels — the 128-character combined limit becomes a
ValueErrorinsideobserve(), which means a monitoring nicety now throws in the checkout path. - Expecting exemplars on a Gauge — the client supports them on Counters and Histogram buckets only; a Gauge is a level, not an event, so there is no observation for a specimen to ride.
- Attach exemplars from the active span context only when the span is sampled, and keep the label set to
trace_idalone — well under the 128-character line. - Enable
--enable-feature=exemplar-storageon obs-01's Prometheus and the exemplar toggle on the Grafana data source in the same change as the code, so the pipeline works end to end on day one. - Put exemplars on the SLO-bearing histogram first —
harborline_checkout_duration_seconds— where a metric-to-trace jump saves the most incident minutes. - Decide the gunicorn question explicitly: one worker per container with replica scaling, or span-metrics from the collector — not multiprocess mode plus wishful thinking.
Knowledge Check
What problem do exemplars solve?
- Histogram quantiles are interpolated, and exemplars restore the exact percentile values
- Aggregation discards individual requests, and an exemplar carries one back out
- High-cardinality labels overload Prometheus, and exemplars replace them with cheaper series
- Scrapes can miss short-lived spikes, and exemplars persist them between scrapes
The bookings team attaches exemplars in code, but Grafana shows no dots and nothing has errored. The most likely cause?
- The exemplar labels exceeded 128 characters and were silently dropped by the client
- Prometheus is running without
--enable-feature=exemplar-storage - Grafana cannot display exemplars on time-series panels at all
- TSDB retention expired the exemplars before the dashboard refreshed
Exemplar storage is a bounded in-memory buffer that drops the oldest specimens. Why is that acceptable?
- Because the buffer is flushed into TSDB blocks before old entries are evicted
- Because evicted exemplars are regenerated by the client on the next scrape
- Because exemplars are pointers into traces, which have their own retention
- Because 100,000 slots is more exemplars than any real system produces
Why do exemplars force an architectural decision for the bookings service?
- Multiprocess mode has no exemplar support, and bookings runs multi-worker gunicorn
- FastAPI cannot serve the OpenMetrics format that exemplars require
- Attaching exemplars to every request would overwhelm Prometheus with sample data and bloat its on-disk blocks
- Enabling exemplar storage requires migrating obs-01 to a clustered Prometheus
You got correct