Trace Backends and Correlation
Spans leaving the Collector need somewhere to live, and Harborline's answer is Grafana Tempo on obs-01:3200 — a backend built on the bet that traces are looked up, not searched like logs. Tempo writes spans to cheap object storage keyed by trace ID and keeps only a thin queryable index, which is what makes keeping 100% of the tail-sampled traces affordable.
The payoff of the whole chapter arrives on this page. With Tempo wired into Grafana next to Prometheus and Loki, one click carries Mara from a latency panel to the exact slow trace to that request's logs — and the Saturday mystery, open since Chapter 1, finally shows its face.
Tempo's Wager
Tempo maintains no full-text index over span contents: trace-ID-keyed blocks in object storage, plus columnar metadata for search. Storage costs a fraction of an indexed store's, which is the same deep-storage philosophy Loki applied to logs in Chapter 7, now applied to the third signal. It ingests OTLP natively, so the Collector's exporter from Topic 41 is three lines of config, and the keep-everything-interesting policy from Topic 42 stays within budget.
Jaeger, the Alternative
Jaeger is the CNCF elder: its own UI, OTLP-native ingest, and Jaeger v2 is itself built on the OTel Collector framework. The honest heuristic — Jaeger when you want a self-contained tracing system with its own frontend, Tempo when Grafana is already your pane of glass and correlation is the point. Either way the Topic 40 instrumentation is untouched, which is the narrow waist doing its job: this is a backend decision, not a re-instrumentation. Zipkin remains the lineage ancestor, still met in older fleets.
TraceQL Basics
TraceQL is Tempo's query language, structured like PromQL for traces: selectors in braces, filtering on span fields and attributes. The resource. prefix scopes attributes describing the emitter, span. scopes per-span attributes, duration and status are intrinsic fields. The first query below finds slow bookings spans; the second, failures anywhere; the third matches on tree structure — >> selects traces where a payments span is a descendant of a web span, a question about shape that no attribute filter can ask.
{ resource.service.name = "bookings" && duration > 2s }
{ status = error }
{ resource.service.name = "web" } >> { resource.service.name = "payments" }
Scoping is not a style preference here — it is the cost model. The columnar search scans blocks rather than consulting a text index, so a query anchored by resource.service.name and a tight time range returns in milliseconds, while an unscoped attribute hunt over a week crawls. Habits from Elasticsearch punish you on Tempo; Topic 42's kept traces are found by asking narrow questions.
The Correlated Stack
Three joins turn three databases into one investigation. Prometheus exemplars stamp a trace_id onto individual histogram observations, so a latency panel shows clickable dots riding the spike. Grafana's trace-to-logs link jumps from any Tempo span to the Loki query for that trace_id — which Chapter 7 put into every log line for exactly this moment. And Loki's derived fields make any logged trace_id a link back into Tempo. Metric to trace to logs, without re-typing a single timestamp.
The Saturday Trace
Saturday, 09:12. The Chapter 6 checkout dashboard shows p99 at 9.4 s, and for the first time the spike carries exemplar dots. Mara clicks one and the waterfall opens: web 9,480 ms total, of which bookings holds 9,310 ms. payments — the suspect everyone liked, the service with the external card processor — took 190 ms. The historical theory dies in one glance at the bars.
Inside bookings, the culprit span is almost comic: the Redis client span for the session lookup ran 9,020 ms before a 2 ms command. The span covers repeated acquisition attempts — each bounded by the pool’s 5-second timeout, retried until one succeeds — plus executing the command — and the pool's ceiling of 10 connections saturates only under weekend concurrency, so checkouts queue for a connection to cache-01 while the command itself stays trivially fast. Seven chapters of narrowing, answered by one bar in one trace: not slow code, not the network, not payments — a connection-pool ceiling nobody knew was there.
The Fix and the Proof
The bookings team raises the pool ceiling from 10 to 50. Before shipping, Mara checks the server side — a client-side fix that just moves the ceiling to the server is no fix. Redis defaults allow 10,000 client connections, and the Chapter 3 host metrics show cache-01 has the memory margin, so the new pool size lands with headroom to spare.
The following Saturday, the same Chapter 6 dashboard holds checkout p99 under 400 ms through the morning peak. The loop closes where it opened: the dashboard that first proved the complaint real now proves the fix real, on the same weekly window that exposed the problem. The customer who said checkout "sometimes takes forever" was right for eight months — and now the system can say why, and show it stopped.
- Enabling tail sampling but skipping exemplars — the trace that explains the spike exists in Tempo, but the on-call path from the dashboard to it is "guess a TraceQL query at 3 a.m." instead of one click on the panel.
- Logging without
trace_idin the structured fields (skipping Chapter 7's enrichment) — trace-to-logs has no join key, and the final hop of the correlation chain dies at exactly the moment a trace shows where and you need the logs to say why. - Treating Tempo like Elasticsearch and searching span attributes with no service or time scoping — the columnar search scans blocks, an unscoped query over a week crawls, and the habit that works on an indexed store punishes you here; scope by
resource.service.nameand a tight time range first. - Stopping at the slowest service instead of the slowest span —
bookingsat 9.3 s is where the time sits, but only the child-span breakdown shows it is pool-wait, not code; "bookings is slow" as a diagnosis would have fixed nothing. - Raising the client pool without checking the server side — 50 connections per gunicorn worker across replicas can slam into Redis's own
maxclientsor memory; the fix that ignorescache-01sizing trades a client-side ceiling for a server-side one.
- Enable exemplars end to end — histogram exemplar support in the SDK, exemplar storage in Prometheus, the exemplar toggle on the Grafana datasource — because the metric-to-trace jump is the correlation chain's front door.
- Configure trace-to-logs and Loki derived fields the day Tempo is wired in, so
trace_idis a two-way link between signals rather than a string to copy-paste. - Anchor every TraceQL investigation with
resource.service.nameand a duration or status predicate —{ resource.service.name = "bookings" && duration > 2s && status = error }is the shape of a query that returns in milliseconds. - Verify a latency fix on the same dashboard and the same weekly window that exposed the problem — Harborline's proof is the next Saturday peak on the Chapter 6 checkout dashboard, not a quiet Tuesday.
Knowledge Check
What does Tempo's storage design trade away, and what does it buy?
- It gives up OTLP ingest for lower storage cost
- It gives up indexed full-text search to make 100% retention of kept traces cheap
- It stores only span metadata, discarding the raw span payloads to save disk space
- It accepts occasional trace loss in exchange for speed
Harborline switches from Tempo to Jaeger next year. What happens to the Topic 40 instrumentation?
- Every service must swap its tracing libraries
- Span names must be rewritten to match Jaeger's naming conventions and format
- Nothing — the Collector's exporter config changes, the services don't
- The traceparent header format must change
What is a Prometheus exemplar, and which jump does it enable?
- A trace_id stamped on a histogram observation, linking a panel spike to a real trace
- A log field that turns a trace_id into a Tempo link
- A sample trace stored inside Prometheus for fast lookup
- A sampling rule that keeps traces referenced by dashboards, evaluated in the Collector
In the Saturday waterfall, why was "the slowest service is bookings" not enough to fix anything?
- Because the time actually sat in payments, mislabeled
- Because the child spans showed the time was pool-wait, not bookings code
- Because clock skew between app-01 and app-02 distorted every bar in the waterfall
- Because sampling had discarded the relevant spans
Why did the fix require checking cache-01 before raising the pool ceiling from 10 to 50?
- Redis must be restarted to accept more than 10 connections, since its default ceiling blocks the larger pool
- To confirm the slowness was actually inside Redis
- Because a bigger client pool spends server connections and memory — the ceiling could just move
- Because cache-01 must approve new pool sizes via config
You got correct