Chapter Eight · Distributed Tracing

Distributed Tracing

Metrics proved Saturday checkout is slow; logs named the error but not where the time goes. What neither can say is where the time goes inside one request crossing five services — and that question is the third signal's whole job. This chapter builds tracing end to end: spans and context propagation, OpenTelemetry instrumentation, the Collector pipeline, a sampling policy that keeps every error, and Tempo with full three-signal correlation — and it ends by answering the question Chapter 1 opened.

6 topics

A 9-second checkout touches web, bookings, db-01, cache-01, and payments before the customer sees anything. The Chapter 5 histogram says the p99 breached; the Chapter 7 logs name a connection pool timeout without saying where the wait happened. Neither can say which hop inside that one request ate the budget, because metrics aggregate away the individual request and logs cannot tie five services' lines to the same one. A trace — one request rendered as a timed tree — answers by construction what the other signals structurally cannot.

Six topics build the signal: why tracing exists and what it alone answers; spans, traceparent, and the three places propagation breaks; OpenTelemetry's API-SDK-OTLP seam that instruments once for any backend; the Collector pipeline that carries all three signals through one process; sampling policy that keeps every error and slow trace while dropping the boring 99%; and Tempo with exemplar-driven correlation — where the Saturday mystery, seven chapters old, finally gets its answer.

One checkout as a trace waterfall — spans left to right in start-time order, not call order
webroot span
bookingschild of web
cache-01the slow hop
payments190 ms

Topics in This Chapter

Topic 38
Why Tracing
A histogram says the p99 hit 9 seconds; it cannot say where in five services the time went, and timestamp-matching logs fails under concurrency. A trace records one request's full path, timed at every boundary — the question the cheap signals structurally cannot answer.
Tracing
Topic 39
Spans and Context Propagation
A trace is an agreement: every operation carries a span with a shared trace_id, handed across services in the W3C traceparent header. The three places that hand-off breaks — proxies, thread pools, queues — are most of the debugging skill.
Propagation
Topic 40
OpenTelemetry
API, SDK, and OTLP split instrumentation from backend, so Harborline instruments once and can change backends by changing an endpoint. Auto-instrumentation delivers the full checkout waterfall with zero code changes; one manual span names the business logic.
OpenTelemetry
Topic 41
The OTel Collector
Receivers in, processors in order, exporters out — one config carries traces, metrics, and logs through the same binary. Alloy agents on every host feed a gateway on obs-01, with memory_limiter and batch as the two processors no pipeline skips.
Collector
Topic 42
Sampling
At 300 rps, search alone produces 150 million spans a day, almost all describing healthy cache hits. Head sampling decides blind at the root; tail sampling decides at the gateway with the outcome in hand — keeping 100% of errors and slow traces.
Sampling
Topic 43
Trace Backends and Correlation
Tempo stores traces in cheap object storage keyed by trace ID, TraceQL queries them, and exemplars link a dashboard spike to the exact slow trace. One click opens the Saturday waterfall: 9,020 ms waiting for a Redis connection — mystery closed, fix proven.
Correlation