Spans and Context Propagation

A trace is not a special object any service holds — it is an agreement. Every piece of work carries a span: a named, timed operation stamped with a trace_id shared by the whole request and a parent_span_id pointing at whoever called it. No single process ever sees the full trace; each one emits its spans independently, and the backend reassembles the tree from the IDs.

The agreement is enforced by context propagation: every outgoing call carries the IDs in a traceparent header, and the receiving side continues the tree instead of starting a new one. Where that hand-off breaks, the trace breaks — and knowing the three places it breaks is most of the debugging skill this topic teaches.

Trace, Span, Parent

A trace is a tree of spans sharing one 128-bit trace_id. Each span carries its own 64-bit span ID, a name (POST /checkout, SELECT bookings), a start time, a duration, and a parent_span_id identifying its caller. The root span — web handling the browser request at Harborline — has no parent. That is the entire data model: the waterfall view in Topic 38 is nothing more than this tree rendered with time on the horizontal axis, and every gap or overlap you read in it is a parent-child relationship plus two timestamps.

Attributes, Events, Status

Three fields turn a timed tree into evidence. Attributes are key-value pairs on the span — http.request.method, db.system, harborline.route_id — the dimensions you will filter and group by later. Events are timestamped moments inside the operation; an exception with its stack trace is the canonical event. Status is one of Unset, Ok, or Error, and it matters more than it looks: the Error status is the single bit that tail sampling (Topic 42) and every status = error TraceQL query keys on. A span that failed but kept status Unset is invisible to both.

The W3C traceparent Header

Between processes, the context travels as one HTTP header standardized by the W3C Trace Context recommendation. Four dash-separated fields: a version byte, the 32-hex-character trace ID, the 16-hex-character ID of the calling span, and a flags byte whose last bit says whether this trace was sampled — the field that lets a sampling decision made at the root propagate to every child service.

# version - trace-id (32 hex) - parent span-id (16 hex) - flags
traceparent: 00-7c3a92d54cf1b8a2e6d90f14ab37c5e8-b7ad6b7169203331-01

A second header, tracestate, rides alongside for vendor-specific data and is safe to ignore until a vendor makes you care. The older B3 headers from the Zipkin lineage still appear in mixed fleets; OTel propagators can emit both formats during a migration, which is the standard escape hatch when half the fleet predates W3C.

Propagation Inside a Process

Between the incoming request and the outgoing calls, the context lives in in-process context — contextvars in Python, thread-locals elsewhere — which instrumentation reads whenever it creates a child span. This works invisibly until work leaves the request's execution context. Hand a job to a bare ThreadPoolExecutor or a fire-and-forget background task and the new thread starts with empty context: the child span has no parent to attach to, so it becomes the root of a fresh, disconnected trace.

Propagation Across the Queue

HTTP headers do not exist on a RabbitMQ message, so the hand-off must be done by hand — or by instrumentation that knows to do it. The producer injects traceparent into the message headers; the consumer extracts it and continues the tree. OTel's pika instrumentation handles both sides for Harborline. Without it, every notifier span exists in its own orphan trace, and "did the confirmation email for this slow booking actually go out" is a question the trace can never answer.

Reading a Broken Trace

Broken traces have a small symptom vocabulary, and each symptom names its cause. A service missing entirely from the waterfall means no instrumentation, or a proxy stripping the header before the service saw it. Two short traces where one long one should be means context died at a queue or a thread pool — the classic split at mq-01. An orphaned span whose parent_span_id points at a span the backend never received means the parent was dropped or unsampled. Learn these three shapes and a broken waterfall reads like an error message.