Sampling

Tracing every request is the one part of this stack that genuinely doesn't scale by default. At Harborline's summer-Saturday peak, search alone takes 300 requests per second, each producing 8–12 spans — over 200 million spans a day from one service, almost all of them describing identical healthy cache hits that nobody will ever look up.

Sampling is the deliberate decision to keep some traces and drop the rest, and the entire craft is which ones. This is the second of the book's three costs from Chapter 1, applied to the most expensive signal — and applied badly, it deletes exactly the traces this chapter exists to capture.

The Bill for Completeness

The arithmetic deserves to be explicit: spans per request, times requests per second, times bytes per span, compounded across five services. Storage is actually the smaller line item — export bandwidth off the hosts, Collector CPU at the gateway, and query-time noise all scale with the traces you keep. And 99% of what a full retention buys is redundant: the ten-thousandth healthy 40 ms search trace teaches nothing the first one didn't.

Head Sampling

Head sampling makes the decision at the root span's birth, inside the SDK, before anything about the request is known. ParentBased(TraceIdRatioBased(0.10)) keeps 10% of new traces, and the ParentBased wrapper makes every downstream service honor the sampled bit arriving in the traceparent flags byte (Topic 39) instead of re-rolling the dice. The root decides once; bookings, payments, and notifier all follow. Cheap, stateless — and blind.

The Blindness, Precisely

Head sampling decides before the outcome exists, so at 10% it discards 90% of the errors and 90% of the slow requests right along with the boring traffic — the sampler cannot favor what it cannot yet see. Harborline's Saturday mystery fires a few dozen slow checkouts an hour; a 10% head sample would have captured a handful, and a cost-driven 1% would likely have captured none. This is the observability-native footgun: sampling away exactly the requests you built tracing to see.

Tail Sampling

Tail sampling moves the decision to the end of the trace, into the Collector gateway. The tail_sampling processor — contrib distribution, as Topic 41 flagged — buffers each trace for a decision_wait window of 10–30 seconds, then applies policies top to bottom, now knowing the trace's duration and status. Harborline's policy keeps every trace with an error status, keeps every trace slower than 500 ms — the checkout SLO threshold, reused deliberately — and takes a probabilistic 5% of the healthy rest.

processors:
  tail_sampling:
    decision_wait: 30s
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 500 }
      - name: keep-baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

Read the policy as keep-rules, in words: 100% of failures, 100% of SLO breaches, a thin residue of healthy baseline for comparison. The interesting traces at Harborline are rare by definition — that is what makes them interesting — and this is the mechanism that guarantees the Saturday trace exists in Topic 43 instead of surviving a dice roll.

What Tail Sampling Costs

The trade is memory and topology. The gateway holds every in-flight trace for the full decision window, which is real memory at peak load — and every span of one trace must reach the same Collector instance, or each instance sees a fragment and decides on incomplete evidence. With Harborline's single gateway on obs-01 the affinity is free. A scaled-out gateway tier needs the load-balancing exporter in front, routing spans by trace ID, before tail_sampling can be trusted.

Composing a Policy

Head and tail stack. A head sample in the SDK caps what leaves the hosts at all — search at Saturday peak is the service that needs it — and tail policies at the gateway spend the surviving budget on errors and outliers. One caveat travels with any sampling at all: kept traces can no longer answer "how many." After keep-all-errors sampling, failures are overrepresented by design, so rates and ratios stay the metrics' job, and traces answer where and why.