OpenTelemetry

For a decade, instrumenting for traces meant marrying a vendor. Zipkin libraries spoke Zipkin, Jaeger clients spoke Jaeger, Datadog's agent spoke Datadog — and switching backends meant re-instrumenting every service, which is why nobody ever switched. OpenTelemetry ended that war by splitting the problem into an API you code against, an SDK that implements it, and OTLP, one wire protocol every backend now accepts.

The consequence for Harborline is the whole sales pitch: instrument the five services once, and the telemetry can go to Tempo today, Jaeger tomorrow, or Datadog next year — by changing an endpoint, not a line of application code.

How the Wars Ended

Two competing open instrumentation standards, OpenTracing and OpenCensus, merged into OpenTelemetry under the CNCF in 2019. All three signals — traces, metrics, and logs — are stable today, and OTel is the second-most-active CNCF project after Kubernetes itself. Seeing either ancestor's name in a library's documentation dates it; adopting OTel in 2026 is the safe default, not a bet.

API, SDK, OTLP

The three layers have a deliberate seam. The API is what application and library code calls — tracer.start_span — and is inert by itself: no exporter, no network, no cost. The SDK is the concrete implementation wired up at process startup, carrying the samplers, processors, and exporters. OTLP is the wire protocol the result travels on — gRPC on port 4317, HTTP/protobuf on 4318. The seam exists so a library can instrument itself by depending only on the API, without dragging an exporter into every application that imports it.

Auto-Instrumentation

The fastest path to a full waterfall requires no code changes at all. The opentelemetry-instrument wrapper starts bookings with instrumentation for every library it recognizes: the FastAPI integration opens a server span per request, and the requests, psycopg, and redis integrations open client spans around every outbound call.

# wrap bookings without touching its code
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install   # detects installed libs, adds integrations
OTEL_SERVICE_NAME=bookings \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
opentelemetry-instrument gunicorn main:app

Two installs, one wrapped launch command, and the entire Topic 38 waterfall exists — including the cache-01 client spans that will matter enormously in Topic 43. The bookings team ships this in an afternoon, which is the point: the expensive part of tracing used to be the code changes, and for the standard library surface that cost is now zero.

Manual Spans

Auto-instrumentation cannot know that "reserve the seat, then charge the card" is one business operation. Between the clean HTTP and database spans sits a void where the business logic lives, so the bookings team brackets it with one manual span carrying the attributes that matter to Harborline.

with tracer.start_as_current_span("checkout.confirm") as span:
    span.set_attribute("harborline.route_id", route_id)
    span.set_attribute("harborline.passengers", len(passengers))
    reserve_seat(booking)
    charge_card(booking)

Unlike metric labels, span attributes are per-request data, not new time series. A booking ID or route ID that would detonate Prometheus's cardinality (Chapter 5) is legitimate — routine, even — on a span. The cost model differs by signal, and this is the signal where high-cardinality context belongs.

Resource Attributes

Resource attributes describe the emitter, stamped once onto all telemetry a process produces. service.name is the one non-negotiable: leave it unset and the backend files everything under unknown_service:gunicorn, service graphs collapse into one blob, and Topic 43's correlation jumps have nothing to key on. Harborline sets it to match the Prometheus job label, adds the service version, and stamps an environment attribute matching the env="prod" convention — so the three signals join on shared identity instead of diverging.

The Narrow Waist

Restate the spine, because it decides architecture arguments: instrumentation is the expensive, invasive part — code changes across five services — so you do it once, against a neutral standard. Backends compete on storage, query, and price behind OTLP. Harborline's choice of Tempo in Topic 43 is a swappable decision, not a marriage, and that asymmetry is the entire reason OTel won.