OTel in Kubernetes

The OpenTelemetry Operator does for telemetry plumbing what the Prometheus Operator did for scraping. Two CRDs — OpenTelemetryCollector and Instrumentation — replace the hand-run Collectors and hand-wired SDK bootstrap of Chapter 8, and a single pod annotation is enough to have auto-instrumentation injected into Harborline's Python services at admission time.

This topic is also the chapter's closing argument. After the migration, discovery is CRDs, agents are DaemonSets, and injection is a webhook — yet the signals, the trace of web → bookings → payments, and the checkout SLO are exactly what they were on the VMs. The platform changed underneath the stack; nothing the stack measures moved.

The Operator and Its Two CRDs

The operator installs by Helm chart, with its admission webhook's TLS backed by cert-manager — and the install order matters, because a webhook that cannot serve TLS never fires and never says so. From there it reconciles OpenTelemetryCollector resources into running Collectors and applies Instrumentation resources to workloads at admission. It is the Chapter 8 architecture restated declaratively, one reconcile loop per moving part.

Auto-Instrumentation by Annotation

An Instrumentation resource names the SDK images to inject and the OTLP endpoint they speak to — declared once per namespace, so no service hardcodes a Collector address.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata: {name: harborline}
spec:
  exporter: {endpoint: "http://otel-agent:4317"}
  propagators: [tracecontext, baggage]

The annotation on a pod template is the trigger: at admission, the webhook adds an init container and the OTEL_* environment, and the pod starts emitting traces with zero code change. search, which never received the Chapter 8 manual instrumentation, gains spans exactly this way; Java, Node.js, and .NET inject the same, and Go gets an eBPF-based variant.

template:
  metadata:
    annotations:
      instrumentation.opentelemetry.io/inject-python: "true"

Auto and Manual, Composed

Injection instruments the frameworks — FastAPI routes, HTTP clients, database drivers — but it cannot know the business. bookings keeps its Chapter 8 manual spans and attributes, and the two compose cleanly because the SDK the webhook injects is the SDK the code already uses. One rule keeps this sane: one SDK owner per service. A service is either annotation-injected or code-instrumented, never both — both means duplicate spans on every request and dashboards reporting a traffic surge that is not there.

Collector Topology, Agent and Gateway

mode: daemonset puts a Collector on every node: the lowest-latency first hop for each pod, the place for kubelet and host receivers, and the home of the k8sattributes processor that stamps k8s.namespace.name and k8s.pod.name onto every span. mode: deployment builds a centrally scaled gateway pool that owns tail-sampling, heavy processing, and the egress credentials to the backends. Harborline runs both, agent to gateway — the Chapter 8 two-tier layout, now expressed as two CRs.

Sidecar Mode, the Exception

mode: sidecar injects a Collector into each pod, which buys hard tenant isolation at the price of one Collector process per replica. The DaemonSet amortizes that cost across the node, which is why it stays the default — per-pod Collectors are a bill, not a baseline. Reach for sidecar only when isolation is a requirement someone can name.

What Changed and What Didn't

The migration's ledger, on one page. Changed: service discovery (ServiceMonitors instead of static scrape configs), log shipping (a DaemonSet instead of per-VM agents), SDK wiring (webhook injection instead of hand bootstrap), and metadata (k8s.* resource attributes on every signal).

Unchanged: the harborline_ metric names, the trace shape across web → bookings → payments, the Loki schema, and the checkout SLO — 99.9% availability, 95% of requests under 500 ms, 28-day window — with its Chapter 10 burn-rate alerts intact. The pipeline of instrument, collect, store, query, visualize, alert never named a platform, and that is why none of its outputs had to move.