eBPF Observability

Everything Harborline has instrumented so far required touching the application: an SDK import in Chapter 5, a client library in Chapter 8, a config change somewhere. eBPF inverts that. Small verified programs load into the kernel, attach to hooks — syscalls, network functions, probes on shared libraries — and observe every socket and TLS call on the host, so telemetry appears for services nobody instrumented. That includes the nginx web container, which has quietly had no RED metrics for twelve chapters.

This is a survey topic, not a build. The questions that matter are what kernel-level telemetry buys, what it structurally cannot see, and where the two tools that define the space — Beyla and Cilium Hubble — fit next to the SDK instrumentation Harborline already owns.

Telemetry from the Kernel

An eBPF program is bytecode the kernel verifies before loading — bounded loops, checked memory access — and then runs sandboxed at its attachment point. Attach at the socket layer and the program sees every read and write a process performs; attach a user-space probe to OpenSSL and it sees plaintext before encryption. Because every HTTP request eventually crosses a socket, the kernel holds the raw material for request rate, error ratio, and latency on every process on the host.

The consequence is breadth no SDK can match. The four Python services took Chapter 5 to instrument; an eBPF agent produces baseline telemetry for all of them, plus nginx, plus any vendor binary, on the day it is deployed — with zero application changes and zero redeploys.

Grafana Beyla and OpenTelemetry eBPF Instrumentation

Beyla attaches to a running process and emits RED metrics and basic HTTP/gRPC spans in OTLP, assembled from kernel and socket data. Grafana donated the codebase to OpenTelemetry, where it continues as OpenTelemetry eBPF Instrumentation (OBI, still pre-1.0 as of mid-2026), with Beyla carrying on as Grafana's distribution of the upstream project. The practical effect: kernel-sourced telemetry speaks the same OTLP the Chapter 8 Collector already ingests.

Configuration is a handful of environment variables — which port to watch, what to call the service, where to ship the output. Pointed at the web container, Beyla produces the request rate, error ratio, and latency histograms nginx never exposed.

# Beyla attached to the nginx container, exporting OTLP to the Collector
BEYLA_OPEN_PORT=80
BEYLA_SERVICE_NAME=web
OTEL_EXPORTER_OTLP_ENDPOINT=http://obs-01:4318

Cilium Hubble

Hubble works one layer down. On Kubernetes clusters running the Cilium CNI, it records every allowed and dropped network flow — which pod talked to which service, DNS lookups, network-policy verdicts — at layers 3 through 7, without touching any pod. It answers a question neither metrics nor traces address: "is bookings actually reaching payments, and if not, what dropped the packets?" On the Chapter 11 cluster that question comes up the first time a network policy misfires; on Harborline's managed cluster it does not apply either, because the cluster ships its own CNI rather than Cilium, and Hubble is a Cilium feature, not a standalone agent.

Breadth Without Code Changes, and the Price

The kernel sees everything that crosses it and nothing that doesn't. An eBPF span carries no business attributes — no booking_id, no cart value — because those exist only inside the application's memory, in variables the socket layer never sees. Trace context propagation rides on rewriting the actual traceparent header in flight rather than reading it from application memory, so it breaks at proxies, load balancers, and protocols the network-level injection doesn't cover — eBPF traces run shallower and less certain than the OTel SDK traces from Chapter 8.

TLS is the permanent cat-and-mouse edge. Visibility into encrypted traffic depends on user-space probes into specific crypto libraries — OpenSSL, Go's crypto/tls, BoringSSL — so a static binary with an unusual TLS stack yields connection counts but opaque payloads. The agents also demand a recent kernel and elevated capabilities in the CAP_BPF/CAP_SYS_ADMIN class, which is a rollout check, not a footnote.

The Division of Labor at Harborline

The four Python services keep their SDK instrumentation — depth wins wherever you own the code, because the checkout trace that names a booking_id cannot come from the kernel. eBPF covers the gaps: web's nginx, a vendor binary, or day-one visibility on an acquisition's uninstrumented stack. Breadth as the complement, never the replacement — and one source of truth per service, because a service reporting RED metrics from both its SDK and Beyla counts every request twice.