Chapter Thirteen · The Wider Ecosystem

The Wider Ecosystem

Harborline's stack is complete and paid for — Prometheus, Grafana, Loki, Tempo, Alertmanager, the checkout SLO holding. This chapter looks around and forward: a profiling experiment that opens a span no trace can, kernel-level telemetry for the services nobody instrumented, probes that measure from where the user stands, the quarterly build-vs-buy conversation armed with actual numbers, and the first deliberate cost-control pass before the disk makes it mandatory.

5 topics

The core stack answers the questions the book set out to answer, and this chapter maps what lies beyond its edges. A fourth signal — continuous profiling — attributes CPU cost to individual functions, the one unit no metric, log, or trace can name. eBPF produces telemetry for code nobody instrumented, including the nginx web container that has gone twelve chapters without RED metrics. And synthetic probes plus real-user monitoring finally measure from the customer's side of DNS, TLS, and the CDN — the path where job="bookings" can read 0.02% errors while every real request dies.

The chapter closes on the two conversations every team eventually has. Managed observability is an exchange rate, not a winner — engineering hours against invoices and lock-in, re-priced yearly as team and telemetry grow. And cost engineering turns the three costs from Chapter 1 — cardinality, sampling, retention — into deliberate per-signal levers, applied before the bill or obs-01's disk applies them for you, with a dashboard that watches the stack itself.

Beyond metrics, logs, and traces — three ways this chapter widens the signal family
Continuous profiling
The fourth signal. Sampled stack traces attribute CPU cost to individual functions — the one unit no metric, log, or trace can name.
eBPF telemetry
Kernel hooks produce RED metrics and basic spans for every process on the host, including web's nginx that nobody instrumented — breadth with zero code changes.
Synthetics & RUM
Probes and real-browser reports measure from the user's side of DNS, TLS, and the CDN — the outside-in path the internal signals never see.

Topics in This Chapter

Topic 65
Continuous Profiling
The fourth signal: always-on sampled stack traces at 1–5% overhead, rendered as flamegraphs with a stored baseline to diff against. Harborline pilots Pyroscope on search and finds 41% of CPU in JSON serialization — a one-line fix no dashboard could attribute to a function.
Profiling
Topic 66
eBPF Observability
Verified programs at kernel hooks produce RED metrics and basic spans for every process on the host — nginx, vendor binaries, anything — with zero code changes. The price is depth: no business attributes, inferred trace context, and TLS visibility only where the crypto library is known.
eBPF
Topic 67
Synthetic Monitoring and RUM
Internal metrics stay green while DNS, the TLS certificate, or the CDN fails outside the instrumented perimeter. blackbox_exporter and a k6 booking flow probe from where the user stands and feed pages; Faro reports what real browsers experienced and feeds planning.
Synthetics
Topic 68
Managed Observability
Datadog-class platforms, Grafana Cloud, and cloud-native monitoring against the self-hosted stack — an exchange rate between engineering hours and invoices, not a winner. Pricing models map straight onto cardinality, sampling, and retention, and OpenTelemetry keeps the exit affordable.
Managed
Topic 69
Cost Engineering
Traffic grew 3×, telemetry grew 10× — volume multiplies along services and cardinality, not just traffic. Apply the three costs as per-signal levers at the cheapest point in the pipeline, and build the observe-the-observability dashboard so the stack's own capacity is a dated ticket, not a 02:00 page.
Costs