Synthetic Monitoring and RUM

Every signal in this book so far is measured from inside Harborline — and the inside can be entirely green while a customer stares at a spinner. DNS, the TLS certificate, and the CDN all live outside the instrumented perimeter, so job="bookings" can report a 0.02% error ratio while the apex record points at a decommissioned load balancer. The trigger for this topic was an old scar: before Chapter 5 deployed the first blackbox probe, Harborline's certificate came within four days of expiring, and nothing then running would have noticed before the browsers did.

Two disciplines close the gap from opposite ends. Synthetic monitoring probes the system from where the user stands, on a fixed schedule, under controlled conditions. Real User Monitoring measures what actual browsers actually experienced. One is controlled and continuous, the other real and messy — and a production service wants both, wired to different destinations.

The Inside-Out Blind Spot

Internal metrics measure the service, not the path to it. A request that dies at DNS resolution, a TLS handshake against an expired certificate, a CDN edge serving stale errors — none of these ever reach bookings, so none appear in its request counters. "Up" is only meaningful measured from the user's side of that path, which means the measurement has to originate outside the network Harborline controls.

blackbox_exporter

Prometheus's outside-in probe is blackbox_exporter: modules for HTTP, TCP, ICMP, and DNS checks, driven by a relabeling pattern in which the target URL becomes a query parameter — Prometheus scrapes the exporter, and the exporter performs the probe. The result comes back as metrics like any other: probe_success feeds availability alerts, probe_duration_seconds tracks the user-side latency of the whole path.

That scar gets its full treatment here — the Chapter 5 probe grows into a synthetic-monitoring practice. probe_ssl_earliest_cert_expiry reports the certificate's expiry as a timestamp, so the rule probe_ssl_earliest_cert_expiry - time() < 14 * 86400 turns expiry into a ticket with two weeks of runway instead of a Saturday outage. The scrape config is one relabeling dance, learned once.

# probe the public booking URL through the full DNS + TLS + CDN path
- job_name: "blackbox-http"
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets: ["https://harborline.example/book"]
  relabel_configs:
    - {source_labels: [__address__], target_label: __param_target}
    - {target_label: __address__, replacement: "blackbox:9115"}

Scripted Synthetic Flows

A single GET proves the homepage answers; it cannot prove a customer can book a ferry. Scripted flows walk the real journey — search a route, hold a seat, reach the payment step — as code. Harborline scripts the flow in k6 and runs it every 5 minutes from outside the network against a sandboxed test account with test-mode processor keys; Grafana Cloud's synthetic checks run k6 under the hood, so the same script scales from a cron job to a managed multi-location service. The flow that earns scripting is the revenue-critical one, and there is exactly one of those.

Real User Monitoring

RUM inverts every property of a probe. The Grafana Faro web SDK rides inside the web SPA and reports what each real browser saw — Core Web Vitals (LCP, INP, CLS), JavaScript errors, frontend traces — shipped to the Faro receiver on Grafana Alloy and stored in the same Loki and Tempo as everything else. A synthetic probe from one clean network will never see the p75 LCP of a phone on ferry-terminal Wi-Fi; RUM sees nothing else. The vitals thresholds are defined at p75, which is the percentile worth tracking.

Both, Deliberately

The two signals route to different destinations by design. Synthetics give a controlled, comparable, alertable baseline that works at zero traffic — a probe fails at 03:00 whether or not any customer is awake, so probe_success == 0 feeds Alertmanager as a page. RUM gives ground truth across real devices, networks, and geographies, but only where traffic exists and only with real-world noise mixed in — so it feeds dashboards and the sprint-planning meeting. Harborline alerts on synthetics and learns from RUM.