Observability Deep Dive
by Sergey Okinchuk
Chapter 5Architecture and Data Sources

Chapter Six · Grafana

Grafana

The metrics built in Chapters 3–5 finally become something a human can read. This chapter provisions Grafana from files, builds the Harborline Checkout dashboard — the symptoms row on top, RED and USE beneath, one $service variable serving all the services — wires CI to post deploy annotations, and ends by putting the dashboards themselves under git review. It closes with the book's first payoff: the Saturday complaint, visible on a graph.

5 topics

Everything Harborline has collected so far lives behind a query prompt, which means it exists only for the person who knows what to type. Grafana on obs-01:3000 changes that: Prometheus, Loki, and Tempo wired in as data sources — provisioned from files, so a rebuilt container comes up already connected — and dashboards that a paged responder at 3 a.m. can read in 30 seconds. Five topics cover the layer: what Grafana is and is not, the mechanics of panels and variables, the design doctrine of symptoms-over-causes, annotations and drill-down links, and finally moving the dashboards themselves into git.

The chapter deliverable is the Harborline Checkout dashboard, and it delivers the first evidence a whole team can see without knowing PromQL: the checkout p95 panel shows a clean latency step every Saturday morning. The Chapter 1 complaint is now a fact on a graph — visible, repeatable, undeniable — with a deploy-annotation overlay proving no release caused it. What nothing on the dashboard can say is why. That answer waits for traces in Chapter 8.

Grafana — one query-and-render layer over three stores
Harborline stack · obs-01
Grafana :3000 — queries and renders every panel, stores no telemetry of its own
PrometheusPromQL · metrics
LokiLogQL · logs
TempoTraceQL · traces
Every panel is a live query in the store's own language. Delete Grafana and no telemetry is lost — the data lives in the three stores below, never inside Grafana.

Topics in This Chapter

Topic 27
Architecture and Data Sources
Grafana owns no telemetry — every panel is a live query against Prometheus, Loki, or Tempo, and its own database holds only dashboards, users, and data source definitions. Provision the sources from files with fixed uids, and a rebuilt container comes up fully wired.
Architecture
Topic 28
Building Dashboards
A dashboard is a grid of panels, and the difference between a demo and a working one is the details: legends, units, thresholds, and a $service variable so one dashboard serves all five services instead of five drifting copies. Includes the two classic breakages — = against a multi-value variable, and a hardcoded [1m] rate window.
Panels
Topic 29
Dashboard Design
The audience is a paged human at 3 a.m., 40 seconds out of bed — and every choice is judged by whether they get an answer in 30 seconds. Symptoms on top, causes below, RED and USE deciding each row, and 15 panels beating 60 every single time.
Design
Topic 30
Annotations and Drill-Downs
A graph shows what changed and when — never why. CI posts a deploy annotation on every release so step changes carry their own suspect list, and data links chain overview to detail without the responder ever retyping a time range.
Navigation
Topic 31
Dashboards as Code
Every dashboard is one JSON document, and left in the UI it becomes the hand-tuned artifact nobody dares touch. Provision it from git instead, and a threshold change arrives as a two-line diff a reviewer approves in 30 seconds.
As Code