Building Dashboards
A dashboard is a grid of panels, and a panel is one visualization fed by one or more queries. The mechanics take an afternoon to learn: pick a panel type, write the PromQL, drag it into place, save. Nothing in that loop is hard, which is why the internet is full of dashboards that were quick to build and useless six months later.
The difference between a demo dashboard and a working one is a handful of details this topic drills: legends that name the series, units that make 0.42 mean 420 ms instead of a bare number, thresholds that turn values into colors, and template variables so one dashboard serves all five Harborline services instead of five copies drifting apart.
Panels and Queries
The time series panel is the workhorse; stat, gauge, table, and heatmap cover most of what remains. Each panel holds one or more queries against a data source, and the panel editor is Explore with a save button — the same query box, plus visualization options wrapped around it. Mara's first panel graphs rate(harborline_bookings_total[$__rate_interval]): booking throughput per second, straight from the counter instrumented in Chapter 5.
Legends and Units
By default a multi-series panel labels each line with its full series identity — every label pair, brace to brace. A legend template like {{job}} {{status}} collapses that into names a human reads at a glance: bookings 200, bookings 500. On any panel with more than one line, the legend is the difference between reading the graph and decoding it.
Units matter just as much. Prometheus convention favors base units — seconds and bytes — so harborline_checkout_duration_seconds produces values like 0.42 — and Grafana has no idea what that means until the panel's unit field says seconds. Set it, and the axis renders 420 ms at one zoom level and 2.1 s at another, scaled correctly every time. Leave it as "none" and every reader supplies their own guess.
Thresholds
Thresholds are numeric breakpoints that color stat panels and draw horizontal lines on graphs. The checkout p95 panel gets a line at 500 ms — the number that becomes the SLO target in Chapter 10, drawn on the graph months before anyone calls it an SLO. A responder who sees the line does not need to remember the number; the panel carries its own definition of bad.
Template Variables
A template variable is a dropdown at the top of the dashboard whose value flows into every query. Harborline's $service is a query variable populated by label_values(up{env="prod"}, job) — Prometheus itself supplies the list, so a sixth service appears in the dropdown the day it is first scraped. An $instance variable chained off $service narrows to the hosts running the selected service.
Turn on multi-value and the "All" option, and the dashboard stops being a static picture of one service and becomes a control panel over all five. That is the whole argument against per-service dashboard copies: one definition, five services, zero drift.
Variables in Queries
In queries, the variable must be matched with =~, not =: a multi-value selection expands to a regex alternation like bookings|payments, and {job="bookings|payments"} matches nothing because no job carries that literal name. So the pattern is {job=~"$service", env="prod"}. This single character is the most common cause of a "broken" variable — the dropdown works, the panels go empty, and the dashboard looks like an outage.
Rows and Repeats
A row groups panels under a collapsible header, and a row set to repeat per $service value stamps out the same panels for every selected service from one definition. Mara builds rate, errors, and duration once; Grafana renders that row once per request-serving service — four of them. Fix the query in the definition and it is fixed everywhere — the repeated rows cannot drift because they do not exist as separate objects.
- Leaving the unit as "none" on a latency panel — one responder reads 0.42 as 420 ms, another as 42%, and the disagreement happens mid-incident, when it is most expensive.
- Using
=instead of=~against a multi-value variable — the moment someone selects two services or "All", every panel goes empty and the dashboard looks like an outage that isn't happening. - Hardcoding
rate(...[1m])in every panel query — the fixed window that looks right at a 1-hour zoom turns into unreadable noise at a 30-day zoom, and no one remembers to retune it.$__rate_intervalexists to compute the right window per zoom level. - Averaging per-instance p95s with
avg()acrossapp-01andapp-02— the average of two percentiles is not the fleet percentile. Sum the histogram buckets first, then takehistogram_quantile, or the panel reports a latency no user experienced. - Building the bookings dashboard, then copy-pasting it for payments and search — three near-copies that drift apart within a month, when one
$servicevariable would have served all five.
- Set an explicit unit on every field — seconds, bytes, requests/sec. A number without a unit is a guess, and Grafana's unit list covers every Prometheus base unit.
- Use
$__rate_intervalin everyrate()andincrease()query so the window tracks the zoom level and never drops below 4× the scrape interval. - Populate variables with
label_values()queries rather than hardcoded lists, so a sixth Harborline service appears in$servicethe day Prometheus first scrapes it. - Template the legend with
{{label}}syntax on every multi-series panel — a legend that readsbookings 500beats one that prints the full series fingerprint.
Knowledge Check
A dashboard's $service variable works with one service selected but every panel goes empty when "All" is chosen. What is the likely cause?
- The label_values() query only returns one service at a time
- Queries use = where the multi-value expansion needs =~
- Prometheus cannot match more than two label values in one query
- Multi-value variables only work on stat panels, not time series
What does $__rate_interval protect against that a hardcoded [1m] window does not?
- Gaps at narrow windows and noise at wide zoom levels
- Counter resets corrupting the rate calculation
- It makes every query cheaper for Prometheus to execute
- High cardinality from too many label combinations
Why is avg() over the per-instance p95s of app-01 and app-02 the wrong fleet latency number?
- Because the instances must be weighted by their individual request volumes first
- Because two instances is too few for percentiles to be valid
- Percentiles are not averageable; combine the buckets, then take the quantile
- Because avg() is deprecated in current PromQL
The bookings team wants dashboards for all five services. What is the maintenance argument for one $service-templated dashboard over five copies?
- The templated dashboard renders faster in the browser
- Grafana limits how many dashboards one organization can store
- One definition cannot drift; five copies will
- Template variables reduce query load on Prometheus fivefold
You got correct