Dashboard Design
The audience for a production dashboard is not the engineer who built it on a calm Tuesday. It is a paged human at 3 a.m., 40 seconds out of bed, deciding whether users are hurting and where to look next. That reader scans; they do not explore. They will not remember which of 40 panels matters, and they will not hover over a legend to work out what an unlabeled axis means.
This topic is the book's design position, and it fits in one sentence: symptoms on top, causes below, a method — RED or USE — deciding what each row contains, and fewer panels beating more every single time. Everything else in the topic is that sentence applied to the Harborline Checkout dashboard.
The 3 a.m. Test
Every design choice on an operational dashboard is judged by one question: can a degraded, adrenaline-soaked responder extract the answer in under 30 seconds? Panels that require context only the author has, axes without units, colors that mean different things on different rows — each one fails the test before the incident starts. The test is brutal precisely because the moment it applies is the moment the dashboard exists for.
Symptoms First, Causes Below
The top row answers exactly one question: are users hurting? For Harborline that is the checkout error ratio and the checkout p95 — the two numbers that measure what a customer feels. Not CPU, not connection counts, not queue depth. If the top row is green, the page can probably wait for coffee; if it is red, everything below exists to explain why.
Each row beneath explains the row above, so reading top-to-bottom is the diagnosis path. The layout encodes the investigation, not the architecture diagram — a dashboard organized by how the system is built serves the author's mental model; one organized by how failures are diagnosed serves the responder's.
RED per Service
RED — Rate, Errors, Duration — is the method for anything that serves requests: how many, how many failed, how long they took. Three panels each for bookings and payments form the dashboard's top rows, all built from the Chapter 5 instrumentation. RED is the user's view of the system, which is why it goes on top: a user cannot feel CPU, but they feel every one of those three numbers.
USE per Resource
USE — Utilization, Saturation, Errors — is the method for resources: how busy, how backed up, how broken. CPU and memory on app-01 and app-02, connections and disk on db-01, memory and evictions on cache-01. USE sits below RED because resources explain services, not the other way around — a saturated database is interesting exactly when a service above it is slow.
Fewer Panels
The wall of graphs is the dominant dashboard anti-pattern. node_exporter alone ships hundreds of series per host, and a dashboard that tries to show them all answers nothing, because every incident renders 55 of its 60 panels irrelevant noise. The checkout dashboard lands at 15 panels — consistent units, consistent colors, one y-axis convention — and everything merely nice to know moves to a linked drill-down dashboard where the curious can find it on a calm day.
The Checkout Dashboard, Assembled
The chapter deliverable in full: a symptoms row on top — checkout error ratio, checkout p95 with its 500 ms threshold line — RED rows for bookings and payments beneath it, USE rows for the hosts, the database, and the cache at the bottom, one $service variable serving all of it.
And it pays off immediately. With three weeks of data on screen, the checkout p95 panel shows a clean step every Saturday morning — up around 09:00, back down by early afternoon, week after week. The Chapter 1 complaint is no longer an anecdote; it is a fact on a graph, visible and repeatable. What the dashboard cannot say is why. The USE rows move — hosts busier, the cache less effective — but nothing crosses saturation, no deploy annotation lines up, and that gap — a symptom with no sufficient cause — is exactly what traces exist to close in Chapter 8.
RED (Rate, Errors, Duration) describes a service from the user's side: how many requests, how many failed, how long they took. Apply it to anything that serves requests — bookings, payments, search, web.
USE (Utilization, Saturation, Errors) describes a resource from the machine's side: how busy, how backed up, how broken. Apply it to hosts, databases, caches, and queues. One dashboard uses both — RED rows on top say whether it hurts, USE rows below suggest where.
- Plotting every metric the exporters emit — node_exporter alone ships hundreds of series per host, and a dashboard that shows them all guarantees the panel that matters is below the fold during the one incident where seconds count.
- Putting cause metrics on top — a red CPU panel at the top tunnels the responder into host archaeology while the checkout error ratio, the only panel that says whether users are hurting, sits three rows down.
- Mixing units across adjacent panels — checkout latency in seconds next to database latency in milliseconds reads as a 1000× discrepancy to a tired brain, and the misread happens exactly when it is most expensive.
- Designing for the author instead of the responder — a layout that mirrors the team's architecture diagram rather than the diagnosis path, so navigating it requires the context of the person who built it.
- One giant dashboard for every audience — capacity planning, business reporting, and incident response crammed together, so each audience gets 5 useful panels and 40 distractions.
- Lay out every operational dashboard top-down as symptoms → services → resources, so scrolling down is the same motion as drilling into causes.
- Apply RED to every request-serving service and USE to every resource, and let the method — not taste — decide what each row contains.
- Cap the incident dashboard near 15 panels and move everything else to linked drill-down dashboards. A panel that never changed a decision is a panel to delete.
- Standardize units, colors, and thresholds across all Harborline dashboards, so green, yellow, and a 500 ms line mean the same thing on every page.
Knowledge Check
Checkout p95 belongs to the top row of the Harborline dashboard, and db-01 connection count sits two rows down. What principle decides that order?
- Panels are ordered to match the system architecture diagram
- Symptoms go on top, causes below
- Cheaper queries render first, so they go on top
- Database metrics are less important than service metrics
Which method applies to the payments service, and which to cache-01?
- USE for payments, RED for cache-01
- RED for both — everything ultimately serves requests
- RED for payments, USE for cache-01
- USE for both — saturation is what pages people
What actually goes wrong with a 60-panel incident dashboard?
- Sixty queries per refresh overload Prometheus
- Grafana cannot render more than 50 panels reliably
- More panels directly increase the false-positive rate of the raw data being collected
- During any incident, nearly all of the panels are noise hiding the one that matters
The 3 a.m. test rejects a dashboard. Who was it most likely designed for instead?
- The engineer who built it, on a calm day, with full context
- Executives who want a high-level business summary rather than incident detail
- Compliance auditors reviewing uptime
- Users of an older Grafana version
You got correct