Dashboards as Code

Every Grafana dashboard is one JSON document — panels, queries, variables, links, thresholds, all of it. Left in the UI, that document becomes the hand-tuned dashboard everyone fears touching: one author who left the company, no history, no review, and no way back after a bad edit on a Friday afternoon.

This topic moves the checkout dashboard into git and provisions it from files, turning dashboard changes into pull requests — the same discipline Harborline's Prometheus rules and Compose files already follow. Prometheus config earned code review in Chapter 3; the dashboards responders stare at during incidents deserve no less.

The Hand-Tuned Dashboard Problem

A UI-edited dashboard is production configuration that escaped version control. When a panel breaks, there is no diff to read, no author to ask, and no revert — just the current state and everyone's fading memory of what it looked like last month. The fear of touching it is rational, and the fix is structural, not behavioral: no policy about careful editing survives contact with a 2 a.m. "quick fix". Put the document in git and the fear evaporates, because any change can be read, reviewed, and undone.

The JSON Model

Export any dashboard — the toolbar's Export icon in the UI, or the HTTP API — and read the document. The checkout dashboard's own JSON is legible top to bottom: a uid identifying it, a title, a panels array where each entry holds its queries, unit, thresholds, and grid position, and a templating section holding the $service variable and its label_values query.

Once the JSON is legible, the UI's status changes: it is no longer where the dashboard lives, just one editor for a document that lives elsewhere. That inversion is the entire topic.

File-Based Provisioning

Dashboard provisioning mirrors the data source provisioning from the start of this chapter: a provider file tells Grafana to load every JSON dashboard from a directory and re-read it on an interval. Harborline mounts its git repo's dashboard directory into the container behind this provider.

# /etc/grafana/provisioning/dashboards/harborline.yaml
apiVersion: 1
providers:
  - name: harborline
    type: file
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards

The provider declares a name, the file type, a 30-second re-read interval, and the directory to watch. With this in place the git repo — not the Grafana database — is where dashboards live: a rebuilt obs-01 container comes up with every dashboard intact, and a merged pull request appears on screen within half a minute of deployment.

The Edit Loop and Review via PR

Provisioned dashboards refuse UI saves by default, which forces the honest loop: edit in the UI, export the JSON, commit, open a pull request. A threshold change from 500 ms to 300 ms arrives as a two-line diff a reviewer approves in 30 seconds, and a bad change is one git revert away. The loop feels slower than clicking Save exactly once — the first time a revert saves an incident, the accounting flips for good.

Parameterize vs Duplicate

The JSON makes a design decision explicit that the UI lets you dodge: one $service-templated dashboard covers all five services as long as their layout is genuinely identical, and duplication is justified only when pages truly diverge. Harborline's generic service view is one templated file; the checkout dashboard is its own file, because its RED-plus-USE composition with a business-symptom top row fits no generic template.

Generators, Briefly

When raw JSON diffs stop being readable — grid-position churn, 2,000-line files, ten teams editing — teams stop writing the JSON and start generating it: Grafonnet (Jsonnet), grafanalib (Python), the Grafana CLI for syncing, or the Grafana Terraform provider alongside the rest of the infrastructure. The output is the same JSON model, produced by code instead of by hand. For Harborline's six dashboards this is machinery without a problem to solve; know the names, reach for them when the diffs demand it.