Annotations and Drill-Downs

A graph shows what changed and when — never why. The leading cause of a step change on any service graph is a deploy, and an annotation is how the graph learns that: a vertical line at the deploy moment, tagged with service and version, overlaid on every panel that subscribes to it. When the line sits two pixels left of the step, the question answers itself.

This topic wires Harborline's CI to post deploy annotations, then builds the other half of navigability: data links and dashboard links, the click path from an overview panel down to the detail view. Together they turn a pile of dashboards into a route a responder can travel without retyping a single timestamp.

The Step Nobody Can Explain

The bookings error ratio steps up at 09:12 on a Tuesday. Without annotations, correlating that with the 09:10 deploy means Slack archaeology across two channels, a scroll through CI job history, and a debate about whose clock is right — 20 minutes of work to establish what one vertical line states at a glance. The correlation is not hard; it is just absent from the place where the question gets asked.

Deploy Annotations via the API

Grafana accepts annotations over HTTP: CI's deploy job POSTs to /api/annotations with a service-account token, and the event appears on every subscribed dashboard within a refresh. The request body carries three fields that matter. time is the deploy moment in epoch milliseconds — taken from the pipeline, not typed by a human. text is what the responder reads on hover: bookings v2.41.0 deployed. tags is what dashboards filter on: deploy, bookings, and the version string.

The cost is one curl line in the deploy pipeline. The return is permanent: every future step change on every Harborline dashboard arrives with its most likely suspect already drawn on the graph, posted by the machine that caused it, at the moment it happened.

Query-Based Annotations

Annotations can also come from a data source query evaluated at render time: a Loki query matching deploy log lines, or Prometheus's ALERTS series overlaying exactly when alerts were firing on the same graph the responder is reading. No API call, nothing stored in Grafana — but the events must already exist as telemetry somewhere. API-posted annotations create the record; query-based ones reuse a record something else already wrote.

Scoping by Tags

A dashboard does not show every annotation in the database; it subscribes to tags. The checkout dashboard requests deploy events tagged bookings or payments and nothing else, so a Saturday of search-team deploys draws zero lines on it. Skip the scoping and every dashboard shows every event from every team — and within a month responders train themselves to ignore vertical lines entirely, which is worse than never having drawn them.

Data Links

A data link is a URL template on a panel that carries context from the clicked point. ${__field.labels.job} resolves to the series the responder clicked; ${__url_time_range} carries the current window. Clicking a spike on the overview lands on the service dashboard already filtered to the right service and the right 30 minutes — the alternative is arriving at a default view and rebuilding both by hand, under pressure, from memory.

Dashboard Links and the Drill-Down Path

Dashboard links do the same job at page level: a header row of links from the overview to the checkout dashboard, from there to the per-service dashboards, from any of them into Explore. Harborline's navigation contract is that every hop preserves var-service and the time range, so the responder who starts at a page and ends four clicks deep in a Loki query is still looking at the same window they were paged about.

That contract is the point of the whole topic. Annotations put the likely cause on the graph; links make the graph a door instead of a dead end. The path from "something is wrong" to "here is the log line" becomes a sequence of clicks — and in Chapter 8 that same path extends one hop further, from a metric spike into the trace that explains it.

Overviewspike clicked

→

Data linkcarries service + window

→

Service dashboardsame window

→

Trace / logsExplore