Alert Quality
More observability programs die of alert fatigue than of missing data. A pager that cries wolf trains its humans within weeks: acknowledge, roll over, check at 09:00 — and that reflex does not distinguish the false page from the outage. The stack built across eight chapters is only as good as the team's willingness to answer it.
Alert quality is not a sensibility; it is a measurable, maintained property. Every page carries a runbook, every fired alert gets reviewed, and the non-actionable ones get fixed or deleted on a schedule. This topic closes the chapter by turning "keep the pager honest" into three concrete mechanisms and three numbers.
Fatigue as the Failure Mode
Each non-actionable page slightly lowers the credibility of the next one. The decline is invisible day to day, and the endpoint is a team that treats the pager as weather — something that happens to you, not something you answer. The cost of the real incident lands there, in the humans, not in the tooling: Prometheus and Alertmanager perform identically whether the team trusts them or not.
A page also taxes the following day. Response quality degrades long before anyone consciously ignores the pager — a responder woken twice loses more than two interruptions' worth of judgment, and the 03:10 disk page from this chapter's Tuesday cost Mara a slow Wednesday on top of the lost hour. Fatigue accounting that only counts pages undercounts the damage.
The Runbook Link
Every severity: page rule carries a runbook_url annotation pointing at one page that answers four questions: what this alert means, how to confirm it is real, the first three things to check, and when to escalate. Harborline's checkout-error runbook links the checkout dashboard from Chapter 6 and the LogQL query from Chapter 7, so the responder's first two moves are one click each.
The value concentrates at 03:10. A rested engineer can improvise an investigation; a woken one executes a checklist, and the runbook is the difference between responding and puzzling. Writing it also works as a filter in the other direction — an alert whose runbook you cannot write has just confessed that it has no action, which is Topic 44's test failed in documentation form.
The Periodic Alert Review
The review is a standing monthly meeting with three inputs: everything that fired — the ALERTS metric in Prometheus makes this a query, not an archaeology project — how each page was resolved, and whether the responder acted or just acked. Every non-actionable alert leaves the meeting with a fix or a deletion assigned. The review's authority to delete is the whole point; a review that can only discuss produces minutes, not quiet.
The Demotion Ladder
A noisy alert descends one rung at a time: raise the threshold, lengthen for:, demote severity: page to ticket, convert to a dashboard panel, delete. Each rung keeps more of the signal than the one below it, so the ladder finds the least destructive fix that actually stops the noise — a silence, by contrast, stops the noise while preserving the broken rule underneath.
The db-01 disk-space alert that woke Mara at 03:10 with 4 days of headroom walks the ladder in this chapter and lands at ticket: same expression, same threshold, but the notification now files a ticket that the bookings team triages with coffee. Same signal, right destination — the demotion changed one label and repaired one relationship with the pager.
Measuring Quality
Three numbers per month tell you whether the program is healthy. Pages per on-call week, where the SRE guideline of at most 2 incidents per 12-hour shift is the ceiling, not the target. Percentage of pages that required action. Percentage where the runbook was followed and sufficient. The first comes from the on-call platform's incident count for the rotation (Alertmanager and Prometheus don't retain 30 days locally at Harborline's 15-day default, and count_over_time counts evaluation ticks, not incidents); the other two from the on-call platform's own analytics plus 5 minutes of honesty in the review.
When actionability drops below roughly 80%, the review has work to do — and the work is named by the ladder, not debated from scratch. Tracking these three numbers makes the pager itself an observed system, which is the fitting place for the chapter to end: the alerting stack now watches Harborline, and the review watches the alerting stack.
- Keeping an alert "just in case" after 3 consecutive non-actionable pages — those 3 pages already taught the team to ack-and-ignore, and the alert is now doing damage while providing nothing.
- Linking a runbook and never maintaining it — after the Chapter 11 move to Kubernetes the old SSH instructions are wrong, and a wrong runbook at 03:10 costs more minutes than no runbook, because it is followed.
- Muting a noisy alert with a silence that gets renewed forever — the rule stays broken, the silence eventually lapses on a holiday weekend, and the noise returns exactly when nobody remembers why it was muted. The fix was one rung on the demotion ladder.
- Measuring alert volume but not actionability — 10 pages a week that are all real is a hard month; 3 a week that are all noise is a dying program. The count alone cannot tell these apart.
- Running reviews where deletion needs unanimous agreement — someone always remembers the one time the alert almost mattered, nothing is ever deleted, and the graveyard compounds monthly.
- Require
runbook_urlon everyseverity: pagerule and enforce it in CI alongsidepromtool check rules— a paging rule without a runbook fails the build. - Hold the alert review monthly with explicit authority to delete, and treat a review that deletes nothing as a review that did not happen.
- Track pages per on-call week and percent-actionable as first-class metrics of the observability program itself, queried from
ALERTSand the on-call platform's reports. - Treat every non-actionable page as a bug with a named fix from the demotion ladder — threshold,
for:, demotion, dashboard, deletion — assigned before the review ends.
Knowledge Check
Why is alert fatigue the program-killer, rather than missing coverage?
- Noise overloads Alertmanager until real alerts queue behind the false ones during the incident itself
- It trains responders to ack without engaging, and that reflex applies equally to the real outage
- False pages waste compute budget that better rules would have used
- It is not — a missing alert is always worse than a noisy one
What must a runbook contain to be useful at 03:10?
- The service's architecture diagram and a history of past incidents
- What the alert means, how to confirm it, the first three checks, and when to escalate
- The alert's PromQL expression and its evaluation history
- The full escalation roster with phone numbers for every engineer and each backup on-call team
The db-01 disk alert paged at 03:10 with 4 days of headroom. Which rung of the ladder fits?
- Raise the threshold from 80% to 95% and keep it paging
- Delete the rule — disk metrics belong on dashboards only, never on a pager or a ticket queue
- Demote severity from page to ticket so it files for morning triage
- Silence it for a month while the team decides
Why does percent-actionable beat raw page count as a health metric?
- It is easier to compute from the ALERTS metric than a raw count, needing no extra recording rules
- Because the SRE guideline is defined in percentage terms
- It requires no human judgment, so it cannot be gamed
- A count cannot distinguish a busy-but-honest pager from a quiet-but-lying one
You got correct