Observability Deep Dive
by Sergey Okinchuk
Chapter 9Prometheus Alerting Rules
Topic 44

Alerting Philosophy

Philosophy

An alert that pages is a claim: a human must act now, or users will suffer. Every page is a person woken at 03:10, and the cost is not just the lost sleep. Each false claim teaches that person to trust the pager a little less, until the real outage gets acknowledged from muscle memory and checked at 09:00.

This topic sets the rule the rest of the chapter implements: page on symptoms users can feel, ticket everything that can wait until morning, and turn the rest into dashboard panels. Chapter 8 closed the Saturday mystery; the alerts built here exist so the next regression pages Mara before a customer has to report it.

The Definition of a Page

A page interrupts a human immediately, at any hour. That interruption is justified only when both halves of a test pass: a user-visible problem is happening or is minutes away, and human action changes the outcome. Disk at 80% with 4 days of headroom fails the first half. A crashed container that Docker restarts in 10 seconds fails the second.

If either half is false, it is not a page. The test sounds strict because it is — the pager's value is exactly proportional to how rarely it lies, and every exception granted "because this one feels important" spends that value down.

Symptoms, Not Causes

Users experience symptoms. Checkout returns errors; checkout takes 8 seconds. No customer has ever experienced "CPU at 92%" — either the high CPU turns into errors or latency, in which case the symptom alert fires, or it does not, in which case there was nothing to wake anyone for.

One symptom alert on the checkout error ratio covers every cause that produces errors: a bad deploy, a dead PostgreSQL on db-01, a saturated Redis connection pool — including the causes nobody predicted. That last clause is the whole argument. Cause alerts enumerate the failure modes someone already imagined, and the Saturday mystery was imagined by no one; a symptom alert on checkout latency would have caught it the first weekend it happened.

The Cause-Alert Graveyard

Every aging monitoring config carries the same graveyard: CPU above 80%, disk at 80%, load average, memory pressure, one replica down behind a load balancer. Each fires routinely with zero user pain. Each has no obvious action at 03:10. Each ends up muted or resented, and the resentment does not stay contained to the noisy rule — it spreads to the pager as a whole.

These signals are not worthless; they are misplaced. CPU, memory, and replica counts belong on dashboards, where they explain a symptom page during the investigation instead of generating pages of their own. Harborline learns this the concrete way in this chapter: a disk-space alert on db-01 fires at 03:10 on a Tuesday, wakes Mara, and turns out to need nothing before Thursday. By Topic 48 it has been demoted to a ticket.

The Three Destinations

Every alert idea gets sorted into exactly one of three destinations. A page interrupts a human now, reserved for act-now situations. A ticket is a real problem on a fuse longer than a night's sleep — the disk fills in 4 days, the queue depth drifts up week over week — handled during working hours. A dashboard holds the context you consult when a symptom fires, and never initiates contact.

The sorting question that settles most arguments: what does the woken human do in the first 5 minutes? A concrete answer means page. "Look at it in the morning" means ticket. No answer at all means dashboard — and "page, just in case" is the answer that destroys pagers, because "just in case" is a confession that no action exists.

Three destinations — sort every alert idea into exactly one
Page
Interrupts a human now, at any hour. Reserved for user-visible pain where minutes matter — checkout errors, checkout latency. The 5-minute action must be nameable.
Ticket
Waits for working hours. A real problem on a fuse longer than a night's sleep — a disk 80% full with 4 days of headroom, a certificate expiring in 2 weeks.
Dashboard
Never initiates contact. The context you consult when a symptom fires — CPU, memory, replica counts that explain a page instead of raising one.

Harborline's Two Pains

Harborline's customers can feel exactly two things: checkout failing and checkout being slow. Those become the book's first two paging rules in Topic 45 — error ratio and p99 latency, measured at the service edge from harborline_checkout_requests_total and the checkout duration histogram, not from the internals underneath them. Chapter 10 sharpens both into multi-window burn-rate alerts against the 99.9%/28-day checkout SLO. The philosophy here is the requirement; the SLO machinery is the implementation.

Page vs Ticket

Page — interrupts a human now, at any hour. Reserve it for user-visible pain where minutes matter: the checkout error ratio, checkout latency.

Ticket — waits for working hours. Real problems on a fuse longer than a night's sleep: a disk 80% full with 4 days of headroom, a certificate expiring in 2 weeks. If you cannot name the action the woken human takes in the first 5 minutes, it is a ticket — or a dashboard.

Common Mistakes
  • Paging on node_cpu above 80% — CPU at 92% with checkout healthy has no action, and after the third such page Mara acks without looking. That reflex carries over, unchanged, to the page that is real.
  • Paging when one bookings replica dies behind the load balancer while the other carries the traffic fine — users see nothing, and the outcome is a 03:10 wake-up for something a morning ticket handles identically.
  • Wiring every cause metric to the pager "for coverage" — one dead PostgreSQL then fires 9 alerts at once (CPU, connections, error ratio, latency, replication lag…), and the on-call spends the first 10 minutes of a real incident triaging their own pager instead of the outage.
  • Marking every alert severity: page because it felt important when written — when everything pages, severity carries no information, and the routing tree in Topic 46 has nothing left to route on.
  • Never deleting an alert — the set only grows, the fraction that is actionable only falls, and within a year the pager is background noise. Topic 48 makes deletion a scheduled activity precisely because it never happens spontaneously.
Best Practices
  • Page only on symptoms measured where users are — error ratio and latency at the service edge, from harborline_checkout_requests_total and the checkout duration histogram — never on the internals underneath them.
  • Sort every alert idea into page, ticket, or dashboard before writing the rule, and record the decision in the severity label so Alertmanager can route on it.
  • Demote, don't mute: when a cause alert proves non-actionable, change its severity to ticket or move it to a dashboard panel. A silence hides the noise; a demotion fixes it.
  • Write the 5-minute action into the alert before it ships — the runbook_url annotation from Topic 48. An alert whose action you cannot write down has just told you it is not a page.
Comparable toolsDoctrine Google SRE Book — the canonical statement of "alert on symptoms" (Ch. 6, Monitoring Distributed Systems)Ancestor Nagios host/service checks — the cause-based model most graveyards descend fromOn-call PagerDuty high/low urgency and Opsgenie priorities — the page/ticket split enforced at the on-call layer

Knowledge Check

During a failure nobody predicted, why does one symptom alert on the checkout error ratio beat ten cause alerts?

  • Symptom expressions evaluate faster, so the page arrives sooner
  • It fires on any cause that produces errors, including unpredicted ones
  • Cause alerts cannot carry severity labels, so Alertmanager cannot route them
  • One alert rule is cheaper for Prometheus to store than ten

The disk on db-01 is 80% full with 4 days of headroom at the current growth rate. Where does this alert belong?

  • Ticket — real, but nothing a woken human must do tonight
  • Page — a full disk takes the database down, so minutes matter
  • Dashboard only — disk usage is a cause metric and never deserves an alert
  • A silence, renewed until the cleanup gets scheduled

What do repeated non-actionable pages actually do to an on-call team?

  • Nothing lasting, provided each one is marked as a false positive afterward in the weekly review
  • They overload Alertmanager until real notifications are delayed
  • They train the responder to ack without looking — a reflex that carries to real pages
  • They only matter once volume exceeds 2 incidents per 12-hour shift

You cannot name any action the woken human would take in the first 5 minutes of an alert. What does that tell you?

  • The rule needs a longer for: duration before it fires
  • It should page a wider group so someone finds an action
  • The threshold is too high; firing earlier would leave more time to act
  • It is not a page — sort it into ticket or dashboard

You got correct