Chapter 8 closed the Saturday mystery; this chapter makes sure the next regression pages Mara before a customer reports it. It sets the philosophy — page on symptoms, ticket the rest — then builds the machinery: Prometheus rules for checkout errors and latency, Alertmanager routing and grouping on obs-01, receivers that reach a phone, and the review discipline that keeps the pager honest.
5 topics
An alert that pages is a claim: a human must act now, or users will suffer. Harborline's stack can finally back that claim — the metrics, dashboards, logs, and traces of Chapters 2 through 8 mean a firing alert arrives with its investigation kit attached. What is missing is the paging layer itself, and the discipline that keeps it from becoming noise: this chapter builds both, and gets one 03:10 wake-up wrong on purpose along the way.
Five topics run the full distance from principle to practice: the page/ticket/dashboard philosophy and why symptoms beat causes; alerting rules — expr, for, labels, annotations — and Harborline's first two pages on checkout errors and p99 latency; Alertmanager's routing tree, grouping timers, silences, and inhibition; receivers and the on-call platforms that own escalation; and alert quality — runbooks, the monthly review, and the demotion ladder that turns the noisy db-01 disk page into a morning ticket.
The life of an alert — a true expression to a woken human