Chapter Twelve · Incidents and On-Call

Incidents and On-Call

The payoff chapter. On an August Saturday at 02:40, a burn-rate page wakes Mara, and she walks the exact path the book has built since Chapter 3 — alert, dashboard, logs, trace — to a payments-processor slowdown amplified by client retries. Thirty-eight minutes from page to resolved, then a blameless postmortem and four concrete improvements fed back into the stack. The same failure in Chapter 1 would have been hours of ssh archaeology.

5 topics

Every layer so far — metrics, dashboards, logs, traces, alerts, SLOs — exists for one moment: a human, woken at 02:40, deciding what to do next. This chapter is about that human. It engineers the rotation that carries the pager (8 people minimum on a single site, pages-per-shift graphed like any other metric), the incident machinery that turns a solo debugging session into a coordinated response with a commander who never touches a terminal, and the communication cadence that keeps stakeholders informed without interrupting the responders.

Then the book pays off its own setup. The August incident runs minute by minute — 02:40 page, 02:44 dashboard, 02:51 logs, 02:58 trace, 03:05 mitigation, 03:18 verified — with every hop landing in a tool built in Chapters 3–10. What follows matters as much as the night itself: a blameless postmortem that finds four contributing factors instead of one scapegoat, and a learning loop that converts each factor into a metric, an alert, a panel, or a runbook line. Failure stops being a war story and becomes configuration.

The incident loop — failure becomes configuration
Page02:40 wake-up
Respondcoordinate
Mitigatestop the bleed
Postmortemblameless
Learnfour changes
Better signalsnext one shorter

Topics in This Chapter

Topic 60
On-Call That Works
On-call is a system to be engineered, not a fact of life to be endured: at least 8 people on a single site, under 25% on-call time each, at most 2 incidents per shift. Pages-per-shift is a metric like any other — graph it, and when it climbs, fix the alerts, not the human.
On-Call
Topic 61
Incident Response
Severity levels defined by dashboard-readable conditions, an Incident Commander who coordinates and never debugs, updates on a fixed cadence, and a status page that shares no infrastructure with the product. Declaring early and loudly beats quiet heroics every time.
Incidents
Topic 62
From Page to Root Cause
The August incident, minute by minute: fast-burn page at 02:40, dashboard localizes payments by 02:44, logs find the processor timeouts, the trace exposes 3 retries amplifying a slow dependency into 4× load. Mitigated, verified against the burn-rate panel, resolved in 38 minutes.
Walkthrough
Topic 63
Blameless Postmortems
A postmortem exists to make the system smarter, and it works only if nobody gets punished for its contents — punished candor becomes silence. Timeline from the incident channel, contributing factors instead of a single root cause, and action items with owners, dates, and tickets.
Postmortems
Topic 64
The Learning Loop
The August incident feeds back into every layer: a retry-budget metric, a reviewed burn-rate alert, a processor-latency panel, a runbook update — each replayed or reviewed before its item closes. Incident count is a signal to read, never a KPI to optimize.
Feedback