On-Call That Works
On-call is the human component of the alerting pipeline. Everything Chapter 9 built — symptom-based rules, Alertmanager routing, runbook links — terminates in one physical event: a specific person's phone makes a specific noise at 02:40, and that person is expected to acknowledge within minutes and act. The pipeline is only as good as what happens after the noise.
Most teams treat on-call as weather: endured, complained about, never engineered. It is actually a system with measurable load, explicit capacity limits, and failure modes as real as a full disk. A rotation that deprives its members of sleep degrades exactly like an unmonitored service — quietly, then all at once, when someone quits.
The Rotation as a Designed Artifact
Two rotation shapes work in practice. Follow-the-sun splits coverage across two or three sites, each handling its own business hours, so nobody is ever paged at night — the gold standard, available only to organizations with teams on multiple continents. Everyone else runs single-site week-long shifts with a primary who gets paged first and a secondary who catches escalations when the primary misses the acknowledgment window.
The sizing math is not negotiable. A single-site rotation needs at least 8 people to keep any individual on call less than 25% of the time — the floor Google's SRE book sets for its own single-site teams; the number only drops to 5–6 per site once coverage splits across two sites. Below that floor the arithmetic turns hostile: three people means each spends a third of their life tethered to a pager, with zero slack for vacation, illness, or the first resignation.
Pager Load as a Measured Quantity
Pages per shift is a metric like any other: count it, graph it, alert on it. Google's working target is at most 2 incidents per 12-hour shift, because each page costs far more than the minutes of response — it costs the follow-up work, the postmortem time, and a broken sleep cycle that degrades the next day's judgment.
A shift averaging 5 or more pages means the alerting from Chapter 9 has failed, not the human. When the pager-load graph crosses the line, the action item goes against the alert rules — tighter thresholds, causes demoted to tickets, symptoms consolidated — never against the on-caller's stamina.
The Handoff
Every shift change gets a written handoff note: open incidents, silenced alerts and exactly when the silences expire, known flaky signals, and in-flight changes that might misbehave. Without it, each new on-caller re-discovers the week's context from scratch, and the discovery reliably happens at 03:00 — the silence on a flapping disk alert expires Tuesday, the new on-caller has never heard of it, and 40 minutes burn on rediscovering last week's decisions.
Compensation and Recovery
Out-of-hours pages are work, and work gets paid or repaid: time-off-in-lieu or an on-call stipend, plus a hard rule that a shift with a night page starts late the next day — as policy, not as a favor the on-caller has to request. Unpaid, unrecovered on-call is how an 8-person rotation quietly becomes a 3-person rotation, and the previous section already showed where 3-person rotations end.
What the On-Caller Actually Owes
The contract is narrower than most people assume: acknowledge within the paging window — 5 minutes at Harborline — then mitigate, not root-cause. The Chapter 9 contract holds: every page has a runbook, and the on-caller's job at 02:40 is to stop the bleeding and escalate if it won't stop. Full diagnosis is daylight work once the symptom is contained.
Mara's Rotation, Concretely
Harborline runs one site: week-long primary/secondary shifts across 8 engineers, with pages-per-shift graphed in Grafana straight from Alertmanager's notification metrics and reviewed monthly. The August Saturday page that opens the next two topics is the first night page in 3 weeks — which is the only reason the rest of this chapter goes well. A rested engineer walked to her laptop; an engineer on her fifth page of the night would have snoozed it.
- Running a rotation with 3 people — each is on call 33% of their life with no slack for vacation or illness, the first resignation collapses the rotation, and the remaining two follow it out the door.
- Letting a noisy shift pass without consequence — 8 pages in a night that all get acknowledged and shrugged off trains the on-caller to sleep through the real one; every page either produces action or produces an alert-tuning ticket, never nothing.
- Skipping the handoff note because the week was quiet — the silence on the flapping
mq-01disk alert expires Tuesday, the new on-caller doesn't know it exists, and Tuesday's page starts with an archaeology dig through last week's context. - Paging the primary and secondary simultaneously for every alert — the secondary exists for escalation after the acknowledgment window, not as a second phone making the same noise; dual-paging doubles the human cost of every page for zero response gain.
- Treating heroic response to a brutal rotation as commendable — praising the engineer who handled 30 pages in a week rewards the broken system that produced them; the fix is fewer pages, not tougher humans.
- Size the rotation before you turn on paging: at least 8 engineers for a single site (5–6 per site once you can run follow-the-sun) with each under 25% on-call time, so night pages simply don't exist.
- Graph pages-per-shift from Alertmanager's notification metrics and review it monthly — when a shift exceeds 2 incidents, file the action item against the Chapter 9 alert rules, not against the on-caller.
- Require a written handoff note at every shift boundary listing open incidents, active silences with their expiry times, and in-flight changes.
- Compensate every out-of-hours page with money or recovery time, and write the morning-after late start into the rotation policy.
Knowledge Check
Why is 8 engineers the floor for a single-site on-call rotation?
- It keeps each person under the 25% on-call ceiling with slack for absence
- Alertmanager cannot route notifications to fewer than five receivers
- Five people are needed so every incident has at least five simultaneous responders working on it
- Fewer people would exceed the 2-pages-per-shift target automatically
A rotation's pages-per-shift graph shows shifts averaging 5+ pages. What does that indicate?
- The on-callers need better incident-handling stamina and training
- The Chapter 9 alerting has failed and the alert rules need tuning
- The rotation simply needs more engineers to absorb the load
- The infrastructure has become dramatically less reliable
What does a missing handoff note actually cost?
- The paging schedule resets to its defaults and every future shift must be manually reassigned
- Active silences are automatically deleted at the shift boundary
- The new on-caller re-learns the week's context from scratch, usually during an incident
- Alerting pauses until the incoming on-caller confirms the handoff
Why is paging the primary and the secondary simultaneously for every alert wasteful?
- Notification systems cannot deliver two pages at once reliably
- Secondaries should never be paged under any circumstances
- It doubles the human cost of every page for zero response gain
- It makes it ambiguous who owns the incident afterward, which delays every response
You got correct