Incident Response
An incident is an unplanned event that threatens the SLO and needs coordinated human response — and coordination is the word that separates a 38-minute incident from a 4-hour one. The framework the industry converged on descends from the Incident Command System — built for 1970s wildfire response and now FEMA's all-hazards standard — imported into software by Google's SRE organization: explicit roles, a single coordinator who does not debug, and communication on a fixed cadence.
The core cultural rule sits on top of the mechanics: declaring an incident early and loudly beats quiet heroics every time. An incident declared, handled, and closed in 10 minutes costs almost nothing. A solo 02:40 debugging session that should have been an incident costs hours, because it forfeits the extra hands, the communication channel, and the timeline the machinery would have provided.
Severity Levels With Teeth
Severity levels only work when they are defined by observable conditions, not vibes. Harborline's ladder: SEV1 — checkout down or the error ratio above 25%; all hands, status page entry, exec notification. SEV2 — an SLO burning fast enough to page, user-visible degradation; an Incident Commander is assigned and a status page entry is considered. SEV3 — degradation with budget to spare and no user-visible impact yet; handled in working hours, no page.
The level dictates the response machinery — who gets woken, what gets posted, who gets told — which is why "is this a SEV2?" must be answerable from a dashboard. A ladder defined as "SEV1: critical, SEV2: major" guarantees that every incident opens with an argument about its own classification while checkout stays broken.
When to Declare
Google's published thresholds adapt cleanly to a team of any size: declare if you need a second team involved, if customers are visibly affected, or if the problem is still unsolved after an hour of focused effort. The fast-burn page from Chapter 10 clears the second condition by construction — it fires only when the error budget is draining at a customer-visible rate — so at Harborline a burn-rate page is a declared SEV2 by default. No deliberation required at 02:40.
The Incident Commander
The Incident Commander coordinates and explicitly does not debug. They assign the investigation to an ops lead, own the incident channel, decide between competing mitigations, and keep a running record of what has been tried. The role exists because a group of engineers all debugging in parallel duplicates work, drops threads, and tells stakeholders nothing.
The moment the IC opens a terminal, nobody is coordinating. In a 2-person night incident the on-caller pulls in the secondary precisely so that one of them can hold the IC role while the other investigates — two hands on keyboards and zero on coordination is the worse configuration, even at 03:00.
Communication Cadence
Updates go out on a fixed clock — every 30 minutes for a SEV2, every 15 for a SEV1 — in one designated channel, even when the update is "no change, still investigating." The fixed cadence kills the constant "any update?" pings that otherwise consume the responders, because everyone knows exactly when the next word arrives. As a side effect, the channel's timestamped messages become the postmortem timeline for free.
The Status Page
The status page is externally hosted — it must not share fate with harborline.example, because the outage that most needs a status page is the one that takes a co-hosted page down with it. Entries are honest and plain-language: what is affected, since when, and when the next update comes. Posting "investigating elevated payment errors" 10 minutes in costs nothing and cuts support load; silence teaches customers to discover outages on social media instead.
Handoff and Closure
Long incidents hand the IC role over explicitly — "you are now IC," said in the channel, so there is never a moment when two people believe the other one is coordinating. Closure is a positive act, not a fade-out: the mitigation is verified against the SLO dashboard, the status page entry is resolved, and a postmortem owner is named before the channel goes quiet.
On-call responder — the hands: paged first, runs the runbook, investigates, applies the mitigation. Choose to stay in this role when a second person is available to coordinate.
Incident Commander — the coordinator: assigns tasks, owns communication, decides between competing mitigations, and touches no terminal. In a small team the first responder starts as both, but the first thing they do when an incident grows is hand one of the two hats to someone else — holding both past the first 15 minutes is how details and communication both drop.
- Debugging alone for 90 minutes before declaring, to avoid "making a fuss" — the incident machinery of extra hands, comms, and a timeline is exactly what compresses resolution time, and the quiet hero forfeits all of it; declaring early and being wrong costs 10 minutes of apology.
- The most senior engineer defaulting to IC and then diving into the logs — now the best debugger isn't debugging and nobody is coordinating; assign IC by role, not seniority, and let the expert investigate.
- Severity levels defined as "SEV1: critical, SEV2: major" with no observable criteria — every incident opens with a 10-minute argument about its own severity while checkout stays broken.
- Hosting the status page on the same infrastructure as the product — the outage that most needs a status page is the one that takes the status page down with it.
- Skipping updates because there's nothing new — subscribers and stakeholders fill silence with worst-case assumptions, and the support team starts pinging the responders directly, which is the interruption the cadence existed to prevent.
- Define each severity level by conditions readable off a dashboard — error ratio, burn rate, user-visible yes/no — so declaring is a lookup, not a debate.
- Declare on any of Google's three triggers (second team needed, customer-visible, unresolved after an hour) and treat every fast-burn SLO page as an automatic SEV2 declaration.
- Assign an Incident Commander who does not debug the moment more than one person is responding, and hand the role over explicitly on shift changes.
- Post to the status page within 15 minutes of a customer-visible incident and update on the declared cadence, hosting the page outside your own infrastructure.
Knowledge Check
Why does declaring an incident early beat quiet solo debugging?
- It establishes early who is at fault for the outage
- The machinery it activates is cheap when wrong and compresses resolution when right
- Every declaration automatically publishes a status page entry, which customers expect
- Early declaration lowers the incident's severity level
During an incident, the Incident Commander's defining constraint is what?
- They coordinate and deliberately do not debug
- They must be the most senior engineer available on the team
- They observe silently and only write the postmortem afterward
- They personally apply every mitigation so responders can keep investigating
Why must severity levels be defined by observable conditions?
- So that incident tooling can auto-create tickets with the right priority labels
- Because compliance audits require documented severity definitions
- So declaring becomes a dashboard lookup instead of a debate
- So every incident can be escalated to all hands immediately
The update cadence says to post "no change, still investigating" every 30 minutes. What does that buy?
- Faster root-cause identification through frequent summaries
- Freedom from interruptions, plus a timeline as a by-product
- It satisfies the status page's external update requirements automatically
- Regular updates allow the severity level to be downgraded sooner
Why does the status page have to live outside Harborline's own infrastructure?
- External hosting is cheaper than provisioning and running the status page on app-01 ourselves
- So customers cannot associate the page with the outage
- Only third-party vendors are allowed to write outage notices
- The outage that most needs the page is the one that would take a co-hosted page down
You got correct