Incidents, On-Call, Postmortems & Maintenance

Things break in production — that's normal, not a sign of failure. What separates good teams is how they respond: someone is on call, they follow an incident process, and afterward they run a blameless postmortem to learn. And all of this lives inside the longest, least-glamorous phase of all: maintenance.

An incident is a serious problem in production that needs urgent attention — the app is down, data isn't saving, reminders have stopped. How a team handles incidents says a lot about its maturity.

A fire department is the model: there's always a crew on duty (on-call), a drilled response for when the alarm sounds (incident response), and a no-blame review after every call to get better (the postmortem).

Detect

→

Respond

→

Mitigate

→

Resolve

→

Postmortem

On-Call

On-call is a rotation that ensures someone is always ready to respond if production breaks, day or night. Team members take turns carrying the responsibility (and the pager) for a stretch — a week, say — so the load is shared and there's always a known person to react. On-call is what makes "we'll fix it fast" a real promise instead of a hope that someone happens to be awake.

Incident Response

Incident response is the practised process for handling a live problem: detect it (usually via an alert), mitigate it to stop the bleeding, communicate clearly to those affected, and then resolve the underlying cause. The first priority is almost always to stop the harm — get the system working again — even with a temporary fix, before hunting down the real root cause. Calm, rehearsed response beats panic every time.

Blameless Postmortems

After an incident, good teams hold a blameless postmortem — a write-up and discussion of what went wrong, why, and how to prevent it. The crucial word is blameless: the focus is on the system that allowed the failure, not on punishing a person. When people aren't afraid of blame, they share honestly, the real causes surface, and the system genuinely improves. Blaming individuals just teaches everyone to hide problems — the opposite of what you want.

Maintenance

Surrounding all of this is maintenance — the long tail of fixing bugs, patching security holes, updating dependencies, and making small improvements for years. It's the least glamorous phase and, as Chapter 2 noted, where most of a system's life and budget actually go. Far from leftover work, maintenance is the job for most of a product's existence, and it loops straight back to planning the next change.

When Cadence's reminders go down at 2 AM, the on-call engineer is paged, logs in, and mitigates fast — switching reminders off via the feature flag to stop the harm. The next day the team holds a blameless postmortem and finds the real cause: a missing alert that would have warned them hours earlier. Nobody is blamed; the fix is to add that alert. Then Cadence settles back into ongoing maintenance — a little better than before, because the failure taught them something.