Blameless Postmortems
A postmortem is a written record of an incident that exists to make the system smarter, and it works only under one condition: nobody gets punished for its contents. The moment a postmortem can hurt the engineer who wrote it, the next one arrives sanitized, late, or not at all — and the organization pays for the same incident twice.
Blameless is not softness. It is the epistemic condition for getting true information out of the people who were there, a principle software imported from aviation and medicine, where post-accident candor is a matter of survival. A pilot who fears prosecution files no near-miss report; an engineer who fears the postmortem edits the timeline.
When a Postmortem Is Owed
Fixed triggers, not mood: user-visible downtime or degradation, any data loss, any paged incident, any SLO budget spend above a set fraction. Harborline's rule: any SEV1 or SEV2, or any single incident consuming over 10% of the 28-day error budget. The August incident qualifies on the SEV2 trigger alone — it also spent roughly 9% of the budget in 38 minutes on top of the month's existing spend — and Mara is named postmortem owner before the incident channel closes.
The Document Format
Four load-bearing parts. A timeline: 02:31 processor latency rises, 02:40 page, 02:44 dashboard, 02:51 logs, 02:58 trace, 03:05 mitigation, 03:18 verified — lifted straight from the incident channel's timestamps. Impact in user terms: 38 minutes of degraded checkout, roughly 9% of the 28-day error budget, and the count of failed payment attempts. Contributing factors — plural, always. And action items, each with an owner and a date.
Why "Human Error" Is Never a Root Cause
If an engineer could take the harmful action, the system permitted it. The retry policy that amplified the August outage was a client-library default nobody had reviewed. "Someone should have reviewed it" is a dead end — it produces the action item "be more careful," which changes nothing. "No retry policy at Harborline has a load-amplification review" is fixable, because a review process can be built and checked.
The analysis therefore asks what made the action possible and reasonable-seeming at the time, never who did it. Names appear in the timeline as actors; they never appear in the causal chain.
Contributing Factors Over Root Cause
The August incident had no single cause. The processor slowed and the retry policy amplified the slowdown and no circuit breaker existed and no alert watched processor latency directly. Remove any one of the four and the night looks different: a healthy processor never triggers the policy, a 1-retry budget never amplifies, a circuit breaker fails fast, a dependency alert fires at 02:33 instead of 02:40.
A postmortem that stops at the first "root cause" — the vendor — produces exactly one action item: open a ticket with the processor. It misses the three factors that are actually in Harborline's control, which are the ones that prevent the whole class of incident rather than this one instance.
Action Items That Close
Each action item gets an owner who is a person (not "the team"), a due date, and a tracker ticket reviewed in the weekly sync until closed. Google's own postmortem literature names the failure mode: beautifully analyzed incidents with action-item lists that quietly rot. An unclosed action item is a scheduled repeat of the incident — the analysis found the fix and the organization declined to apply it.
Review and Circulation
The draft is reviewed by engineers who were not on the incident, because they catch the leaps of logic the responders can no longer see — then published internally, unredacted. A postmortem read by 40 engineers prevents more incidents than one filed in a folder, and the next team about to ship an unreviewed retry default is exactly who needs to read this one.
Blameless — the analysis never terminates at a person: no names in the causal chain, no punishment for honest accounts, because punished candor becomes silence. Choose this posture for every incident, without exception.
Consequence-free it is not — the system faces consequences: action items, changed defaults, new alerts. Repeated refusal to do the follow-up work is a performance issue handled outside the postmortem. Blame the fence, fix the fence; the fence-fixing is mandatory.
- Writing "root cause: engineer deployed the misconfigured retry policy" — the analysis stops at a person, the fix becomes "be more careful," and the identical incident recurs with a different name in the blank.
- Filing action items with no owner or date — "improve retry handling" assigned to nobody is a wish, and Google's own postmortem literature identifies unclosed action items as the most common failure of otherwise healthy postmortem cultures.
- Writing the timeline from memory 5 days later — reconstructed timelines drift toward the narrative the author already believes; the incident channel captured the real one at zero cost, use it.
- Stating impact only in internal units ("p99 exceeded 8 s") — the document's readers include future engineers deciding what to prioritize; "38 minutes of degraded checkout, 9% of the 28-day error budget" carries the weight.
- Restricting the postmortem to the people who were there — the unread postmortem teaches no one, and the next team to ship an unreviewed retry default learns the lesson the expensive way.
- Trigger postmortems by fixed criteria — any paged incident, any user-visible impact, budget spend over a set threshold — and name the owner before the incident closes.
- Build the timeline from the incident channel's timestamps the same day, while the artifacts (dashboard links, LogQL queries, trace IDs) are one click away.
- List contributing factors until removing any one would have changed the outcome, and reject any causal chain that terminates at a human.
- Track every action item with an owner, a date, and a ticket, and review open items weekly until the list is empty.
Knowledge Check
Why is blamelessness an information-quality mechanism rather than kindness?
- It keeps team morale high and helps engineers feel supported after stressful incidents
- Punishing authors corrupts the next report; blamelessness keeps the data true
- It means nobody is responsible for anything that happened
- It protects the company from legal liability for outages
Why is "human error" never an acceptable root cause?
- Because trained engineers do not actually make errors
- Because naming humans violates the postmortem template's formatting and style rules
- It terminates analysis where it should begin: at what the system permitted
- Because incidents are never actually triggered by human actions
The August postmortem lists four contributing factors instead of one root cause. What does that buy?
- Three of the four factors are fixable by Harborline; a single root cause would hide them
- A longer factors list makes the document look more thorough and rigorous to postmortem reviewers
- Multiple factors spread the blame so no single team feels singled out
- It proves the vendor was not actually involved in the outage
What separates a real action item from a decorative one?
- It is ranked by severity in the postmortem document
- It is assigned to the team collectively so anyone can pick it up
- A named owner, a due date, and a ticket reviewed until closed
- It describes the technical fix in enough implementation detail
You got correct