From Page to Root Cause
This is the 38 minutes the whole book has been building toward. On an August Saturday at 02:40, the checkout SLO's fast-burn alert pages Mara; at 03:18 she posts the resolution. Every minute in between is spent inside tools built in Chapters 3–10: the dashboard from Chapter 6, the Loki queries from Chapter 7, the trace view from Chapter 8, the burn-rate math from Chapter 10.
Hold the counterfactual while you read. In the Chapter 1 world — two VMs, ssh, docker logs, htop — the same failure would have been hours of archaeology across app-01 and app-02, with the actual mechanism likely never found. Walk it minute by minute.
02:40 — The Page
Alertmanager fires the checkout SLO's fast-burn alert: the error budget is burning at a rate that would exhaust the entire 28-day budget in under 2 days. The page carries the alert name, the runbook link, and the dashboard URL — Mara acknowledges within the 5-minute window and opens the runbook before she is fully awake.
A fast-burn page is customer-visible by construction, so per the previous topic there is nothing to deliberate: she declares a SEV2 in the incident channel at 02:42 and starts the clock on the 30-minute update cadence.
02:44 — The Dashboard
The checkout dashboard from Chapter 6 localizes the symptom in one look. The bookings request rate is normal for a summer Saturday night; the checkout error ratio is climbing; and the payments p99 has exploded from its usual 300 ms to over 8 s. The RED-method rows rule out search, web, and the hosts in seconds — every other panel is flat.
Four minutes after the page, the scope has narrowed from "something is wrong with checkout" to "the problem lives on the payments path." That narrowing is the dashboard's entire job, and it happened without a single terminal.
02:51 — The Logs
Next hop down the ladder: what is payments actually saying? A LogQL query scoped to {job="payments"} and filtered to errors shows a wall of structured timeout entries on calls to the third-party card processor, starting at 02:31 — nine minutes before the page. The service is healthy; its dependency is not answering in time.
Every one of those log lines carries a trace_id field, because Chapter 7 wired trace IDs into the structured log format before anyone needed them. The next hop costs one click, not one re-typed timestamp.
02:58 — The Trace
The Tempo trace makes the mechanism visible in a way no log line could. Checkout requests show 8 s spans on the outbound processor call, and stacked beneath each failed call sit 3 retry spans — the client retries every timed-out call 3 times, so a slow processor receives 4× the request volume, which keeps it slow. The processor degraded on its own; Harborline's retry policy amplified that degradation into an outage. Two contributing factors, visible in one waterfall.
03:05 — Mitigation, Not Root Cause
Mara cannot fix the processor at 03:00, so she stops the amplification instead: cap the retry budget to 1 and trip the circuit breaker so payment attempts fail fast instead of holding connections for 8 s. Checkout now degrades gracefully — "payment temporarily unavailable, retry shortly" — instead of hanging, and the processor stops receiving 4× load it cannot handle.
The status page gets its entry, and the incident channel gets the running timeline. The vendor ticket about the processor's own degradation can wait until Monday; every minute spent on diagnosis while the budget burns is a minute of user-facing failure the mitigation could have stopped.
03:18 — Verification and Close
The burn-rate panel on the SLO dashboard from Chapter 10 bends back under 1× — the budget stops draining, which is the only evidence of mitigation that counts. The p99 recovers on its own as the processor sheds the amplified load. Mara posts the resolution, resolves the status page entry, and is named postmortem owner before the channel goes quiet.
Total wall-clock time from page to verified resolution: 38 minutes. Each hop — alert, dashboard, logs, trace — was a purpose-built tool answering exactly one question: is a human needed, where is it, what happened, why. None of the four could have carried the incident alone.
- Starting the investigation by ssh-ing into
app-01and tailing raw container logs — the Chapter 1 reflex; the dashboard localizes the problem in 4 minutes precisely so that grep-across-hosts never happens. - Restarting the
paymentscontainer as the first move — the classic cause-guess; the service is healthy, its dependency is slow, and a restart destroys the evidence while fixing nothing. - Hunting for the root cause before mitigating — the processor's vendor ticket can wait until Monday; every minute spent diagnosing while the budget burns is a minute of user-facing failure the mitigation could have stopped.
- Trusting the retry default because it never hurt before — 3 retries is invisible at 300 ms and catastrophic at 8 s; retry policies are load multipliers on exactly the days dependencies are slowest.
- Declaring victory off the p99 graph alone — latency recovering does not prove users are transacting; the SLO burn-down returning under 1× is the verification that maps to the promise made in Chapter 10.
- Follow the alert → dashboard → logs → trace ladder in order — each hop narrows scope (something is wrong → where → what happened → why), and skipping levels means guessing.
- Wire trace IDs into structured logs before you need them, so the log-to-trace hop during an incident is a click instead of a timestamp hunt.
- Mitigate first and diagnose second: cap retries, trip the circuit breaker, shed load — anything that stops the burn — and leave root cause to daylight.
- Verify every mitigation against the SLO burn-rate panel, because "the graph I was staring at improved" and "the promise to users is being kept again" are different claims.
Knowledge Check
What does each hop of the alert → dashboard → logs → trace ladder contribute?
- Each hop independently confirms the same diagnosis for certainty
- Progressive narrowing: something is wrong → where → what → why
- The hops are ordered from cheapest tool to most expensive tool
- Each hop escalates the incident to a higher severity level
How did a 3-retry client policy turn a slow processor into an outage?
- The retries inflated the p99 latency statistic on the dashboard without affecting any real users
- The retry spans overwhelmed Tempo and hid the real trace data
- Each timed-out call became 4 calls, so the struggling processor got 4× the load
- The retries exhausted the Redis connection pool from Chapter 8 again
Why did Mara cap retries and trip the circuit breaker instead of diagnosing the processor at 03:05?
- Mitigation stops the burn now; root cause is daylight work she couldn't fix anyway
- Because the root cause was impossible to determine during the incident
- Circuit breakers hide failures from users until the vendor recovers
- The runbook forbids any diagnosis before a mitigation is applied
The payments p99 graph recovered. Why is that not the verification that closes the incident?
- Percentile graphs are statistically unreliable during incidents
- Only the burn-rate panel proves the budget stopped draining — the promise itself
- The 15 s scrape interval makes the p99 panel too stale to trust
- An incident cannot close until the postmortem is written
What made the 38-minute resolution possible, compared with the Chapter 1 version of Harborline?
- Mara's debugging skills improved over the course of the book
- The failure happened to be a simple one with an obvious cause
- Having more engineers awake and responding at 02:40
- The Chapters 3–10 stack: each hop was a purpose-built tool answering one question
You got correct