The Maturity Path

Observability is adopted in rungs, and the order matters more than the speed. The ladder a healthy adoption climbs — structured logs and host metrics, then service RED metrics and dashboards, then symptom alerting, then tracing, then SLOs, then SLO-driven operations — is not arbitrary, even though this book covered tracing (Chapter 8) before alerting (Chapter 9) for teaching reasons. Each rung produces the baseline the next rung needs, and each rung's payoff funds the instrumentation cost of the one above it.

Skipping rungs is how teams end up with traces nobody reads and SLOs nobody believes. The failure mode is always the same shape: a signal was bought before the signal underneath it existed, so there was nothing to compare it against and no habit of using it. This topic walks the ladder rung by rung and names what each one requires from the one below.

Rung 1: Structured Logs and Host Metrics

node_exporter on every host and JSON logs flowing into Loki — Chapters 3 and 7 — cost near-zero code changes and immediately replace ssh, htop, and docker logs. Harborline climbed this rung first because it covers every service at once, including the ones you cannot modify. It also establishes what "normal" looks like on app-01 and app-02, and every threshold decision made later leans on that record.

Rung 2: Service RED Metrics and Dashboards

Rate, errors, and duration per service (Chapter 5), plus one owned overview dashboard per service (Chapter 6). This is the first rung that requires touching application code, and it is affordable precisely because rung 1 already proved the stack's value — the incident that host metrics shortened is the argument that buys RED-instrumentation time from the bookings team's sprint.

Rung 3: Symptom Alerting

Alerts on the RED signals users actually feel — checkout latency, checkout error ratio — routed through Alertmanager with a runbook per page (Chapter 9). Alerting cannot come earlier, because a threshold is a claim about abnormality, and without rung 2's weeks of baseline data every threshold is a guess. Guessed thresholds fire wrong, and wrong-firing alerts become the muted channel from topic 70 within a quarter.

Rung 4: Tracing

OpenTelemetry across the request path (Chapter 8). Tracing is fourth because it carries the highest per-service instrumentation cost: every hop must propagate context before the first trace is whole, and a half-propagated rollout produces orphaned spans that look like broken tooling. Its value is also dependent — a trace localizes a problem that metrics have already detected. Detection before localization is the ladder's core dependency.

Harborline's payoff came exactly in that order: the Saturday checkout complaint from Chapter 1 was quantified by metrics in Chapter 5, and only then did the Chapter 8 trace pin it to the Redis connection-pool ceiling in bookings. The trace without the metric baseline would have been a waterfall of numbers with nothing to say whether 340 ms in a span is a problem or a Tuesday.

Rung 5: SLOs

The checkout SLO — 99.9% availability over 28 days, 95% of requests under 500 ms (Chapter 10) — needs months of RED history to set a defensible target. An error budget declared without baseline data is either trivially safe or instantly bankrupt, and either way the team stops believing it. The 500 ms number was defensible because rung 2 had been recording checkout latency for months before anyone negotiated it.

Rung 6: SLO-Driven Operations

Burn-rate paging, error budgets gating release pace, blameless postmortems feeding instrumentation gaps back into the backlog (Chapters 10 and 12). The top rung is organizational, not technical — no new software gets installed, and the change is that reliability decisions now cite budget numbers instead of feelings. This is the rung where the August incident's 38-minute resolution stops being luck and becomes process.