Anti-Patterns
Every anti-pattern in this gallery is an observability deployment that technically works. The metrics scrape, the logs flow, the dashboards render — and when the pager fires, none of it answers anything. Each one is a position taken, usually by accident, on the three costs from Chapter 1: cardinality nobody budgeted, sampling nobody chose, retention nobody questioned.
The pattern behind all six is the same. Telemetry got added — a dashboard here, an alert there, ten INFO lines per request "for visibility" — and the decision it was supposed to inform never got named. Eighteen months in, what keeps Harborline's stack useful is not the five containers on obs-01 but the habit of asking, for every panel and every rule, what a human would do differently because of it.
Dashboard Sprawl
The org has 300 Grafana dashboards and the on-call uses 5. The other 295 were built for one incident, one demo, or one manager's question, and never deleted. Individually each was reasonable; collectively they mean that during a page the responder scrolls a graveyard looking for the one checkout overview that matters while p95 keeps climbing. The cost is not storage — dashboard JSON is tiny — it is attention at the worst possible moment.
Chapter 6's rule is the fix, enforced as a calendar event: one owned overview per service, folders by team, and a standing delete pass for anything unviewed in 90 days. A scheduled script over the Grafana HTTP API, cross-referenced against Loki and Prometheus query logs, shows which dashboards nobody opened, so the pass is a sort, not an investigation. Expect the first one to remove more than it keeps.
Alert Noise
The #alerts channel carries 400 messages a day and everyone has muted it. Each alert seemed reasonable when it was added — disk at 80%, a single 500, a container restart — and together they trained the team that alerts mean nothing. The burn-rate page from Chapter 12 arrived at 02:40 and was acted on in minutes; in a muted channel it would have scrolled past unread until customers started calling. Chapter 9's discipline is the cure: a page is a claim that a human must act now, everything else is a ticket or a dashboard panel, and any alert that fires twice without action gets rewritten or deleted.
Log-Everything-at-INFO
Every request through bookings logs 8–10 INFO lines "for visibility," so Loki ingests gigabytes a day of lines nobody has ever queried, and the one WARN that preceded the outage sits buried 40,000 lines deep in the same stream. The arithmetic is unforgiving: at search's 300 requests per second, a single line per request is already about 26 million lines a day. Chapter 7's contract holds here — INFO for state changes, WARN for survivable anomalies, ERROR for failed work, and request-scoped detail belongs in a span, not a log line.
Vanity Metrics
Dashboards of totals: requests served since launch, days of uptime, cumulative registered users. The graphs only go up, and nobody has ever changed anything because of them. The test for every panel is blunt — what would I do differently if this line moved? When the honest answer is nothing, the panel is decoration, and it taxes attention every time someone scans past it during an incident.
The Unwatched Stack
Observability deployed as a compliance checkbox. Prometheus runs on schedule, nobody wrote alert rules, the dashboards came from a community pack whose metric names do not match what the services emit, and the first real incident gets debugged over ssh anyway — the stack existed for the audit, not for 02:40. Chapters 9 and 10 compress to one sentence here: the stack exists to change what humans do during failure. A stack that pages nobody and informs no decision is spend, not observability.
Tool Sprawl
Three logging systems, because each team picked its own and nobody forced the question. Correlating one incident now means three query languages, three retention policies, and three bills — and the trace ID Chapter 8 threaded through the stack links into exactly one of them. The consolidation move is one system per signal with a named owner and a migration deadline for the stragglers. Chapter 13's platform survey exists so that choice gets made once, deliberately, instead of three times by default.
- Adding telemetry without naming the decision it informs — every dashboard, alert, and log line that answers "what would change if this fired?" with "nothing" is negative value: it spends cardinality and attention and pays nothing back.
- Copying dashboards and alert packs from a community repo without adapting thresholds and metric names — Harborline's 500 ms checkout target means nothing for a batch pipeline, and unadapted defaults are how the muted channel is born.
- Treating observability as a project with an end date — the stack declared finished in Q1 is the unwatched stack by Q3, because the services changed and nobody owned keeping the signals true.
- Only ever adding, never deleting — dashboards, alerts, and log statements accumulate monotonically unless deletion is a scheduled activity, and every anti-pattern on this page is an accumulation problem.
- Measuring maturity by tool count — three logging systems is not three times the observability, it is one third of it. The real metric is time-to-resolution during an incident, which Chapter 12 clocked at 38 minutes.
- Run a quarterly deletion pass driven by a script over the Grafana HTTP API and your datasource query logs — delete or archive anything unviewed in 90 days, and expect to remove more than you keep.
- Enforce Chapter 9's action rule on every alert: it pages a human who acts now, or it becomes a ticket or a panel. Audit Alertmanager's notification history monthly for alerts that fired with no action taken.
- Assign a named owner to every dashboard and alert rule — ownerless telemetry is dead telemetry, and the owner field is where the next deletion pass starts.
- Consolidate to one system per signal — one metrics store, one log store, one trace store — and treat any second system for the same signal as technical debt with a retirement date.
Knowledge Check
Why are 300 Grafana dashboards worse than 30, when storage for dashboard JSON is nearly free?
- Grafana slows down measurably once it stores that many dashboards
- The responder pays the cost in search time during a page
- Each dashboard adds standing query load on Prometheus
- Licensing costs scale with the dashboard count
The #alerts channel gets 400 messages a day. What is the actual damage?
- Alertmanager cannot reliably deliver at that volume
- Chat retention costs grow until the channel must be purged
- The team learns alerts are ignorable, so the real page is missed
- The alerts arrive ungrouped, which routing configuration would fix
Where does per-request detail — arguments, timings, downstream calls — belong, and why not in INFO logs?
- In INFO logs, because Loki makes high-volume ingest cheap enough
- On metrics, using a request-ID label so each request is queryable
- In a trace span, where sampling controls the cost
- At DEBUG level in production, filtered at query time
What makes a metric a vanity metric?
- It is monotonically increasing, like any counter
- No decision changes when it moves
- It measures a business quantity instead of a technical one
- It is stored outside Prometheus, so it cannot be alerted on
You got correct