Monitoring vs Observability
Monitoring answers questions you wrote down in advance: is the disk over 90%, is the process running, does the health check return 200. Observability is the ability to answer questions you didn't write down — to interrogate a live system about a failure nobody predicted, using telemetry it already emitted, without shipping new code and waiting for the failure to come back.
The distinction sounds like vendor wordplay, and vendors have done their best to make it one. It is actually the difference between catching last year's outage again and diagnosing this Saturday's new one. This book builds an observability stack from bare metal to SLOs, and every layer of it exists to widen the set of questions you can answer after the fact — so the definition has to be nailed down first.
Known Unknowns
Monitoring is a checklist of predicted failure modes. Each check — disk full, host down, queue depth over 10,000 — exists because someone anticipated that exact failure, usually after living through it once. The checklist grows one incident at a time: an outage happens, a postmortem action item adds a check, and the next identical outage gets caught. That loop is genuinely useful, and nothing in this book asks you to abandon it. Its limit is structural: the checklist is always exactly one incident behind reality.
Unknown Unknowns
The failures that matter in a distributed system are novel combinations. A connection pool that saturates only under load. A retry storm that amplifies a slow third party into a full outage. Two services deployed an hour apart that disagree about a timeout. No pre-written check fires for any of these, because nobody wrote one — and by definition nobody could have. You cannot enumerate the failure modes of five services, three data stores, and a payment processor in advance; the combinatorics are against you. What you can do is collect telemetry rich enough to reconstruct what happened once it does.
The New-Question Test
Here is the working definition this book uses. Can you ask the system a question you did not anticipate — "which endpoint got slower, for which requests, starting when?" — and get an answer from data already collected, without deploying code and waiting for the failure to recur? If yes, the system is observable. If the honest answer is "we'd have to add some logging and hope it happens again," you are monitoring.
The test has teeth because both conditions bite. "Data already collected" rules out the instrument-after-the-incident reflex. "A question you did not anticipate" rules out the dashboard that answers exactly the twelve questions someone thought of last quarter and nothing else. Apply the test to your own stack with a real incident from last quarter and the result is usually uncomfortable — which is the point.
Why "We Have Zabbix" Falls Short
Check-based host monitoring reports whether machines are healthy, not whether checkouts complete. At Harborline — the ferry-booking service this book instruments, introduced properly in topic 05 — every host check is green: CPU at 40%, disk at 60%, all five containers up. Meanwhile a customer waits 9 seconds for checkout to finish, because nothing anywhere measures the transaction the customer actually cares about. The hosts are fine. The business is not. A wall of green host checks and a broken product coexist without contradiction, because they measure different things.
Instrumentation Is the Price of Admission
The ability to answer new questions is bought at emit time, not query time. Telemetry must carry its dimensions — endpoint, status code, method, host — when the code produces it, because no backend on earth can slice by a label that was never recorded. A $500,000 analytics platform pointed at telemetry that says only "request handled" can tell you how many requests were handled, and nothing else. This is why Chapter 5 of this book is about editing application code: the raw material of every future answer is created there, and nowhere downstream.
A Property, Not a Product
Vendors sell "observability platforms," and some of them are good. But observability is a property of the system, the way security is — you cannot buy it, only build toward it, and the measure is operational: how fast does a novel question get answered during a real incident? Ten minutes means you have it. Three days and a code deploy means you don't, whatever the invoice says.
Monitoring evaluates predefined checks against thresholds and tells you that a known condition occurred — cheap, essential, and blind to anything off its list. Keep it as the floor, especially for hardware and capacity.
Observability means the telemetry is rich enough to answer unanticipated questions during a novel failure. Invest here when the system is distributed enough that most incidents are new. They stack: an observable system is still monitored; a monitored system is not automatically observable.
- Treating a wall of green host checks as coverage — CPU, disk, and ping all pass while the checkout error ratio sits at 4%, because no check measures the user's transaction. The customer becomes the alerting system.
- Responding to every incident by adding one more threshold check — the new check catches an exact repeat and nothing else, and after two years the config holds 400 checks nobody can explain while novel failures still arrive unannounced.
- Buying an "observability platform" and pointing it at unstructured logs and host metrics — the tool cannot slice by endpoint or customer tier because the telemetry never carried those fields. The invoice changed; the answerable questions didn't.
- Equating an uptime ping with health —
webreturns 200 on/whilepaymentstimes out on every charge. Synthetic reachability of one path says nothing about the other twelve. - Skipping instrumentation because "we'll add it when we need it" — the need arrives mid-incident, when adding instrumentation means deploying code and waiting for the failure to happen again.
- Apply the new-question test before buying anything: pick last quarter's worst incident and check whether today's telemetry could answer its key question in under 10 minutes.
- Instrument the business transaction first — checkout latency and error ratio at Harborline — and treat host metrics as the floor underneath it, not the goal.
- Keep the existing check-based monitoring running while you build observability. Disks still fill and hosts still die, and a Zabbix-style check catches those fine.
- Emit dimensions at instrumentation time — endpoint, status code, service — even before anyone asks for them. Labels are the raw material every future question is made of.
Knowledge Check
A team responds to every incident by adding a new threshold check. Why does this never catch the next novel failure?
- Each added check slows the monitoring system until it misses events
- Each check catches only the exact failure it was written for
- Too many checks cause alert fatigue, so engineers ignore the real one
- Threshold values drift out of date as traffic grows
What exactly does the new-question test require, beyond the question being unanticipated?
- That the answer appears on a pre-built dashboard
- That the failure can be reproduced on demand so new logging can capture it
- That the answer comes from data already collected, with no new code
- That the team uses a tool marketed as an observability platform
Every host check at Harborline is green while a customer waits 9 seconds for checkout. What explains the contradiction?
- The checks are misconfigured and reporting stale data
- Nothing measures the checkout transaction itself
- The check interval is too long to catch a 9-second delay
- The CPU metric is averaged in a way that hides the spike
Where is the ability to answer new questions actually paid for?
- At query time, by buying a faster analytics backend
- At storage time, by retaining raw events longer
- At visualization time, by building richer dashboards that add the missing fields
- At emit time, when the code attaches dimensions to telemetry
You got correct