Chapter Thirteen · Observability From the Inside
Observability From the Inside
On the second on-sale night the answer to "why are the ticket emails late" took 4 minutes instead of 40, because the service reported it: one JSON line per event with the request id on every line, a histogram per endpoint instead of an average, a gauge for the queue's age, and a trace that followed one buyer's checkout from the load balancer through Postgres and Redis into the worker's render. Four topics are the instrument in the code, what to emit, in what shape, with which ids, and the fourth turns the numbers into a promise with an alert on it.
Every chapter so far has ended a mechanism with a sentence that began "Chapter 13 puts this on the dashboard": the pool's acquire wait from Chapter 6, the breaker's state and the outbox's age from Chapter 7, the queue's age per kind from Chapter 8, the cache hit ratio from Chapter 9, the reconciliation's last success from Chapter 10, the loop lag and the memory from Chapter 11. This chapter is where those promises are kept, and the rule for keeping them is short: the person who built the mechanism chooses the number that shows it working, and the service exposes that number itself. Nothing here is a tool bolted on afterwards. The log line is written by the code that did the thing, the counter is incremented by the code that knows the event happened, the span is started by the library that made the call.
The chapter is three signals and one promise. Structured logs are one JSON object per event, with a stable name, typed fields, the request id and the trace id stamped by the logger from the context of Chapter 4, a redaction processor that no call site can bypass, and sampling that keeps every error and 1 in 10 successes on a night that writes 3,000 lines a second. Metrics are Rate, Errors and Duration per route template, with duration as a histogram whose buckets include the 800 milliseconds the SLO is stated on, the gauges behind them that name a cause, business counters from the domain layer, and one scraped endpoint. Traces are one request's path as a tree of spans across two hosts and four minutes, propagated by one header into the Payrail call and the job payload, produced almost entirely by four instrumentation libraries, and kept at the tail for the errored and the slow. The SLO is where the numbers become a decision: an indicator from the histogram and the counter, an objective agreed with the product's owner, an error budget of 4,320 bad checkouts a month, and two burn-rate alerts that stay quiet through a blip and page within 5 minutes of a broken deploy.
No new wound opens here. The third one, the emails 40 minutes late, was seen at 4 minutes on the second night because Chapter 8's age gauge was on a dashboard somebody watched; this chapter is why every other mechanism in the book is watched the same way. The collector that scrapes the endpoint, the store that keeps the traces, the dashboards and the paging are Observability Deep Dive's subject, and this book hands them over by name at the end of every topic. Chapter 14 is where the numbers from here and from the load test of Chapter 12 turn into a plan for the night.