What to Instrument

A team handed a metrics library instruments what is easy: function call counts, cache hits, whatever the last incident touched. Six months later there are 200 metrics and still no way to say whether checkout is healthy. The Google SRE book's four golden signals are the antidote — latency, traffic, errors, saturation, measured at the service boundary. Four well-placed metrics on the checkout path answer more questions than 200 scattered ones.

This topic turns the instruments from topic 22 into a plan: which metrics the bookings service actually needs, in what order, and what each one will be asked during an incident.

The Four Signals on the Checkout Path

Latency is harborline_checkout_duration_seconds, already declared. Traffic is a counter, harborline_http_requests_total, read with the Chapter 4 pattern rate(harborline_http_requests_total[5m]) — now pointed at a metric the team owns instead of a host's. Errors are the 5xx fraction of that same counter family, one PromQL division away, which is why errors and traffic must live in the same metric.

Saturation is how full the limiting resources are: gunicorn workers busy, Postgres connections in use against db-01, Redis connections in use against cache-01. It is the least glamorous of the four and the one this chapter keeps circling back to.

Latency Honestly

A latency histogram lies unless failed requests are distinguishable from successful ones, so the histogram carries a status label. The failure mode is concrete: payments goes down, every checkout returns 500 in 20 ms, and a histogram that mixes them — or worse, records only successes — shows p95 improving while the service burns. Fast failures are the cheapest requests a service ever serves.

Bucket edges sit around the threshold that becomes the SLO in Chapter 10: 95% of checkouts under 500 ms. A histogram is blind between its edges, so precision is spent where the SLO question will be asked, not spread evenly from 5 ms to 10 s.

Boundaries First

One middleware on inbound HTTP instruments every route in a single move — latency, traffic, and errors for the whole service before anyone writes a bespoke metric. The route label comes from the framework's route template, never the raw path, for reasons topic 24 prices out in full.

@app.middleware("http")
async def observe_request(request, call_next):
    start = time.perf_counter()
    response = await call_next(request)
    route = request.scope["route"].path  # "/bookings/{id}"
    labels = (route, request.method, response.status_code)
    REQUESTS.labels(*labels).inc()
    DURATION.labels(*labels).observe(time.perf_counter() - start)
    return response

The middleware clocks the whole request, including time spent before the handler runs — exactly the component that grows when workers saturate. The same move repeats on the three outbound edges: one wrapper each around HTTP calls to payments, queries to db-01, and cache operations against cache-01, with the target as a bounded label. Interior functions wait; a boundary metric will point at the interior worth instrumenting when the time comes.

Saturation, the Signal Everyone Skips

For every bounded pool, two Gauges: connections in use beside the pool maximum, for both Postgres and Redis. Saturation is the only leading indicator among the four — latency and errors report damage already done, while a pool at 96% predicts it. Teams skip these gauges because nothing is on fire when they are written, which is precisely when they can still be written calmly.

The Saturday complaint has survived four chapters because nothing measured this. Host graphs looked healthy in Chapter 3 and PromQL got sharper in Chapter 4, but if the misbehaving resource lives inside bookings itself, it is invisible to every exporter. These pool gauges are the first instrumentation that can see inside the service at all — the proof has to wait for Chapter 8's traces.

Business Counters Ride Along

harborline_bookings_total and a counter of payment failures cost two lines each and answer the questions the business asks first. "Bookings per minute dropped 40% at 09:47" lands harder in an incident channel than any CPU graph, and it is the number the postmortem's impact section needs anyway. Add them the day the service is instrumented, not the day someone asks.