Chapter Two · Metrics Fundamentals

Metrics Fundamentals

Nothing gets installed in this chapter — it exists to make the words precise before the tools arrive. Time series, labels, the four instrument types, pull versus push, the USE and RED checklists, and the statistics of percentiles: five topics that turn 'checkout sometimes takes forever' into a question a database can answer. Every design decision in Chapters 3 through 10 traces back to a definition made here.

5 topics

Harborline is still flying blind. The Saturday complaint is still just a complaint — no number attached, no way to confirm it, no way to say whether last weekend was worse than the one before. What Mara is missing is not a tool; it is vocabulary: what a metric actually is, why one label can multiply storage costs a thousandfold, why a summary recorded today forecloses a question asked next month, and why the average — the one statistic everyone reaches for first — is precisely the one that hides a slow checkout.

By the last topic the complaint has become a measurable question: is the p95 of harborline_checkout_duration_seconds above 500 ms on Saturday mornings? That question has a yes/no answer — one that ssh, htop, and docker logs on app-01 and app-02 structurally cannot produce, no matter how many terminals are open. It stays unanswered here; Chapter 3 starts building the stack that answers it.

The four instrument types — the choice fixes which questions the data can answer

Counter

Only ever increases and resets to 0 on restart. Query it as a per-second rate(), never the raw value.

Gauge

Goes up and down. Read the current value directly — queue depth, pool usage, connections in use.

Histogram

Counts observations into buckets; percentiles are computed at query time and buckets aggregate across instances.

Summary

Computes quantiles inside one process — precise there, but impossible to merge across the fleet.

Topics in This Chapter

Topic 06

What a Metric Is

A metric is a named, labeled number sampled over time, and every unique name-plus-label combination is its own series. That identity rule decides what you can query, what you can aggregate, and what one careless user_id label does to your storage bill.

Time Series

Topic 07

Metric Types

Counter, gauge, histogram, summary — the choice fixes at instrumentation time which questions the data can answer forever after. The checkout SLO is a fleet-wide p95, and that single fact dictates a histogram before a line of code is written.

Types

Topic 08

Pull vs Push

Prometheus scrapes targets it knows about; push systems receive whatever senders transmit. The difference looks like plumbing and decides who controls cadence, how a dead service is detected, and why the up metric comes free in one model and must be hand-built in the other.

Collection

Topic 09

USE and RED

USE interrogates every resource — utilization, saturation, errors; RED interrogates every request-driven service — rate, errors, duration. Two checklists cover the whole Harborline fleet with a dozen questions instead of a thousand series, and set up the rule the book alerts by: page on RED, diagnose with USE.

Methods

Topic 10

Percentiles over Averages

Ninety-nine requests at 100 ms plus one at 5 seconds averages to 149 ms — a number describing none of them. Percentiles report latency honestly, and their one inconvenience — they cannot be averaged — dictates how latency must be recorded all the way down.

Percentiles