Topic 06

What a Metric Is

Metrics

A metric is a numeric measurement attached to a name and a set of key-value labels, sampled repeatedly over time. harborline_bookings_total{job="bookings",env="prod",status="confirmed"} is one; so is the CPU-idle percentage that node_exporter reports for app-01. Each unique combination of metric name and label set is one time series — a stream of timestamped values that grows by one sample every scrape.

That identity rule is the single most consequential fact in this chapter. It determines what you can query, what you can aggregate across, and what your storage bill looks like. Everything else about metrics — types, collection, dashboards, alerts — sits on top of it.

The Time Series

A series is an append-only stream of samples, and each sample is nothing more than a timestamp plus a 64-bit float. At Harborline's 15-second scrape interval, one series accumulates 5,760 samples a day. Prometheus's TSDB compresses consecutive samples to under 2 bytes each, so that full day costs roughly 11 KB on disk.

This is why keeping thousands of series costs almost nothing, and why a metrics stack can afford to record CPU, memory, disk, and network for every host around the clock. The economics only break when the series count explodes — not when any individual series gets old.

Series Identity: Name Plus Label Set

harborline_bookings_total{status="confirmed"} and the same name with status="cancelled" are two entirely separate series. They share nothing but a naming resemblance: separate sample streams, separate storage, separate entries in the TSDB index. Adding, removing, or changing any label value never annotates an existing series — it mints a new one. When the bookings container restarts with a new instance label value, every metric it exposes starts a fresh set of series, and the old ones simply stop receiving samples.

Series identity — name plus label set is one series, and a series is a stream of samples

Metric name

harborline_bookings_total

+ Label set

{status="confirmed"}

change any value → a brand-new series

= One time series

this exact name and label set, on its own

Samples

timestamp + 64-bit float, one appended per scrape

Labels as Query Dimensions

Labels exist to be filtered and aggregated. A selector like {job="payments"} isolates one service out of the fleet; sum by (status) collapses every other dimension and answers "confirmed versus cancelled bookings, per second" in one expression. That is the entire payoff of the label model, and it defines the test for adding one: a label you will never group or filter by in a real query is pure cost with no query it ever improves.

The Cardinality Multiplication

Total series count is the product of label value counts, not the sum. Five services times two hosts times four statuses is 40 series — harmless. The danger is that one unbounded label multiplies everything behind it: put a value with 10,000 possibilities on that same metric and 40 series becomes 400,000.

This product is cardinality, the first of the three costs that rule the whole book — cardinality, sampling, retention. Every label decision in every chapter that follows is a position taken on it. Prometheus holds the index of active series in memory, so cardinality is not an abstract accounting concern: it is the number that decides whether the TSDB on obs-01 runs comfortably or falls over.

A Metric Is Not a Log Line with a Number in It

A metric is pre-aggregated at the source. The counter behind harborline_bookings_total costs exactly the same at 10 requests per second as at 10,000, because the application only ever holds one number per label set and the scrape only ever transfers the current value. A log line costs per event, forever. The trade is brutal and permanent: the aggregation that made the metric cheap also destroyed every individual request. A metric can say checkout is slow; it can never say which checkout was slow, or for whom. That question belongs to logs and traces — Chapters 7 and 8 — and knowing the boundary is what stops you from torturing the wrong signal for answers it does not contain.

Common Mistakes

Putting user_id, an email address, or a request ID in a label — every distinct value mints a new series, so 200,000 customers detonate one well-meaning counter into 200,000 series and take the TSDB's memory with them. This is the canonical cardinality footgun, and it recurs through the entire book.
Expecting a metric to identify the slow request — the aggregation that made it cheap discarded the individual events, so hours spent slicing a latency metric for "which customer" produce nothing a log query would not have answered in seconds.
Assuming the graph shows everything that happened — at a 15-second scrape interval, a 5-second CPU spike that starts and ends between two scrapes never existed as far as the series is concerned. Resolution is bounded by collection cadence.
Encoding a dimension into the metric name — harborline_bookings_confirmed_total and harborline_bookings_cancelled_total can no longer be summed or ratioed in one query, which is exactly the job a status label exists to do.

Best Practices

Name metrics with the unit as a suffix and _total for counters — harborline_checkout_duration_seconds, harborline_bookings_total — following the Prometheus naming conventions, so no query ever has to guess whether a value is seconds or milliseconds.
Bound every label before adding it: enumerate its possible values, and if you cannot enumerate them, it does not belong in a label. status with 4 values, yes; booking_id, never.
Prefix application metrics with the product name (harborline_) so a single selector separates your own instrumentation from the hundreds of series every exporter emits.
Add a label only for dimensions you will group or filter by in a real query; any detail that fails that test belongs in a log line instead.

Comparable toolsGraphite hierarchical dot-paths (prod.bookings.confirmed.count) — the dimension-in-the-name model, with tags retrofitted laterInfluxDB splits key-value pairs into indexed tags and unindexed fieldsOpenTelemetry calls labels attributes — they map directly onto Prometheus labels at export time

Knowledge Check

How does Prometheus relate harborline_bookings_total{status="confirmed"} to the same metric name with status="cancelled"?

They are one series, with the status value stored as an annotation on each sample
They are two entirely separate time series
They are merged into one combined series at scrape time
The second one is rejected, because a metric name may carry only one label value

Why does a counter cost the same at 10 requests per second as at 10,000?

The TSDB compresses high-traffic counter series far more aggressively than quiet ones
Prometheus scrapes busy services less often to compensate
The application holds one running number, and the scrape transfers only that
Counters are sampled probabilistically, so most increments are dropped

A developer adds a user_id label to a request counter on a service with 200,000 registered customers. What happens?

Prometheus rejects the label because IDs are not allowed as label values
One series grows 200,000 times larger on disk
Old user values are evicted so only recent users keep series
The counter fans out into up to 200,000 separate series

A CPU spike lasts 5 seconds, starting and ending between two scrapes at Harborline's 15-second interval. What does the series show?

Nothing — the spike falls between samples and is never recorded
A smoothed version of the spike, reconstructed from neighboring samples
The full spike, because scrapes fire automatically when values change
The spike appears, but shifted 15 seconds later

You got correct