Topic 24

Label Discipline

Cardinality

Every distinct combination of label values on a metric is a separate time series that Prometheus indexes, holds in head memory, and walks on every query that touches the metric. Labels are not decoration — they are the pricing model. Chapter 1 named cardinality the first of the three costs, and almost all of that cost is set in one place: the moment someone types a label name into a metric declaration.

The bookings team can now mint metrics, which means the bookings team can now mint outages. One thoughtless label on one histogram can take Prometheus down harder than any traffic spike Harborline will ever see, so Mara spends a full day of the pairing week on nothing but labels. The whole discipline reduces to one multiplication and one test.

Series Arithmetic

The series count of a metric is the product of its labels' value counts. A histogram multiplies once more: every label combination carries one series per bucket, plus _sum and _count. Nothing else about a metric's cost matters as much as this product — and it grows multiplicatively while every intuition about it stays additive.

Run it for the middleware's all-routes histogram, harborline_http_request_duration_seconds. A route label with 12 route templates × a method label with 3 values × a status label with 8 codes is 288 combinations; ten buckets plus _sum and _count make 12 series per combination, so 3,456 series. Scraped from both app-01 and app-02, that doubles to roughly 7,000 series for one well-behaved histogram. This is the multiplication the team repeats, on paper, for every metric it ever ships.

The user_id Outage

Now put user_id on that histogram. The existing 7,000-series family times 100,000 users is roughly 700 million series from a single metric family. At a few kilobytes of head memory per series, that is terabytes of RAM for one metric: Prometheus on obs-01 either OOMs outright or every query slows to a crawl while it labors under an index built for a hundred thousand times fewer series.

The trap is that it never stops. New users keep arriving, so the series set grows without bound; the memory graph climbs for weeks and then Prometheus dies on a Tuesday because of a label added in June. An unbounded label is not a performance problem — it is an outage on a delay timer, and booking_id is the same bomb with a faster fuse.

Series count is a product, not a sum — one label decides

Bounded — the histogram that ships

route (12 templates) × method (3) × status (8) = 288 combinations. Each combination is 12 series (10 buckets + _sum + _count) = 3,456. Scraped from app-01 and app-02 → roughly 7,000 series.

+ user_id — the outage

Add one user_id label: the existing 7,000 series × 100,000 users = roughly 700 million series from a single metric family. Terabytes of head memory, growing without bound — Prometheus OOMs on obs-01.

Bounded Values Only

The review test is one question: can the label's full value set be written down before deploy? Route templates pass — /bookings/{id}, never the raw path /bookings/8843. HTTP methods, status codes, and dependency names (payments, db, cache) pass. Anything minted per user, per request, or per entity automatically fails: IDs, emails, session tokens, raw error strings.

Estimating Before Shipping

The pre-deploy ritual is section one's multiplication on paper, checked against a written budget — Harborline caps any single metric family at 10,000 series. A proposal that lands above the cap redesigns its labels or does not ship. The arithmetic takes ninety seconds in code review and replaces a 3 a.m. OOM.

The post-deploy check is live. prometheus_tsdb_head_series reports the total series count in head memory, count({__name__=~"harborline_.*"}) prices everything under the prefix — narrow the regex to price one family at a time — and Prometheus's TSDB status page keeps a top-10 cardinality table that names offenders directly. Together they catch whatever the paper estimate missed.

Where High Cardinality Belongs

The questions an ID label was trying to answer are real; they are just not metric questions. "What happened to booking 8843?" is a per-event question, and per-event detail is what logs (Chapter 7) and traces (Chapter 8) are built for. The booking id goes into the log line and the span attribute, where one more distinct value costs bytes on disk, not a permanent series in memory.

What metrics keep is the aggregate; what the aggregate loses is the individual. Exemplars — the next topic — bridge the two, attaching a single traceable specimen to a histogram bucket without ever minting a series per user. Labels stay reserved for the dimensions a human would put on a dashboard axis.

Common Mistakes

Labeling by raw URL path instead of route template — every distinct booking id mints a fresh series set, which is the user_id explosion wearing a different name.
Putting the exception message in a label value — error strings embed IDs, addresses, and timestamps, so each occurrence is a new series, and the "errors by type" panel becomes unreadable while cardinality climbs.
Adding one more label to an existing histogram without redoing the multiplication — a 6-value label on a 7,000-series family is 42,000 series in a single deploy, and nobody notices until Prometheus's memory graph does.
Copying labels from request headers like user-agent or referer — attacker-controlled and effectively infinite; a bot with a randomized user-agent can DoS the monitoring system without ever touching the application.
Shipping first and checking cardinality never — by the time queries slow down, weeks of bloated blocks are already on disk, and the cleanup means deleting series, not just fixing code.

Best Practices

Use route templates from the framework's router as the route label, never the raw request path.
Multiply out the series count for every new metric and every new label in code review, against a written per-family budget — 10,000 series is Harborline's line.
Watch prometheus_tsdb_head_series on the obs-01 dashboard and alert on its growth rate, so an explosion is caught in hours instead of at OOM.
Route per-entity questions to logs and traces, and reserve labels for the dimensions a human would put on a dashboard axis.

Comparable toolsScale-out Mimir / VictoriaMetrics — per-tenant series limits reject the explosion at ingest instead of dying from itSaaS Datadog — bills per distinct custom-metric series, turning this topic into literal invoice linesStandard OpenTelemetry metric views — drop or re-aggregate attributes at the pipeline edge, a second chance the Prometheus client does not offer

Knowledge Check

A histogram has a route label with 10 templates and a method label with 4 values, uses 8 buckets plus _sum and _count, and is scraped from two hosts. How many series?

400
800
80, because labels add up: 10 + 4 + 8 + _sum + _count, doubled for two hosts
40

Why is a user_id label an outage rather than merely a slow query?

Prometheus rejects the scrape once a label exceeds its allowed value count
The series set grows without bound, and head memory grows with it until the process dies
User ids are long strings, and long label values overflow the sample encoding
The client library slows the application down as the label's children accumulate

Which proposed label passes the bounded-set test?

The raw request path, like /bookings/8843
The exception message, so errors can be grouped by cause
The name of the outbound dependency being called
The session token, to trace a user's journey across requests

A support ticket asks what happened to booking 8843. Which signal answers, and why not metrics?

Metrics, by querying the histogram filtered to the booking's route template
Metrics, but only if the team switches the instrument to a Gauge
Logs and traces, which carry the booking id per event
Exemplars, since each one carries an id that can be searched for the booking

You got correct