The Three Costs

Every design decision in observability — every label, every scrape interval, every line on a vendor's pricing page — is a position taken on three costs. Cardinality: how many distinct series or streams exist. Sampling: what fraction of events you keep. Retention: how long you keep them.

The three trade against each other under a finite budget, and this book returns to them in every chapter, because "why is this designed this way" almost always resolves to one of the three. This topic hands you the lens; the rest of the book looks through it.

Cardinality — The Cost of Distinctness

Every unique combination of label values is a separate time series, held in memory and indexed for as long as it exists. Take harborline_checkout_duration_seconds with an endpoint label (12 values), a status label (5 values), and a host label (2 values): 12 × 5 × 2 = 120 series. Completely fine. Now add a user_id label with 50,000 values and the same metric is 6 million series.

The operative word is multiplies. Labels never add to the series count; they multiply it, which is why one unbounded label detonates a metric that ten bounded labels left healthy. Cardinality is the dominant cost of metrics — the whole reason metrics are cheap is that many events collapse into few series, and an unbounded label un-collapses them.

Sampling — The Cost of Completeness

Logs and traces price per event, so at search's 300 requests per second on a summer Saturday, nobody stores every trace. Keeping 1% cuts the cost 100× and leaves the statistics honest — p95 computed over a uniform 1% sample tracks the true p95 closely at that volume. The catch: the naive version also discards 99% of the rare events, and the rare events are the failures you bought tracing to see. Deciding what to keep before you know how the request ends is the head-versus-tail sampling problem, and Chapter 8 spends a topic solving it.

Retention — The Cost of History

Prometheus keeps 15 days locally by default, at roughly 1–2 bytes per compressed sample, and for incident work 15 days is usually plenty. The 13-month question — "how does this August compare to last August?" — needs a long-term store (Thanos, Mimir) and almost always downsampling, because raw 15-second resolution held for a year is the most expensive possible answer to a question nobody asks at 15-second resolution. Capacity planning wants the daily p95, not 2 million samples per series per year.

The Triangle

The three costs convert into one another, because all three multiply into the same bill: bytes stored and indexed over time. You can afford millions of series for a day, or thousands of series for a year. Full traces for the 0.1% of requests that failed, or a 1% sample of everything. When the budget overruns — and it will; telemetry volume grows faster than traffic — the fix is always to cut one of the three, and the engineering skill is choosing which cut loses the least answering power. That skill, more than any tool knowledge, is what this book is trying to install.

Reading Any Tool Through the Costs

Once you hold the three costs, tool architectures stop being arbitrary. Loki indexes only labels and brute-forces the log text at query time — that is a cardinality decision: full-text indexing of every line is a cost Loki refuses to pay, and its speed and price follow from the refusal. Tempo stores whole traces cheaply in object storage because it assumes you already sampled upstream. Grafana Alloy exists partly so dropping and relabeling can happen before bytes cross the network.

Pricing pages read the same way. Datadog bills per custom metric and per ingested gigabyte — a cardinality-and-sampling bill with a logo on it. Grafana Adaptive Metrics and Datadog's Metrics without Limits are productized cardinality control: tooling that finds series nobody queries and aggregates them away. Every invoice in this industry is one of the three costs wearing a brand name.

The Recurring Question

From Chapter 3 on, each design choice gets the same interrogation: what does this label, this interval, this pipeline stage do to cardinality, sampling, and retention? Ask it about the 15-second scrape interval, about every label the bookings team proposes, about why Chapter 7 forbids request_path as a Loki label. The answers differ; the question never does.