Topic 69

Cost Engineering

Costs

Harborline's traffic grew 3× since Chapter 1; the telemetry grew closer to 10×, and obs-01's disk noticed before anyone else did. That ratio is not an accident or a misconfiguration — telemetry volume is roughly traffic × services × cardinality, and all three multiply. Every new service brings new series, new log streams, and new labels on everything that touches it.

Cost engineering is applying the book's three costs — cardinality, sampling, retention — as deliberate levers, per signal, before the bill or the disk applies them for you. Chapter 1 introduced the three as vocabulary; this topic operates them as controls, and closes with the stack pointed at itself.

Superlinear Growth, Mechanically

Follow one request through the arithmetic. In Chapter 1 it produced 1 log line and a handful of metric samples; now it crosses 5 services and produces 5 log lines, 25 samples, and a trace. Then multiply by labels: add one label with 20 values to a histogram whose 12 series are ten buckets plus _sum and _count, and the series count multiplies by 20 across all 12. Cost grows along three axes at once while traffic grows along one, which is why "we grew 3×" and "the disk grew 10×" describe the same quarter.

Metric Levers: Cardinality and Recording Rules

The hunt starts with a query: topk(10, count by (__name__)({__name__=~".+"})) ranks metrics by series count, and the offenders are rarely a surprise — histograms with a label that should never have existed. Drop the label at relabel time, before it becomes series; a label discarded at the scrape edge costs nothing downstream, while the same label dropped at query time already paid for storage and memory.

The retention lever pairs with recording rules. Dashboards only ever query a dozen aggregates — the p95, the error ratio, the per-service rates — and Chapter 4's recording rules store those cheaply as one series each. That is what makes short raw retention survivable: keep 15–30 days of full-resolution samples for incident forensics, and let the recording rules carry the year-long view at a fraction of the storage.

Log Levers: Levels, Sampling, and Tiers

Level discipline is the cheapest lever in the entire stack: DEBUG in production is paying to store noise, and turning it off is a config change. Next comes sampling the repetitive — 1 in 100 health-check log lines still proves liveness, and nobody has ever read the other 99. Then retention tiers per stream, which Loki supports natively: payments logs keep 90 days because audit questions arrive months late, while search cache-hit chatter keeps 7 because no question about it survives a week.

Trace Levers: Tail Sampling

The Chapter 8 sampling decision returns wearing a budget hat. Tail sampling at the Collector keeps every error trace and every trace over the 500 ms SLO threshold, plus 1–5% of the boring rest — preserving exactly the traces worth money while cutting Tempo ingest by 10–20×. The alternative, cutting cost with 1% head sampling, decides before the outcome is known and throws away errors at the same rate as noise: the Chapter 8 footgun, recommitted for budget reasons.

Three signals, three levers — cut cost at the cheapest point in the pipeline

Metrics — cardinality

Drop high-cardinality labels at relabel time, before they become series, and let recording rules carry the long view so raw retention can stay short (15-30 days).

Logs — levels, sampling, retention

Turn off DEBUG in production, sample the repetitive (1 in 100 health checks), and set per-stream tiers — payments 90 days, search chatter 7.

Traces — tail sampling

Sample at the Collector after the outcome is known: keep every error and every trace over the SLO threshold, plus 1-5% of the rest, cutting ingest 10-20×.

The Stack's Own Footprint

The stack must measure itself with the same rigor it applies to bookings. Prometheus exposes prometheus_tsdb_head_series for live series count; Loki reports per-stream ingest bytes; Tempo counts spans per second; and predict_linear over obs-01's disk turns "when do we go blind" into a dated forecast. The forecast belongs in an alert, because the observability host filling up is the one outage the tools for seeing outages cannot report.

# a dated forecast, not a 02:00 surprise: disk full within 14 days
- alert: ObsDiskFullIn14Days
  expr: |
    predict_linear(node_filesystem_avail_bytes{
      instance="obs-01", mountpoint="/data"}[7d], 14 * 86400) < 0
  for: 6h
  labels:
    severity: ticket

The Observe-the-Observability Dashboard

The levers converge on one Grafana dashboard: head series with a marker at the last capacity review, log GB per day per service, trace ingest rate, disk days remaining, and week-over-week deltas on all of them. Mara checks it after every deploy — not on a schedule — because cardinality regressions ship inside application changes, not observability changes. The customer_id label that detonates the series count arrives in a feature branch that never mentions monitoring.

Common Mistakes

A deploy adds customer_id as a label on harborline_bookings_total and nobody reviews it — series count goes from hundreds to tens of thousands overnight, Prometheus memory follows series count, and the observability host becomes the incident.
Keeping raw 15-second samples for a year "just in case" — the year-old questions anyone actually asks are answerable from a handful of recording-rule aggregates at roughly 1/100th the storage.
Cutting trace cost with 1% head sampling instead of tail sampling — the decision is made before the outcome is known, so errors and slow requests are discarded at the same rate as the noise.
Flipping production to DEBUG during an incident and never reverting — Loki ingest triples, the retention tiers fill early, and the cost shows up weeks after the incident that caused it.
Running no forecast alert on obs-01's disk — when the observability host fills, Prometheus and Loki stop ingesting, and the outage arrives exactly where the tools for seeing outages live.

Best Practices

Review cardinality on every deploy — alert on a step change in prometheus_tsdb_head_series, say +20% week-over-week, so a new label is caught in hours instead of on the invoice.
Set retention per signal deliberately: short full-resolution metric retention backed by recording rules for the long view, per-stream log tiers in Loki matched to each stream's audit value, and tail-sampled traces sized to the questions you actually replay.
Sample at the cheapest point in the pipeline — drop labels at relabel time, sample logs at the Alloy agent, tail-sample traces at the Collector — because telemetry discarded early costs nothing downstream.
Build the observe-the-observability dashboard and put a dated predict_linear disk alert behind it, so the stack's own capacity is a ticket with a deadline instead of a 02:00 page.

Comparable toolsProducts Chronosphere and Grafana Adaptive Metrics/Logs — cardinality and volume control sold as a productStorage VictoriaMetrics — competes partly on per-series storage efficiencyControl point OpenTelemetry Collector processors — filter and tail sampling in open sourceDiscipline Usage dashboards in Datadog / New Relic serve the same role — no direct equivalent for the practice itself

Knowledge Check

Traffic grew 3× but telemetry grew 10×. What explains the gap?

A telemetry leak somewhere in the collection pipeline is silently duplicating data
Volume multiplies along services and cardinality, not just traffic
The scrape interval was accidentally shortened during growth
Storage compression degrades as data volume rises

Why is dropping a label at relabel time better than dropping it at query time?

Discarded at the edge, the label never becomes series or storage
Query-time aggregation produces mathematically wrong results in the panels
Relabeling is the only place labels can be removed at all
Relabel rules run faster than PromQL aggregations

What makes 15–30 days of raw metric retention survivable?

Incidents never require data older than a few days
Prometheus automatically migrates older samples to cheaper archival storage
Recording rules carry the long view at a fraction of the storage
Regulations forbid keeping metrics longer than 30 days

As a cost measure, what does tail sampling preserve that 1% head sampling destroys?

The complete set of all traces ever produced, kept at full ingest volume
The ability to avoid sampling decisions entirely
Every error trace and every trace over the SLO threshold
The span attributes inside whichever traces are kept

Why does the observe-the-observability dashboard get checked after every deploy rather than on a weekly schedule?

Because the dashboard is too expensive to leave open continuously
Because Loki retention tiers rotate on the deploy cadence
Because cardinality regressions ship inside application changes
Because the disk forecast only updates when a deploy runs

You got correct