Topic 21

Recording Rules

Rules

Once Chapter 5 instruments checkout, its dashboard will run histogram_quantile over summed bucket rates across every checkout series — recomputed on every refresh, by every viewer, the same expensive answer thrown away and rebuilt each time. A recording rule evaluates a PromQL expression on a schedule and writes the result back into the TSDB as a new series, so dashboards and alerts read a precomputed number instead of re-deriving it.

This is the first time the book spends one of the three costs to buy relief from another: a rule trades a standing purchase of cardinality and retention for query speed. Made deliberately, that trade is the foundation Chapter 10's burn-rate alerting stands on. Made reflexively, it fills the TSDB with series nobody can name.

What a recording rule buys

Heavy queryrecomputed each refresh

→

Recording ruleruns on a schedule

→

Precomputed serieslevel:metric:operations

→

Dashboard / alertreads one number

The Mechanism

Rules live in files listed under rule_files: in prometheus.yml, organized under groups:. Prometheus evaluates each group every evaluation_interval — 1 minute by default, settable per group with interval:. Each evaluation runs the expression once and appends the result as ordinary samples under the rule's name.

# rules/harborline.yml
groups:
  - name: harborline-bookings
    interval: 1m
    rules:
      - record: job:harborline_bookings:rate5m
        expr: sum by (job) (rate(harborline_bookings_total[5m]))

This group writes one new sample per job every minute. The recorded series is real TSDB data with real retention — indistinguishable from scraped data except by its name.

The Naming Convention

Recorded series follow level:metric:operations — colons are reserved for rule output, and Prometheus's own naming guidance says exporters and direct instrumentation should never use them in scraped metric names. job:harborline_bookings:rate5m reads as "the 5-minute rate of bookings, aggregated to the job level" — the name alone tells a reader which labels survive and what math was done. The convention is the only thing standing between a rule file and fifty mystery metrics.

When a Rule Beats a Dashboard Query

Three honest triggers. The expression touches thousands of series or long windows — the overview dashboard's fleet-wide p99 qualifies. The result feeds an alert, which must evaluate cheaply and identically every interval. Or the result is an ingredient other rules build on — Chapter 10's multi-window burn rates are recording rules stacked on recording rules, and this topic is where that stacking becomes possible. A query that hits none of the three stays a query.

Record the Reaggregatable Form

Record sum by (job, le) (rate(harborline_checkout_duration_seconds_bucket[5m])) as job_le:harborline_checkout_duration_seconds_bucket:rate5m — not the histogram_quantile output. Recorded bucket rates can still be re-summed and re-quantiled at query time; a recorded p99 is a dead end that can never be correctly combined with anything. The previous topic's law, now with persistence.

Ordering and Chaining

Rules within one group evaluate sequentially in file order, so a rule may safely reference the output of the rule above it. Across groups, evaluation is concurrent — a cross-group dependency reads the previous interval's value, one evaluation stale. Keep each derivation chain inside a single group, in dependency order, and the subtlety never bites.

The Bill

Every rule emits at least one series per output label combination, stored for the full retention window — 15 days by default. A rule file is a standing purchase of cardinality and retention, which is why each rule should trace to a named dashboard panel or alert, never to "might be handy". The two rules this chapter defines — the bookings rate above and the checkout bucket rates — are the precomputed shape the Chapter 6 panels can switch to the day their raw queries get heavy.

Common Mistakes

Recording an average or a quantile and aggregating it later — job:checkout_latency:p99_5m averaged across jobs reproduces the exact composition errors of the last two topics, now laundered through an innocent-looking metric name.
Naming recorded series like scraped ones — harborline_bookings_rate_5m with no colons; six months later nobody can tell which metrics come from code and which from rules, and an instrumentation cleanup deletes a rule's input while its dashboards keep rendering stale-looking mysteries.
A rule group evaluated every 5 minutes feeding an alert checked every minute — the alert re-reads the same stale sample five times, adding up to 5 minutes of detection latency on top of the alert's for: duration. Keep the rule interval at or below the alert's evaluation interval.
Editing a rule's expression in place and trusting old graphs — samples recorded under the previous expression stay in the TSDB under the same name, so the panel silently splices two different definitions at the deploy moment. Rename the rule when its semantics change.
Recording rules as a reflexive optimization for every dashboard query — fifty rules nobody audits, each a permanent series family; the cheap queries never needed help, and the series budget from Chapter 3's cardinality lessons quietly absorbs the cost.

Best Practices

Name every rule level:metric:operations exactly — job:harborline_bookings:rate5m — so the aggregation level and the math are readable from the name alone.
Record sums, rates, and bucket rates — the reaggregatable forms — and compute averages, ratios, and quantiles from them at query time.
Keep each derivation chain inside a single rule group, in dependency order, so every rule reads fresh inputs from the same evaluation.
Tie every recording rule to a consumer — the dashboard panel or alert that reads it, noted in a comment — and delete rules whose consumer is gone, because each one is a standing cardinality and retention purchase.

Comparable toolsLoki the ruler runs the same rule syntax over LogQL, recording log-derived metrics into PrometheusInfluxDB continuous queries and Flux tasks precompute downsampled seriesTimescaleDB continuous aggregates — the SQL materialized-view equivalentDatadog generate-metrics pipelines derive metrics from logs and traces at ingestion

Knowledge Check

What does a recording rule physically produce?

An in-memory cache entry that speeds up matching queries
A lazy view evaluated only when a dashboard requests the rule name
Ordinary samples appended to the TSDB under the rule's name
A compressed rollup stored in a separate long-term archive

What does the name job:harborline_bookings:rate5m communicate?

Aggregation level, source metric, and the operation applied, in that order
That the rule evaluates every 5 minutes
That the series was scraped from an exporter named job
That samples are kept for only 5 minutes and then downsampled into a coarser rollup

Which result is safe to record for later re-aggregation?

The histogram_quantile output, so dashboards never recompute it
A per-instance average latency, ready for a fleet-wide avg()
sum(harborline_bookings_total), rated at query time
sum by (job, le) of the bucket rates

An alert evaluates every minute against a rule recorded every 5 minutes. What is the effect?

Prometheus refuses to load an alert that reads a slower rule
Up to 5 extra minutes of detection latency from re-reading stale samples
The alert sees no data at all and fires as absent
The alert silently downgrades itself to a 5-minute schedule

Why does every recording rule need a named consumer?

Unreferenced rules are garbage-collected after 15 days
A rule without a consumer is a standing cardinality and retention purchase with no return
The rule file syntax requires a mandatory consumer: field and refuses to validate the group without it
Unused rules slow down every other query in the group

You got correct