Prometheus Alerting Rules

An alerting rule is where the philosophy of Topic 44 becomes a machine's commitment. Rules live in YAML files that Prometheus loads through rule_files and evaluates on a fixed schedule — every minute by default — and each rule is four parts: expr, a PromQL expression that defines "wrong"; for, how long wrong must persist before anyone hears about it; labels, which classify the alert for routing; and annotations, the human-readable text templated in at fire time.

Prometheus's responsibility ends at deciding that something is wrong. Who hears about it, how often, and bundled with what is Alertmanager's job in the next topic. Keeping the two jobs straight explains most of this chapter's structure: thresholds get edited in one file, routing in another, and the person changing one does not need to understand the other.

Rule File Anatomy

Rules sit in named groups inside YAML files listed under rule_files in prometheus.yml. A group evaluates on the global evaluation_interval — 1 minute unless overridden — and can set its own interval when a rule is expensive enough to deserve a slower cadence. promtool check rules validates a file before Prometheus ever loads it, and the failure mode of skipping that step is forgiving in one specific way: a reload with a broken file is rejected and the old rules keep running. The trap is that "keeps running the old rules" also means your new checkout page silently never armed.

The Expression

expr is ordinary PromQL, and the rule fires one alert per series the expression returns. That single sentence sets your page volume. Harborline's first rule divides the rate() of checkout 5xx responses by the rate() of all checkout requests over 5 minutes, and wraps both sides in sum() so the two series from app-01 and app-02 collapse into one — one series, one alert, regardless of how many replicas are involved.

# rules/checkout.yml — loaded via rule_files: in prometheus.yml
groups:
  - name: checkout
    rules:
      - alert: HarborlineCheckoutErrorRatio
        expr: sum(rate(harborline_checkout_requests_total{code=~"5.."}[5m]))
                / sum(rate(harborline_checkout_requests_total[5m])) > 0.02
        for: 5m
        labels: {severity: page, team: bookings}
        annotations: {summary: "Checkout error ratio at {{ $value | humanizePercentage }}"}

Read the expression from the inside out. The numerator is the per-second rate of checkout requests that returned a 5xx status, averaged over the last 5 minutes; the denominator is the same rate for all checkout requests; the division yields the error ratio as a number between 0 and 1. The > 0.02 comparison then filters: PromQL drops every series at or below the threshold, so the expression returns data only while more than 2% of checkouts are failing — and "returns data" is exactly the condition the alerting machinery watches for.

Pending vs Firing

The moment expr returns data, the alert enters pending. Nothing is sent anywhere. Only after the condition has held continuously for the full for duration does the alert become firing and reach Alertmanager. With for: 5m, a single bad scrape interval — a garbage-collection pause, a transient network hiccup — can never page anyone, because the condition clears before 5 minutes elapse. The price is 5 extra minutes of detection latency on the real outage, and that trade is worth making on nearly every paging rule.

keep_firing_for covers the mirror case. A series that dips briefly below the threshold mid-incident would otherwise resolve the alert and re-fire it minutes later, generating a fresh notification each round trip. Setting keep_firing_for holds a firing alert through short gaps in the data, so one incident stays one alert instead of becoming a flapbook.

Labels and Routing Classification

Rule labels merge with the labels of the series that fired, and the merged set is the alert's identity. Harborline stamps severity: page and team: bookings — which is everything Topic 46's routing tree needs to send this to the right receiver. Because labels define identity, they must be static: template a changing value into a label and every evaluation mints a brand-new alert, each one greeted by Alertmanager as a fresh incident.

Annotations and Templating

Annotations carry the human-readable payload. summary and description are Go-templated at fire time — {{ $labels.job }} pulls a label value, {{ $value }} the expression's number, so the page reads "checkout error ratio is 3.4%" instead of naming a rule and leaving the arithmetic to a person at 03:10. runbook_url rides along as a third annotation, and Topic 48 makes it mandatory. Annotations are display-only: they can change between evaluations without changing which alert this is, which is precisely why volatile values belong here and never in labels.

expr > 0.02one alert per series

→

Pendingfor: 5m

→

Firing+ labels & annotations

Harborline's First Two Pages

Mara ships two rules. The first is the error ratio above: more than 2% of checkout requests failing, sustained for 5 minutes. The second targets the other pain customers can feel — checkout p99 latency over 1 second for 10 minutes, computed with histogram_quantile over sum(rate(...)) of the harborline_checkout_duration_seconds buckets. Both are symptoms measured at the service edge, straight out of Topic 44.

Both are also deliberately blunt. Why 2% and not 1.5%? Why 5 minutes? The honest answer is that these are defensible starting guesses, and that is fine for chapter nine. Chapter 10 replaces the guesses with multi-window burn-rate alerts derived from the 99.9%/28-day checkout SLO — thresholds that follow from an explicit reliability target instead of a gut feeling.