rate, increase, irate
A counter's raw value is a number that only ever grows — 4,182,993 keyspace hits since cache-01 last restarted tells you nothing. Every question you actually have is about speed: hits per second, bytes per second, bookings per second. The instant table from the last topic was 32 monotone ramps; this topic turns ramps into rates.
rate() is the single most-typed function in PromQL, and using it correctly — over counters only, over a window wide enough to hold samples — is the difference between a truthful graph and confident garbage. Its two siblings, increase() and irate(), are the same machinery pointed at different questions, and each has one legitimate job.
What rate() Computes
The query below returns the per-second average increase of the Redis hit counter over the trailing 5 minutes, one value per series. It is the shape every throughput panel in the book is built from.
# cache hits per second on cache-01, averaged over 5 minutes rate(redis_keyspace_hits_total[5m])
Under the hood, rate() takes the samples in the window and extrapolates to the window boundaries from the first and last of them — capped when those samples sit farther than about 1.1× the average sample interval from the edge, and clamped so a counter is never extrapolated below zero. Extrapolation is why the output is smooth rather than stair-stepped, and why the results are estimates by design, not exact tallies.
Counter Resets, Handled
When bookings restarts, harborline_bookings_total drops back to 0. rate(), increase(), and irate() all treat any decrease between adjacent samples as a counter reset and add the pre-reset value back in before computing. A restart costs you at most the handful of requests that landed between the last scrape and the crash — not a negative spike, not a gap. This reset handling operates per raw series, a detail that becomes law two sections from now.
The Reset Logic as a Trap
That same "any decrease is a reset" rule is why rate() over a gauge produces garbage with no error. Every ordinary dip in node_memory_MemAvailable_bytes gets "compensated" as if the kernel had restarted, and the output is a plausible-looking positive line that corresponds to nothing in reality. Prometheus 3.x surfaces an info annotation in the UI when rate() is applied to a metric without a counter-style _total suffix, but the query still runs and still returns numbers. For gauges, the honest tools are delta() and deriv().
increase(), rate() in a Costume
increase(harborline_bookings_total[1h]) is exactly rate(...[1h]) * 3600 — same extrapolation, same reset handling, multiplied back into natural units. Because of the extrapolation it happily reports 1,283.7 bookings in an hour even though bookings arrive in whole units. That fraction is correct behavior and a perpetual source of support tickets; increase() belongs where a human reads the number, not where a machine consumes it.
irate() and Its One Job
irate() uses only the last two samples in the window, so it reacts instantly and jitters wildly. That makes it a magnifying glass for zoomed-in debugging of a fast-moving counter, and precisely wrong everywhere else. On a dashboard, the panel step samples the expression every 30 or 60 seconds and throws away everything between the two scrapes each evaluation happens to land on; in an alert, a sustained burn can evaluate as calm because the last two samples were quiet.
rate() times the window, in natural units like bookings this hour. Stat panels and reports — extrapolation makes fractions, so keep it where a human reads the number.Sizing the Window
rate() needs at least 2 samples in the window to compute anything, so the hard floor is 2× the scrape interval — [30s] on Harborline's 15-second scrapes. The working rule is 4× or wider: at [1m] a single failed scrape cannot blank the graph, and [1m] to [5m] is where dashboards and alerts should live. Wider windows smooth more and react slower; the window is a smoothing knob, not a formality.
# fraction of CPU busy, per host, 5-minute window
1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))
This expression rates the idle counter per CPU, averages the per-CPU idle fractions within each host, and subtracts from 1 to get busy fraction. Mara runs it across two weeks of Ch3 data and the Saturday story gains its second number: app-01 sits at 35% busy on a Tuesday morning and 78% at 09:30 on Saturday. Busier, measurably — but 78% is not saturation, so CPU alone does not explain a 3× latency multiple. The evidence narrows; the cause stays open.
rate() — the per-second average over the whole window: smooth, reset-safe, and the default for dashboards, alerts, and recording rules. When in doubt, this one.
irate() — the per-second slope of the last two samples only: maximally responsive, maximally noisy. Zoomed-in live debugging of fast-moving counters, nothing else.
increase() — rate() multiplied by the window: human units like "bookings this hour" for stat panels and reports, with extrapolated fractions as the price.
rate(node_memory_MemAvailable_bytes[5m])— rate over a gauge; every downward wiggle triggers reset compensation and the graph shows a plausible positive line that means nothing, with no error to warn you. Usedelta()orderiv()for gauges.- A range window under 2× the scrape interval —
rate(...[15s])against a 15-second scrape usually finds one sample per window and returns nothing; the panel renders as intermittent dots and the on-call reads it as flapping. - Summing counters before rating them — a rule that stores
sum(harborline_bookings_total)and rates the result turns every single-instance restart into a huge apparent reset of the whole sum. Rate first, then sum; per-series reset handling depends on it. irate()inside an alert expression — the alert reads two adjacent scrapes and nothing else, so a sustained 10-minute error burn can evaluate as calm if the last two samples happened to be quiet. Alerts wantrate()over a multi-minute window.- Filing a bug because
increase(...[1h])reports 1,283.7 requests — the fraction is extrapolation working as designed, not data corruption; PromQL deliberately does not privilege the exact raw counter delta.
- Rate then aggregate, never the reverse —
sum by (job) (rate(harborline_bookings_total[5m]))— because reset detection only works on individual raw counter series. - Size every rate window at 4× the scrape interval or wider —
[1m]minimum on 15-second scrapes — so a single failed scrape cannot blank the graph. - Use
increase()only where a human reads the number in natural units — stat panels, weekly reports — and keeprate()everywhere machines consume the result. - Confine
irate()to interactive debugging at short time ranges, and delete it from any dashboard JSON you find it in; the panel step silently discards the responsiveness it pretends to add.
Knowledge Check
Why must rate() be applied before sum(), never after?
- Summing counters first is a type error that PromQL rejects
- Reset detection is per-series, and a pre-summed counter turns one restart into a giant fake reset
- rate() over a sum is too slow to evaluate at dashboard refresh rates and often times out on large fleets
- rate() can only accept a single series as input
What does rate() over a gauge like MemAvailable actually return?
- An immediate query error, since gauges are not valid input
- Always zero, because a gauge has no monotonic increase for rate() to measure over the window
- A meaningless positive line, because every dip is treated as a counter reset
- The true rate of change of the gauge, same as deriv()
With Harborline's 15-second scrape interval, why does rate(x[15s]) render as scattered dots?
- The window usually contains one sample, and rate() needs two to compute
- The TSDB has already dropped samples older than 15 seconds
- The exporters are flapping and failing every other scrape, leaving gaps in the series
- Grafana throttles panels whose windows are under one minute
An alert built on irate() misses a sustained 10-minute error burn. What is the mechanism?
- irate() ignores counter resets, so the burn was subtracted away
- Alertmanager rejects irate() expressions outright and never evaluated the rule at all
- Each evaluation sees only the last two samples, which happened to be calm
- irate() lags behind by one full scrape interval
increase(harborline_bookings_total[1h]) returns 1,283.7. What should you conclude?
- The counter is corrupted and stores fractional bookings
- Extrapolation is working as designed; increase() is rate() times the window
- A counter reset occurred mid-window and split a booking in two
- The scrape interval drifted, and fixing scrape timing yields whole numbers
You got correct