Topic 19

Aggregation

rate(node_cpu_seconds_total[5m]) across the two app hosts returns 64 series — 8 CPU modes × 4 vCPUs × 2 instances — and nobody's question is about series 41. Aggregation operators collapse a vector across its label dimensions: sum, avg, min, max, count, topk, and friends, steered by by() and without().

The skill is not making numbers smaller. It is deciding — explicitly, in the query text — which labels still matter and which instance-level detail you are deliberately throwing away. Every aggregation is a decision about what you will no longer be able to ask.

The Operator Set

sum, avg, min, max, and count do what they say across series, not across time — the time direction belongs to the separate _over_time() family from the last topic's range vectors. The distinction matters constantly: avg(x) averages the current values of many series at one instant; avg_over_time(x[1h]) averages one series over an hour.

Two operators break the collapsing pattern. topk(3, ...) and bottomk keep whole series instead of merging them — the 3 largest, labels intact. quantile(0.99, ...) takes a quantile across series at one instant, and count counts series, not events; a counter of requests counted with count() tells you how many series exist, not how many requests arrived.

by() Keeps, without() Drops

Both clauses answer the same question — which labels survive — from opposite directions. by (mode) keeps only mode and collapses everything else; without (cpu) drops exactly cpu and keeps whatever else exists, including labels you did not know were there, like env.

# 64 series -> 8: one per CPU mode, fleet-wide
sum by (mode) (rate(node_cpu_seconds_total[5m]))

# 64 series -> 16: drop only the per-core detail, keep host identity
sum without (cpu) (rate(node_cpu_seconds_total[5m]))

The two results answer different questions: the first is "what is the fleet's CPU doing by mode", the second is "what is each host's CPU doing by mode". Because without() survives labels you have not met yet, it is the safer habit for exploratory work; by() states the exact output contract, which is what a published dashboard wants.

Aggregation collapses a vector — sum by (job)

Series setmany series

→

sum by (job)job kept

→

One per jobrest dropped

The One-Way Door

After sum without (instance), the question "is it one host or both?" is unanswerable from this query. That is a feature when the panel is fleet throughput and a bug when it is a saturation hunt. The raw series still exist in the TSDB — nothing is destroyed — but every aggregation is a one-way door within the query, so decide which door on purpose and leave a per-instance panel next to any fleet panel someone will debug from.

The Hit-Ratio Query

Mara's cache-01 evidence is two counters composed into a ratio. The rule for every ratio in this book: build it from rates, never from raw counter values, and never as an average of pre-computed per-shard ratios.

# Redis cache hit ratio over 5 minutes
rate(redis_keyspace_hits_total[5m])
  / (rate(redis_keyspace_hits_total[5m])
     + rate(redis_keyspace_misses_total[5m]))

The numerator is hits per second; the denominator is total lookups per second; the quotient is the fraction of lookups served from cache. On a Tuesday it reads a healthy 0.97. On Saturday between 09:00 and 11:00 it sags to 0.71 — the same two-hour window where the checkout probe runs 3× slow. Two independent metrics now point at the same window, and neither says why.

topk's Dirty Secret

In a graph panel, topk(3, ...) re-evaluates at every step, so the membership of the top 3 changes across the time axis. Series flicker in and out, the legend fills with ghosts, and during an incident nobody can tell whether a line vanished because it recovered or because it fell to 4th place. topk belongs in instant contexts — tables, "right now" stat rows — while graphs want an explicit selector naming the series you decided to watch.

count and Presence

count(up{job="bookings"}) is the number of targets; sum(up{job="bookings"}) is the number of healthy targets, because up is 1 or 0. The difference between the two is your down count — a three-token availability check built from one metric. Confusing count-of-series with sum-of-values is the beginner aggregation error, and this pair is the cleanest place to burn the distinction in.

Common Mistakes

Averaging averages — avg() over per-instance latency averages weighs an idle instance handling 3 req/s equally with one handling 300; correct fleet latency is sum(rate(..._sum[5m])) / sum(rate(..._count[5m])), total time over total events.
A typo inside by() — sum by (instanse) (...) throws no error; every series lacks the misspelled label, everything collapses into one series with an empty label set, and the dashboard renders one confident, wrong line.
sum(rate(harborline_bookings_total[5m])) with no grouping in an environment that also scrapes staging — dropping env merges prod and staging into one number, and the throughput panel reads 20% high until someone asks why.
topk(5, ...) in a timeseries panel — membership flips per step, the legend fills with ghost series, and nobody can tell recovery from demotion mid-incident.
Aggregating histogram _bucket series without preserving le — sum without (instance, le) (...) destroys the bucket structure and any histogram_quantile downstream returns NaN; le is data, not metadata, and the next topic lives on this.

Best Practices

Default to without() when exploring and by() when publishing — without survives labels you have not met yet, by states the exact contract a dashboard depends on.
Keep job in every by() clause on shared metric names — sum by (job, instance) — so two services emitting the same exporter metric never merge silently.
Compute ratios from summed rates, not from averaged ratios — sum numerators and denominators separately, divide once.
Use topk and bottomk in table panels and instant queries only, and name explicit series in graphs; a graph legend should be stable across its whole time range.

Comparable toolsLogQL reuses sum by() / without() unchanged over log-derived metricsSQL GROUP BY is the direct ancestor — TimescaleDB does exactly this over hypertablesGraphite sumSeries() and groupByNode() aggregate by path segment instead of labelsKusto summarize ... by is the same idea in different clothes

Knowledge Check

You are exploring an unfamiliar metric that may carry labels you don't know about. Which grouping clause is safer, and why?

by(), because it explicitly names what to keep and drops nothing by surprise
without(), because unknown labels survive instead of being silently merged
Either — the two clauses always produce identical output
Neither works without listing every label the metric carries

Why does averaging per-instance latency averages produce a misleading fleet number?

avg() cannot be applied twice, so the query errors out
Floating-point rounding accumulates across the two averaging passes, biasing the fleet mean upward
Each instance counts equally regardless of how much traffic it serves
Stale series contaminate the outer average with old samples

sum by (instanse) (...) — with the label misspelled — produces what?

A parse error naming the unknown label
An empty result set, since no series carries that misspelled label to group on
One merged series, because every input lacks the label equally
Per-instance results under a blank legend entry

What is the difference between count(up{job="bookings"}) and sum(up{job="bookings"})?

count is total targets; sum is healthy targets
Nothing — for the up metric they always return the same number
count looks across series while sum looks across time
count skips targets that are down, so it is always the smaller value

You got correct