Selectors and Vectors

Every PromQL query returns one of two shapes, and every confusion in the language traces back to mixing them up. An instant vector is a table: one row per matching series, each carrying a single value as of one evaluation moment. A range vector is that same set of series, but each row holds a window of raw samples instead of one value — raw material for functions like rate(), never something you graph directly.

The distinction is not bookkeeping. Functions demand one shape or the other, graph panels accept only one of them, and the most common error message in the language — "expected type instant vector" — is the engine telling you which shape you handed it. Two weeks of Ch3 telemetry sit in the TSDB on obs-01; this topic is the grammar for asking it anything.

The Instant Vector

Query up{job="bookings"} at 10:00:00 and you get one row per bookings target, each with its most recent sample. "Most recent" has a precise meaning: Prometheus searches backward up to the lookback delta — 5 minutes by default — and returns the newest sample it finds in that window. With Harborline scraping every 15 seconds, a healthy target always has a sample well inside the lookback, so the table reads as live data.

The lookback cuts both ways. When a target dies, Prometheus normally writes a staleness marker — on the failed scrape, or when the target drops out of service discovery — and the marker ends the series immediately instead of letting it coast. Where no marker lands, the dead series keeps answering instant queries with its last value for up to the full 5 minutes. A "current value" panel can show a healthy number from a corpse.

Label Matchers

Selectors filter by label with four operators: = exact, != negated exact, =~ regex, !~ negated regex. Multiple matchers inside one pair of braces are ANDed, so a selector narrows with every matcher you add.

# four matchers, four behaviors
up{job="bookings"}                          # exact match
node_cpu_seconds_total{mode!="idle"}        # everything except idle
up{instance=~"app-0.*"}                     # regex: app-01 and app-02
{__name__=~"harborline_.+", env="prod"}     # all harborline metrics in prod

Two rules save hours of confusion. First, regexes are fully anchored: {instance=~"app"} matches nothing, because the engine reads it as ^app$ — substring intent needs .* on both sides. Second, a selector must contain at least one matcher that cannot match the empty string, which is why {job=~".+"} is accepted where {job=~".*"} alone is rejected. The guard exists to stop a careless selector from sweeping every series in the database.

The Range Vector

Append a duration in square brackets — node_cpu_seconds_total[5m] — and each row swaps its single value for every raw sample from the trailing 5 minutes. Nothing is computed yet; you are holding raw samples, per series, waiting for a function. rate(), increase(), and the _over_time() family all take exactly this shape as input.

Since Prometheus 3.0 that window is left-open: a sample sitting exactly 5m00s back is excluded, one at 4m59.999s is in. The change sounds pedantic but fixes a real inconsistency — under the old closed window, whether a [5m] range returned 5 or 6 samples depended on pure luck: whether the query happened to land exactly on a scrape timestamp.

Range Query vs Range Vector

The names collide; the concepts don't. A Grafana graph is a range query: the same instant-vector expression evaluated over and over, once per step across the time axis, each evaluation producing one point per series. A range vector is a per-series window of samples inside a single evaluation. Hand a graph panel a bare [5m] selector and it fails with "expected type instant vector" — the window shape belongs inside a function, not on an axis.

Time Travel with offset

The offset modifier shifts one selector's evaluation time into the past, independent of the panel's time range. That makes week-over-week comparison a one-line query instead of two browser tabs and a squint: the metric now, divided by the same metric a week ago, on one axis.

# this Saturday's checkout probe vs the same moment last Saturday
probe_duration_seconds{job="blackbox"}
  / probe_duration_seconds{job="blackbox"} offset 7d

The division pairs each series with its own past self, so the panel draws a ratio: 1.0 means this week matches last week. Mara's version of this query is the book's first hard evidence. Offset by 7 days, the Saturday 09:00–11:00 hump in checkout probe latency lines up with an identical hump the Saturday before; offset to line Saturday up against Tuesday, the same window runs roughly 3× the Tuesday value. Saturday mornings are measurably different — the customer was right, and the graph finally agrees.

First Contact with the Harborline Data

Run node_cpu_seconds_total{instance="app-01:9100"} as an instant query in the Prometheus table view and 32 rows come back: 8 CPU modes across the 4 vCPUs, every one a counter that has been climbing since boot. The raw values — hundreds of thousands of accumulated seconds — answer nothing on their own. What the host data cannot yet say is why Saturday differs, or even how busy the CPU is right now; turning monotone ramps into truth is the next topic's entire job.