LogQL

LogQL is deliberately PromQL's sibling: the same label matchers, the same range windows, the same aggregation grammar. A query starts by selecting streams with labels, filters lines like grep, optionally parses fields out of the line at query time, and can finish by turning the whole thing into a metric.

Which means the mental toolkit you built in Chapter 4 already covers most of this topic. What is genuinely new is small: line filters, query-time parsers, and the discipline of ordering stages so the expensive ones see the fewest lines. Everything else is PromQL wearing a log costume.

Stream selectorthe only index

→

Line filtergrep raw bytes

→

Parserjson · label filters

→

Metrics from logsrate · quantile

Stream Selectors

Every query opens with a stream selector: {job="bookings", env="prod"}, using PromQL's matcher syntax — =, !=, =~, !~ — against stream labels. The selector is the only part of a query the index accelerates, so it does the heavy narrowing before anything touches chunk data. Topic 34 said selectors are the only index Loki has; this is where you spend that index.

Line Filters

Next come line filters: |= "timeout" for substring match, != "healthcheck" to exclude, |~ "pool.*exhausted" for regex. They run over raw line bytes and are the cheapest per-line operation LogQL has. Chain them — |= "error" != "retrying" — and millions of lines narrow to hundreds before any expensive stage runs.

Parsers and Label Filters

The stage | json (or | logfmt) extracts fields from line content into labels at query time. This is the other half of topic 34's bargain: structure paid for on read instead of indexed on write. Extracted labels then feed label filters — | level="error", | wait_ms > 3000 — and lines that fail to parse do not vanish; they get an __error__ label you can inspect or drop deliberately. Ignore __error__ over mixed-format streams and your "all errors" panel quietly undercounts.

Metrics from Logs

The last stage turns logs into numbers. Wrap a filtered stream in rate() with a range window and you get a per-second rate; count_over_time counts raw occurrences in the window; sum by (job) aggregates across services. And unwrap pulls a numeric field out of each line so quantile_over_time can compute percentiles from it — a latency distribution conjured from logs, no instrumentation change required.

# per-second rate of pool timeouts, computed from logs alone
rate({job="bookings"} |= "connection pool timeout" [5m])

# p95 of the wait_ms field — a percentile with no code change
quantile_over_time(0.95,
  {job="bookings"} |= "conn_pool_timeout" | json | unwrap wait_ms [5m])

The first query is the log-derived twin of a counter rate from Chapter 4. The second filters to the pool-timeout events, parses each line, unwraps the numeric wait_ms field, and computes the 95th percentile of it over 5-minute windows — Mara now knows not just how many requests waited, but how long the slow ones waited.

One Toolkit with PromQL

Because rate, range vectors, by/without, and the arithmetic operators behave the same in both languages, a Grafana panel can graph harborline_bookings_total errors and log-derived timeout rates side by side, and your instincts transfer whole. Mara's Saturday-morning query takes her 30 seconds to write, because Chapter 4 already taught its grammar.

{job="bookings", env="prod"}
  |= "connection pool timeout"
  | json
  | level="error"

Selector first, cheap substring filter second, parser third, label filter last — the cost-ordering rule applied. The result: hundreds of matching lines between 09:00 and 11:30, all from bookings, every one a request that waited more than five seconds for a connection. For the first time the Saturday complaint has a name and a count. What the lines still cannot say is where those five seconds went inside each request — that answer needs the third signal, and it arrives in Chapter 8.