Chapter Seven · Logs

Logs

The second signal comes online. Harborline's services switch to structured JSON lines, Grafana Alloy ships every container's stdout to Loki on obs-01, and for the first time all of Saturday morning is greppable from one query box — where the errors turn out to be sitting in plain sight. This chapter builds that pipeline end to end and prices every decision in it.

6 topics

Metrics said checkout was slow; they could not say why. Logs are the signal that carries the actual words — and until now Harborline's words were printf-style free text scattered across five containers, reachable only by ssh and docker logs. Six topics turn that into a queryable system: JSON events with request_id on every line, Alloy tailing every container into Loki, and LogQL to ask the questions.

The payoff lands mid-chapter: one query over Saturday morning surfaces hundreds of connection pool timeout errors in bookings between 09:00 and 11:30 — the mystery's first hard evidence. The chapter closes on the two halves nobody advertises: hygiene (levels, sampling, retention, and the iron rule that card numbers never reach a log) and the wider market — Elasticsearch, OpenSearch, and the SaaS platforms — mapped by what each bet actually costs.

The pipeline this chapter builds — one container's stdout to a query box
Container stdoutJSON per line
Alloytail · label · push
Lokiindex labels, store chunks
LogQL in Grafanaone query box

Topics in This Chapter

Topic 32
Structured Logging
A line written for humans is nearly useless to a machine reading ten million of them. One JSON object per event — timestamp, level, service, request_id on every line — costs 2–3x the bytes and buys every query in the rest of the chapter.
Logging
Topic 33
The Log Pipeline
Between stdout and a query box sits a pipeline: Docker's json-file driver, Alloy tailing and labeling, a push to Loki on obs-01:3100. Every stage has a buffer, every buffer has a limit, and the honest question is which lines get dropped when the pipe clogs.
Pipeline
Topic 34
Loki Architecture
Loki indexes the labels, not the text — a few kilobytes of index per stream, chunks compressed into object storage. The bet makes it dramatically cheaper than Elasticsearch and measurably slower at needle-in-haystack search; this topic prices both halves.
Loki
Topic 35
LogQL
PromQL's sibling: select streams by label, filter lines like grep, parse JSON fields at query time, and finish by turning logs into a metric. Mara's Saturday query takes 30 seconds to write because Chapter 4 already taught its grammar.
LogQL
Topic 36
Log Hygiene
Levels that mean the same thing in every service, sampling for the noisiest streams, retention priced per stream — and the iron rule that secrets and card numbers never reach an immutable chunk store. Working is not the same as affordable, trusted, and legal.
Hygiene
Topic 37
The Wider World
Elasticsearch and OpenSearch index every word and answer any term in milliseconds — at 50–100% storage surcharge plus a cluster to run; SaaS platforms meter every gigabyte ingested. The map of the log-store market, and why Harborline's query shapes keep it on Loki.
Ecosystem