The second signal comes online. Harborline's services switch to structured JSON lines, Grafana Alloy ships every container's stdout to Loki on obs-01, and for the first time all of Saturday morning is greppable from one query box — where the errors turn out to be sitting in plain sight. This chapter builds that pipeline end to end and prices every decision in it.
6 topics
Metrics said checkout was slow; they could not say why. Logs are the signal that carries the actual words — and until now Harborline's words were printf-style free text scattered across five containers, reachable only by ssh and docker logs. Six topics turn that into a queryable system: JSON events with request_id on every line, Alloy tailing every container into Loki, and LogQL to ask the questions.
The payoff lands mid-chapter: one query over Saturday morning surfaces hundreds of connection pool timeout errors in bookings between 09:00 and 11:30 — the mystery's first hard evidence. The chapter closes on the two halves nobody advertises: hygiene (levels, sampling, retention, and the iron rule that card numbers never reach a log) and the wider market — Elasticsearch, OpenSearch, and the SaaS platforms — mapped by what each bet actually costs.
The pipeline this chapter builds — one container's stdout to a query box