Anatomy of the Stack
Strip the logos off any observability product — open-source stack or a $500,000-a-year SaaS — and the same six-stage pipeline is underneath: instrument → collect → store → query → visualize → alert. The pipeline runs once per signal, three times in total, and every chapter of this book is a stop on it.
Learn the shape once and every tool you meet afterward is just an implementation of some stage. Prometheus is collect-plus-store-plus-query for metrics. Loki is store-plus-query for logs. Grafana is visualize for all three. A vendor pitch that sounds novel usually resolves to "we do stages two through six behind one agent and one bill."
Instrument
Telemetry is born in code and exporters. A client library increments harborline_bookings_total inside bookings; node_exporter translates /proc into metrics on port 9100; OpenTelemetry auto-instrumentation wraps FastAPI and emits a span per request. This stage has a property no other stage shares: whatever is not emitted here does not exist downstream, at any price. What instrumented output actually looks like for metrics is a plain-text HTTP page — the exposition format below is what Prometheus reads when it scrapes a target.
# excerpt of GET /metrics on bookings (app-01:8000)
# HELP harborline_bookings_total Total booking requests handled.
# TYPE harborline_bookings_total counter
harborline_bookings_total{endpoint="/checkout",status="200"} 18734
harborline_bookings_total{endpoint="/checkout",status="500"} 121
Two lines of data, and already the shape of the whole discipline: a metric name, labels carrying the dimensions (endpoint, status), and a running count. Everything Prometheus does in Chapters 3–4 starts from text like this. Chapters 5, 7, and 8 cover producing it — and its log and trace equivalents — from your own code.
Collect
Collection moves telemetry off the host, and the two transport models split by signal. Prometheus pulls metrics: it scrapes each target's HTTP endpoint on an interval — 15 seconds at Harborline. Logs and traces are pushed, typically over OTLP. In both cases an agent — Grafana Alloy in this book, the OpenTelemetry Collector generically — runs on every host as the local funnel where batching, relabeling, and dropping happen before anything crosses the network. The agent is also the shock absorber: when the backend is down, it buffers and retries so the application never feels the outage.
Store
Each signal gets a purpose-built backend: a time-series database for metrics (Prometheus), a label-indexed log store (Loki), an object-storage trace store (Tempo). A general-purpose database serves all three badly, because the write pattern — millions of tiny appends, compression that exploits the regularity of timestamped samples — is nothing like OLTP. Postgres storing Prometheus's workload would spend more on write amplification than the data is worth; a TSDB gets a compressed sample down to 1–2 bytes because it is built for exactly this shape.
Query
PromQL, LogQL, and TraceQL are where stored bytes become answers. Every stage upstream exists to make these queries fast, and a stack is worth exactly what its query layer can answer during an incident — which is why Chapter 4 is the longest metrics chapter in the book, and why "write down the queries first" appears in the best practices below as a purchasing rule, not a platitude.
Visualize and Alert
Grafana reads all three stores and renders dashboards — one visualization layer over PromQL, LogQL, and TraceQL alike, built out in Chapter 6. Dashboards are for humans who are already looking.
Alerting is the stage that works while nobody is looking, and its defining property is where it runs: server-side, against the store. Prometheus evaluates alert rules on its own schedule and fires them to Alertmanager, which routes, deduplicates, and silences (Chapter 9). A red threshold line on a dashboard panel is decoration — at 02:40 it alerts precisely the zero people watching it.
The Map of the Book
The chapters ahead walk the pipeline: Chapters 3–4 collect and query metrics, Chapter 5 instruments application code, Chapter 6 visualizes, Chapter 7 runs the whole pipeline again for logs, Chapter 8 for traces, Chapters 9–10 turn stored telemetry into alerts and SLOs, and Chapter 11 re-plumbs all six stages on Kubernetes. When you feel lost anywhere in the book, locate the current topic on the six stages — it is always on exactly one of them.
- Letting applications push telemetry straight to the storage backend with no agent between — there is nowhere to batch, retry, or drop under backpressure, so a storage outage propagates backward into application latency on
bookings. - Shipping logs into the metrics store or metrics into the log store — a log line rendered as high-cardinality series melts a TSDB, and metrics stored as text lines cost 100× per query. One backend per signal is the design, not an inconvenience.
- Treating a dashboard threshold as an alert — the red panel at 02:40 alerts nobody, because visualization is stage five and alerting is stage six, evaluated server-side whether or not anyone is watching.
- Choosing a storage backend before writing down the queries it must answer — the migration lands, and then the team discovers the query language cannot express "p95 by endpoint excluding health checks," which was the whole point.
- Scraping targets directly across sites or WANs with no collection tier in between — every network blip becomes a gap in the data, indistinguishable afterward from the target actually being down.
- Run one agent per host — Alloy on every Harborline VM — as the single local funnel for all three signals, so batching, relabeling, and credentials live in one place instead of five services.
- Keep one purpose-built store per signal — TSDB, log store, trace store — and resist the one-database-for-everything consolidation until you have actually priced it.
- Write the ten queries the stack must answer — starting with "checkout p95 over the last hour, by service" — before selecting any storage, and select storage by whether it answers them.
- Evaluate every alert rule server-side in Prometheus and Alertmanager, version-controlled as code, and treat dashboard thresholds as decoration.
Knowledge Check
Which signals are pushed and which are pulled in this book's stack?
- Metrics are pushed; logs and traces are pulled by the backends on a fixed schedule
- Metrics are pulled by scraping; logs and traces are pushed over OTLP
- All three signals are pulled on the scrape interval
- All three signals are pushed by the application
What does the per-host agent buy you that direct application-to-backend shipping does not?
- Lower latency, because the agent forwards events faster
- Encryption, which applications cannot do themselves
- A buffer where batching, relabeling, and backpressure handling live
- The agent generates the telemetry so the app doesn't have to, inventing dimensions the code never emitted
Why does each signal get its own purpose-built store instead of one shared database?
- The write and compression patterns differ radically from OLTP and from each other
- Licensing terms forbid mixing signal types in one database, and vendors enforce it per signal
- Separate stores are easier to operate than one database
- Grafana can only connect to one data source per signal
A dashboard panel turns red at 02:40 and nobody responds until morning. Which stage was missing?
- Collect — the metrics never reached the store
- Visualize — the panel was misconfigured
- Alert — no server-side rule existed to page a human
- Query — the query language could not express the condition
You got correct