Capstone: The Full Picture
One request, every signal, the whole pipeline. It is 09:14 on a Saturday morning — the exact conditions of the Chapter 1 complaint — and a customer clicks "Pay now" on harborline.example. Eighteen months ago that click was invisible: five services, two VMs, and a diagnostic toolkit of ssh, docker logs, and htop that could not say whether checkout was slow, for whom, or where.
This topic follows that single checkout request through everything built since Chapter 3, signal by signal, stage by stage. Every hop below is one chapter of this book doing its job on one ordinary request, and the walkthrough ends with the questions Mara's team now answers in seconds that they once could not answer at all.
The Request Arrives
web's nginx on app-01 serves the POST and writes exactly one JSON access log line: the route, status 200, 411 ms, and the trace ID. That single line is Chapter 7's discipline in miniature — one structured event per request, carrying the identifier that ties it to everything below.
# the one access line web writes for this request
{"time":"09:14:07", "method":"POST", "route":"/checkout/pay",
"status":200, "duration_ms":411,
"trace_id":"9f4e21c8a7d3b5f0e6a1d2c4b8f73e90"}
Grafana Alloy tails the container log and pushes the line to Loki on obs-01 (:3100) under the labels {service="web", env="prod"}. About a second after the customer's click, the LogQL query {service="web"} |= "/checkout/pay" finds it — one event among Saturday's thousands, queryable by anyone.
The Trace Is Born
The OpenTelemetry instrumentation in web starts the trace and injects a traceparent header into the outbound call. bookings on port 8000 continues the context and calls payments on port 8002, each hop adding spans — including the Redis session lookup on cache-01, the very call whose connection-pool ceiling Chapter 8 caught saturating on Saturdays.
The spans land in Tempo (:3200) as one waterfall: 411 ms total, 287 ms of it inside the payments-processor call. That is the localization no other signal provides — not that checkout was slow, but which hop in web → bookings → payments held the time.
The Metrics Tick
The same request becomes one observation in the harborline_checkout_duration_seconds histogram — 411 ms falls into its bucket — and one increment on harborline_bookings_total{status="200"}. Prometheus (:9090) collects both on its next 15-second scrape. This is Chapter 5's cheap, aggregated accounting: the request costs a few counter increments, indistinguishable in price from the thousands around it, which is exactly why metrics are the signal that watches everything all the time.
The Dashboard and the SLO Consume Them
On the Chapter 6 checkout dashboard the RED panels re-render on refresh: p95 sits at 430 ms against the 500 ms objective, the request rate climbing into the Saturday peak. The Chapter 10 recording rules fold the request into the 28-day error budget as one more good event — status 200, under 500 ms.
Had it failed, nothing would page yet. One bad request is a blip; the multi-window burn-rate rules would need roughly 45 more minutes of failures like it before Alertmanager (:9093) pages Mara. The gap is deliberate — it is the distance between one flake and actual user pain, and Chapter 9 spent a whole topic defending it.
The Chapter 1 Diagram, Now Real
Chapter 1's six-stage pipeline — instrument, collect, store, query, visualize, alert — just ran three times. The log's run: application → Alloy → Loki → LogQL → dashboard. The metric's run: client library → scrape → TSDB → PromQL → panel → alert rule. The trace's run: SDK → OTLP → Tempo → trace view. Three runs joined by the shared trace ID and timestamps, so Chapter 8's metric → exemplar → trace → logs jump works in either direction — from a latency panel to this request's waterfall, or from a log line to its spans.
What the Team Can Now Answer
Chapter 1's question — is Saturday checkout actually slow, for whom, and where — takes one dashboard and one trace: yes, p95 rises 18% on Saturday mornings; the time sits inside the payments-processor call; and it is within SLO. In Chapter 1 the same question produced a shrug and an ssh session on app-01.
The questions behind it fall the same way. How much error budget is left — the Chapter 10 panel. What changed at 09:12 — the Chapter 6 deploy annotation. Which requests failed and why — Chapter 7's logs and Chapter 8's traces, joined by trace ID. Should anyone be paged — Chapter 9's answer this morning is no. Each answered from obs-01 in under a minute, and none of them answerable at all eighteen months ago. That distance is what this book built.
- Reading the capstone as a tool inventory rather than a pipeline proof — the point is not that five containers run on
obs-01, it is that one request's evidence is findable in all three signals and cross-linked. A stack where this walkthrough breaks at any hop is topic 70's unwatched stack with better branding. - Forgetting that every stage was a cost decision — this request's trace survived sampling because checkout is a kept path (Chapter 8), its log line was one line rather than ten (Chapter 7), its histogram added no new label values (Chapter 5). Remove those disciplines and Saturday's traffic detonates cardinality, sampling, or retention.
- Expecting the correlated jump to work without shared identifiers — the metric → trace → log chain rides on the trace ID in the log line and the exemplar on the histogram. Skip either during instrumentation and the walkthrough degrades into three disconnected searches by timestamp.
- Treating the finished stack as finished —
searchgets a rewrite next quarter and enters at rung 0 of topic 71's ladder. The capstone describes a moment; only the topic-70 practices — owners, deletion passes, alert audits — keep it true next year.
- Run this walkthrough as a drill twice a year: pick one production request at random and trace it through all three signals by hand — every hop that requires tribal knowledge or hits a missing link is an instrumentation bug to file.
- Keep the Chapter 1 pipeline diagram posted with the real component names on it — Alloy, Prometheus, Loki, Tempo, Alertmanager — so a new engineer can place any signal question at a pipeline stage in their first week.
- Verify the cross-signal links — trace IDs in log lines, exemplars on histograms — in CI or a synthetic check, because they decay silently: nothing alerts when a log-format change drops the trace ID field.
- Answer "should we add this telemetry" with the three costs from Chapter 1 — what it does to cardinality, sampling, and retention. Eighteen months of this stack surviving two rewrites and a Kubernetes move is the compound interest of asking that question every time.
Knowledge Check
The 09:14 request produced an access log line, spans, and a histogram observation. Which pipeline stage do all three artifacts belong to?
- Collect — they exist once Alloy tails the log, Prometheus scrapes them, and OTLP forwards them onward
- Instrument — they are what the application emits before anything else happens
- Store — Loki, the TSDB, and Tempo create them on write
- Query — they materialize when someone runs LogQL or PromQL
What carries the correlation that lets Mara jump from a latency panel to this exact request's trace and logs?
- The shared service names carried on the metric series, the log lines, and the span attributes
- Grafana's data-source links configured between panels
- The trace ID in the log line plus the exemplar attached to the histogram
- The env label, which all three signals share
If the 09:14 request had failed, why would nobody be paged for another 45 minutes?
- Prometheus would not scrape the failure until the next cycle
- The burn-rate rules require sustained failures before the budget spend justifies a page
- Alertmanager batches every page and delivers the whole queue on a fixed 45-minute flush schedule
- Saturday-morning pages are suppressed by the on-call calendar
Why does answering Chapter 1's Saturday question take all three signals rather than any single one?
- Redundancy — any one of the three signals could answer it alone, and keeping three copies survives a store outage
- Cost — splitting the data three ways keeps each store's bill smaller
- Vendor design — no single tool ships more than one signal
- Each signal answers a different part: metrics detect, traces localize, logs hold the per-request evidence
You got correct