Observability Deep Dive
by Sergey Okinchuk
Ch. 1

Observability Deep Dive

Welcome

Four courses in this catalog teach you to build and run systems. This one teaches you to see them. Metrics, logs, and traces — collected by Prometheus, Loki, and OpenTelemetry, read through Grafana, and turned into decisions by alerting, SLOs, and a working incident practice. One fictional company, Harborline, goes from flying blind to answering any question about production in minutes — and you build every piece of that stack with it.

14 chapters 74 topics covered 10 hours audio Knowledge check on every topic

About This Course

Your Kubernetes cluster exposes metrics, your Linux hosts write logs, and your load balancer counts requests — and none of it answers the only question that matters at 2 a.m.: why is checkout slow, and for whom? Observability is the discipline that turns raw telemetry into answers. This course teaches the whole pipeline — instrument, collect, store, query, visualize, alert — run once for each of the three signals: metrics, logs, and traces. The tools are the open-source standard stack: Prometheus and PromQL, Grafana, Loki, and OpenTelemetry with Tempo.

To keep it concrete, the whole book follows one company. Harborline is a ferry-booking service with five services, a PostgreSQL database, a Redis cache, and a complaint nobody can verify: checkout "sometimes takes forever" on Saturday mornings. In Chapter 1 the only tools are ssh and docker logs, and the complaint is unanswerable. Chapter by chapter you instrument the system with Mara, Harborline's SRE, until the mystery falls in Chapter 8 — to a single trace no metric or log could have produced. The examples accumulate instead of resetting, so by the end you have built one coherent stack, not fifty disconnected demos.

The second half turns data into decisions: alerts that page on symptoms rather than causes, SLOs and error budgets that make reliability a negotiable number, the same stack re-plumbed on Kubernetes, and a full 02:40 incident run from page to blameless postmortem in 38 minutes — because of everything built before it. The book closes with the wider ecosystem: continuous profiling, eBPF, synthetics, the managed platforms, and the cost engineering that keeps the observability bill from chasing the compute bill.

Who This Is For

Engineers who run something in production — or are about to — and want to stop debugging by ssh-archaeology. It assumes you are comfortable with Linux and a shell (the Linux Deep Dive covers that), know what containers are (the Docker Deep Dive covers that), and understand HTTP and basic networking (the Networking Deep Dive covers that). Kubernetes is needed only for Chapter 11, and that chapter leans on the Kubernetes Deep Dive rather than re-teaching it. No prior Prometheus, Grafana, or tracing experience is assumed — the stack is built from zero.

What You Should Already Know

  • Comfort with Linux and the command line — running commands, editing configs, reading output
  • What a container is and how to run one — the whole lab runs on Docker
  • HTTP basics — requests, responses, status codes, and what a load balancer does
  • Reading simple Python — the instrumentation examples use it, but never cleverly

How the Course Is Built

The fourteen chapters build in a deliberate arc. First the foundations: what observability is, the three signals, and the three costs — cardinality, sampling, retention — that shape every design decision in this field. Then the metrics stack, one layer per chapter: Prometheus, PromQL, code-level instrumentation, and Grafana. Then logs into Loki and traces through OpenTelemetry into Tempo — where the correlated stack pays off and the Saturday mystery falls. Then decisions: alerting, SLOs and error budgets, Kubernetes, and incident response. The last two chapters look outward — profiling, eBPF, synthetics, build-versus-buy, cost — and close with the anti-patterns and the capstone walkthrough of the whole pipeline.

Every topic has the same shape: an opening that states what the thing is and why it exists, the mechanics explained with real configs and queries, a comparison box where two things are genuinely confused, the specific mistakes that cause real outages and real bills, the practices that prevent them, and a short knowledge check. It is written senior-engineer to peer — direct, specific, and honest about tradeoffs.

One stack, built end to end
Prometheus, Grafana, Loki, Tempo, and OpenTelemetry wired together on real hosts — every config on the page runs against Harborline's own lab. You leave able to stand up the stack, not just name its parts.
One company, all the way through
Harborline is instrumented from nothing, chapter by chapter. The metric you create in Chapter 5 feeds the dashboard in Chapter 6, the SLO in Chapter 10, and the incident in Chapter 12 — the story compounds instead of resetting.
Signals answer different questions
Metrics say something is wrong. Logs say what happened. Traces say where in the request path it happened. The book teaches when to reach for which — and how the correlated stack lets you jump between them without re-typing a timestamp.
Data is not the goal — decisions are
Alert on symptoms, page only on user pain, and let error budgets settle the argument between shipping and reliability. Half this book is about what to do with telemetry, because collecting it is the easy half.

Chapter Map

Chapter 1
Foundations — What Observability Is
Monitoring versus observability, the three signals and what each can answer, the instrument-to-alert pipeline the book follows, the three costs that rule every design — and Harborline, flying blind with a Saturday-morning mystery.
Chapter 2
Metrics Fundamentals
What a time series actually is, the four metric types and why histograms win, pull versus push collection, the USE and RED methods for deciding what to measure, and why percentiles — never averages — describe latency.
Chapter 3
Prometheus
The server that runs the metrics world: architecture and the scrape loop, prometheus.yml, exporters for hosts and stores, service discovery and relabeling, what the TSDB writes to disk, and when one server is genuinely enough.
Chapter 4
PromQL
The query language behind every graph and alert: selectors and vectors, rate and increase done right, aggregation that keeps the labels you need, quantiles from histogram buckets, and recording rules that precompute the expensive questions.
Chapter 5
Instrumenting Applications
Metrics from inside the code: the client library in Harborline's bookings service, the four golden signals applied to checkout, label discipline and the cardinality budget, exemplars linking metrics to traces, and exporters for what you can't change.
Chapter 6
Grafana
From queries to a screen someone can read mid-incident: data sources and provisioning, building the checkout dashboard, the design doctrine for 3 a.m. readability, deploy annotations and drill-downs, and dashboards as code in git.
Chapter 7
Logs
From printf to a queryable log stack: structured JSON logging, the agent pipeline with Grafana Alloy, Loki's index-the-labels architecture, LogQL, log hygiene — levels, sampling, PII — and an honest comparison with Elasticsearch.
Chapter 8
Distributed Tracing
One request followed across five services: spans and context propagation, OpenTelemetry and the Collector, head versus tail sampling, Tempo — and the correlated metrics-logs-traces jump that finally solves the Saturday mystery.
Chapter 9
Alerting
Symptoms not causes, page versus ticket: alerting rules on the checkout path, Alertmanager routing, grouping and silences, the notification and on-call layer, and the alert-quality practice that keeps the pager credible.
Chapter 10
SLOs and Error Budgets
Reliability as a number you can argue with: SLI versus SLO versus SLA, choosing good-over-valid indicators for checkout, error budgets as negotiation currency, multi-window burn-rate alerts, and the culture that makes SLOs stick.
Chapter 11
Observability in Kubernetes
The same signals, re-plumbed: what the cluster gives you for free, kube-prometheus-stack and the operator, ServiceMonitors, logs and OTel on Kubernetes, and the dashboards that matter when the cluster itself hurts.
Chapter 12
Incidents and On-Call
The payoff under pressure: on-call as an engineered system, incident roles and severity, the August Saturday incident walked from a 02:40 page to root cause in 38 minutes, blameless postmortems, and the learning loop that feeds it all back.
Chapter 13
The Wider Ecosystem
Beyond the core stack: continuous profiling as the fourth signal, eBPF telemetry without code changes, synthetics and real-user monitoring, the managed platforms weighed honestly, and the cost engineering every stack eventually needs.
Chapter 14
Best Practices and Anti-Patterns
The discipline distilled: the anti-pattern gallery, the maturity path and why traces don't come first, the brownfield instrumentation playbook, testing in production with canaries and flags, and the capstone walkthrough of the whole pipeline.

Disclaimer

This course is an independent educational project created and maintained by Sergey Okinchuk. It is provided for learning and reference purposes only.

No affiliation. This course is not affiliated with, sponsored by, endorsed by, or officially connected to any company, product, or project mentioned — including Prometheus, Grafana Labs, the OpenTelemetry project, the Cloud Native Computing Foundation, Datadog, or any cloud provider. All opinions, interpretations, and recommendations expressed are those of the author.

Trademarks. Product and project names referenced — including "Prometheus", "Grafana", "Loki", "Tempo", "OpenTelemetry", "Kubernetes", "Datadog", "New Relic", and "Dynatrace" — are the property of their respective owners. Use of these names is for identification and educational purposes only and does not imply any endorsement.

Not operational advice. This material teaches durable concepts and practices for observing systems, not turnkey instructions for any specific environment. Configuration examples are simplified for learning. Always consult official documentation and test in a safe environment before making changes to production systems.

Accuracy and currency. The observability ecosystem evolves continuously — tools, defaults, and details drift over time. Facts in this course reflect the author's understanding at the time of writing and may not be current. Always verify against authoritative sources before acting.

No warranty. This material is provided "as is" without warranty of any kind. The author accepts no liability for any loss or damage arising from reliance on the content.