Observability Deep Dive
by Sergey Okinchuk
Chapter 10What Kubernetes Gives You

Chapter Eleven · Observability in Kubernetes

Observability in Kubernetes

Harborline lands on a managed Kubernetes cluster — same five services, same containers, new substrate. This chapter re-plumbs the stack from Chapters 3–10 the Kubernetes way: an operator-managed Prometheus, ServiceMonitors instead of static configs, Alloy as a DaemonSet, and webhook-injected OpenTelemetry. The checkout SLO crosses the migration byte-for-byte unchanged — the platform changed, the discipline didn't.

6 topics

Harborline's five services move to a managed Kubernetes cluster, and every layer of the observability stack has to answer the same question: what does this look like when pods churn by the minute? The cluster arrives with telemetry of its own — cAdvisor inside every kubelet, control-plane /metrics, events — and with a hard boundary: none of it knows a checkout from a health check. This chapter assumes the Kubernetes Deep Dive course and never re-teaches the platform; it builds the durable stack underneath kubectl top and kubectl logs.

Six topics make the migration: the built-in telemetry surface and its application-shaped hole; kube-prometheus-stack and the operator pattern; ServiceMonitors and the three selector layers that fail silently; node-level log shipping with Alloy as a DaemonSet; the dashboards that separate a starved workload from a slow application; and the OTel Operator, where the chapter's thesis lands — the checkout SLO survives the replatform unchanged.

The migration ledger — the plumbing changed, the discipline didn't
Changeddiscovery, agents, plumbing
Discovery becomes ServiceMonitors instead of static scrape configs. Agents become DaemonSets — one Alloy and one Collector per node. Plumbing becomes webhook-injected SDKs and operator-generated config, and every signal gains k8s.* metadata.
Unchangedsignals, SLOs, dashboards
The signals and the harborline_ metric names, the dashboards, and the checkout SLO — 99.9% availability, 95% under 500 ms, 28-day window — cross the migration byte-for-byte.

Topics in This Chapter

Topic 54
What Kubernetes Gives You
Every kubelet embeds cAdvisor, the control plane speaks Prometheus natively, and kube-state-metrics turns object state into series — yet none of it knows a checkout from a health check. Map the built-in surface precisely, so the rest of the chapter knows what still has to be added.
Built-ins
Topic 55
kube-prometheus-stack
One Helm install deploys the operator, Prometheus, Alertmanager, Grafana, and the exporters — and shifts the interface from editing prometheus.yml to declaring CRDs. What it cannot generate is anything about Harborline: the SLOs and recording rules remain your work.
Operator
Topic 56
Service Discovery in K8s
A ServiceMonitor per service replaces static scrape configs, and a new pod is scraped within one interval of appearing. The price is three layers of label selection whose only failure mode is silence — plus the fixed debugging walk that finds the broken layer.
Discovery
Topic 57
Logs in Kubernetes
The runtime writes every container's stdout to node files that vanish with the pod; kubectl logs is a 50Mi window, not a log system. One Alloy per node ships everything to Loki with Kubernetes metadata attached — and a multiline stage keeps tracebacks whole.
Logging
Topic 58
Kubernetes Dashboards That Matter
Is the workload starved or the application slow? The requests/limits/usage triangle, throttling ratios, OOM-kill signatures, and pending pods answer that in a fixed order — resource pressure first, application blame second.
Dashboards
Topic 59
OTel in Kubernetes
The OTel Operator injects auto-instrumentation with one pod annotation and runs Collectors as node agents feeding a tail-sampling gateway. The chapter's closing ledger: the plumbing changed everywhere, and the checkout SLO didn't move a byte.
OTel