What Kubernetes Gives You
Kubernetes arrives with a telemetry surface already running before you install anything: every kubelet embeds cAdvisor and serves per-container resource metrics, the control-plane components expose Prometheus-format /metrics endpoints, and the cluster records an event for every scheduling decision, restart, and OOM kill. Harborline's new cluster can report what every container consumes from the moment the first pod lands.
What it cannot report is anything about Harborline. No request rates, no error ratios, no harborline_bookings_total — the built-in surface observes containers, not checkouts. This topic maps that surface precisely, so the rest of the chapter adds exactly what is missing and re-plumbs nothing that is already there.
cAdvisor Inside the Kubelet
Every kubelet embeds cAdvisor and serves its measurements at /metrics/cadvisor: container_cpu_usage_seconds_total, container_memory_working_set_bytes, filesystem and network counters, each labeled with namespace, pod, and container. That is resource truth for every container on the node with nothing to install — the role node_exporter played per-VM, absorbed into the platform. Prometheus scrapes each kubelet and the cluster-wide resource picture assembles itself.
Control-Plane Metrics
The platform also observes itself. The kubelet's own /metrics, the API server, the scheduler, the controller-manager, etcd, and CoreDNS all speak Prometheus exposition natively; apiserver_request_duration_seconds and the scheduler's queue depths are how you notice the control plane degrading before workloads do. kube-prometheus-stack, coming in the next topic, scrapes all of them out of the box — this layer costs you nothing but retention.
kube-state-metrics: Objects as Metrics
cAdvisor measures usage; nothing so far reports state. kube-state-metrics is a separate deployment that watches the API server and re-publishes object state as metrics: kube_pod_status_phase, kube_deployment_status_replicas_available, kube_pod_container_status_restarts_total, kube_pod_container_resource_requests. It measures nothing itself — it translates what Kubernetes already knows into something PromQL can graph and alert on.
Restart counts are the canonical payoff. The query below turns a container that dies quietly every few hours into an alertable number, something no amount of cAdvisor data reveals because usage looks normal right up until the kill.
# containers restarted in the last hour, per workload
increase(
kube_pod_container_status_restarts_total{namespace="harborline"}[1h]
) > 0
Events: the Ephemeral Signal
Events explain why something happened: FailedScheduling names the constraint that could not be satisfied, Backoff shows the crash loop, OOMKilling names the process the kernel shot. They live in etcd with a default TTL of 1 hour (--event-ttl), so anything you want after lunch must be exported — as log lines through the pipeline in the logs topic, or as metrics via an events exporter. During an incident, kubectl get events shows the last hour, not the history.
metrics-server Is Not Monitoring
metrics-server is the component most often mistaken for a monitoring system. It feeds the Metrics API that kubectl top and the HorizontalPodAutoscaler read — current values only, no history, no PromQL. It is an autoscaling input, not an observability backend, and installing it changes nothing about which questions the cluster can answer about last night.
The Hole in the Middle
Nothing above knows a checkout from a health check. A pod at 20% CPU returning HTTP 500 to every customer looks perfectly healthy to cAdvisor, kube-state-metrics, and metrics-server alike. Harborline's request rates, error ratios, and harborline_checkout_duration_seconds still come from the Chapter 5 instrumentation, unchanged — Kubernetes observes containers; you observe the application.
cAdvisor (inside the kubelet) measures resource usage: CPU seconds, working-set bytes, per container. Scrape it into Prometheus for resource truth.
kube-state-metrics re-publishes object state from the API server — phases, replica counts, restart counts, requests and limits — and measures nothing itself. Scrape it too; it is the raw material of Kubernetes alerting.
metrics-server serves live usage to kubectl top and the HPA with zero history. Leave it to autoscaling; it has no role in your monitoring stack.
- Treating metrics-server as the monitoring system and skipping Prometheus — it retains nothing, so the first "what did memory look like before the crash at 02:40" question has no answer.
- Alerting on
container_memory_usage_bytesinstead ofcontainer_memory_working_set_bytes— usage includes reclaimable page cache, so pods look near their limit while the kernel would free those pages long before an OOM kill fires on the working set. - Reading
kubectl get eventsduring an incident and assuming the history is complete — with the 1-hour default TTL, this morning'sFailedSchedulingburst is already gone unless it was exported. - Expecting pod restart counts or OOM kills to appear in application metrics — they only exist in kube-state-metrics; a cluster without it is blind to the most common Kubernetes failure signatures.
- Assuming the built-in surface covers the app because "Kubernetes has metrics" — it has container metrics; a pod idling at 20% CPU while failing every request looks healthy to all of it.
- Run kube-state-metrics in every cluster and scrape it into Prometheus — restarts, phases, and requests/limits are the raw material of every Kubernetes alert worth having.
- Scrape
/metrics/cadvisorfrom every kubelet for resource truth, and keep application metrics as a separate, explicitly instrumented concern. - Export events to a durable store — one log line per event through the log pipeline — so the why-did-scheduling-fail trail outlives the 1-hour TTL.
- Alert memory pressure on
container_memory_working_set_bytesrelative to the limit — it's the non-reclaimable estimate the kubelet's own eviction manager keys off, and the closest metric you have to what will actually run a container out of headroom.
Knowledge Check
A payments pod restarts every three hours, but its CPU and memory panels look normal. Which component makes that visible as an alertable number?
- cAdvisor, via the kubelet's /metrics/cadvisor endpoint
- kube-state-metrics
- metrics-server
- Kubernetes events, read with kubectl get events
Why does alerting memory pressure on container_memory_usage_bytes produce false alarms?
- It only reports node totals, not per-container values
- It is sampled too infrequently to catch memory spikes
- It counts reclaimable page cache that the kernel would free before an OOM kill
- It is measured in pages rather than bytes, so limits never match on the memory dashboard
This morning's FailedScheduling burst is missing from kubectl get events at noon. Why?
- Events are only shown for pods currently in a Running phase
- The scheduler deletes its events once the pod is finally placed
- The events were stored on the node that emitted them, and it was drained
- They expired — events have a default 1-hour TTL unless exported
Every built-in panel for a bookings pod is green — CPU 20%, memory well under limit, zero restarts — yet customers see errors on every request. What does this demonstrate?
- Built-in telemetry has no application signal; request-level data still requires instrumentation
- cAdvisor is misconfigured and failing to collect error metrics
- kube-state-metrics is lagging behind the API server
- The liveness probe interval is too long to catch the errors, so the failing requests slip past the check
You got correct