Kubernetes Dashboards That Matter

"The cluster feels slow" is an accusation without coordinates, and Kubernetes gives it a specific shape: is the workload starved — throttled, OOM-killed, unschedulable — or is the application actually slow? The two look identical from a latency graph, and they have opposite fixes.

The dashboards that matter answer that question in order — resource pressure first, application blame second — built from the requests/limits/usage triangle that kube-state-metrics and cAdvisor together make visible. Mara builds three views on top of it: capacity, saturation, and failure signatures.

The Requests/Limits/Usage Triangle

Three numbers exist per container: what it asked for (kube_pod_container_resource_requests), what it may not exceed (kube_pod_container_resource_limits), and what it actually uses — cAdvisor's CPU rates and container_memory_working_set_bytes. The relationships carry the meaning, not any single line: usage far below requests is cluster money burning quietly, and usage pressed against the limit is an incident forming. Panel all three on one graph per service, because the gaps are the story.

The Throttling Ratio

A pod at its CPU limit is not killed — it is stalled. The CFS quota inserts pauses into request handling, and the tell is the ratio of throttled periods to total periods, not the usage graph: sustained above roughly 25%, throttling is adding latency to checkout while every CPU panel stays comfortably green. The query below draws that ratio per bookings pod.

# share of CFS periods in which the container was throttled
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{container="bookings"}[5m]))
  /
sum by (pod) (rate(container_cpu_cfs_periods_total{container="bookings"}[5m]))

OOM Kills and Restart Signatures

Working set against the limit is the memory headroom gauge, because working set is the non-reclaimable estimate Kubernetes' own eviction logic — and, in practice, an OOM kill — both key off. The failure signature is a pair: kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} plus a rising kube_pod_container_status_restarts_total. A restart count that increments every few hours while probes keep the Service green is how a memory leak hides on Kubernetes — the platform keeps repairing the symptom, and every repair drops the requests in flight.

Pending Pods and Headroom by Requests

kube_pod_status_phase{phase="Pending"} held for more than a couple of minutes means the scheduler cannot place workloads, and that is a capacity fact, not a glitch. It belongs on the board as a stat panel that is normally zero.

Its companion panel is per-node allocatable minus the sum of requests, because the scheduler places on requests, not on usage. A cluster can idle at 40% CPU and still refuse to schedule anything — the requests are spoken for even where the cycles are not. Headroom by requests is the only headroom that predicts whether the cluster can take Saturday's traffic.

Mara's Order of Interrogation

The "cluster feels slow" runbook is one dashboard walk in a fixed order: node saturation first (load, memory pressure), then throttling ratios per service, then OOM kills and restart rates, then pending pods — and only then the Chapter 6 checkout dashboard. Ten minutes in that order prevents the hour spent blaming bookings for a starved node. The ordering works because resource pressure mimics application slowness perfectly, and an application cannot prove its innocence from inside a throttled container.

Mixin Boards and Your Own

The bundled kubernetes-mixin boards already draw the triangle well — compute resources by cluster, namespace, and pod — so Mara adopts them for capacity and builds only the Harborline-specific saturation view: throttling ratio, OOMKilled terminations, restart rate, and pending pods for the five services. Adopt the commodity views, own the specific ones — the Chapter 6 dashboard economy, applied to a platform that ships its own commodity.