Metrics and Monitoring
Metrics are the numeric time series that tell you how a cluster and its workloads are behaving — request rate, latency, error count, resource use. The de facto standard is Prometheus, which scrapes metrics from targets, stores them, and answers queries; Grafana visualizes them and Alertmanager fires alerts.
There is a common point of confusion to clear first: the metrics-server that the autoscaler uses is not Prometheus. They solve different problems, and conflating them leads to surprises.
The Prometheus Model: Pull and Scrape
Prometheus is pull-based: it periodically scrapes an HTTP /metrics endpoint exposed by each target. Applications expose their own metrics; exporters expose metrics for things that can't (node-exporter for hardware, kube-state-metrics for Kubernetes object state). Prometheus discovers targets via the Kubernetes API and stores everything as time series labeled by dimensions. Pull means Prometheus controls the cadence and can tell when a target is down (the scrape fails).
metrics-server vs Prometheus
These are different tools. metrics-server is a lightweight component that provides current CPU/memory for kubectl top and for the HPA — it keeps no history and is not a monitoring system. Prometheus is a full monitoring system with storage, a query language, and alerting. You need metrics-server for autoscaling; you need Prometheus (or equivalent) for actually observing the system. Installing one does not give you the other.
| Tool | Purpose | History |
|---|---|---|
| metrics-server | Current CPU/mem for HPA and kubectl top | None — point-in-time |
| Prometheus | Monitoring: scrape, store, query, alert | Yes — time-series storage |
PromQL, Recording Rules, and Alerting
PromQL is Prometheus's query language for slicing and aggregating time series — error rate per service, p99 latency, saturation. Recording rules precompute expensive queries; alerting rules fire when a condition holds, routed by Alertmanager to the right channel with grouping and silencing. Good alerting targets symptoms users feel (high error rate, latency) over causes, so you are paged for problems, not noise.
Cardinality: The Trap
Every unique combination of label values is a separate time series, and high-cardinality labels — user IDs, request paths, anything unbounded — can explode the series count and overwhelm Prometheus's memory and storage. This is the classic way to take down a monitoring system. Keep labels to bounded, meaningful dimensions; never label metrics with unbounded values. kube-state-metrics is worth knowing too: it turns the state of Kubernetes objects (Deployment replicas desired vs ready, Pod phase) into metrics you can alert on.
metrics-server — current CPU/memory only, for the HPA and kubectl top. No history, not monitoring.
Prometheus — scrape + time-series storage + PromQL + alerting. The actual monitoring system.
- Confusing metrics-server with Prometheus — installing one expecting the capabilities of the other.
- Labeling metrics with high-cardinality values (user IDs, paths), exploding series and crashing Prometheus.
- Alerting on causes and internal counters instead of user-facing symptoms, producing noise.
- No retention or federation plan, so metrics either fill disk or disappear too soon.
- Scraping nothing useful — deploying Prometheus without instrumenting apps or running exporters.
- Use metrics-server for autoscaling and Prometheus (or a managed equivalent) for monitoring.
- Keep metric labels to bounded, meaningful dimensions; never use unbounded label values.
- Alert on symptoms users feel (errors, latency, saturation), and route with Alertmanager.
- Use kube-state-metrics and node-exporter to cover Kubernetes object state and node hardware.
- Plan retention and, at scale, long-term storage or federation for historical metrics.
Knowledge Check
How does Prometheus collect metrics?
- It pulls — periodically scraping an HTTP /metrics endpoint on each target
- Targets push their samples to it over a UDP statsd protocol
- It reads the metric series straight out of the etcd datastore
- The kubelet on each node collects and forwards every Pod's metrics automatically
What is the difference between metrics-server and Prometheus?
- metrics-server gives current CPU/mem for the HPA with no history; Prometheus adds storage, PromQL, and alerting
- They are the same underlying component, just shipped under two different project names, charts, and install paths
- Prometheus is used only for driving horizontal autoscaling decisions
- metrics-server stores long-term history for trend queries
What causes a Prometheus cardinality explosion?
- Labeling metrics with high-cardinality, unbounded values like user IDs or request paths
- Scraping too few targets at an overly long interval
- Using recording rules to precompute frequent queries
- Defining alerting rules on a request latency histogram with several fixed bucket boundaries
You got correct