kube-prometheus-stack

kube-prometheus-stack is the Helm chart that turned "assemble Prometheus on Kubernetes by hand" into one install: the Prometheus Operator, a Prometheus and an Alertmanager it manages, node_exporter as a DaemonSet, kube-state-metrics, and Grafana pre-loaded with dashboards and alert rules for the cluster itself. Everything Mara ran via Compose on obs-01 arrives as a single release.

The operator's real contribution is a shift in interface. You stop editing prometheus.yml and start declaring intent through CRDs; the operator generates the config and reloads Prometheus for you. What it cannot generate is anything about Harborline — your recording rules, your checkout SLO, your pages remain your work.

The One-Install Stack

The chart lives in the prometheus-community Helm repository and one helm install deploys the whole lineup: the operator, a Prometheus that can run 2 replicas, Alertmanager, Grafana, node_exporter on every node, and kube-state-metrics. Scraping of the kubelet, cAdvisor, and the control-plane components is preconfigured, so the entire built-in surface from the previous topic flows into the TSDB within one scrape interval of the install finishing.

The Operator Pattern, Aimed at Monitoring

The Prometheus Operator watches a family of CRDs — Prometheus, Alertmanager, ServiceMonitor, PodMonitor, PrometheusRule, Probe, ScrapeConfig, AlertmanagerConfig — and reconciles reality toward them. It generates the actual prometheus.yml from those resources, and a config-reloader sidecar hot-reloads Prometheus on every change. This is the same declare-and-reconcile loop the Kubernetes Deep Dive course teaches, aimed at monitoring.

Declare CRDsintent, in git

→

Operator watchesreconcile loop

→

Generates configprometheus.yml

→

Reloads Prometheushot, no restart

Reconciliation cuts both ways. Because the operator owns the generated config, editing it directly — or the Secret that holds it — accomplishes nothing: the operator writes its own version back within moments, without an error. The CRDs are the only interface that sticks, which is exactly what makes monitoring config reviewable in git instead of drifting on a server.

The Prometheus CRD as the Server Spec

Replicas, retention, resource requests, external labels, and which ServiceMonitors this server selects all live in one Prometheus resource. Extending retention from 15 to 30 days is a field edit and a reconcile, not an SSH session — the snippet below declares a 2-replica server keeping 30 days on a 100 Gi volume.

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata: {name: harborline}
spec:
  replicas: 2
  retention: 30d
  storage:
    volumeClaimTemplate:
      spec:
        resources: {requests: {storage: 100Gi}}

Bundled Dashboards and Rules

The chart ships the kubernetes-mixin dashboards — compute resources by namespace, pod, and node — plus dozens of alert rules: KubePodCrashLooping, CPUThrottlingHigh, NodeFilesystemSpaceFillingUp. One of them, Watchdog, fires permanently on purpose: it is a dead man's switch, and its absence at the receiving end is the signal that the alerting pipeline itself has died. That is a platform-monitoring baseline Mara does not have to write.

The CRD Upgrade Trap

Helm installs CRDs on first install but does not touch them on helm upgrade. Each major chart version therefore requires applying the new CRDs manually — kubectl apply --server-side from the matching release — before upgrading the chart. Skip it and the operator hits unknown fields and quietly stops reconciling parts of the config while every pod still reports Running.

What You Still Own

The operator automates config plumbing, not judgment. The harborline_ recording rules, the checkout SLO burn-rate alerts from Chapter 10, and the bookings and payments dashboards arrive as PrometheusRule resources and dashboard ConfigMaps that Mara writes, reviews, and version-controls exactly as before. The chart monitors the platform; monitoring the product is still your job.