Service Discovery in K8s

On the VMs, adding a scrape target meant editing prometheus.yml and reloading — workable for six hosts, hopeless for a cluster where pods appear and vanish by the minute. The operator replaces hand-written scrape configs with two CRDs: a ServiceMonitor says "scrape the Services matching these labels on this named port," a PodMonitor says the same about Pods directly, and Prometheus's Kubernetes service discovery keeps the target list current as pods churn.

The price of the indirection is a new failure mode. There are now three layers of label selection between Prometheus and a pod, and a mismatch at any one of them produces not an error but silence — a target list that is simply, quietly empty.

From Static Config to Selectors

One ServiceMonitor per Harborline service replaces the old static scrape_configs stanza. Its spec.selector.matchLabels picks the Service, endpoints[].port names the port to scrape, and every pod behind that Service becomes a target automatically — deploy a fourth bookings replica and it is scraped within the 15 s scrape interval, no config change anywhere. The resource below is the entire scrape configuration for bookings.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: bookings
  labels: {release: kube-prometheus-stack}
spec:
  selector:
    matchLabels: {app: bookings}
  endpoints:
    - port: metrics

The Named-Port Contract

A ServiceMonitor references the Service's port name — metrics above — never its number. An unnamed port, or a name that does not match, means the endpoint is skipped without complaint: the pod is fine, the Service is fine, the target simply does not exist. That is why every Service port exposing /metrics gets a name as a matter of policy, not taste.

Three Layers of Selection

Selection stacks three deep. Prometheus selects ServiceMonitors, via serviceMonitorSelector and a namespace selector on the Prometheus CRD; the ServiceMonitor selects Services by label; the Service selects Pods. Each layer is an independent label match, and each can break independently of the other two.

The top layer carries the single most common trap. kube-prometheus-stack defaults to selecting only ServiceMonitors that carry the Helm release: <name> label — the release: kube-prometheus-stack line in the snippet is load-bearing. Apply a ServiceMonitor without it and Prometheus never selects it: zero targets appear, and nothing anywhere reports an error.

PodMonitor for the Serviceless

notifier consumes RabbitMQ and has no HTTP port worth a Service. A PodMonitor scrapes its metrics port by pod labels directly, no Service in between — the right shape for queue workers, batch jobs, and sidecars, where inventing a dummy Service would misdescribe the workload just to satisfy the tooling.

Relabeling as Metadata Policy

Discovery attaches namespace, pod, service, and container labels to every series it creates. metricRelabelings on the ServiceMonitor can drop high-cardinality series at ingestion, before they ever reach the TSDB — the Chapter 3 cardinality discipline, now enforced per-service at the CRD instead of globally in one config file.

The Silent-Target Debugging Path

Because every layer fails silently, debug in a fixed order, starting from the Prometheus UI: Status → Targets. First question: is the target absent, or present but down? If absent, walk the selector chain top-down — does the ServiceMonitor carry the release label, does its namespace fall inside the namespace selector, do its matchLabels match the Service's labels, does the named port exist on that Service?

Present but down is a different family: the selection chain worked and the scrape itself fails. That is a wrong metrics path, a TCP connection refused, or a NetworkPolicy blocking traffic from Prometheus's namespace — network problems, not label problems, and the Networking Deep Dive instincts apply.