Service Discovery and Relabeling

A static target list is a lie waiting to happen. The moment hosts autoscale, move, or multiply, prometheus.yml drifts from what is actually running, and the drift is invisible — nothing alerts on a host that was never added. Service discovery makes the target list a live query against the infrastructure itself, and relabeling is the machinery that shapes what discovery returns: which targets to keep, what to call them, and which labels they carry into every series they produce.

Harborline's six named VMs do not autoscale, and honesty about that matters as much as the mechanism. This topic climbs the discovery ladder only to the rung that fits — a file Ansible writes — and then spends its real effort on relabeling, the rule language that runs everywhere in Prometheus and finally cracks the blackbox_exporter puzzle topic 13 left open.

The Discovery Ladder

static_configs is the bottom rung: honest for a handful of stable VMs, which is exactly what Harborline has. One rung up, file_sd_configs points Prometheus at JSON or YAML files it watches and re-reads on change, no reload required — anything that can write a file becomes a discovery mechanism, which in practice means Ansible, Terraform output, or a cron job. At the top sit the native mechanisms — ec2_sd, gce_sd, consul_sd, dns_sd, kubernetes_sd — where the platform's own API is the source of truth and targets appear and vanish as the platform sees them.

Mara picks file_sd, generated by the same Ansible that provisions node_exporter, and the choice is the point: the target list becomes a build artifact of provisioning rather than a second source of truth maintained by hand. Six hosts do not justify Consul.

# targets/node.json — written by Ansible, watched by Prometheus
[
  {
    "targets": ["app-01:9100", "app-02:9100", "db-01:9100",
                "cache-01:9100", "mq-01:9100", "obs-01:9100"],
    "labels": {"env": "prod"}
  }
]

What Discovery Returns

Discovery does not return addresses; it returns labeled candidates. Each discovered target arrives carrying __address__ plus a pile of __meta_* labels describing it — the EC2 instance's tags, the Consul service's metadata, the custom labels from the file above. Every label whose name starts with __ is dropped before storage, so this metadata exists only during relabeling. Use it there or lose it: anything from discovery you want to see at query time must be copied into a normal label, deliberately.

Shaping Targets Before the Scrape

relabel_configs is a pipeline of rules applied to every candidate target before any scrape happens. keep and drop filter on a regex — scrape only targets whose __meta_ec2_tag_env is prod. replace rewrites labels, including __address__ itself. labelmap copies whole families of __meta_* labels into real ones in one rule. Every rule has the same shape — source labels, regex, action — and a target dropped here is never contacted at all, which makes this the place where scrape cost is controlled.

One behavior bites everyone once: Prometheus relabel regexes are fully anchored. regex: web matches exactly the string web and nothing else — not web-01, not web-search. The partial match you assumed needs web.*, and this single fact explains most "my keep rule keeps nothing" tickets ever filed.

The Blackbox Trick, Solved

Topic 13 ended on a cliffhanger: scrape blackbox_exporter directly and you monitor the exporter process while zero probes run. The fix is the canonical three-step relabel. Copy __address__ — which the target list filled with the probe destination, https://harborline.example — into the __param_target URL parameter; copy it again into instance so the stored series name the probed site; then replace __address__ with obs-01:9115, where the exporter actually listens.

# the canonical three-step blackbox relabel
  - job_name: "blackbox-http"
    metrics_path: /probe
    static_configs:
      - targets: ["https://harborline.example"]
    relabel_configs:
      - {source_labels: [__address__], target_label: __param_target}
      - {source_labels: [__address__], target_label: instance}
      - {target_label: __address__, replacement: "obs-01:9115"}

Read the flow once more, because it generalizes to every exporter that probes on behalf of others: Prometheus scrapes the exporter, the exporter probes the site named in the URL parameter, and the resulting series carry instance="https://harborline.example". The middle step is the one people skip. Without rewriting instance, every probe target would end up labeled with the exporter's address — add a second probed site and the two become indistinguishable duplicates.

The Last Gate Before Storage

metric_relabel_configs applies the same rule language at a different point: after the scrape, to every sample, just before ingestion. drop kills a known-noisy metric family; labeldrop strips a high-cardinality label an exporter attached that you cannot fix at the source. It is the final defense against a cardinality detonation — but the scrape already happened, so the savings are storage and memory, never scrape cost. Filter targets in relabel_configs; filter metrics here.

The Action Inventory

The full action list: replace, keep, drop, labelmap, labeldrop, labelkeep, hashmod for sharding targets across servers, lowercase and uppercase, and keepequal/dropequal for label comparison. Six of these cover 95% of configs in the wild. When a rule misbehaves, the UI's Service Discovery page shows every target's labels before and after the pipeline — reading both columns beats re-deriving regex behavior in your head.