Configuration — prometheus.yml

Everything Prometheus does is declared in one YAML file. prometheus.yml holds the global defaults, the list of scrape jobs, and pointers to rule files and Alertmanager — change what Prometheus monitors and this file is what you change. There is no database of checks and no UI wizard; the config is text, which means it diffs, reviews, and versions like any other code.

Mara's first version is a dozen lines: one job, six statically listed targets. It takes Harborline from zero telemetry to six hosts reporting every 15 seconds, and the two labels it establishes — job and instance — are how you will identify every series for the rest of the book.

File Anatomy

Four top-level blocks. global sets defaults every job inherits: scrape_interval, scrape_timeout, evaluation_interval, and the external_labels that stamp every sample leaving this server. scrape_configs is the heart of the file — the list of jobs. rule_files and alerting point at recording rules (Chapter 4) and Alertmanager (Chapter 9); both stay empty this chapter. A minimal working config is global plus one entry in scrape_configs.

A scrape_config, Line by Line

job_name becomes the job label on every series the job produces — Harborline names jobs after the service or exporter, job="node", job="postgres". metrics_path defaults to /metrics and scheme defaults to http, so most jobs set neither. static_configs lists targets as host:port strings, and each target's address becomes its instance label. Between them, job and instance pin down where any sample came from.

Intervals and Timeouts

The shipped default scrape_interval is 1 minute with a 10-second scrape_timeout; 15 seconds is the common production interval and what Harborline sets in global. The timeout must be shorter than the interval — Prometheus rejects the config otherwise. And 200 targets do not get hit in the same millisecond: Prometheus deliberately staggers each target's scrape start across the interval, spreading load on both the network and itself.

Mara's First Scrape

One job, six hosts, and the env="prod" label attached once in the config instead of once per host. This is the entire file that gives Harborline its first telemetry:

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ["app-01:9100", "app-02:9100", "db-01:9100",
                  "cache-01:9100", "mq-01:9100", "obs-01:9100"]
        labels:
          env: "prod"

Sixty seconds after the reload, up{job="node"} returns six series with value 1 and the Targets page shows six green rows. That is Harborline's first real data: CPU, memory, disk, and network for every host, every 15 seconds. Nothing about the five services yet — they stay dark until Chapter 5.

Validation and Reload

promtool check config prometheus.yml parses the file and catches both syntax and semantic errors — an unknown field, a timeout longer than the interval — before anything reaches the server. The running server picks up changes via SIGHUP or, with --web.enable-lifecycle set, an HTTP endpoint:

promtool check config prometheus.yml
curl -X POST http://obs-01:9090/-/reload

The difference between reload and restart matters most when the file is broken. A reload validates the new file and, if it is bad, rejects it while the old config keeps serving. A restart parses the broken file at startup and exits — the same typo a reload shrugs off becomes a monitoring outage.

The Limits of static_configs

A hardcoded list is the right call for six long-lived VMs, and exactly wrong for fleets where targets churn — autoscaling groups, containers, anything with a lifespan shorter than a config review. Topic 14 replaces the static list with service discovery. Everything else in this file — intervals, labels, the job structure — carries over unchanged.