The Prometheus Architecture

Prometheus is a single Go binary with one job, executed on a loop: every few seconds it issues an HTTP GET to each target on its list, parses the plain-text metrics that come back, and appends the samples to a time-series database embedded in the same process. No message queue, no external database, no agent framework. The whole collect-and-store stage of the metrics pipeline is one process reading and writing local disk.

This chapter it lands on obs-01. One Compose file brings up Prometheus on :9090 alongside Grafana, Loki, Tempo, and Alertmanager, but only Prometheus does real work for now — the goal is host- and store-level telemetry, the first real data Harborline has ever had. The pull model is the design decision everything else follows from, so start there.

The Pull Model

Prometheus scrapes; targets never send. A target's entire job is to serve its current state at /metrics when asked — it needs no knowledge of where Prometheus lives, no retry logic, no buffering, no delivery guarantees. The scraper owns the schedule, the timeout, and the target list, which concentrates every collection decision in one config file instead of scattering it across the fleet.

The model's sharpest edge is liveness. Because Prometheus initiates every request, a target that stops answering is visible as a failed scrape rather than silently absent — a push system cannot tell "crashed" from "nothing to report." The cost runs the other way: Prometheus must be able to reach every target over the network, so firewalls become monitoring concerns, and every stored value is up to one scrape interval old.

One Binary, Four Jobs

Inside the process sit four subsystems: the scrape manager running the loops, the TSDB storing samples (topic 15), the PromQL engine answering queries (Chapter 4), and the HTTP API with the web UI — rewritten in the 3.x line, including a query tree view that breaks a slow expression into its parts. There is no daemon farm to operate and no cluster to bootstrap. docker compose up on obs-01 is the entire deployment.

The Text Exposition Format

A scrape response is plain text a human can read: a # HELP line describing the metric, a # TYPE line naming its type, then one metric_name{label="value"} 42 line per series. Run curl http://app-01:9100/metrics and you are looking at exactly what Prometheus sees — the format itself is a debugging tool.

# curl http://app-01:9100/metrics (excerpt)
# HELP node_filesystem_avail_bytes Filesystem space available.
# TYPE node_filesystem_avail_bytes gauge
node_filesystem_avail_bytes{device="/dev/sda1",mountpoint="/"} 3.94e+10
# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 812345.67

The excerpt shows the shape: a gauge reporting the bytes available on app-01's root filesystem, then a counter of CPU seconds by mode, each line one series. OpenMetrics is the standardized descendant of this format, and Prometheus speaks both. For everything in this book, what curl shows you is what gets stored.

The Scrape Loop, Mechanically

Per target, per interval: GET the metrics path, parse the text, attach the identifying job and instance labels, append the samples to the TSDB. Prometheus also synthesizes one extra series per target — up{job,instance} is 1 when the scrape succeeded and 0 when it failed, generated even when the target is dead, because Prometheus produces it, not the target. That makes up the cheapest health check in the ecosystem, available before you write any config beyond the target list.

What Prometheus Refuses to Do

The list of missing features is deliberate: no clustering, no replication, no built-in long-term storage, no dashboards, no log or trace storage. The reasoning is operational — the monitoring system must stay up when everything else is down, and a single node reading local disk has the shortest possible failure chain. Every dependency Prometheus doesn't have is a dependency that can't take monitoring down with it.

The gaps are filled outside the binary, on purpose. Grafana owns dashboards (Chapter 6), Alertmanager owns notification routing (Chapter 9), and the remote-storage ecosystem owns durability past one disk (topic 16). Prometheus stays a scrape-store-query engine and nothing more.

Where It Sits in the Pipeline

In the pipeline — instrument, collect, store, query, visualize, alert — Prometheus is the collect and store stages for the metrics signal, plus the query engine on top. It is one lane of three; Loki and Tempo sit idle on obs-01 waiting for Chapters 7 and 8. And every design choice you just met is a position on the three costs: pull plus labels sets the cardinality model, the local TSDB with 15-day default retention sets the retention model, and the scrape interval is the sampling knob for the whole lane.