Topic 08

Pull vs Push

Collection

Prometheus pulls: every target exposes its current numbers on an HTTP endpoint — /metrics, in a simple line-based text format (also expressible in the related OpenMetrics standard) — and the server fetches them on a schedule it controls. Push systems invert this: the application or an agent sends measurements to a collector, on the sender's schedule.

The choice looks like plumbing. It is actually a stance on who knows what should exist — and that stance decides how you discover targets, how you detect death, and what a firewall does to your monitoring.

Pull, Mechanically

Each Harborline service will expose /metrics — bookings on :8000, search on :8001, payments on :8002 — once Chapter 5 instruments them, with node_exporter on :9100 on every host from Chapter 3 onward. Prometheus on obs-01 will GET each endpoint every 15 seconds. The configuration below is the shape that collection contract will take for two of those jobs — the server owns the cadence, and one file on obs-01 declares what the fleet should be.

# prometheus.yml on obs-01 — the server's model of the fleet (the shape it will take)
scrape_configs:
  - job_name: "payments"
    static_configs:
      - targets: ["app-01:8002", "app-02:8002"]
  - job_name: "node"
    static_configs:
      - targets: ["app-01:9100", "app-02:9100", "db-01:9100"]

That declared model is the load-bearing idea. A pull server does not passively receive whatever shows up; it holds an explicit list of what exists and goes checking. Debugging is correspondingly plain: curl app-01:8002/metrics shows you exactly what the next scrape will see, no tooling required.

Service Discovery

The target list can come from static config, DNS, a cloud provider's API, or Kubernetes. Harborline starts with six hosts and five services declared by hand in prometheus.yml, which is fine at this size and wrong at any larger one — hand-edited lists drift from reality. The general point stands regardless of mechanism: a pull system must be told, or must discover, what exists. That knowledge is not overhead. It is precisely what makes the next section possible.

The up Metric

Every scrape writes a synthetic series: up{job="payments",instance="app-01:8002"} — 1 if the scrape succeeded, 0 if it failed. Because the server knows what it expected to reach, an unreachable target is a first-class, unambiguous signal, recorded in the same database as everything else and alertable with a one-line rule.

Push systems cannot have this for free. When a sender goes quiet, silence is undecidable between dead, idle, and firewalled — and any absence detection must be hand-built, per sender, by someone who remembered the sender exists. The up metric is the single strongest argument for pull, and it costs nothing.

Pull vs push — who initiates, and who knows what should exist

PullPrometheus

The server initiates — it scrapes targets it already knows about on a cadence it owns. Because it holds an explicit list of what should exist, an unreachable target records up == 0 for free.

PushStatsD, Graphite, InfluxDB, OTLP

The sender initiates — it transmits on its own schedule. The receiver knows nothing about what should exist, so a silent sender is undecidable between dead, idle, and firewalled, and absence detection must be hand-built.

Push and Its Legitimate Territory

StatsD, Graphite, InfluxDB, and OTLP are push-native, and push wins wherever the server cannot reach the target: processes behind NAT or a customer's firewall, IoT fleets, agents reporting out of third-party networks. Outbound HTTPS traverses what inbound scrapes never will. The consequences travel with the model — the sender owns the cadence, the receiver must absorb whatever burst arrives, and liveness is nobody's job by default. Push is not wrong; it is a different contract, and you should sign it only for what pull cannot reach.

The Pushgateway's Narrow Niche

One shape of workload genuinely defeats pull. Suppose Harborline runs a nightly booking-data cleanup cron: it lives for 40 seconds and exits — gone before any 15-second schedule catches it mid-flight. So it pushes its final counters to the Pushgateway, and Prometheus scrapes the gateway.

The niche is deliberately this narrow, because the gateway inverts pull's guarantees: pushed metrics persist until explicitly deleted, and up now monitors the gateway, not the job. Route a normal service through it and you have rebuilt push's silence problem inside a pull system — with extra steps.

Pull vs Push

Pull (Prometheus) — the server scrapes targets it knows about. You get the up metric, central control of cadence, and /metrics endpoints you can inspect with curl; the price is that the server must be able to reach every target. Default here for long-running services.

Push (StatsD, Graphite, InfluxDB, OTLP) — senders transmit on their own schedule. It crosses NAT and firewalls and fits short-lived processes; the price is that target death is silence and ingest load is the senders' choice, not yours. Choose it for what pull cannot reach.

Common Mistakes

Routing a normal long-running service's metrics through the Pushgateway — the service dies, its last pushed values persist indefinitely, dashboards keep rendering a healthy service that no longer exists, and up cheerfully reports that the gateway is fine.
Never deleting stale Pushgateway groups — metrics from a job decommissioned months ago keep evaluating in alert rules until someone eventually gets paged by a ghost.
Treating silence as health in a push pipeline — "no data" is not "no problem"; without an explicit absence alert (absent() in PromQL), a dead sender is indistinguishable from a quiet one for as long as nobody happens to look.
Hand-maintaining a static target list that drifts from reality — the app-03 added under load next summer serves traffic for weeks while being scraped by nobody, which is precisely the failure service discovery exists to prevent.
Exposing /metrics on a public interface — the endpoint enumerates your internal architecture, software versions, and hostnames for anyone who asks. Bind it to the internal network or put auth in front.

Best Practices

Default to pull for every long-running service, and reserve push for what genuinely cannot be scraped — short-lived jobs, NAT-bound agents, third-party networks.
Alert on up == 0 per job from day one — it is the cheapest alert in the entire stack and catches whole categories of failure before any application metric can.
Use the Pushgateway strictly for service-level batch jobs, and alert on push_time_seconds going stale, so a cron that silently stops running is caught within a day.
Drive the target list from service discovery — file-based at minimum, API-backed when the platform offers it — rather than hand-edited host lists.

Comparable toolsPush lineage StatsD/Graphite and InfluxDB — senders transmit on their own scheduleOpenTelemetry Collector pipelines are push-based OTLP end to endDatadog the agent pushes to SaaS — which is why it works from anywhere with outbound HTTPSZabbix supports both directions: passive (pull) and active (push) checks

Knowledge Check

What does up == 0 assert, and why can a push system not offer the equivalent for free?

An expected target was unreachable — push receivers do not know what should exist
The target's metrics contained a literal zero value where a positive one was clearly expected
Push systems detect dead senders faster, just less precisely
The target exposed an up series set to 0 in its /metrics output

Under a traffic surge, who decides how much metric data flows into the collection system in each model?

Pull: each target decides its own cadence; push: the central server decides
Pull: the server's scrape schedule; push: whatever the senders transmit
Both models scale ingest linearly with application traffic
Both models negotiate a rate between sender and receiver

Which workload is the Pushgateway actually for?

A long-running API whose team simply prefers pushing to being scraped directly
High-traffic services like bookings, to reduce scrape pressure
A short-lived batch job that exits before a scrape could reach it
Every process behind NAT, as the standard NAT traversal layer

Metrics must come from agents inside customers' networks, behind firewalls you do not control. Which model, and why?

Pull, with each customer target manually added to the server's scrape_configs
Pull, over a dedicated VPN into each customer network
The Pushgateway, as a relay for all customer agents
Push — outbound HTTPS crosses the firewall; inbound scrapes never will

You got correct