Scaling Prometheus

The first scaling move is a bigger machine, and it goes further than most teams believe: one well-provisioned Prometheus carries millions of active series and hundreds of thousands of samples per second. Harborline's tens of thousands of series occupy a rounding error of that ceiling. So this topic is a survey in escalation order — sharding, federation, remote_write, the clustered rebuilds — and an argument for recognizing, honestly, which rung you are actually standing on.

The survey matters even at Harborline's size, because one early choice decides whether escalation stays cheap later. external_labels set today, before the first sample ever leaves the server, keep every rung on the ladder reachable; history without them cannot be retro-labeled.

Vertical Limits

Memory scales with active series — the head keeps recent chunks and index entries for every live one — and CPU scales with samples per second plus query load. The number to watch is prometheus_tsdb_head_series against RAM, and the ceiling is high enough that most teams retire the server before they reach it.

The common failure at this rung is not outgrowing the machine. It is detonating cardinality — a user_id label, coming in Chapter 5 — and mistaking the explosion for organic growth. Per-job scrape_samples_scraped separates the two: real traffic grows across the fleet, a label accident grows in one job. The fix for the second is relabeling from topic 14, not architecture.

Functional Sharding

Before any clustering, run more Prometheus servers: split by team, environment, or failure domain, each one small, independent, and blast-radius-contained. A shard's death blinds only its own patch, and each shard alerts on what it scrapes. The cost is the global picture — no single server can answer a query spanning shards — and the arrangement is acceptable exactly as long as nobody needs one.

Federation

Federation is a parent Prometheus scraping /federate on the leaves, pulling only series that match explicit match[] selectors. It is built for hierarchies of aggregates: the leaves keep raw detail and evaluate recording rules (Chapter 4), and the parent scrapes the resulting roll-ups to serve the global dashboard.

Used to copy everything, it fails on schedule: the parent re-ingests the fleet's entire output, the /federate response grows until the scrape exceeds its timeout, and you have built a slower, lossy imitation of what remote_write does properly. Federation is a roll-up channel, not replication.

remote_write and Agent Mode

remote_write streams every sample to an external system as it is ingested, buffered through the WAL, so a receiver outage is survivable for hours rather than seconds. Remote-Write 2.0 extends the protocol with metadata, exemplars, and native-histogram support. Agent mode — the --agent flag, stable in the 3.x line — strips the server down to scrape, relabel, forward: no local queries, no rule evaluation, no alerting, built for edge collectors feeding one central store.

# prometheus.yml — samples stream out as they are ingested
global:
  external_labels:
    env: prod
    replica: A
remote_write:
  - url: "https://metrics-store.example/api/v1/push"

The two external labels are load-bearing. env keys every aggregation the backend does, and replica is how an HA pair avoids double counting — the receiver deduplicates on the one label that distinguishes the two replicas. The buffer has a limit, too: WAL segments are truncated as the head flushes, so the prometheus_remote_storage_* lag metrics are worth an alert before any receiver ever goes down.

The Clustered Rebuilds

When one dataset must outgrow one node, three architectures wait. Thanos bolts onto the Prometheus servers you already run: a sidecar ships their immutable blocks to object storage, a global query layer fans out across all of them, and old data finally gets the downsampling topic 15 refused. Mimir — Cortex lineage — inverts the model: a horizontally scalable, multi-tenant backend fed purely by remote_write, with Prometheus reduced to an agent at the edge.

VictoriaMetrics is the pragmatic third path: a Prometheus-compatible engine with its own storage format, run as a single binary or a cluster, with a markedly lower resource appetite. Three tools, three verbs — extend, centralize, substitute — and the comparison box pins down which requirement justifies each.

When One Server Is Enough

Most of the time, and specifically: series count in the low millions or less, retention needs measured in weeks, no hard multi-region or multi-tenant query requirement. Harborline meets every criterion, so the honest architecture is the one already running — one server on obs-01, sized with topic 15's arithmetic, external labels set so the ladder stays climbable. The chapter ends where it began, with one binary; the difference is that it now watches six hosts and three stores, and the next Saturday-morning complaint will land on a system that finally has data.