TSDB Internals

Everything Prometheus scrapes ends up under --storage.tsdb.path on obs-01's local disk, and the layout is worth understanding before you size it, back it up, or trust it. The design has two moving parts: an in-memory head block journaled to a write-ahead log, and behind it a growing shelf of immutable two-hour blocks. That shape is why a restart — or a crash — hands you back your data.

Nothing in this topic is Harborline-specific, but the numbers are: six node_exporters plus the store exporters push roughly 10,000 active series through this machinery every 15 seconds. Small numbers, deliberately — the point of walking the arithmetic now is that it stays linear when Chapter 5 multiplies the series count.

The Head Block and the WAL

Incoming samples land in memory, in the head block, and are simultaneously appended to a write-ahead log on disk, segmented into 128 MB files. The WAL is never queried. It exists for exactly one reader: a restarting Prometheus, which replays it to rebuild the head as it was.

Kill the process mid-scrape and you lose essentially nothing — you wait through the replay instead. That replay is the real cost of a restart: minutes on a large head, during which nothing is scraped. Plan around the gap, not around data loss.

Two-Hour Blocks

Roughly every two hours, the oldest head data is written out as a block: a directory named by ULID containing compressed chunks, a full inverted index, and a meta.json describing its time range. Once written, a block is never modified. Queries read the head plus whichever blocks the time range touches, and immutability is what keeps the engine simple to reason about — and, as the backup discussion below shows, cheap to snapshot.

# ls /prometheus — the whole database is this directory tree
01JZX0Q8Z3N9K5T2W7E4R6M1BC/   # one immutable ~2 h block
  chunks/000001                # compressed samples
  index                        # inverted index: labels -> series
  meta.json                    # time range, stats, compaction level
chunks_head/                   # the head's memory-mapped chunks
wal/00000012                   # 128 MB write-ahead segments

The listing is the whole contract: immutable block directories, the head's memory-mapped chunks, and the WAL segments that make the head crash-safe. Everything Prometheus knows lives in this one tree on local disk — which is also why the storage rules later in this topic are unforgiving about what kind of disk.

Compaction

A background compactor merges small blocks into larger ones, deduplicating index entries and applying deletion tombstones as it goes. Block size has a ceiling: 10% of the retention window or 31 days, whichever is smaller — at the default 15-day retention Harborline runs, that computes to 36 hours, but Prometheus only compacts in fixed 2h→6h→18h→54h steps, so blocks actually top out at 18 hours. Compaction is why a query spanning a week touches a handful of block indexes instead of eighty-four.

Retention

Two limits, enforced together: --storage.tsdb.retention.time, default 15 days, and --storage.tsdb.retention.size — whichever trips first wins. Deletion is coarse. Entire expired blocks are dropped at compaction time, so data leaves in multi-hour slabs, and there is no per-series or per-age policy inside one server.

The Sizing Arithmetic

Compression brings a sample to roughly 1–2 bytes on disk, and the sizing formula is one line: needed disk ≈ ingestion rate × bytes per sample × retention. Ingestion rate is active series divided by scrape interval — every term is observable on a running server.

Harborline's chapter-end state: call it 10,000 active series every 15 s, which is about 670 samples per second, under 100 MB a day, roughly 1.5 GB for the full 15 days. A rounding error on any disk. The reason to run the arithmetic at this size is that it is linear — when Chapter 5 instruments five services and the series count multiplies, the same three numbers produce the new answer.

Full Resolution to the Cliff

Prometheus never downsamples. A 14-day-old sample sits at the same 15-second resolution as one from this minute, and on day 15 its block is deleted whole — full fidelity, then the cliff. The graceful thinning of old data that Graphite's Whisper and RRDtool did is deliberately absent, and it reappears only in the long-term systems of topic 16, where it belongs to someone else's storage engine.