The Log Pipeline

Between a container writing to stdout and Mara querying Loki sits a pipeline: Docker persists the stream to disk, an agent tails it, attaches labels, batches lines, and pushes them to Loki over HTTP. Every stage has a buffer, and every buffer has a limit.

So the honest questions about a log pipeline are not "does it work" — on a quiet Tuesday, everything works. They are "what happens when Loki is down for 20 minutes" and "which lines get dropped when the pipe clogs." This topic answers both, stage by stage.

The Pipeline Shape

The chain runs application → stdout → Docker log driver → file on disk → agent → Loki on obs-01:3100. The agent does four jobs in one pass: tail the files, parse and label what it reads, batch lines into requests, and push them over HTTP. It is the same instrument → collect → store spine you built for metrics, with one inversion — logs are pushed by an agent instead of scraped, because a discrete event cannot sit around waiting to be polled.

Capturejson-file → disk

→

Parse / enrichtail · label

→

Bufferdurable, bounded

→

Shipbatch · push

Grafana Alloy, the Taught Agent

The agent on every Harborline host is Grafana Alloy, Grafana's collector. Promtail, Loki's original agent, is deprecated in Alloy's favor — commercial support ended in March 2026, so new deployments start on Alloy and this book teaches nothing else. An Alloy config wires components into a graph: one component discovers containers over the Docker socket, one tails their logs, one ships the result.

// config.alloy on app-01: tail every container, push to Loki
discovery.docker "local" { host = "unix:///var/run/docker.sock" }
loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.docker.local.targets
  forward_to = [loki.write.loki.receiver]
}
loki.write "loki" {
  endpoint { url = "http://obs-01:3100/loki/api/v1/push" }
}

Read it as plumbing: discovery.docker asks the Docker socket which containers exist, loki.source.docker tails the log stream of each one and forwards every line to loki.write, which batches and pushes to Loki's ingest endpoint. Add a container to the host and it is being shipped within seconds, with zero per-service configuration — the discovery component notices it the same way Prometheus service discovery noticed new scrape targets in Chapter 3.

Docker Log Capture

Underneath, Docker's default json-file driver writes each container's stdout and stderr to /var/lib/docker/containers/<id>/<id>-json.log, one JSON envelope per line. Those files are what Alloy actually tails, and container metadata — name, image, labels — is what it maps onto stream labels, container name becoming job. The files are also a liability nobody configures until it hurts: rotation is not a default.

// /etc/docker/daemon.json — cap log history per container
{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" }
}

This caps each container at three 10 MB segments, oldest overwritten first. Without it Docker keeps unlimited history, and one debug-heavy weekend from a chatty container fills a 40 GB disk until Docker itself starts failing writes. Harborline sets 10m x 3 on every host.

Parsing and Labeling at the Agent

Between source and write you can insert a loki.process component: a chain of stages that sets the static label set (job, host, env="prod"), extracts a timestamp, or redacts a pattern before anything leaves the host. The discipline that matters here: label streams, not lines. The agent must never promote fields from log content — request IDs, URL paths, status codes — into labels, for cardinality reasons topic 34 makes brutal. At this stage in the pipeline, one config line commits that mistake for every log line the host ever ships.

Buffering and Backpressure

When Loki is unreachable, Alloy retries with exponential backoff, and Docker's json-file segments act as the durable buffer — the agent remembers its read position and resumes where it stopped. A 20-minute Loki outage loses nothing. Sustained pressure is a different animal: rotation keeps overwriting the oldest segment, and once it laps the agent's read position, those lines are gone, with no error surfacing in your application anywhere.

The other cliff is at the Docker driver itself. In the default blocking mode, a driver that cannot keep up eventually stalls the application's writes to stdout — no line is ever lost, but your service slows down because of its logging. In non-blocking mode, writes never stall; instead a fixed buffer (1 MB by default) silently drops new lines once it fills, discarding whatever the container writes next instead of blocking it. Every log pipeline chooses between losing lines and slowing the service — the only real decision is making that choice deliberately, per service.

Fluent Bit, the Ecosystem Alternative

The agent you will meet most often outside Grafana shops is Fluent Bit: a CNCF-graduated collector written in C, with an input/output plugin matrix covering Elasticsearch, Kafka, S3, Loki, and a hundred more, and ubiquitous as a Kubernetes DaemonSet. Its config vocabulary differs; its concepts do not. Tail, parse, buffer, backpressure — everything this topic taught on Alloy transfers one-to-one, so learning one agent properly is learning the category.