Logs in Kubernetes
Kubernetes made the Chapter 7 collection decision for you. Containers log to stdout and stderr, the runtime writes those streams to files on the node, and kubectl logs reads them back — until the pod is deleted or the node dies, at which point they are gone. There is no opt-in and no configuration: the platform decided where logs go the moment you containerized.
Durable logging on Kubernetes is therefore a node-level shipping problem: one agent per node tails every container's log files, attaches the Kubernetes metadata that makes them queryable, and pushes to Loki. It is the same pipeline Chapter 7 built — collector, labels, backpressure — running on new plumbing.
The stdout Convention
The runtime captures stdout and stderr into /var/log/pods/<ns>_<pod>_<uid>/<container>/, with symlinks under /var/log/containers/, one entry per write, in the CRI format: timestamp, stream name, a continuation flag, then the message. The snippet below shows what two lines of a bookings traceback actually look like on disk. Applications that write log files inside the container instead of stdout opt out of this entire pipeline — the runtime never sees those bytes, and neither does anything downstream.
# /var/log/pods/harborline_bookings-7d9f..._0/bookings/0.log 2026-07-12T09:14:03.912Z stderr F Traceback (most recent call last): 2026-07-12T09:14:03.913Z stderr F File "/app/checkout.py", line 88, in charge
Rotation and the kubectl logs Ceiling
The kubelet rotates each container's log at containerLogMaxSize (10Mi by default) and keeps containerLogMaxFiles of them (default 5), and kubectl logs reads only the live pod's current container plus, with -p, the previous one. That is roughly 50Mi of history per container, and it evaporates the moment the pod is deleted or the node is drained. Centralization is not an upgrade over this ceiling; it is the only durability there is.
Alloy as a DaemonSet
One Alloy pod per node, deployed by the Grafana Helm chart, replaces the per-VM installs. Its discovery.kubernetes component finds the pods scheduled on its own node, it tails their log files from a hostPath mount — or reads them through the kubelet API with loki.source.kubernetes — and it pushes to the same Loki, on the same schema Chapter 7 built. The DaemonSet shape is a requirement for the hostPath-tailing path — an agent reading node-local files can only see the node it runs on — though loki.source.kubernetes, which reads through the API instead, does not share that constraint.
Labels Under Churn
Discovery attaches namespace, pod, container, and whichever pod labels you select. The Chapter 7 cardinality rule now has teeth, because pods churn in a way VMs never did: promote pod to a Loki label and every rollout mints a complete new set of streams per service, so the index grows with deploy frequency instead of log volume.
Keep Loki labels to namespace, app, and container, and leave pod names as structured metadata or filter-time content. A LogQL line filter finds one pod's output in milliseconds; it never needed a label for that.
The Multiline Problem
The runtime emits one CRI entry per newline, so a 15-line Python traceback becomes 15 separate entries, interleaved with whatever else the node wrote in the same second, and lines over 16KB are additionally split and flagged P for partial. A stage.multiline block in loki.process, anchored on a firstline regex such as the timestamp prefix, reassembles the pieces into one entry before they reach Loki — or the application logs single-line JSON and the problem never exists at all.
loki.process "pods" {
stage.multiline {
firstline = "^\\d{4}-\\d{2}-\\d{2}"
max_wait_time = "3s"
}
forward_to = [loki.write.default.receiver]
}
Harborline's Cutover
The five services already log single-line JSON to stdout — Chapter 7 made that call — so the migration touches zero application code. The Alloy DaemonSet lands via Helm, stream labels shift from host="app-01" to namespace, app, and container, and the saved LogQL queries need exactly one label rename to keep working.
- Applications writing to log files inside the container instead of stdout — invisible to
kubectl logs, invisible to the DaemonSet, effectively unlogged until someone execs into the pod mid-incident. - Treating
kubectl logsas the log system — a crash-looping pod that gets replaced, or a drained node, takes its history with it, and "what did it log before the OOM kill" has no answer without shipping. - Promoting
pod(or worse,pod_uid) to a Loki label — every deployment rollout mints a full new set of streams per service, and the index grows with deploy frequency instead of log volume. - Skipping multiline handling for stack-trace-producing services — each frame becomes its own entry, interleaved across streams, and the trace that explains the crash is unreadable exactly when it matters.
- Running the hostPath log agent as a Deployment instead of a DaemonSet — a node-local tailer can only read its own node's files, so every node without a replica is a blind spot that looks like "some pods just don't log".
- Log single-line JSON to stdout from every service — it satisfies the platform convention, kills the multiline problem at the source, and keeps parsing in the agent trivial.
- Deploy the log agent as a DaemonSet with a
/var/log/podshostPath and the RBAC for pod discovery, so node coverage is structural rather than hoped-for. - Restrict Loki labels to low-cardinality Kubernetes identity —
namespace,app,container— and query everything else by content, exactly the Chapter 7 rule with churn added. - Configure a
stage.multilinefor any workload that can emit stack traces, anchored on your timestamp prefix, so a traceback lands as one entry.
Knowledge Check
A bookings pod crash-looped overnight, was deleted during the morning rollout, and a fresh pod replaced it. Where are the logs the old pod wrote?
- Still under /var/log/pods on the node, readable until the kubelet's rotation schedule eventually catches up with them
- Gone from the node; only what Alloy already shipped to Loki survives
- In etcd alongside the pod's events, for one hour
- Readable with kubectl logs -p from the replacement pod
Why must the log agent run as a DaemonSet rather than a Deployment?
- A Deployment cannot mount hostPath volumes
- DaemonSets receive log streams from the kubelet at higher priority
- A Deployment would collect each pod's logs twice when replicas share a node, duplicating every line
- The agent reads files on its own node's filesystem, so every node needs its own replica
Which label set keeps Loki's index healthy across daily rollouts?
- namespace, app, container — pod names stay out of the label set
- namespace, pod, container, since pod is the most precise identity Kubernetes offers and precision improves queries
- host and filename, keeping the Chapter 7 VM schema unchanged
- No labels at all; query everything by content
A Python traceback from search lands in Loki as 15 separate interleaved entries. What happened?
- Loki split the entry because it exceeded the ingestion line limit
- The runtime emits one CRI entry per newline; a stage.multiline block would reassemble them
- Alloy tailed the file faster than Python wrote it, breaking the trace mid-write
- The kubelet rotated the log file mid-write, splitting the trace across files between two writes
You got correct