Log Hygiene

A log pipeline that works is not the same as a log pipeline you can afford, trust, and legally operate. Hygiene is the unglamorous half of logging: levels that mean the same thing in every service, sampling that tames the noisiest streams, retention matched to what each stream actually answers during an incident, and an iron rule that secrets and cardholder data never reach the log store.

The iron rule earns its name from Loki's own architecture. Chunks are immutable — once a card number is compressed into a chunk on obs-01, "just delete that line" is not an operation Loki offers. Removal means an async compactor delete job — scoped by a stream selector, time window, and at best a line filter, not a single-row DELETE — and the number sat readable in the meantime. The other three disciplines are about cost and trust; that one is about the law.

Level Discipline

A level is a contract, not a mood. ERROR means an operation failed and someone should eventually look. WARN means degraded but recovering on its own. INFO records business events — booking created, payment captured. DEBUG is off in production. The contract matters because the previous topic built queries on it: when search logs cache misses at ERROR, | level="error" counts routine cache behavior as failure, the error-rate panel becomes a lie, and the twelve real failures hide inside forty thousand fake ones.

An operation failed and someone should eventually look→ERROR

Degraded but recovering on its own→WARN

A business event — booking created, payment captured→INFO

Diagnostic detail, off in production→DEBUG

Card numbers, passwords, bearer tokens, session cookies→Never log

The Volume Problem

At its 300-requests-per-second Saturday peak, search logging one INFO line per request produces about 26 million lines a day, nearly all of them reporting that everything is fine. Log volume follows traffic, not information. The streams that grow fastest are almost never the ones that answer incident questions: last Saturday's investigation turned on a few hundred connection pool timeout lines from bookings, while search's heartbeat INFO paid full storage price to say nothing.

Sampling the Noise

The fix is Chapter 1's sampling cost applied to logs. Keep 1 in 100 of a high-frequency INFO event and tag the kept lines (sampled=100) so queries can rescale counts, or demote the event to DEBUG and let production drop it entirely. Never sample ERROR or WARN — errors are rare, cheap to store, and exactly the lines an incident needs complete. Spend the budget on signal, not heartbeat.

Retention Per Stream

Loki's compactor applies a global retention default with per-stream overrides. Harborline keeps 30 days by default, payments streams for 90 because disputes and chargebacks arrive late, and web access logs for 14 because nobody has ever queried them past day three. Retention is the third cost lever from Chapter 1, and paying 90-day prices for access-log noise is a choice, not a default you have to accept.

# loki.yml — 30-day default, per-stream overrides via the compactor
limits_config:
  retention_period: 720h            # 30 days
  retention_stream:
    - selector: '{job="payments"}'
      priority: 1
      period: 2160h                 # 90 days
    - selector: '{job="web"}'
      priority: 1
      period: 336h                  # 14 days

PII and Secrets

Card numbers, passwords, bearer tokens, and session cookies never get logged — not on failure paths, not "temporarily, for debugging." A single primary account number in payments logs drags the entire pipeline — Alloy, Loki, every backup of obs-01 — into PCI DSS scope, and the incident stops being an engineering problem. The defense is layered, because any single layer will someday fail: payments never logs raw request or response bodies at the source, and a redaction stage in Alloy's loki.process stands behind that promise on every host.

// Alloy backstop: Luhn-shaped digit runs never leave the host
stage.replace {
  expression = "\\b(?:\\d[ -]?){13,16}\\b"
  replace    = "[REDACTED]"
}

The backstop replaces anything shaped like a 13-to-16-digit card number with [REDACTED] before the line reaches Loki. And when a real secret does land in a log — a bearer token in a stack trace, a password in a captured URL — treat it as compromised and rotate it. Deleting the line removes the evidence, not the exposure; the token was readable by everyone with Grafana access for as long as it sat there.

Logs as an Interface

Alerts and dashboards now parse these lines, which makes message wording and field names load-bearing. The connection pool timeout string bookings emitted hundreds of times between 09:00 and 11:30 last Saturday is exactly what an |= filter matches on, and a refactor that rewords it to pool wait exceeded silently blinds every query and alert built on the old text. A log message an alert matches on is an API — changes to it get reviewed like API changes, not waved through as copyediting.