Topic 45

journald

systemdLogging

journald — the daemon systemd-journald — is systemd's logging service. It captures stdout and stderr from every unit, the kernel ring buffer, structured messages from the native API, and anything still written to /dev/log, and it stores them in indexed binary journal files. Each entry carries metadata fields the originating process never had to format: which unit emitted it (_SYSTEMD_UNIT), the PID (_PID), the UID, the boot it belongs to (_BOOT_ID), and the priority. That replaces the old syslog model, where log lines were free-form text appended to flat files and any structure had to be parsed back out with regular expressions.

The single fact that bites operators most: on Debian and Ubuntu the journal is volatile by default. It lives in /run/log/journal, which is tmpfs, so it is gone on reboot. A server that panicked, rebooted, and came back clean has thrown away the evidence unless you turned on persistent storage first — and you find that out at exactly the moment you need the logs.

Structured Logging Model

Every journal entry is a set of FIELD=value pairs, not a line of text. Some fields come from the application; most are stamped on by journald itself and are trusted because the daemon, not the process, sets them. The trusted fields begin with an underscore — _PID, _UID, _GID, _SYSTEMD_UNIT, _BOOT_ID, _HOSTNAME, _COMM — which is why a process cannot forge the unit it claims to belong to. The message body lives in MESSAGE, and the severity lives in PRIORITY using the same 0–7 syslog scale (0 emerg, 7 debug).

Because the fields are indexed, you filter by exact match instead of scanning text. That is the whole point of the binary format: journalctl _SYSTEMD_UNIT=nginx.service _PID=4821 jumps straight to matching entries without a substring search across gigabytes of file. The trade-off is that you cannot tail -f or grep the file directly — the bytes are not text — so the journal is only as useful as your fluency with journalctl.

# one entry, shown as fields with -o verbose
_SYSTEMD_UNIT=nginx.service
_PID=4821
PRIORITY=3
MESSAGE=connect() failed (111: Connection refused)

Applications running under systemd get this for free: anything a service prints to stdout/stderr is captured with the right _SYSTEMD_UNIT attached, so most software needs no logging configuration at all to land in the journal correctly labeled.

Querying with journalctl

journalctl is the only supported reader, and the flags that matter narrow a flood to the lines you want. -u scopes to one unit; -b shows the current boot and -b -1 the previous one; -p err filters by priority and below; -k shows only kernel messages; -f follows live like tail -f; and --since/--until take human times such as "2 hours ago" or "2026-05-30 09:00".

For machine consumption, -o json emits one JSON object per entry that pipes cleanly into jq, and -o verbose dumps every field for an entry when you need to know exactly what is queryable. Bare field matches compose with the flags, so a single command can pin down a unit, a boot, and a severity at once.

# errors from one unit, since the last boot, followed live
journalctl -u ssh.service -b -p err -f
# everything from the previous boot, as JSON for jq
journalctl -b -1 -o json | jq .MESSAGE

Reading other users' logs and the full system journal needs membership in the systemd-journal group (and on Debian/Ubuntu the adm group is added to it by default). An unprivileged user outside those groups sees only their own session entries, which surprises people who expect journalctl to show everything the way cat /var/log/syslog once did under sudo-less read access.

Volatile and Persistent Storage

The Storage= directive in /etc/systemd/journald.conf decides where the journal lives. The default on Debian and Ubuntu is Storage=auto, which writes to /var/log/journal only if that directory already exists, and otherwise falls back to volatile /run/log/journal. Out of the box that directory does not exist, so the journal is in RAM and dies on reboot. Set Storage=persistent (or simply create the directory) and restart the daemon to keep logs across reboots.

Persistent storage is the difference between post-incident forensics and a shrug. On any server where you might have to explain what happened before a crash, persistent is the only defensible setting — and the cost is bounded disk, not unbounded, because the retention limits below cap it.

# turn on persistent storage
mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald
journalctl --disk-usage

Retention and Rotation

journald rotates and prunes on its own once you set bounds in journald.conf. SystemMaxUse caps total journal size (default is the smaller of 10% of the filesystem or 4 GB); SystemKeepFree guarantees free space is left for everyone else (default 15% of the filesystem); MaxFileSec forces rotation after a time even if size limits are not hit; and MaxRetentionSec deletes entries older than a cutoff. The journal also splits per boot, so old boots age out as whole files.

When you need to reclaim space immediately rather than wait for the daemon, journalctl --vacuum-size=500M trims to a size, --vacuum-time=2weeks trims to an age, and --vacuum-files=10 keeps only the most recent N files. These act on the persistent journal and are the right tool inside a cleanup runbook.

Setting	Bounds	Default
`SystemMaxUse`	Total journal size	min(10% of FS, 4 GB)
`SystemKeepFree`	Free space to leave on the FS	min(15% of FS, 4 GB)
`MaxFileSec`	Time before a file is rotated	1 month
`MaxRetentionSec`	Max age before entries are deleted	off (size-bound only)

A journal that grows until the disk fills is almost always a journal where nobody set SystemMaxUse on a filesystem large enough that 10% is a lot — set an explicit cap rather than trusting the percentage default on a 2 TB volume.

Forwarding and rsyslog Coexistence

journald can hand entries onward. ForwardToSyslog=yes in journald.conf pushes everything to a classic syslog socket, which is how rsyslog (no longer installed by default since Debian 12 — install it if you want it) keeps writing /var/log/syslog and friends. For centralization, systemd-journal-upload ships entries to a remote systemd-journal-remote collector, preserving the structured fields end to end rather than flattening them to text.

The architectural decision is who owns persistence, and the wrong answer is "both." If rsyslog is writing /var/log/syslog and you also enable persistent journald, you store every line twice and double the disk cost for the same data. Pick one as the durable store: keep journald persistent and let rsyslog forward to a remote aggregator without local files, or keep rsyslog as the local store and leave journald volatile. Running both as full persistent copies is a configuration left on autopilot, not a design.

journald vs rsyslog

journald — a structured, indexed binary store with rich filtering by unit, priority, and boot, and trusted daemon-stamped metadata. Choose it as the default on any modern systemd box; it captures every service's output with no per-application config.

rsyslog (syslog) — flat-text files (/var/log/syslog) and a mature, battle-tested forwarding stack (TCP, TLS, RELP) to remote collectors. Choose it when you need reliable network shipping, downstream tools that expect plain text, or a long-established central log pipeline — and let journald feed it via ForwardToSyslog=yes rather than persisting both.

Common Mistakes

Assuming logs survive a reboot on a stock Debian/Ubuntu box — the default journal is volatile in /run/log/journal, so the crash you needed to investigate is wiped clean the moment the machine comes back up.
Never setting SystemMaxUse, then watching the journal grow until SystemKeepFree kicks in or the disk fills — the percentage default is fine on a small root volume and a disaster on a large data volume.
Trying to grep or tail -f the journal files directly — they are indexed binary, not text, so the only correct reader is journalctl with field matches and -u/-p/--since.
Trusting a journal that may be damaged after an unclean shutdown without running journalctl --verify — a corrupted file can silently truncate your view of what happened.
Filing a bug because journalctl "shows nothing" while running as an unprivileged user not in the systemd-journal or adm group — that user only ever sees their own session entries.
Leaving both rsyslog writing /var/log/syslog and journald set to persistent — every message is stored twice and disk usage doubles for no added durability.

Best Practices

Set Storage=persistent in journald.conf on any server where you might need post-incident logs, and restart systemd-journald so it takes effect immediately.
Set explicit SystemMaxUse and SystemKeepFree values rather than trusting the percentage defaults — bound the journal to a number you chose for that disk.
Query with field matches and -u/-p/-b/--since instead of dumping everything and piping to grep — the index is the reason the binary format exists.
Use journalctl -o json | jq for any programmatic parsing so you read trusted structured fields rather than re-parsing a formatted message string.
Run journalctl --verify periodically and after unclean shutdowns to catch journal corruption before you rely on the data.
Decide deliberately whether journald or rsyslog is the durable store, and put journalctl --vacuum-size/--vacuum-time in your cleanup runbooks instead of editing limits ad hoc.

Comparable toolsWindows — Event Log, a binary, structured, queryable event store filtered through Event Viewer or Get-WinEventsyslog / rsyslog — the flat-text logging model journald replaces, and still coexists with via ForwardToSyslogmacOS — Unified Logging (os_log), also a binary indexed store read with the log command

Knowledge Check

A Debian server panicked and rebooted overnight. You run journalctl -b -1 and get nothing. What is the most likely reason?

The journal is volatile by default — it lived in tmpfs at /run/log/journal and was lost on reboot because persistent storage was never enabled
The -b -1 relative offset is invalid syntax on Debian; previous boots can only ever be addressed there by their full 32-character boot ID passed to --boot
journald deletes the previous boot's entries automatically the instant the next boot writes its first message
A kernel panic disables journald before it can flush, so nothing was ever written for that boot at all

Why filter with journalctl _SYSTEMD_UNIT=nginx.service -p err instead of journalctl | grep nginx | grep error?

The journal is indexed binary; field and priority matches use the index and trust daemon-stamped fields, while grep does a substring scan over decoded text and matches the wrong lines
grep cannot read the binary .journal files at all, so the second command always returns empty no matter what pattern you give it or how many messages the unit may itself have logged to disk
The two return identical results; the field-and-priority match is only fewer keystrokes to type at the prompt
grep over the journal silently rotates and deletes the very .journal files it reads as it scans them

On a 2 TB data volume you leave SystemMaxUse unset. What is the consequence?

The journal can grow toward the percentage default — up to 4 GB capped by the 10%-of-filesystem rule — bounded only by SystemKeepFree, far more than you intended to spend on logs
journald refuses to write any logs at all until a numeric SystemMaxUse size limit is set in journald.conf first
The journal is hard-capped at a fixed 100 MB ceiling regardless of how large the underlying disk happens to be
Logs are written normally but rotation and vacuuming are disabled entirely when the directive is unset, so the very oldest entries are retained on disk forever and the journal never shrinks

rsyslog is writing /var/log/syslog and you also set journald to Storage=persistent. What is the trade-off?

Every message is now stored durably twice, doubling disk usage for the same data without adding redundancy you designed for
journald and rsyslog deadlock competing for ownership of the /dev/log socket and both stop logging entirely until one of the two is disabled by hand
rsyslog automatically stops writing /var/log/syslog once journald becomes persistent, so there is no extra cost
The structured journal fields are silently dropped from both stores, leaving only plain text

You got correct