Topic 66

Auditing

SecurityForensics

Auditing is the layer that answers two forensic questions after the fact: who did what, and did anything change that should not have. The kernel audit subsystem with the auditd daemon records security-relevant events — system calls, file access, privileged commands — straight from the kernel as they happen, so a process cannot decline to log its own activity. File-integrity tools such as AIDE take the complementary tack: they hash a known-good baseline of the filesystem and report every deviation on the next scan.

Without this layer you cannot reconstruct an incident — the evidence was never recorded, and a backdoored binary or an edited /etc/passwd looks identical to the legitimate file. Compliance regimes including PCI-DSS, HIPAA, and the CIS Benchmarks mandate exactly this kind of trail, which is why auditing is a baseline requirement on a production server rather than an optional extra.

The Audit Subsystem

The audit subsystem lives in the kernel and is paired in userspace with auditd. The kernel evaluates each event against a loaded rule set; matching events are queued and handed to the daemon, which writes them to /var/log/audit/audit.log per /etc/audit/auditd.conf. Rules are managed with auditctl for the running kernel and stored as files under /etc/audit/rules.d/ for persistence. On Debian and Ubuntu the daemon comes from the auditd package; on Red Hat it is the audit package, but the tooling and file layout are identical.

There are two kinds of rules. A file watch records access to a path: -w /etc/passwd -p wa -k identity logs any write (w) or attribute change (a) to the password file and tags the record with the key identity. A syscall rule matches by system call and architecture: -a always,exit -F arch=b64 -S execve -k exec records every program execution on a 64-bit host. The single most important hardening line is -e 2, which makes the rule set immutable until the next reboot — once set, even root cannot run auditctl -D to wipe the rules without a reboot, and a reboot is itself loud and auditable.

# Debian/Ubuntu: install and enable the daemon
sudo apt install auditd

# persistent rules: /etc/audit/rules.d/hardening.rules
-w /etc/passwd -p wa -k identity
-w /etc/sudoers -p wa -k privesc
-a always,exit -F arch=b64 -S execve -k exec
-e 2

# compile rules.d into audit.rules, load, and confirm
sudo augenrules --load
sudo auditctl -l

What to Audit and How to Read It

The hard part is noise, not capture. A broad rule like auditing every execve on a busy host can generate millions of records a day, bury the events that matter, and fill /var. Scope rules to what actually carries weight: privileged commands and sudo use, changes to account files (/etc/passwd, /etc/shadow, /etc/group, /etc/sudoers), and access to the data your compliance scope cares about. Key every rule with -k so you can retrieve by intent later.

Raw audit.log is dense, and one logical action spans several records — a single execve emits a SYSCALL record, one or more PATH records, and an EXECVE record holding the arguments, all sharing an event ID. Rather than read it by hand, use ausearch to query and aureport to summarize. The keys are what make this tractable: ausearch -k privesc returns exactly the events your -k privesc rule tagged.

# every event tagged with a given key, interpreted
sudo ausearch -k privesc -i

# authentication failures since midnight
sudo ausearch -m USER_LOGIN --start today -sv no

# summary reports: by key, and by user
sudo aureport --key
sudo aureport -u --summary

File Integrity Monitoring with AIDE

AIDE (Advanced Intrusion Detection Environment) answers the "did anything change" question. You initialize a baseline database that records, for each file you tell it to watch, a set of attributes — permissions, owner, inode, size, mtime, and one or more cryptographic hashes (SHA-256, SHA-512). A later aide --check rehashes the same files and reports anything added, removed, or modified. Where auditd catches the act of changing a file in real time, AIDE catches the result on a schedule, which means it still flags a change made while auditing was somehow bypassed.

The workflow is two commands and a config under /etc/aide/ on Debian and Ubuntu (/etc/aide.conf on Red Hat). The config selects which trees to monitor and which attribute groups to check — system binaries in /bin, /sbin, and /usr warrant full hashing, while a churning directory like /var/log is excluded or watched only for permission changes, because hashing a file that legitimately changes every second produces nothing but false positives.

# Debian/Ubuntu: build the initial baseline
sudo aideinit
sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

# scan the filesystem against the baseline
sudo aide --check

# after a legitimate package update, regenerate and promote the db
sudo aide --update
sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

Baseline Discipline and Logs as Evidence

A file-integrity baseline is only trustworthy if an attacker cannot rewrite it. If aide.db sits on the same writable host as the files it protects, an intruder who plants a backdoor simply runs aide --update and the next check reports nothing wrong. Store the baseline read-only — on offline media, a remote pull target, or at minimum a path made immutable with chattr +i — and regenerate it deliberately after legitimate changes, never automatically. The same logic applies to audit.log: anything an attacker who reaches root can read, they can also alter or delete.

The defense for both is to get the evidence off the box in real time. Forward audit.log and the systemd journal to a central collector over rsyslog or the audisp remote plugin, so a compromised machine cannot erase its own history, and set max_log_file, num_logs, and max_log_file_action in auditd.conf to keep local retention bounded without dropping records. Run aide --check on a timer rather than by hand — a check nobody runs is a check that does not exist — and ship its output to the same place the audit trail goes.

Common Mistakes

Storing the AIDE baseline on the same writable host it protects — an attacker who plants a backdoor just runs aide --update, and the next scan reports a clean system.
Editing files under /etc/audit/rules.d/ but never running augenrules --load, so the kernel keeps enforcing the old rule set while you believe the new one is active.
Keying every execve on a busy host, generating millions of records a day that fill /var and make ausearch crawl — the events that matter are buried in the noise.
Forgetting the -e 2 immutable flag, so an intruder runs auditctl -D to disable auditing, does their work, and restores it with no reboot and no trace.
Pointing AIDE at /var/log or other churning paths with full hashing, drowning every report in false positives until people stop reading them.
Never reviewing audit logs or running aide --check until an incident — the trail exists but nobody looked, so the breach ran for weeks unnoticed.
Leaving max_log_file_action at a value that stops logging on a full disk, so a flooded /var silently halts the audit trail exactly when an attacker is generating the most events.

Best Practices

Scope audit rules to privileged commands, account-file changes, and sensitive data access — then key each with -k so ausearch -k and aureport --key slice the log by intent.
Set -e 2 as the last line of the rule set to make auditing immutable until reboot, and confirm with auditctl -l after every augenrules --load.
Store the AIDE baseline read-only and off the host — offline media, a remote pull, or chattr +i — and regenerate it deliberately with aide --update only after legitimate changes.
Run aide --check on a systemd timer or cron schedule and forward its output off-host, rather than relying on someone to run it by hand.
Forward audit.log and the journal to a central collector in real time over rsyslog or the audisp remote plugin, so a compromised box cannot destroy its own trail.
Bound local retention with max_log_file, num_logs, and a max_log_file_action of rotate or keep_logs so the disk never fills, and set retention to match your compliance window.
Baseline your audit rules from a vetted set such as the CIS Benchmark or Neo23x0 rules rather than writing them from scratch.

Comparable toolsWindows — the Security Event Log with audit policy (auditpol), the OS-native who-did-what trailTripwire / Osquery — file-integrity monitoring and SQL-queryable host state, alternatives or complements to AIDESIEM — Splunk, Elastic, Wazuh and similar, the aggregation layer that audit and FIM logs feed into

Knowledge Check

Why set -e 2 as the final line of an audit rule set?

It makes the rule set immutable until reboot, so an attacker who reaches root cannot quietly run auditctl -D to disable auditing without a visible, auditable reboot
It compiles every fragment under /etc/audit/rules.d/ into the single audit.rules file in lexical order so the complete, merged rule set loads automatically at every boot of the host
It enables auditing of 32-bit syscalls alongside the 64-bit ABI so older statically linked binaries running on the b32 syscall path are covered too
It rotates audit.log automatically and opens a fresh file once the partition approaches full so the daemon never stalls on writes

What is the consequence of keeping the AIDE baseline database on the same writable host as the files it monitors?

An attacker who modifies a file can run aide --update to fold the change into the baseline, so the next check reports a clean system
AIDE refuses to run any check at all unless the baseline database lives on a separate, dedicated read-only filesystem mounted elsewhere
The baseline hashes are recomputed from scratch on every boot of the host, roughly doubling the disk I/O cost of each scheduled scan
The database silently disables itself and stops reporting once the monitored files exceed its configured maximum size on disk

How do auditd and AIDE differ in what they detect?

auditd records the act of access or change in real time via the kernel; AIDE detects the resulting filesystem change on a scheduled scan against a baseline
auditd hashes files on a fixed schedule and stores a baseline; AIDE streams syscall events live from the kernel as they happen
Both read the same audit.log on disk, but AIDE merely parses, summarizes, and reports on the syscall events that auditd has already captured
auditd watches only loaded user-space shared libraries as they are mapped into a process, while AIDE watches only the kernel modules that were loaded during early boot

Why forward audit and integrity logs to a central host in real time rather than relying on the local copies?

An attacker who reaches root can alter or delete anything on the box, so a remote copy preserves the trail a compromised machine would otherwise erase
Central storage is the only place from which the ausearch and aureport query tools are ever able to read, filter, and correlate the stored audit records at all
The kernel stops writing any further records to the local audit.log entirely once the -e 2 immutable flag is appended to the rule set
Remote logging removes the local disk-fill risk entirely, because once forwarding is enabled nothing is ever written to the box itself

You got correct