Patching and Secrets

Patching is keeping installed packages current with the security fixes the distribution publishes; secret management is keeping the credentials those services need — database passwords, API tokens, TLS private keys — out of source control, out of process listings, and off world-readable disk. They are separate disciplines that fail the same way: a known CVE that sat unpatched for weeks, or a password committed to a Git repository, is the most common root cause in real server compromises, and both are entirely preventable with a routine.

The operational consequence is that a server is never "done." On Debian and Ubuntu, security updates ship continuously through a dedicated -security archive, and the gap between a public CVE and an attacker scanning for it is now measured in hours, not days. A fleet you patch by hand once a quarter is a fleet that runs known-exploitable code most of the year — and the secrets on those boxes are exactly what the exploit is after.

The APT Update Model

On Debian and Ubuntu, patching is two distinct operations. apt update refreshes the package index from the configured sources — it changes nothing on disk except the cache. apt upgrade then installs newer versions of already-installed packages, and apt full-upgrade additionally removes packages where that is required to complete an upgrade (the old dist-upgrade). Running upgrade without a preceding update just reinstalls what you already have, which is the most common reason "I patched it" turns out to be false.

Security fixes come from a separate archive — $RELEASE-security on Debian, $RELEASE-security on Ubuntu — kept distinct from the general updates pocket so you can apply only security content if you want a smaller, lower-risk change window. The contract is that within a stable release the distribution backports the fix to the existing version rather than bumping you to a new upstream major version, so openssh-server stays at the same feature set and only the vulnerability is closed. That stability is the whole point of running stable: you patch without your config or behavior shifting underneath you.

# refresh the index, then see exactly what security has waiting
sudo apt update
apt list --upgradable 2>/dev/null | grep -security

# apply only security updates, non-interactively
sudo unattended-upgrade --dry-run -d

# a kernel was updated if this file exists
ls /var/run/reboot-required && echo "reboot needed"

Unattended Upgrades

The unattended-upgrades package applies security updates automatically on a timer, and on a server it should be on. The default configuration in /etc/apt/apt.conf.d/50unattended-upgrades restricts it to the -security origin only, so it closes vulnerabilities without dragging in feature changes that might surprise you. Pair it with /etc/apt/apt.conf.d/20auto-upgrades, which controls how often the index is refreshed and the upgrade run fires via the apt-daily and apt-daily-upgrade systemd timers.

Two settings decide whether automation is safe to leave running. Unattended-Upgrade::Automatic-Reboot defaults to false — a kernel or glibc update lands but the new code is not active until you reboot, which on a server you usually want to schedule rather than have happen mid-traffic. Unattended-Upgrade::Mail sends a report so a failed or held-back upgrade is visible instead of silent. The failure mode to avoid is the opposite of negligence: an unattended reboot at an unplanned hour because Automatic-Reboot was set to true with no Automatic-Reboot-Time.

Live Kernel Patching and Reboots

A user-space fix — openssl, bash, nginx — takes effect when you restart the affected service; only a kernel update genuinely needs a reboot to activate. The tool that tells you which running processes still link an outdated library is needrestart, installed by default on recent Ubuntu, which after every apt run lists the services holding stale code and can restart them. Relying on a clean reboot to "make sure everything is patched" wastes downtime on fixes that a service restart would have applied immediately.

For the kernel itself, live patching closes the reboot gap for critical CVEs. Ubuntu's Livepatch and the equivalent kpatch/kgraft mechanisms inject fixed code into the running kernel without a reboot, buying time until a maintenance window. They are not a replacement for rebooting onto the new kernel — a livepatch covers a curated set of high-severity fixes, not every change, and the running kernel is still the old binary on disk. Treat livepatch as the thing that lets you reboot on your schedule instead of the attacker's.

Secrets at Rest and in the Environment

The baseline for a secret on disk is a file owned by the service account with mode 0600 (or 0640 with a dedicated group), never world-readable and never in a directory anyone else can traverse. A TLS private key at 0644 is readable by every account on the box, which defeats the certificate; check with find /etc -name '*.key' -perm /044 to catch the ones that slipped. Secrets do not belong in shell history, in command arguments (anyone can read them in /proc and ps), or committed to a repository where git log preserves them forever even after a later deletion.

Passing secrets through environment variables is convenient and leakier than it looks: a child process inherits the whole environment, /proc/PID/environ exposes it to anyone who can read the process, and crash handlers and error trackers routinely serialize the environment into logs. systemd's LoadCredential= and SetCredential= are the modern answer — the unit receives the secret as a file under a ramfs path in $CREDENTIALS_DIRECTORY — ramfs, not tmpfs, so it is never swapped to disk — visible only to that service, never to its children or to other units. For encryption keys that must be wrapped at rest, systemd-creds encrypt ties the ciphertext to the host TPM so a stolen disk image yields nothing usable.

# lock down an existing key file and its directory
sudo chown app:app /etc/app/tls.key
sudo chmod 0600 /etc/app/tls.key
sudo chmod 0750 /etc/app

# hand a secret to a service via systemd, not the environment
# in the unit's [Service] section:
LoadCredential=db-password:/etc/app/db.pass
# the app reads $CREDENTIALS_DIRECTORY/db-password at runtime

For anything beyond a single host, a dedicated secret store — HashiCorp Vault, or a cloud KMS-backed store — adds rotation, audit logging, and short-lived dynamic credentials, so a leaked token expires on its own instead of living until someone notices. The principle is the same at every scale: the secret is encrypted at rest, access is logged, and the application fetches it at start time rather than carrying a static copy in its image or its repo.