The Kernel and User Space

Linux splits every running machine into two zones separated by a hardware privilege boundary. Kernel space runs in CPU ring 0 with unrestricted access to memory, devices, and the instruction set. User space — every daemon, shell, and application — runs in ring 3, sandboxed, unable to touch hardware or another process's memory without asking the kernel first. The CPU enforces the split; it is not a convention the kernel can wish away.

The operational consequence is the difference between an annoyance and an outage. A crash in user space kills one process: Nginx segfaults, systemd restarts it, the rest of the box keeps serving. A crash in kernel space — a null dereference in a driver, a bad loadable module — has nothing above it to catch the fault, so it panics and takes the whole machine down. That is why you treat anything running in ring 0 with far more suspicion than anything in ring 3.

Protection Rings and Mode Switches

x86-64 CPUs define four privilege rings, 0 through 3, but Linux uses only two of them: ring 0 for the kernel and ring 3 for everything else. Rings 1 and 2 exist in the hardware but no mainstream operating system bothers with them, because a clean two-level split maps directly onto the kernel/user divide and is simpler to reason about. Ring 0 can execute privileged instructions — loading page tables, talking to I/O ports, disabling interrupts. Ring 3 traps and faults if it tries.

Crossing from ring 3 to ring 0 is a mode switch, and it is not free. A modern syscall instruction plus the kernel entry and exit work costs on the order of a few hundred CPU cycles even before the kernel does anything useful — and mitigations like Spectre and Meltdown page-table isolation have pushed that higher on affected CPUs. A function call inside one ring costs a handful of cycles. That gap is why batching work into fewer, larger system calls beats making many small ones.

You rarely think in rings day to day, but the cost shows up the moment a workload is syscall-bound. A program that reads a file one byte at a time pays the mode-switch toll on every byte; the same program with a 64 KB buffer pays it once per block. The boundary is real, and it has a price tag.

The System Call Interface

User space cannot open a file, send a packet, or fork a process on its own — it asks the kernel through a system call. Linux exposes roughly 350 syscalls on x86-64, each identified by a number, each a documented contract between user and kernel space. Almost no application invokes them directly; the C library wraps them in friendlier functions, so a call to fopen() bottoms out at the openat syscall, and a call to printf() eventually reaches write.

strace makes the boundary visible by tracing every syscall a process makes. It works through ptrace, which stops the target on entry and exit of each call — useful for debugging, but it adds two extra context switches per syscall, so a strace'd program can run an order of magnitude slower. That slowdown is the reason you reach for strace in development, not on a hot production path.

# Summarize syscalls a process makes, with time spent in each
strace -c -f curl -s https://example.com > /dev/null
# -c aggregates counts; -f follows forked children
# Output: openat, connect, sendto, recvfrom, read, write, close ...

In production, prefer eBPF-based tooling. bpftrace and the bcc tools attach probes inside the kernel and aggregate in place, so you observe syscalls with a fraction of strace's overhead and without stopping the process on every call.

Kernel Modules

Not all kernel code is compiled into the boot image. Loadable kernel modules — device drivers, filesystems, netfilter hooks — load and unload at runtime with modprobe and rmmod, and lsmod lists what is currently resident. A module runs in ring 0 with the same privileges as the core kernel, which is exactly why a buggy module is dangerous: there is no sandbox around it. A null pointer in a third-party driver panics the box just as surely as a bug in the scheduler would.

# Inspect and load modules
lsmod | grep nvme        # is the NVMe driver loaded?
modinfo nvme              # version, signer, parameters
modprobe dummy            # load a module by name (resolves deps)

On a machine with UEFI Secure Boot enabled, the kernel refuses to load modules that are not signed by a trusted key, and it logs a tainted-kernel warning. On Debian and Ubuntu this trips up out-of-tree drivers built locally — Nvidia, VirtualBox, ZFS — whose DKMS-rebuilt modules must be signed with a Machine Owner Key (MOK) enrolled through mokutil, or the load fails with Operation not permitted after a kernel upgrade. Red Hat behaves the same way; the signing tooling differs but the policy is identical.

Virtual Memory and the Kernel/User Split

Each process gets its own virtual address space, and the kernel maps itself into the top of every one of them. On x86-64 with a 48-bit canonical address space, user space owns the lower half and the kernel occupies the upper half. Mapping the kernel into every process is what makes a system call cheap to enter — there is no full address-space switch on the way into ring 0 — but the user-space half of that mapping is marked privileged in the page tables, so a ring-3 read of a kernel address faults immediately.

That protection is structural, not advisory. A user process literally cannot read kernel memory or another process's memory; the MMU rejects the access and the kernel delivers a SIGSEGV. The Meltdown vulnerability mattered precisely because it broke this guarantee through speculative execution, which is why current kernels add page-table isolation that unmaps most of the kernel during ring-3 execution — at the cost of more expensive syscalls.

Process Context and Scheduling

The scheduler runs in ring 0 and decides which thread occupies each CPU core. It manages both ordinary user processes and kernel threads — tasks like kworker and ksoftirqd that do work on the kernel's behalf and show in brackets in ps output. Linux is preemptive: the kernel can interrupt a running task to give the CPU to a higher-priority one, so a CPU-bound loop in one process does not starve the rest of the system.

The kernel/user split is visible in any CPU breakdown. The us column in top is time spent executing user-space code; sy is time in the kernel on behalf of processes; wa is time idle waiting on I/O. A web server pinned at 40 percent us and 50 percent sy is spending most of its CPU inside the kernel — usually on network or filesystem syscalls — and no amount of application tuning fixes that until you cut the syscall volume.

# Watch the user/kernel/iowait split over 1-second samples
vmstat 1
# columns: us = user, sy = system (kernel), wa = iowait, id = idle
pidstat -u 1            # per-process %usr vs %system

When you profile, read sy first. High system time points at the kernel — syscalls, lock contention, interrupt load — and tells you the fix lives in how the program talks to the kernel, not in its arithmetic.