eBPF

eBPF runs small, verified programs inside the running kernel, attached to events — a syscall entry, a packet arriving on a NIC, a function call deep in the kernel, a tracepoint. You compile a program to eBPF bytecode, the kernel's in-kernel verifier checks it cannot loop forever, dereference a wild pointer, or read memory it shouldn't, then a JIT compiles it to native code and runs it on every hit of that event. No kernel module, no reboot, no patched kernel — and if your program is unsafe, the verifier rejects it at load time instead of panicking the box at runtime.

That safety property is the whole story for a server operator. Before eBPF, instrumenting the kernel in production meant a custom module that ran with full ring-0 privileges and could take the machine down with one bad pointer, so nobody did it on a box that mattered. eBPF turns "instrument the kernel" from an outage risk into something you run on a loaded production host with single-digit-percent overhead — which is why the modern observability, networking, and runtime-security stack (bcc, bpftrace, Cilium, Falco, Pixie) is built on it.

The Core Idea

eBPF is a tiny virtual machine in the kernel: eleven 64-bit registers, a 512-byte stack, a fixed instruction set, and no unbounded loops in the early designs (bounded loops arrived in kernel 5.3). A program is attached to a hook and runs in the context of whatever triggered it, reading the event's arguments and the kernel state around it. Because it runs in kernel space, it sees everything the kernel sees — but it cannot do everything the kernel can, and that constraint is enforced, not trusted.

The verifier is what makes this safe rather than reckless. At load time it walks every possible path through the program, proves it terminates, proves every memory access is in-bounds, and rejects anything it cannot prove. A program that passes is guaranteed not to crash the kernel; a program the verifier can't reason about is refused, even if it would have been fine. The result is a genuinely new capability: arbitrary, operator-supplied code running in ring 0 that is safe by construction, not by careful review.

What It Powers

Observability is the entry point most people meet first. Tools like bpftrace and the bcc collection attach to tracepoints, kprobes, and uprobes to answer questions that no /proc file exposes — per-process disk latency histograms, which files a process opens, off-CPU time, syscall error rates — all without changing the application or restarting anything. The data is gathered in the kernel and only summaries cross into user space, so the overhead stays low even under load.

Networking is the second large domain. eBPF programs at the XDP hook run on the network driver before the kernel even builds an sk_buff, which is how DDoS scrubbers and load balancers drop or redirect millions of packets per second per core. Cilium replaces iptables-based Kubernetes networking and policy entirely with eBPF, and the third domain — runtime security — uses the same hooks to observe and block syscalls in real time, which is what tools like Falco and Tetragon do.

Hooks and Maps

A hook is where a program attaches. The common ones are kprobes (any kernel function entry or return, fragile across kernel versions because internal symbols change), tracepoints (stable, kernel-maintained instrumentation points — prefer these), uprobes (user-space function entry, for tracing applications), XDP and tc (network ingress/egress), and LSM hooks (security decisions). The choice of hook decides both what you can see and how portable your program is across kernel versions.

Maps are the other half of the model: typed key/value stores that live in the kernel and are shared between the eBPF program and user space, and between separate eBPF programs. A program writes counts, histograms, or per-PID state into a map on every event; a user-space process reads the map periodically to render output. This split — heavy per-event work in the kernel, cheap aggregated reads from user space — is exactly why eBPF tools stay cheap where strace does not.

# count system calls by name across the whole machine for 10s
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[probe] = count(); }'

# histogram of block I/O latency in microseconds (bcc tool)
biolatency

# which processes are opening which files, live
opensnoop

The Tooling

bpftrace is the awk of kernel tracing: a one-line language for ad-hoc questions, ideal when you don't yet know what you're looking for. The bcc project ships dozens of ready-made tools — execsnoop, opensnoop, biolatency, tcpconnect, runqlat — that answer specific production questions out of the box. On Debian and Ubuntu both arrive via apt install bpftrace bpfcc-tools; on RHEL family it is dnf install bpftrace bcc-tools. Both need root or CAP_BPF plus CAP_PERFMON on recent kernels.

The reason these tools beat strace under load is mechanical, not cosmetic. strace uses ptrace, which stops the target twice per syscall to hand control to the tracer — that can slow a syscall-heavy process by 100x or more, enough to change its behavior or trip timeouts. An eBPF program runs in-kernel at the hook with no context switch to a tracer and aggregates in a map, so tracing every syscall on a busy server costs a few percent. For a one-off look at a single process, strace is still faster to reach for; for anything running on a host under real traffic, eBPF is the only safe option.

Kernel Version and Portability

eBPF features are kernel-version-gated, and the gap between distributions is large. The verifier, the available hook types, bounded loops (5.3), BPF LSM (5.7), and the CO-RE / BTF machinery that lets one compiled program run across kernels all landed in specific releases. A Debian 12 kernel (6.1) or Ubuntu 22.04 (5.15) has a rich feature set; an old enterprise 4.x kernel has eBPF but a far narrower one. Check the running kernel before assuming a tool will load.

Portability used to mean compiling the eBPF program on the target host against its exact kernel headers, because kprobe-attached programs read kernel structs whose layout shifts between versions. CO-RE (Compile Once, Run Everywhere), built on BTF type information and libbpf relocations, removed that: a single compiled object adapts to the running kernel's struct layout at load time. Tools built on libbpf with CO-RE ship as one binary; older bcc tools still carry a compiler and kernel headers and recompile on first run, which is why they pull in clang and the headers as dependencies.