The Kernel and the System Call

Every program you have written lives in a sandbox. It can compute anything, but it cannot touch a disk, a network card or another program's memory. To do any of those it has to ask the kernel, and asking is not a function call. It is a deliberate trap that switches the processor into a privileged mode, runs kernel code and switches back.

That round trip costs on the order of a hundred times a function call before any work is done. Most of the input and output advice you have heard, from "buffer your writes" to "batch your queries" to "avoid tiny packets", is this one price paid over and over. This topic shows where the price comes from and how to stop paying it per item.

Two Modes on One CPU

A processor runs in one of two modes. In user mode, where your program runs, the instructions that talk to devices, change memory mappings or reconfigure the processor are forbidden: attempting one causes a fault. In kernel mode everything is allowed. The mode is one bit of processor state, not a second computer, and the kernel is ordinary machine code running on the same cores as your program.

Kernel code runs only when something sends the processor across the line, and there are exactly three ways to do that. The program asks, with a system call. A device signals, with an interrupt, such as a network card announcing that a packet has arrived. Or an instruction fails, with a fault, such as a division by zero or a touch of memory that is not mapped. Every entry into the kernel is one of these three.

The Trap

A system call is a numbered request. The program puts the call's number and its arguments into registers, the processor's own small set of named slots that Chapter 3 introduced, and executes one special instruction. The processor switches to kernel mode and jumps to a single fixed entry point that the kernel chose at boot. The kernel looks the number up in a table, checks every argument, does the work, and returns a result or an error code, switching the mode back on the way out.

The checking is not optional. A pointer handed in from user space might point at kernel memory, at another part of the program that the caller is not allowed to overwrite, or at nothing at all, and the kernel treats every such value as hostile until it has verified it. The program cannot jump into the kernel anywhere else, and that single guarded door is all that keeps one program out of another.

What a Crossing Costs

A function call inside your program costs a nanosecond or two. The simplest possible system call, one that asks the kernel for a number it already knows, costs on the order of a hundred nanoseconds on current hardware, and a few hundred on machines running the default protections against the processor attacks disclosed in 2018. The crossing saves and restores registers, switches stacks, and disturbs the caches and branch predictors from Chapter 3 that your program had warmed up. The program then runs cold for a while after it returns.

An empty crossing costs that much. A call that waits on a device adds whatever the device costs, from Chapter 4's ladder: up to a hundred microseconds for a solid-state drive, milliseconds for a spinning disk. A few calls are so common and so simple, reading the current time being the usual example, that mainstream kernels answer them without a trap at all, from a page of memory the kernel keeps updated inside every process. That exception exists because the trap is worth avoiding.

Buffering Is Amortization

Write 100 MiB one byte per call and the program makes about 105 million system calls. At a hundred nanoseconds each, that is roughly ten seconds spent entering and leaving the kernel, with no useful work counted at all. Write the same bytes in 64 KiB chunks and it makes 1,600 calls, less than a fifth of a millisecond of crossing overhead. The bytes are identical. The number of crossings differs by a factor of about 65 thousand.

A buffer is how programs get the second number while writing code that looks like the first. Small writes go into a block of the program's own memory, and only when the block fills does the program cross into the kernel, once, with all of it. This is the same trade as the amortized append in Chapter 1: an occasional expensive operation, spread across many cheap ones. Python's file objects buffer by default, and output to a terminal is buffered one line at a time, so a print statement's output appears a line at a time on a screen and in large bursts when the same program writes to a file.

with open("out.bin", "wb", buffering=0) as f:   # no buffer: every write crosses
    for _ in range(1_048_576):
        f.write(b"x")

with open("out.bin", "wb") as f:                # default: a buffer in the process
    for _ in range(1_048_576):
        f.write(b"x")

The two loops above write the same mebibyte, one byte at a time. The first opens the file with buffering turned off, so every write becomes a system call. The second uses Python's default, a buffer of at least 128 KiB in Python 3.15, so the same million writes turn into a handful of crossings. On the machine used to check this page, running Python 3.15, the first loop took about two seconds and the second about a twentieth of a second: forty times slower for the same bytes. Each unbuffered write there cost about two microseconds, well above the empty-crossing price, because a write to a file also does file-system work inside the kernel every time.

Crossings Hidden in Ordinary Code

The same arithmetic hides in ordinary code. A logger that flushes after every line makes one crossing per line. A loop that sends each small message as its own network write makes one crossing per message, and usually one tiny packet per message as well. A database driver that makes one round trip per row pays the crossing, the network and the database's own overhead a thousand times for a thousand rows. A web server that reads a large file in 4 KiB pieces makes sixteen times as many calls as one reading 64 KiB pieces.

The fix is the same move in every case: fewer, larger crossings. Batch the rows, buffer the log, send the messages together, read in big chunks. To count the system calls a real process makes, the tracing tools in Linux Deep Dive, in its chapter Performance and Troubleshooting, show every crossing as it happens.