strace, lsof, and Tracing

When a process is stuck and the metrics tell you that it is stuck but not why, you stop guessing and watch what it actually does. strace intercepts every system call a process makes — every open(), read(), connect(), futex() — and prints the arguments, the return value, and the errno when it fails. lsof answers the complementary question: which files, sockets, pipes, and devices does this process currently hold open. Between the two you can see the exact syscall a hung daemon is blocked on and the exact path it could not read.

These tools answer questions logs and top cannot: why a service that "started fine" is permission-denied on a config it can see, why a port is already in use when no listener is obvious, why disk space stays full after you deleted the log. The catch is cost — strace stops the target on every syscall through ptrace, which can slow a busy process by an order of magnitude or more. That is acceptable for an ad-hoc investigation and dangerous on a hot production path, which is why the eBPF tools exist alongside it.

Tracing Syscalls with strace

strace works through the kernel's ptrace interface: it attaches to a process, and the kernel halts that process and notifies strace on entry to and exit from every system call. You either launch a command under it or attach to a running PID with -p. Attaching needs the same privilege as signalling the process — usually sudo — and on Ubuntu the Yama LSM restricts non-child ptrace by default, so attaching to an unrelated PID requires root or a relaxed kernel.yama.ptrace_scope.

Raw output is a firehose, so the filters earn their keep. -e trace= narrows to a class or to named calls — -e trace=openat,read for file access, -e trace=network for sockets. -f follows children across fork and clone, which matters because the process you launched is often a wrapper and the real work happens in a child. -c drops the per-call trace for a summary table — count, time, and errors per syscall — the fastest way to see that a process spent its time in futex or burned a million failing stat calls.

# Attach to a running PID, follow children, timestamp each call
sudo strace -f -tt -p 4821

# Trace only file-opening syscalls of a command you launch
strace -e trace=openat,access myapp --config /etc/myapp.conf

# Count syscalls instead of printing them — the time/error summary
sudo strace -f -c -p 4821

Reading the errno

The most useful thing in a trace line is the failure code at the end. A line like openat(AT_FDCWD, "/etc/myapp/cert.pem", O_RDONLY) = -1 EACCES (Permission denied) names the syscall, the exact path, and the reason in one place — no log statement required, because the kernel told you. The errno is the diagnosis: ENOENT means the file is genuinely missing (often a typo or a relative path resolved from the wrong directory), EACCES means it exists but the process lacks permission, EADDRINUSE means the port is taken, ECONNREFUSED means nothing is listening on the other end.

That turns "the service won't start and the log just says failed" into a specific, actionable fact. A process looping on EACCES for a directory it can list is missing the x traverse bit, not r. A daemon blocked forever in read() on a socket is waiting on a peer that never replies — a network problem, not a CPU one. Skipping past the errno to "restart it and see" throws away the one piece of evidence the trace was run to collect.

Open Files and Sockets with lsof

lsof — list open files — leans on the Unix premise that almost everything is a file: regular files, directories, sockets, pipes, and devices all show up the same way. lsof -p 4821 lists everything one process holds open; lsof /var/log/app.log lists every process holding that specific file, which is how you find what is keeping a filesystem busy or blocking an unmount. For networking, lsof -i :443 names the process bound to a port — the answer to "address already in use" — and lsof -i TCP -nP shows established connections without the slow DNS and port-name lookups.

The case that surprises people is reclaimed-but-not-freed space. When a process holds a file open and someone deletes it, the directory entry is gone but the inode and its blocks survive until the last descriptor closes — so df shows the disk full while du finds nothing to blame. lsof is the only tool that connects the two: it lists the descriptor with a (deleted) marker and names the PID still holding it. The fix is to restart or signal that process, not to hunt for files that no longer have names.

# Which process is bound to port 443 — the "address in use" answer
sudo lsof -i :443 -nP

# Find deleted-but-held files still eating disk space
sudo lsof -nP +L1          # files whose link count is below 1
sudo lsof -nP | grep '(deleted)'

# Everything a single process has open
sudo lsof -p 4821

The Overhead Caveat

strace is not a passive observer. Every traced syscall causes two context switches into strace and back, so a process that makes millions of small reads can slow to a fraction of its normal speed — syscall-heavy workloads routinely show 10x to over 100x slowdowns under strace. On a quiet process you will not notice; on a database under load, attaching strace can push latency past timeouts, trip health checks, and cause the cascading failure you were trying to debug.

Treat it as a scalpel, not a monitor. Attach briefly, capture what you need, and detach. Narrow with -e trace= so you are not stopping the process for syscalls you do not care about. When the target is a production service handling real traffic, reach for an eBPF tool instead — it reads the same events through in-kernel probes without halting the process on every call.

Beyond strace — ltrace, perf, and eBPF

strace sees the kernel boundary; it cannot see calls that stay in user space. ltrace fills that gap by intercepting library calls — it shows the malloc, strcmp, or SSL_read a program makes against shared libraries, which is where the logic often lives. perf goes the other way: perf trace is a lower-overhead syscall tracer, and perf top and perf record sample the CPU to show which functions actually burn cycles, kernel and user space together.

The production answer is eBPF — small, verified programs the kernel runs at trace points and kprobes with minimal cost. bpftrace gives you awk-like one-liners over those events, and the bcc tool collection (opensnoop, execsnoop, tcpconnect, biolatency) packages the common investigations. The defining difference is safety and overhead: the in-kernel verifier rejects any program that could loop forever or touch bad memory, so a bad probe cannot crash the box, and events are aggregated in the kernel instead of stopping the process on each one. Debian and Ubuntu ship kernels recent enough that bpftrace installs from apt with no custom kernel; the same holds for current RHEL-family kernels.

# Library calls a program makes — user space, where strace cannot see
ltrace -e 'malloc+free' myapp

# Lower-overhead syscall trace, tolerable under more load than strace
sudo perf trace -p 4821

# eBPF one-liner: every file opened, system-wide, near-zero overhead
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

# bcc tool: trace new processes as they exec
sudo execsnoop-bpfcc