CPU and Memory Analysis

CPU and memory analysis is the work of deciding whether a slow server is starved of cores, blocked on I/O, or quietly being killed by the kernel's out-of-memory reaper — and proving it with numbers rather than guessing. The toolset is small and stable: uptime and vmstat for the system-wide pulse, top and htop for the live per-process view, mpstat for per-core breakdown, and free plus /proc/meminfo for where the RAM went. Every one of these reads from /proc, so the data is the same numbers the kernel sees; the skill is reading the columns correctly.

The two questions that trip people up live in this topic. "Is the CPU busy?" is not answered by load average — load counts everything waiting, including processes blocked on disk. "Is memory full?" is not answered by the first line of free — Linux deliberately spends almost all free RAM on page cache and hands it back on demand. Misreading either number sends you scaling the wrong resource, and the bill or the outage follows.

Load Average versus CPU Utilization

Load average is the number of processes in the run queue averaged over 1, 5, and 15 minutes — and on Linux specifically it also counts processes in uninterruptible sleep (state D), which is almost always disk or network I/O wait. A load of 8.0 on an 8-core box can mean every core is pegged, or it can mean two cores are idle and six processes are blocked on a slow NFS mount. Load alone cannot tell the two apart, which is why a high load average with low CPU usage is the classic signature of an I/O problem, not a CPU one.

To know whether the CPU itself is the bottleneck, divide load by core count and then confirm with actual utilization. nproc gives the core count; vmstat 1 and mpstat -P ALL 1 give the real split between user time, system time, I/O wait, and idle. The single most useful column is wa (iowait): if it is high while us and sy are low, the cores are sitting idle waiting for storage, and adding vCPUs buys you nothing.

# Run-queue and CPU breakdown, sampled every second
vmstat 1 5
# r  b   swpd   free   buff  cache   si so   bi bo  in cs  us sy id wa st
# 9  0      0 1.2G    88M  5.4G    0  0   12 40 900 2k 71  6  1 22  0

# us=71 sy=6  -> CPU-bound;  wa=22 -> a fifth of the time waiting on I/O
mpstat -P ALL 1   # per-core: find one saturated core hiding behind a low average

System Time, User Time, and Steal

CPU time splits into categories that point at different culprits. High us (user) time is your application code doing arithmetic and is fixed in your code, not the OS. High sy (system) time means the kernel is doing the work — usually a syscall storm: a process hammering small reads and writes, opening and closing files in a tight loop, or thrashing on locks. When sy rivals us, reach for strace -c on the offending PID to see which syscall dominates before you touch the code.

On any VM — and that includes most production servers today — watch the st (steal) column. Steal is CPU time the hypervisor gave to another tenant while your vCPU was ready to run; it is the cloud-VM symptom of a noisy neighbor or an oversubscribed host. Sustained steal above a few percent means your guest is not getting the cores you are paying for, and the fix is a larger instance or a different host class, not anything inside the guest. A bare-metal server always shows st at 0.

Resident, Virtual, and Shared Memory

Every process reports several memory sizes and they measure different things. VSZ (virtual size) is the entire address space the process has mapped, including memory it reserved but never touched and shared libraries it barely uses — it routinely reads as gigabytes and is almost never what you want. RSS (resident set size) is the physical RAM the process actually occupies right now, which is closer to the truth but double-counts shared pages: a shared library mapped by 40 processes is added into all 40 RSS figures, so summing RSS across a process list overstates real consumption.

The honest per-process number is PSS (proportional set size), which divides each shared page by the number of processes sharing it, so the column does sum correctly to the machine's real usage. PSS lives in /proc/<pid>/smaps_rollup and is what smem reports. For a quick which-process-is-eating-RAM check, sort by RSS; when you need the figures to add up to the truth — capacity planning, a memory-leak hunt across a fleet of identical workers — use PSS.

Reading free and the Page Cache

Linux treats unused RAM as wasted RAM. It fills almost all otherwise-idle memory with page cache — copies of recently read files — and evicts that cache instantly the moment a process needs the pages. This is why the used figure on a healthy server looks alarmingly high and free looks alarmingly low. The number that actually matters is available, the kernel's own estimate of how much a new process could allocate without swapping, computed from reclaimable cache plus genuinely free pages.

On any kernel from the last decade, read the available column and ignore free for capacity decisions. The buff/cache figure is reclaimable, not lost. The signals of real memory pressure are different: rising si/so (swap in/out) in vmstat meaning the system is paging to disk, and entries in journalctl -k from the OOM killer naming a process it terminated to reclaim memory.

# Human-readable, with the column that matters
free -h
#               total   used   free  shared  buff/cache  available
# Mem:           7.8Gi  1.9Gi  220Mi   140Mi       5.7Gi      5.5Gi
# free=220Mi looks scary; available=5.5Gi is the real headroom

# Did the kernel kill something for memory?
journalctl -k --since "1 hour ago" | grep -i "out of memory\|oom-kill"