The Stack and the Heap
A program gets memory in two ways. The stack hands it out by moving one pointer and takes it back when the function returns, at almost no cost, but only for data whose life ends with the call. The heap hands out any size for any lifetime, and pays for that with an allocator that must search, split, merge and now and then ask the operating system for more.
Nearly every object in a Python program lives on the heap, integers and strings included. So allocation is a cost a Python service pays on every request, and it explains why the service's memory graph after a traffic spike looks the way it does.
Two Ways to Get Memory
The stack grows and shrinks with function calls, one frame per call, as Chapter 3 drew it. Allocating space on it is one adjustment of a register, and freeing it is the return itself: the frame is gone, and so is everything in it. The catch is the rule that comes with it. Nothing on the stack may outlive the call that made it.
The heap is a large region carved up on request. A block on the heap lives until someone frees it, whichever function asked for it and however long ago. So the choice between the two is a lifetime decision first and a speed decision second: data that must survive the call has to go on the heap.
What an Allocator Does
An allocator keeps free blocks sorted into size classes, such as 16, 32, 48 and 64 bytes and upward. A request is served from the smallest class that fits: a request for 40 bytes gets a 48-byte block. When a class runs dry, the allocator carves a fresh chunk out of memory it obtained from the kernel (Chapter 10) and cuts it into blocks of that size.
Large requests take a different path. In the common Linux C library, requests above about 128 kibibytes by default skip the size classes and get a mapping of their own straight from the kernel (Chapter 10), and that mapping goes back to the kernel the moment it is freed. Small blocks are pooled and mostly stay with the process after they are freed, ready for the next request of the same size.
Fragmentation
Rounding a 40-byte request up to a 48-byte block wastes 8 bytes inside the block. That is internal fragmentation. Freeing blocks in a different order from the one they were allocated in leaves free memory in pieces too small or too scattered to serve a larger request. That is external fragmentation.
The third effect is the one engineers notice. A chunk goes back to the operating system only when every block in it is free, so one long-lived object pins a whole chunk. Hours of mixed allocation leave the heap as a patchwork of mostly empty chunks, each held by a few survivors.
Where Python Puts Things
In Python, a variable is a reference, and almost every value it refers to is an object on the heap; the exceptions are a few shared constants built into the interpreter, such as the small integers below. CPython serves objects of up to 512 bytes from its own small-object allocator, which takes memory in arenas of 1 mebibyte on 64-bit builds, cuts each arena into pools, and cuts each pool into blocks of one size. An arena goes back to the operating system only when every block in it is free.
The commonest integers cost nothing to allocate: CPython keeps one shared object for each integer from minus 5 to 1,024 in version 3.15, a range that stopped at 256 in earlier versions. Every other integer a calculation produces is a new heap object of 28 bytes or more. On CPython 3.15, creating a million integers above that range took about 36 nanoseconds each, and a million small three-key dictionaries about 220 nanoseconds each.
records = [{"id": i, "title": str(i) * 3} for i in range(2_000_000)]
keep = records[::1000] # 2,000 survivors, scattered through the heap
del records # 99.9% of the records are now garbage
The listing builds two million small records, keeps every thousandth one, and deletes the list that held the rest. Measured on CPython 3.15 on macOS, the process grew from 16 to 586 megabytes while building the records. After the delete, with 99.9% of them freed, it still held 586 megabytes, because the 2,000 survivors were spread across nearly every arena. Only when the survivors were dropped too did it fall to 36.
Other Languages Decide Differently
C and C++ leave the choice to the programmer, who picks the stack or the heap for every value. Go's compiler runs escape analysis: it places an object on the stack when it can prove that no reference to it outlives the call. Java's just-in-time compiler does a similar analysis and can remove a provably local allocation altogether. Rust puts values on the stack by default and on the heap only when the code asks.
A garbage collector with a young generation (see Garbage Collection) changes the arithmetic again. There, a heap allocation is a pointer bump almost as cheap as the stack's, and the bill is settled later, at collection time, which is the next topic.
Allocation in a Long-Running Service
Serializing a response of 10,000 records in Python creates tens of thousands of small objects per request, each one a trip through the allocator now and a free later. A long-running worker's resident memory climbs after a traffic spike and stays near its peak, because the allocator keeps the fragmented chunks rather than returning them. That plateau is not a leak, and restarting the worker is often the right fix.
Threads that allocate at the same moment contend for the allocator's shared state unless it keeps a cache per thread. The GNU C library, jemalloc, tcmalloc and mimalloc all keep per-thread caches for this reason, and swapping allocators is a real tuning step for services that allocate heavily from many threads.
- "Stack memory is faster memory." Both are the same RAM. What is cheap on the stack is the allocation, one register adjustment, and the locality, since the top of the stack is almost always in L1. The memory itself is no faster.
- "Deleting a variable gives the memory back to the operating system." It removes a name. The object is freed when its last reference goes (see Garbage Collection), the block returns to CPython's allocator, and the process shrinks only if a whole arena empties. Measured on 3.15, keeping one record in a thousand kept the process at its full 586-megabyte peak.
- "Allocation is free in a modern language." A fast-path allocation is tens of instructions, a slow path searches, splits or calls the kernel, and every freed object is work for someone later. In Python, every intermediate integer above 1,024 is a new heap object.
- "Fragmentation is a C programmer's problem." Python, Ruby and Node services all run on a C allocator underneath, and hours of mixed-size allocation leave resident memory well above the live data.
- "Large and small allocations behave the same." A large block gets its own mapping and goes back to the kernel on free. Small blocks are pooled and mostly stay with the process.
- Allocate outside hot loops and reuse buffers. Each allocation costs allocator work now and reclamation work later.
- Store large homogeneous data in one contiguous allocation, such as a bytes object, an array or a NumPy buffer, rather than millions of objects. One allocation replaces millions, and the data stays together (Chapter 5).
- Track live data and resident size as two separate numbers, and expect resident size to plateau above live data. The gap is the allocator holding fragmented chunks, not a leak.
- Recycle long-running worker processes on a schedule when fragmentation, not a growing object count, is the diagnosis. A fresh process starts with an unfragmented heap.
Knowledge Check
Why is allocating on the stack cheaper than allocating on the heap, and what is the catch?
- Stack memory is a faster kind of RAM, but it is limited to a few kilobytes
- It is one pointer adjustment, but the data cannot outlive the call
- The stack never needs freeing, but it is shared by all threads at once
- The operating system prepares stack blocks in advance, but only for integers
A Python worker builds a large list of records during a spike, then deletes the list but keeps a few hundred of the records. Resident memory stays near its peak. Why?
- The list is still referenced by the interpreter until the process restarts
- Python never frees any object until the cycle collector runs later on
- The kept records are scattered, so few arenas are empty enough to release
- The operating system delays taking memory back until the machine runs low
An allocator rounds every request up to a size class: 16, 32, 48, 64 bytes and so on. What does that design trade?
- It trades speed for memory: lookups are slower, but nothing is ever wasted
- It wastes a few bytes per block to make finding and reusing blocks fast
- It prevents fragmentation entirely, at the cost of calling the kernel often
- It makes large requests cheaper, since they are served from the same classes
A Go function creates a small struct and uses it only inside the function. What does escape analysis let the compiler do?
- Delete the struct's fields that the function never happens to read
- Move the struct into a global variable that all calls can share
- Allocate the struct on the heap early, before the function is ever called
- Place the struct on the stack, since no reference outlives the call
Resident memory of a long-running service rises after a spike and then stays flat for days, while the count of live objects returns to normal. What is the most likely diagnosis?
- Fragmentation: the allocator holds chunks that a few survivors pin
- A leak: some code keeps adding objects that are never released
- A cache that the operating system is filling with the service's files
- Stack growth from deep recursion during the spike that never unwound
You got correct