Resource Limits and the Process Model

A process has a memory footprint, a number of open file descriptors, a number of threads and a share of the CPU, and the platform enforces a limit on each whether or not the service knows what its own numbers are. Stagedoor found out its numbers the expensive way. An organizer uploaded a 40-megapixel venue photograph as a seat-map background, the PDF render rasterized it at print resolution, the render allocated 3 GB on a host with 4 GB, and because the worker's container had no memory limit the kernel's out-of-memory killer took the host's processes down one after another, the two api processes among them. With a limit, the same render would have killed only the worker, which is the point of the limit: a limit turns "the host is down" into "one process restarted."

This topic is the four resources, what the platform does at each limit, where each of Stagedoor's numbers comes from, and how the limits fit the process model of Chapter 1: one event loop per api process, four processes on a four-core host, and a thread pool of 8 in the worker. The numbers are the measured ones, and the last section is the line where the orchestrator's vocabulary begins and Kubernetes Deep Dive takes over.

Memory

An api process at rest is 150 MB: the interpreter, the imported code, the pool's 20 connections with their buffers, the clients, the config. Each in-flight request adds a few hundred kilobytes for its parsed body, its response and its stack, so the 400 in flight on api-01 at the on-sale peak are 100 per process and about 25 MB each. Each connection the balancer holds open to the process keeps its buffers too, a few kilobytes as Topic 10 of Chapter 2 counted them, and the 50 in its share are a rounding error beside the requests. The arithmetic therefore comes to about 175 MB, and the measured peak of one api process over a week that included the load test is 220 MB. The gap is what a measurement always holds and a calculation never does, allocator fragmentation and the odd large response among it, and it is the measurement that the limit is set from: four processes on api-01 are 880 MB. The container's limit is 1.25 GB, above the peak with 40 percent of headroom, and under an orchestrator its request to the scheduler is 1 GB.

The worker is where the outliers live. Its steady state is the same 150 MB, and each of its 8 render threads peaks at 400 MB on a normal seat map, but the eight do not peak at the same instant; the load test with all eight rendering measured 1.6 GB. Its limit is 2.5 GB. The 3 GB render exceeds that on its own, and what happens then is precise: the kernel kills the process, the container's status reads OOMKilled with exit code 137, and the platform restarts it. That is a restart, not a graceful stop; no drain runs, no SIGTERM handler fires, and the job that was rendering is left pending in the stream. Chapter 8's poison-message wrapper never sees the failure, because the failure was outside the process. What catches it is the delivery count from Topic 46: the entry is reclaimed after 60 seconds idle, kills the worker again, is reclaimed again, and on its third delivery goes to jobs:dead without being run. Two extra kills and a few minutes of delayed tickets is the cost of a limit set right; the host down is the cost of no limit.

CPU

A CPU limit does not kill; it throttles. The kernel gives the container a quota per scheduling period, 200 milliseconds of CPU time in every 100-millisecond period for a limit of two cores, and a container that has used its quota waits for the next period. An api process on the event loop uses one core, so four processes on api-01 want four cores, and a limit of two means that for part of every 100 milliseconds all four loops are paused. Nothing errors. Every request just takes longer, and the latency graph shows P99 doubling on every endpoint at once with no change in the code, no error in the log and no dependency slower than yesterday.

That signature, latency up everywhere and errors flat, after a platform change, is the first thing to check against the throttling counter the runtime exports, which counts the periods in which the container wanted CPU and was made to wait. Stagedoor's api container is limited to four cores for its four processes, and requests four; the worker on the free-threaded build renders on all 8 of its threads at once and is limited to 8, because a render pool of 8 on a limit of 4 is eight renders each taking twice as long, which the P95 of 4 seconds for a ticket PDF does not have room for.

File Descriptors

Every open socket is a file descriptor, and so is every open file. For one api process the arithmetic is the listening socket, the 20 connections to Postgres, the Redis client's connections, the Payrail client's keep-alive pool of 20, stdout and stderr, and one descriptor for every inbound connection the process is holding, which is its share of the balancer's pool of 200 rather than the clients' own keep-alives, about 50 at on-sale across the host's four processes. That comes to roughly 100 for the api, comfortably inside any default, which is exactly why nobody had looked at the number until the worker ran out. The default soft limit that most Linux distributions hand a process is 1,024. Some container runtimes raise it and some do not, and the value inside the container is the one that counts, so it is read there with ulimit -n rather than assumed. Stagedoor's is set to 65,536 on every container, from the arithmetic and not from a guess.

Running out looks like nothing else, which is what makes it slow to diagnose. The 1,025th open descriptor fails wherever it happens to be requested: the pool trying to reopen a broken connection, the logger reopening stdout after a rotation, a template file the handler reads once, the accept call on the listening socket. The error is "too many open files" on a code path that has nothing to do with connections, and on the night the on-sale traffic took the worker past 1,024 it was the job stream's reconnect that hit it first, so the symptom was ticket emails stopping while the api, whose own descriptor count never left a hundred, carried on serving every buyer normally.

Threads and Processes

Stagedoor has three sources of threads. The worker's render pool of 8, sized to its cores on the free-threaded build as Topic 04 arranged. The loop's default executor, which asyncio.to_thread uses for the occasional synchronous call and which the interpreter sizes to the core count plus four. And the framework's pool for plain def handlers, 40 threads by default, of which Stagedoor uses none because every handler is async. Each thread reserves an 8 MB stack by default, reserved rather than resident, so 40 sleeping threads cost the memory limit almost nothing and 8 rendering threads cost it 8 times the render's peak. The thread count is an input to the memory arithmetic, not a limit of its own.

Processes are the other multiplier. Four uvicorn processes are four copies of the 150 MB steady state, 600 MB on api-01 before a request arrives, and four pools of 20, which with api-02 is Chapter 6's 160 connections against a max_connections of 200. Both numbers are reasons Topic 62 runs one process per container under an orchestrator: the memory limit, the CPU limit and the pool size then all describe one loop, and the replica count is where four becomes eight.

Measuring

The process reports its own numbers: resident memory, open descriptors against the limit, thread count, CPU seconds, event-loop lag, and the pool's in-use count, as gauges on the metrics endpoint that Topic 69 of Chapter 13 builds. The standard process collector supplies resident memory, the descriptor counts and CPU seconds for free; the thread count, the loop lag and the pool gauge are Stagedoor's own. The limits are set from a week of those gauges under real traffic plus the load test of Topic 67 at 3,000 requests a second, never from a guess, and they are revisited after every load test, because the numbers move: a dependency upgrade added 30 MB to the steady state in July and nobody would have known without the graph.

process_resident_memory_bytes    2.31e+08     # 220 MB: steady 150 + requests + keep-alives
process_open_fds                 104         # balancer pool share + pools + stdout, stderr
process_max_fds                  65536       # raised from 1,024 after the worker ran out
stagedoor_threads                9           # the loop, the default executor, nothing else
stagedoor_loop_lag_seconds       0.004       # 4 ms behind: healthy
stagedoor_pool_in_use            17          # of 20

The snapshot says what the limit has to cover and how close the process is to it. Resident memory is 220 MB against a per-process share of 312 MB in the 1.25 GB limit. Open descriptors are 104 against 65,536, and the gap is the point: the api was never near the old limit of 1,024, while the worker beside it went through it in one night. Threads are 9 because the api renders nothing. Loop lag at 4 milliseconds says no request is blocking the loop. The pool at 17 of 20 says checkout is busy and not yet queueing. A limit set at twice a guess and never revisited is one of two things: a host half-wasted, or one render from a kill, and the graph is the only way to know which.

What the Orchestrator Adds

An orchestrator separates what the scheduler reserves from what the kernel enforces. The request is the reservation: the scheduler places the container on a node with that much memory and CPU unclaimed. The limit is the enforcement: the cgroup the container runs in, the kill at the memory limit, the throttle at the CPU quota. A container that exceeds its request but not its limit is running on the node's spare capacity, and when the node itself runs short of memory, the containers most over their requests are evicted first, which is a stop from outside that no probe predicts and the drain of Topic 60 has to survive. The throttling metric is the orchestrator's, the eviction is the orchestrator's, and the placement is the orchestrator's.

The service's job is to be honest about its numbers and to survive a kill. Honest means the request is the measured steady state plus the per-request share, the limit is the measured peak plus headroom, and the descriptor limit is the connection arithmetic. Surviving a kill means every job is idempotent and every request that is cut off leaves the database in a state a retry can complete, which is Chapters 7 and 8 again. Everything past that, the node's capacity, the scheduler's bin-packing, the quality-of-service classes, is Kubernetes Deep Dive's subject, and the checklist of Topic 62 is what the service hands it.