Processes, Threads and the Event Loop

A service serves many requests at once, and there are three ways to do it: a process per request, a thread per request, or one loop that switches between requests whenever one of them is waiting on the network. Stagedoor's api uses the loop. Its worker uses a thread pool for rendering PDFs. Knowing which is which explains why a slow database call stalls one of them and not the other, and why the 35 unexplained milliseconds in the previous topic grow under load.

The choice is not a matter of taste. It is decided by one question about the work: when a request is in flight, is it mostly waiting for something else, or mostly computing? The seat hold waits for 130 of its 180 milliseconds. The PDF render computes for four seconds straight. Those two workloads want different machinery, and putting either on the other's machinery is a ceiling that shows up under load rather than in tests.

Three Models, One Question

A process per request isolates completely and costs megabytes of memory each; it is how Postgres itself serves connections, and it is why Postgres needs a connection pooler in front of it. A thread per request shares memory and costs a stack each, eight megabytes of address space reserved on Linux by default, of which only the pages the thread touches become resident; and in Python the interpreter's global lock decides how much of that parallelism is real. An event loop runs one thread and switches between requests at every await, so a request that is waiting costs almost nothing, and a request that is computing stalls every other request on the instance for as long as it computes.

Where a Request Waits

The database round trip, the Redis round trip, the Payrail call, the buyer's slow upload of a seat-map image. For the hold request, 130 of 180 milliseconds are spent waiting for a socket to have something to say. During those 130 milliseconds the loop runs other requests, which is why one process on one core can hold thousands of in-flight requests and why the loop wins for the api. It is also why one call that does not await (a synchronous driver, a time.sleep, a CPU-bound loop) freezes every request on the instance until it returns.

The Blocking Call That Freezes the Loop

Stagedoor's first version rendered ticket PDFs inside the POST /orders handler. The render took four seconds of pure CPU. For those four seconds the event loop ran nothing else: not the seat map, not the health check, not the other buyer's hold. The symptom on the dashboard was P99 latency spiking on endpoints that had done nothing wrong, which is the signature of a blocked loop and the reason Chapter 14 puts loop lag on the dashboard as a first-class metric.

# blocks the loop for 4 s; every other request on the instance waits
async def place_order(...):
    pdf = render_tickets(order)            # plain CPU work, no await inside

# hands the CPU work to a thread; the loop keeps serving
async def place_order(...):
    pdf = await asyncio.to_thread(render_tickets, order)

# what Stagedoor actually does: not in the request at all
async def place_order(...):
    await svc.outbox.add("render_tickets", order.id)   # the worker renders it

The three versions do the same work and differ only in who waits for it. The first freezes the instance. The second moves the render to a thread so the loop keeps serving, which is right for a short computation the response depends on. The third is Chapter 8's answer: the buyer does not need the PDF to see the order confirmed, so the render leaves the request entirely and the response returns in 90 milliseconds instead of 4,000.

Threads for the CPU Work

The worker renders PDFs in a thread pool sized to the machine's cores, and hands only its I/O to its own event loop. Before Python 3.14 that pool was less parallel than it looked: the interpreter's global lock let threads interleave CPU work rather than run it side by side, so a pool of eight threads rendering PDFs was one core doing eight things slowly. Python 3.14 is the version where the free-threaded build became a supported way to run the interpreter, with the lock removed and the threads genuinely parallel. Stagedoor's worker runs that build; the book names the version because it is the version where the mental model changed.

Workers Times Instances

Uvicorn runs one event loop per process, and a loop uses one core. api-01 has four cores, so it runs four processes, and the service has eight loops across its two hosts. Each loop holds its own connection pool of twenty, which is where Chapter 6's arithmetic starts: 8 loops × 20 connections is 160, against a Postgres max_connections of 200, leaving forty for the worker, the migrator and a human with psql. The pool size that looks like twenty in the config is 160 on the database, and a change to either number has to be made with the other one in view.

Choosing

I/O-bound work with many concurrent requests wants the loop. CPU-bound work wants threads or processes. Mixed work wants a loop that offloads its computation to a pool, which is what Chapter 14 tunes. The wrong choice is not a bug that a test catches. It is a ceiling: a service that works perfectly at 100 requests a second and falls over at 800, with nothing in the logs but latency.