Async Where It Pays

Chapter 1 chose the event loop for the api because a request mostly waits, and it was right: the hold path spends 130 of its 180 milliseconds waiting on a socket, and one loop on one core holds thousands of those in flight for almost nothing. This topic closes that decision with what a year of running it taught. The loop wins exactly as long as the work is waiting, and it loses the moment something computes. The PDF render, the argon2 hash, the seat-map serialization and a large jsonb parse each stalled every other request on the instance for as long as they ran, and each was found the same way, as a spike in the P99 of endpoints that had done nothing, and moved the same way, off the loop.

What came out of the year is not "async everywhere." It is: await every socket on the loop, put every millisecond of CPU in a pool or in the worker, and measure loop lag to know which is which. The keyword changes where a function waits. It does not change where the CPU is spent, and a year of P99 graphs is the evidence.

The Signal

Event-loop lag is the time between when a task was ready to run and when the loop actually ran it, and it is the one number that says whether the loop is doing what Chapter 1 promised. Stagedoor measures it the simplest way: a task sleeps for 100 milliseconds and records how much later than 100 it woke up, every 100 milliseconds, as the stagedoor_loop_lag_seconds histogram of Chapter 13. At idle it reads 0.1 milliseconds. Under the on-sale's load it reads 2, which is the cost of 3,000 requests a second taking turns. And on the night the render ran on the loop it read 200, because a 4-second render is 4 seconds of lag for everything scheduled behind it, and the histogram's bucket boundaries only go so high.

Lag is the P99 of every request on the instance, added. A request that would have taken 8 milliseconds takes 8 plus whatever the loop was doing when its socket became readable, and that "whatever" is shared by every request in flight, which is why a blocked loop shows up as unrelated endpoints failing together and never as the endpoint that blocked it. The trace of Chapter 13 sees the lag as gaps between spans; the profiler of Topic 72 sees the function that caused it; the lag gauge is the number that says to look. Chapter 13 alerts on it at 50 milliseconds for 2 minutes, in the buyer's words, "every request on api-01 is slow," and the alert has fired three times in a year, each time for a computation someone put on the loop.

What Belongs on the Loop

Every call that awaits a socket: the query through the async driver, the Redis command, the Payrail call through the async client, the read of the client's own request body, the write of the response. And the handler's own logic, when that logic is a few microseconds of Python between awaits. The hold path is the model. It parses a request, checks the admission token, takes a connection from the pool, runs five statements and a commit, deletes a cache key, sends an outbox row, and serializes a small response, and the CPU it spends on all of that is 200 microseconds per request; the other 130 milliseconds are sockets. At 300 holds a second the loop is busy for 60 milliseconds of every second on the hold path, and idle, which is to say available, for the rest.

async def hold_seat(req: HoldRequest, ctx: Ctx) -> ORJSONResponse:
    admission.verify(ctx.headers, event_id=req.event_id, user=ctx.principal)   # 20 µs: one signature check
    async with ctx.pool.connection() as conn:                                   # await: the pool
        async with conn.transaction():
            seat = await seats.lock(conn, req.seat_id)                             # await: FOR UPDATE
            if seat.status != "available":
                raise SeatTaken(seat.label)                                     # 409, Chapter 6
            hold = await holds.create(conn, seat, ctx.principal, minutes=10)     # await: two statements
            await outbox.add(conn, "seatmap.invalidate", {"event_id": req.event_id})
    await ctx.redis.delete(f"seatmap:v2:{req.event_id}")                        # await: one round trip, after commit
    return ORJSONResponse(hold.as_public(), status_code=201)                    # 40 µs: 300 bytes

Every line that leaves the process is an await, and between them the handler does almost nothing: a signature check on the admission token, a comparison, and the serialization of a 300-byte response. That is the shape of a handler that belongs on the loop, and it is the shape of most of Stagedoor's handlers, which is why the loop was the right choice. The loop's cost per request here is the 200 microseconds of Python between the awaits, and the loop's benefit is that during the 130 milliseconds of waiting it is running four hundred other requests.

What Does Not

Four things in Stagedoor compute, and each one stalled the loop before it was moved. The argon2 hash on the login route is 100 milliseconds of CPU by design, as Chapter 5 set it, and on the loop it froze the instance for 100 milliseconds per login; a credential-stuffing run at 40 attempts a second was a denial of service against every buyer on that host, and the rate limiter of Topic 74 was only half the fix. The PDF render is 4 seconds and was the first stall the book met, in Chapter 1, and Chapter 8 moved it to the worker because the buyer does not need it in the response. The seat-map serialization was 30 milliseconds before orjson, and Topic 72 took it to 4, which is under the line but was over it. And the organizer's bulk import parses a jsonb document of 2,000 seats, 60 KB, in 40 milliseconds of pure Python, on the loop, on a Tuesday afternoon while a buyer's checkout waited behind it.

The move for each is one of two. Work whose result the response needs goes to a thread pool through asyncio.to_thread, which hands the function to the loop's executor and awaits the result, so the loop keeps serving while a thread computes; the hash and the import went this way. Work the response does not need goes to the worker through the outbox and the stream, which is Chapter 8's answer and the render's. The executor is bounded, 4 threads per process on the four-core hosts, because an executor sized to the request rate is a thousand threads for a thousand hashes and the memory limit of Chapter 11 ending the process; a login that finds all 4 threads busy waits its turn in the executor's queue, which is the right place for it to wait, since it is one buyer waiting instead of every buyer.

# before: 100 ms of CPU on the loop; every request on the instance waits behind each login
async def login(req: Login, ctx: Ctx):
    ok = hasher.verify(user.password_hash, req.password)         # no await inside: the loop is frozen

# after: the same call in the executor; the loop serves 400 other requests meanwhile
async def login(req: Login, ctx: Ctx):
    ok = await asyncio.to_thread(hasher.verify, user.password_hash, req.password)

# at startup: a bounded executor, sized to the cores, not to the request rate
loop.set_default_executor(ThreadPoolExecutor(max_workers=4, thread_name_prefix="cpu"))

The two versions of the login handler differ by one call. The first runs the hash on the loop's thread, and for the 100 milliseconds it takes, no other task on that process runs. The second hands the same function to a thread and awaits its result, so the loop is free between the handoff and the return, and the 100 milliseconds are spent on another core. The third line, run once at startup, sizes the pool to 4 threads so that the number of hashes in flight has a ceiling the memory arithmetic of Chapter 11 can count. After the change the login route's own latency was unchanged at 100 milliseconds, and the P99 of every other endpoint on the instance stopped having a login-shaped spike in it.

Threads, Free-Threaded Python, and Processes

A thread pool is the right home for two kinds of work. Synchronous I/O in a library that has no async form, a driver or a client that blocks on its socket, belongs in a thread because the thread sleeps on the socket and the interpreter lock is released while it does. CPU work also belongs in a thread when the code doing the computing releases the lock, which the argon2 binding does, which the compression and image libraries do, and which pure-Python code did not until Python 3.14's free-threaded build, the version Chapter 1 named as the one where the mental model changed. On the standard build, a pool of 4 threads running pure-Python parsing is one core doing four things slowly, with the loop's thread competing for the same lock; on the free-threaded build the four threads run on four cores, and the worker's render pool, which runs that build, is the proof.

CPU work in pure Python on the standard build wants a process pool: the same run_in_executor call with a ProcessPoolExecutor, at the cost of pickling the arguments and the result across the process boundary, which for a 60 KB document is under a millisecond and for a 2 MB one is not. And work that should not be in the request at all wants the worker, where Chapter 8 already built the queue, the retries, the dead letters and the age gauge, and where a 4-second job is a 4-second job rather than a 4-second stall. Stagedoor's api runs the standard build with a thread executor of 4 for the hash and the compression, and sends everything else to the worker; the choice between thread and process was made per function by reading whether the library releases the lock, which its documentation says and a 10-second test with py-spy confirms.

The Sync Handler Escape

A FastAPI handler written as plain def is run by the framework in its own thread pool, 40 threads by default, and the loop is never involved in the body at all. It is the right choice for a handler that calls a synchronous library end to end and awaits nothing: the whole body blocks in a thread, the loop keeps serving, and the cost is one thread per request in flight. It is the wrong choice for a handler that would have awaited five sockets, because that handler now holds a thread for 130 milliseconds of waiting that the loop would have done for free, and 40 threads is a ceiling of 40 concurrent requests on a route the loop could have served 4,000 of.

When Chapter 11 counted the API's threads, every handler was async and the framework's pool of 40 sat unused. Since then three def handlers have arrived, each with a comment above it saying why. The venue-image resize on upload runs an image library end to end for 300 milliseconds and touches no socket of its own. The organizer's ticket preview runs the same renderer the worker uses, for one ticket, 400 milliseconds, and the response is the image. The calendar export builds an iCalendar file from rows already fetched, through a library that is synchronous throughout. None of the three awaits anything, none is on a buyer's path, and each would have been a 300-millisecond stall on the loop or a pointless to_thread around a body that was entirely sync. The comment is the rule: a sync handler is a decision, written down, never a habit.

Measuring, Not Believing

Three practices keep the loop honest. Loop lag is on the dashboard beside the RED lines and alerts at 50 milliseconds, so a computation that lands on the loop is found by its lag and not by a buyer's complaint. A profile under load, as Topic 72 does it, is re-run after every change to a hot path, and any function that spends more than a millisecond of CPU on the loop's thread is a candidate for the executor; the seat-map serialization was found at 30, the import at 40, and the hash was known. And the third is the one that surprises people: async has a cost of its own, because every await is a scheduling point where the loop may run something else, and a handler that awaits one tiny thing at a time, one Redis command per seat instead of one pipeline for 2,000, pays 2,000 scheduling points and 2,000 network round trips for what one command could have done. Async is a tool for waiting well. It is not a badge, and a function does not get faster by being declared async; it gets scheduled.