Chapter Fourteen · Performance and Scale

Performance and Scale

The load test said two instances carry 4,500 requests a second and the forecast for the biggest event of the autumn is 6,000, with 20,000 buyers arriving in the same second at 19:00:00. Six topics turn the numbers of Chapters 12 and 13 into the plan for that night: a profile that finds the time in the encoder rather than the database, a third and fourth instance that did nothing until a sticky-session setting was deleted, a token bucket per buyer and a bound per path, a waiting room built from a sorted set, the CPU moved off the loop, and an honest last page about the day one service stops being enough.

6 topics

Everything in this chapter starts from a number somebody measured. The run of Chapter 12 found the knee at 4,500 requests a second with api-01's CPU at 100 percent, and the dashboard of Chapter 13 says which gauge moves first when it happens again. Marek profiles one checkout and finds that the 90 milliseconds everyone had charged to the database were 25 milliseconds of queries and 65 of serialization, validation and log formatting; three fixes take it to 32 without touching a query. He adds a third and a fourth instance and watches them idle for a day at 5 percent each, because the load balancer still pinned every returning buyer to the host they had started on, a setting that outlived the in-memory sessions Chapter 5 removed by two chapters. The pool arithmetic of Chapter 6 is redone for sixteen loops and does not fit, and PgBouncer goes in front of the primary.

Then the chapter turns to the night itself. A token bucket in Redis, atomic in one Lua script, limits each buyer to 10 holds a minute and each scanner key to 50 in a burst, keyed on the identity whose fairness it protects and placed in the rings on the right side of the password hash; a bound of 20 checkouts per loop refuses the twenty-first at the door with a 503 that arrives in a millisecond instead of after a 5-second pool wait. A waiting room puts the 20,000 buyers in a sorted set by arrival time, admits 300 a second because 300 is what the hold path measured, and hands each one a 10-minute token that is the only key to the seat map, so that the seats are sold in order rather than by whoever's retry landed. The CPU that had been stalling the loop, the 100-millisecond hash, the 4-second render, the 40-millisecond parse, goes to a bounded thread pool, a process pool or the worker, and loop lag on the dashboard is how the next one is found.

No wound opens here and none closes; the three from Chapter 1 were closed by Chapters 6, 7 and 8, and this chapter is what keeps them closed at 6,000 requests a second. On Stagedoor's first waiting-room night the sorted set at 20,000 members cost nothing, the primary held at 40 percent, and the ceiling was Payrail's 100 charges a second, which nobody had load-tested because the fake had no limit. The last topic is the book's handoff: the four signals that justify splitting one service, the render pipeline as the safe first split along the stream that already exists, the data boundary as the cost to price before any other, and the two courses, Middleware Deep Dive and System Design, that begin where this one stops.

From the load test's number to the on-sale night: what each topic adds
Profile90 ms → 32, queries unchanged
Scale out4,500/s → 8,400/s
Limit and shed429 per caller, 503 per service
The waiting room20,000 in, 300 a second through
CPU off the looplag 200 ms → 2
The nightthe ceiling, named
Beyond one servicethe handoff

Topics in This Chapter

Topic 72
Where the Time Goes
The trace for the calls, py-spy for the gaps between them, pg_stat_statements for the many and the slow. The checkout's 90 milliseconds were 25 of queries and 65 of encoder, validator and logger, and three fixes take it to 32 with the database untouched.
Performance
Topic 73
Scaling Out — Stateless Is the Price
N instances behind a balancer, and the sticky-session setting that left two of them at 5 percent for a day. The pool arithmetic at sixteen loops, the three singletons that N cannot touch, the autoscaler's numbers, and why scaling down needs the drain.
Performance
Topic 74
Rate Limiting and Backpressure
A token bucket in Redis as one atomic script, keyed on the buyer, the API key or the trusted address, placed on the right side of the password hash, answered with 429 and a true Retry-After. Backpressure as the same idea for the service's own capacity, answered with 503.
Reliability
Topic 75
The On-Sale Day
Twenty thousand buyers in one second for two thousand seats, and the waiting room that admits 300 a second from a sorted set with a 10-minute token as the only key to the seat map. Where N comes from, what fair means, and the ceiling the first night found.
Performance
Topic 76
Async Where It Pays
Loop lag as the P99 of every request on the instance, what belongs on the loop and what does not, the thread pool, the free-threaded build and the process pool, and the three sync handlers that say why beside each. Async is a tool for waiting well, not a badge.
Architecture
Topic 77
What Is Beyond One Service
The four signals that justify a split and the one that does not, the render service as the safe first cut along the stream that exists, the data boundary as the price, and the handoff to Middleware Deep Dive and System Design with one warning.
Architecture