The On-Sale Day

At 19:00:00 twenty thousand buyers who have been refreshing the event page for a minute send their first request inside the same second, for two thousand seats. No number of instances serves that as ordinary traffic: four hosts carry 8,400 requests a second across every endpoint, the hold path serves 300, and 20,000 holds in one second is 66 seconds of hold-path work arriving at once. It does not need to be served as ordinary traffic. The seats will take ten minutes to sell whatever happens, because a buyer takes a minute to choose and pay and the holds that expire come back, and the job is to let buyers into the seat map at the rate the hold path can serve, in the order they arrived, with an honest estimate of the wait.

The waiting room is that mechanism, and this topic is how Stagedoor built one from the parts of the book: a sorted set in Redis for the queue, a short-lived token from Chapter 5 as the key to the seat map, the number N from the load test of Chapter 12, the rate limiter of the previous topic to keep the bots out, and the hold transaction of Chapter 6 at its shortest. It is the spine's last appearance: every mechanism here exists because the network delivers 20,000 requests in a second and drops the answers to most of them.

The Herd

Without a gate, the first second looks like this. Twenty thousand requests arrive for the seat map and then for holds. The seat map is cached and survives. The hold path's own bound, the concurrency limit the previous topic put on every hot path, admits a few hundred across the sixteen loops and sheds the rest with fast 503s, which is backpressure doing its job; the 300 holds a second the path can serve are served. But 19,700 buyers now have a refusal and a Retry-After of 2, and at 19:00:02 they all come back, joined by the buyers whose browsers retried anyway and the ones who pressed the button twice. The second wave is 30,000. By 19:00:05 the retries are the storm of Chapter 7, the load balancer's connection table is full, the outer bound of 400 in flight per instance is shedding seat-map reads along with holds, and the buyers who are being served are the ones whose retry happened to land in a free slot, which is a lottery, not a queue.

Every mechanism in the book is present in that picture and every one of them is working. The pool holds, the bound sheds, the cache serves, the breaker stays closed, and the night is still a failure, because the mechanisms protect the service from the herd and nothing protects the buyers from each other. The missing piece is not capacity. It is order.

The Waiting Room

A buyer who opens the event page at 18:59 lands on the waiting room, not the seat map. The page's first request joins the queue: the service adds the buyer to a Redis sorted set keyed on the event, ZADD queue:8812 NX 1791054000123 4471, with the arrival time in milliseconds as the score, and answers with the buyer's position, which is one sorted-set rank lookup. Once a second, a job running under the scheduler's lock of Chapter 8 pops the next N members from the head of the set and marks each admitted in Redis with a 10-minute expiry. The page polls its position every 3 seconds, and the poll that finds the buyer admitted returns an admission token: a signed token from Chapter 5, 10 minutes of life, bound to this event and this buyer. The seat map for an event that is on sale is served only to a request that carries a valid admission token, and so is the hold path. A buyer without one sees the waiting room and nothing else.

POST /events/8812/queue                              # join: ZADD queue:8812 NX <now_ms> <user>
202 Accepted
{ "position": 14812, "estimated_wait_seconds": 50 }     # ZRANK, and position / 300

GET /events/8812/queue                               # poll, every 3 s
200 OK
{ "position": 6200, "estimated_wait_seconds": 21 }

GET /events/8812/queue                               # the poll that finds admit:8812:<user> set
200 OK
{ "admitted": true, "admission_token": "eyJhbGciOi..." }   # 10 min, sub = user, evt = 8812

GET /events/8812/seats                               # the seat map, only with the token
Admission-Token: eyJhbGciOi...
200 OK                                               # without it: 403, and the waiting room again

The exchange is three requests and then the seat map. The join adds the buyer to the set if she is not already in it and answers with her position and an estimate. The poll repeats the rank lookup, which is 200 microseconds against a set of 20,000, and the poll that finds the admission mark answers with the token instead of a position. From then on the seat map and the hold path check the token's signature, its event and its expiry the way Chapter 5 checks any token, and a request without it gets a 403 and a pointer back to the queue. The page never sees the seat map early, a script cannot skip the queue by calling the seat-map URL directly, and the only way to a hold is through the head of the sorted set at N a second.

The Number N

N is 300 admissions a second, and it is not a guess. The load test of Chapter 12 ran 300 holds a second for ten minutes at the plateau with the pool acquire wait flat and the hold transaction at 5 milliseconds, and that is the rate the hold path is known to serve with the rest of the night's mix beside it. The arithmetic behind N includes what an admitted buyer does next: each one reads the seat map, 300 a second more against a cache the early refresh of Chapter 9 keeps warm, and roughly a third of them check out within the minute, 100 checkouts a second, each 400 milliseconds of Payrail and one of the 20 checkout slots per loop. Three hundred admissions, three hundred seat-map reads, three hundred holds and a hundred checkouts a second is the on-sale mix of Chapter 12 exactly, which is the point: the waiting room turns 20,000 arrivals into the traffic the service was tested for.

On the night the dashboard of Chapter 13 shows admissions, holds and checkouts a second as three lines on one panel, with the queue's length beside them. Admissions at 300 and holds at 300 is the room working; holds falling to 200 while admissions stay at 300 is the hold path slower than the test said, and the person watching lowers N by config before the buyers notice. An N from a guess protects nothing: 1,000 admissions a second against a hold path that serves 300 is the herd again, 3 seconds later and with a token.

Fair, and Seen to Be Fair

Fair means first-come by arrival, and the sorted set is ordered by the arrival timestamp, so the buyer who joined at 18:59:00 is admitted before the one who joined at 18:59:01 and no amount of activity changes that. One place per buyer: the set's member is the buyer's id, and the join uses NX, so a second join from the same buyer, a refresh, a second tab, a script, does not update her score and does not move her, which is what neutralizes the refresh that decided the old lottery. The estimate on the page is the position divided by N, in seconds, and it is honest because N is real: position 14,812 is 50 seconds, and a buyer told 50 seconds who waits 50 seconds trusts the number the next time. A page that says "you are in the queue" with no number is a page that gets refreshed, and a page that lies about the number is worse.

Bots get three obstacles and no special code. The per-buyer hold limit of the previous topic caps a script at 10 holds a minute whatever position it bought; a script with 200 accounts is 200 places in a 20,000-place queue, admitted in their turn; and the admission token is the only key to the seat map, so a script that calls the seat-map URL from a thousand addresses gets a thousand 403s. What the room cannot do is tell a script from a fast human, and Stagedoor does not try; the limit on what any one buyer can hold is what makes the difference not matter.

Holding Under Load

Behind the room every mechanism runs at its sharpest, and each is a chapter's. The hold transaction is the shortest in the book: lock the seat row with FOR UPDATE, check that it is still available, update it to held with the version incremented, insert the hold row with its 10-minute expiry, commit. Five statements, 5 milliseconds, no call to anything outside Postgres inside the transaction, which is what lets one pool of 24 per loop serve 300 a second with room to spare. The seat map's cache uses the early refresh of Chapter 9 for every event on sale within the hour, so the key is rebuilt in the background 5 seconds before its logical expiry and there is never an instant when 300 admitted buyers find it missing; the delete after every hold's commit keeps it honest, and the rebuild lock bounds the miss that the crash between commit and delete can still cause to one query.

The hold expiry job of Chapter 8 runs every 30 seconds during an on-sale instead of every minute, by the same config that turns the room on, so that a seat abandoned at 19:04:00 is available to the buyer admitted at 19:04:30 rather than at 19:05:00; on a night when 2,000 seats are held and re-held several times each, a minute of dead time per abandoned hold is minutes of buyers looking at a full map. The bulkhead on Payrail and the breaker around it from Chapter 7 are unchanged, and they are the reason a slow Payrail at 19:03 costs 20 slots per loop and not the seat map. Nothing here is new. The room is what makes the load arrive at a rate the old mechanisms were built for.

After the Night

At 19:11 the queue is empty and the room is turned off by config, which for Stagedoor is a deploy with the room's event list emptied and a 30-second drain, as Chapter 4 arranged; a buyer arriving at 19:12 goes straight to the seat map. The metrics from the night are the next load test's target: the mix that actually happened, admissions, holds and checkouts a second, replaces the mix that was guessed six weeks before, and the run before the next on-sale replays it. And the postmortem lists what was closest to its limit, because the ceiling on this night is the thing that fails on the next one if the event is bigger.

On Stagedoor's first waiting-room night the list was short. The sorted set at 20,000 members was nothing to Redis: a rank lookup in a set of 20,000 is 15 comparisons, and the 6,000 polls a second cost less than the seat-map reads did. The primary held at 40 percent, the pool acquire wait never left zero, and loop lag stayed under 20 milliseconds on all four hosts. The ceiling was Payrail: its rate limit is 100 charges a second per merchant, the checkouts reached it at 19:02, and for four minutes the 101st charge each second got Payrail's own 429, which the client of Chapter 10 honoured with the buyer waiting and the hold intact. Nobody had load-tested the provider's limit, because the fake of Chapter 12 does not have one, and it does now. The SLO's error budget of Chapter 13 was 40 percent spent in ten minutes, on the checkouts that waited behind Payrail's limit, and it recovered over the month, which is what a budget is for; a night that spends none of it was a night that admitted too slowly.