Load Testing the On-Sale

The autumn on-sale will bring 3,000 requests a second for ten minutes, and the only way to know what breaks first is to send them before the buyers do. A load test is a script that behaves like the buyers, browse, load the seat map, hold, pay, at a controlled rate against a staging environment sized like production, while the dashboards of Chapter 13 are watched by a person. It is not a benchmark of one endpoint and it is not a pass-or-fail test in the sense of the rest of this chapter. It is an experiment with a question, and the question is which number moves first.

What moved first for Stagedoor, in the run six weeks before the autumn on-sale, was the pool of Topic 31 at 800 requests a second. Then the seat-map rebuild of Topic 50 at 2,000. Then nothing until 4,500, when api-01's CPU reached 100 percent and the run answered the question Chapter 14 asks: how many instances the night needs. Three findings, three fixes, and a knee that the suite of the previous four topics could never have found, because a functional test sends one request and the pool has never run out under one request.

The Scenario, Not the Endpoint

A hammer on GET /events/8812/seats at 10,000 requests a second measures one query and proves the seat map is fast, which was already known. The on-sale scenario is the mix the night will bring: 2,600 seat-map reads a second, 300 holds, 100 checkouts, with the fake Payrail of Topic 65 answering at its real 400 milliseconds, and a 10 percent share of buyers who retry a request that took longer than 2 seconds, because on the night they will. The ratio is what makes the pool arithmetic real. Holds and checkouts are the requests that take a connection from the pool and keep it for the transaction, and 400 of them a second against 160 connections is the number the seat-map hammer could never have produced.

export const options = {
  scenarios: {
    onsale: {
      executor: 'ramping-arrival-rate',          // requests per second, not virtual users
      timeUnit: '1s', startRate: 100,
      preAllocatedVUs: 500, maxVUs: 6000,
      stages: [
        { duration: '5m',  target: 3000 },       // ramp: find the knee
        { duration: '10m', target: 3000 },       // plateau: find the leaks
        { duration: '1m',  target: 6000 },       // spike: 2x, does it recover?
        { duration: '2m',  target: 0 },
      ],
    },
  },
  thresholds: {
    'http_req_duration{name:checkout}': ['p(95)<800'],   // the SLO of Topic 71, as a gate
    'http_req_failed': ['rate<0.001'],
  },
};

export default function () {
  const r = Math.random();
  if (r < 26 / 30)      seatMap(8812);           // 2,600 of every 3,000
  else if (r < 29 / 30) holdSeat(8812);          // 300
  else                 checkout(8812);          // 100, each one a 400 ms fake Payrail call
}

The script asks the tool for an arrival rate rather than a number of simulated users, because buyers arrive at a rate and do not wait for each other. It ramps from 100 to 3,000 requests a second over five minutes, holds there for ten, doubles for one minute, and stops. Every iteration is one buyer action, picked at the night's ratio: 26 of every 30 load the seat map, 3 hold a seat, 1 pays. Two thresholds turn the run into a gate: the checkout P95 must stay under the 800 milliseconds of Topic 71's SLO, and fewer than one request in a thousand may fail. The retry share and the fake's latency live inside the three functions, so the mix is the night's and not a flattering version of it.

What to Measure

The latency per endpoint, P50, P95 and P99, and the error rate by status, are what the tool reports and what everyone looks at first. They are the symptoms. The causes are the internal gauges of Topic 69, read from the same dashboard at the same time: pool acquire wait, the Redis hit ratio, event-loop lag, the outbox and queue age of Topic 47, and CPU per instance. The point of the run is the moment one of those leaves its resting value, because that moment is the number the test was for and it arrives before the latency does. On the six-week run the pool acquire wait left zero at 780 requests a second, the checkout P95 crossed 800 milliseconds at 840, and the first 503 from the acquire timeout appeared at 900. A team watching only P95 gets the number 840 and no idea why.

Ramp, Plateau, Spike

Each phase answers a different question. The ramp, five minutes from 100 to 3,000, finds the knee: the rate at which P95 starts climbing without the throughput rising with it, which is the rate at which some resource is full. The plateau, ten minutes at 3,000, finds what grows slowly: a connection that leaks one per checkout, a memory footprint that climbs 4 megabytes a minute, a queue age that rises because the worker is 5 percent behind and will be 40 minutes behind by midnight. A test that ramps to the target and stops has never held it long enough for those to show. The spike, one minute at 6,000, asks two things: what fails at twice the target, and whether the service recovers when the spike ends, which is the recovery assertion of Topic 66 applied to load. A service that survives the spike and does not come back is the same incident as the breaker that never closes.

What Broke, in Order

The six-week run found three things, and each one had a topic waiting for it. At 800 requests a second the pool ran out: acquire wait climbed from zero to 2 seconds in 40 seconds of ramp, then the 5-second acquire timeout of Topic 31 started turning checkouts into 503s. The fix was two changes from Chapter 6: the hold path's transaction shortened so the Payrail call of Topic 53 happens after the commit and not inside it, which gave the connection back 400 milliseconds sooner, and the pool resized from 20 to 24 per loop after the arithmetic against max_connections showed it still left room for the worker and the migrator. At 2,000 the seat map stampeded: the hit ratio dipped to 40 percent every 30 seconds as the key expired and 2,600 readers found nothing, and the primary saw a burst of 2,000 identical queries. The fix was the rebuild lock and early refresh of Topic 50, and on the second run the hit ratio stayed above 99 percent through the plateau. At 4,500, during the spike, api-01's CPU reached 100 percent and loop lag crossed 200 milliseconds, and that is not a bug; it is the number that says two instances carry 4,500 requests a second and the night's 3,000 with a 2× spike needs three, which is the arithmetic of Topic 73.

Nothing on that table is a surprise to a reader of Chapters 6 and 9. Every mechanism had been built and every number had been reasoned about, and the run's contribution was to say which limit was reached first and at what rate, which is what reasoning cannot do. The order also matters: without the first fix the second could not be seen, because the pool fell over before the cache got a chance to.

Staging Must Be Honest

A load test against a staging with an empty database and a bigger instance finds nothing, and a load test that finds nothing is worse than none, because it is believed. Stagedoor's staging has the same instance sizes as api-01 and api-02, the same PostgreSQL 18 major with a copy of production's row counts, 40 million seats rows among them, anonymized so that no buyer's email leaves production, the same Redis, and the fake Payrail run as a small HTTP process with its 400-millisecond latency built in, so that PAYRAIL_URL in staging's config points at it during the run and at the sandbox otherwise. The query that is an index scan on 10,000 rows and a sort on 40 million is the query the empty staging would have blessed.

The one thing staging is allowed to differ in is the buyers, and the script is the buyers. Toxiproxy sits between the service and the fake Payrail on runs that want to add what the fake cannot: 200 milliseconds of jitter, a slice of connections reset, a full timeout on 2 percent of calls, injected at the network without touching either side. The dependency table of Topic 42 gets its load-time exercise this way, one row at a time during the plateau, with the same assertions Topic 66 made at one request a time now made at 3,000.

Making It Routine

The run happens before every on-sale and after every change to the hot path, from CI on a schedule, monthly at least, and its four numbers are kept: the knee, the first gauge that moved, the growth rate on the plateau and the recovery time after the spike. This month's knee is compared with last month's, and a knee that moved from 4,500 to 3,000 after a deploy is a regression the functional suite could not see, because every request still returned the right answer, only later and with a connection held 200 milliseconds longer. The comparison is the assertion. A single run, ever, answers the question once and then ages; the on-sale after next is served by whatever the code has become since.

The cost is 18 minutes of a staging environment that exists anyway and one person watching a dashboard, and the person is the part not to automate away. The thresholds in the script catch the SLO breach; the person catches the gauge that moved at 780 when nothing had breached yet, and that is the number the whole exercise exists to find.