Scaling Out — Stateless Is the Price

More instances of a stateless process is the simplest way to serve more requests, and the only way if the process is truly stateless. The load test of Chapter 12 put two instances' knee at 4,500 requests a second, the forecast for the biggest event of the autumn is 6,000, and the arithmetic says a third and a fourth instance. Stagedoor's third and fourth instances added nothing for a day. The load balancer was still configured for sticky sessions from the era when sessions lived in a dictionary inside the process, so every returning buyer went to the instance they had started on, and the two new instances saw only buyers who had never visited before: 5 percent of the traffic each, while api-01 and api-02 carried 70 percent each and were as full as they had been the day before.

This topic is what horizontal scaling asks of the process, which is that it hold nothing between requests, what it does to the dependencies the instances share, which is multiply their load, and what is left that cannot be scaled this way at all. The scaling that looks free is the scaling that pushes the load onto something that was already the ceiling.

The Shape

N identical instances behind a load balancer that may send any request to any of them. Each instance holds nothing between requests, which Chapter 1 stated as the rule and Chapter 11 turned into the twelve-factor discipline: sessions in Redis, uploads in object storage, the seat-map cache in Redis and not in a process dictionary, no in-memory counter that any second instance would disagree with. Under that rule the instances are interchangeable, and capacity is N times the per-instance throughput. Two instances carried 4,500 requests a second before api-01's CPU reached 100 percent, so one instance is worth 2,250, three should be 6,750 and four 9,000, and the forecast's 6,000 with a 2 times spike wants four.

The arithmetic holds exactly as long as nothing the instances share is the limit. On the three-instance run the knee came at 6,200 rather than 6,750, and the gap between those two numbers is what the third section is about: a fourth instance is not a fourth of the capacity, it is a fourth pool of connections against one primary, a fourth set of readers against one Redis, a fourth client of the same load balancer. Scaling out multiplies the process's capacity and multiplies the load on everything behind it in the same step, and the instance count that fits is the one where the shared thing still fits too.

Sticky Sessions Are the Anti-Pattern

Session affinity, routing every request from one client to the same instance by a cookie the balancer sets, was the workaround for a process that kept sessions in memory: api-02 did not know the session api-01 had created, so the balancer made sure api-02 never saw that buyer. Chapter 5 moved sessions to Redis and Chapter 11 called the affinity a lie about scaling, and this is the topic that shows the lie's shape. With no state in the process, affinity buys nothing and costs three things. The load is uneven, because the balancer is distributing clients rather than requests, and one buyer who refreshes the seat map 40 times is 40 requests on one instance. New instances sit idle, because every existing client is pinned elsewhere and only new arrivals reach them, which is the 70 and 70 against 5 and 5 that Marek watched for a day. And when an instance drains for a deploy, every buyer pinned to it loses a "session" that was never state at all, only a routing preference, and their next request lands on an instance that serves it perfectly well, which proves the affinity was protecting nothing.

The fix is one line removed from the balancer and one word changed: no affinity cookie, and least-connections instead of round-robin, so that a request goes to the instance with the fewest in flight rather than to the next in a fixed order. Round-robin is fine when every request costs the same; least-connections is better when a checkout holds a slot for 500 milliseconds and a seat-map read for 8, because it sends the next request to the instance that has the most room rather than the instance whose turn it is. Within a minute of the change the four instances were at 35 percent each, and on the next run, with PgBouncer already in front of the primary for the reason the next section gives, the knee moved from 4,500 to 8,400: 600 short of the paper figure, and the shared things are where the 600 went.

backend stagedoor_api
    balance leastconn                              # was: roundrobin
    # cookie SDSRV insert indirect nocache          <- the sticky line, deleted
    option httpchk GET /readyz                     # Chapter 11: not ready means no traffic
    server api-01 10.0.1.11:8000 check inter 2s fall 2 rise 2
    server api-02 10.0.1.12:8000 check inter 2s fall 2 rise 2
    server api-03 10.0.1.13:8000 check inter 2s fall 2 rise 2
    server api-04 10.0.1.14:8000 check inter 2s fall 2 rise 2

The block is the balancer's whole opinion of Stagedoor: four servers, a health check against the readiness endpoint every 2 seconds so that a draining instance stops receiving traffic within 4, and a distribution rule. The commented line is the one that had been there since the in-memory sessions and had outlived its reason by two chapters. Nothing in the service changed; the service had been stateless since Chapter 5, and the balancer had been routing as if it were not.

The Shared Things Scale Differently

Postgres first, because the pool arithmetic of Chapter 6 is the shared limit that arrives soonest. Each host runs four processes, one loop each, and each loop has its own pool, which Chapter 12 raised from 20 to 24 after the six-week run. Two hosts is 8 loops and 192 connections against a max_connections of 200, with 8 left for the worker, the migrator and a person with psql. Four hosts is 16 loops and 384, and even at the original 20 it is 320. The fourth host's processes cannot connect, and the failure is not gradual: the pool that cannot open its connections at startup fails readiness, and the autoscaler's new instance is an instance that never takes traffic. Scaling the API out therefore means one of two things done before the instance count changes: shrink the per-loop pool to 12 so that 16 loops fit in 192, and accept more pool wait per instance, or put the pooler in front of the primary, PgBouncer in transaction mode as Chapter 6 named it, so that 16 pools of 24 on the outside are 40 server connections on the inside.

Redis is one instance, and the instance count multiplies its readers but not the work per read: 2,600 seat-map reads a second from two API hosts is the same 2,600 from four, because the buyers did not multiply, and a single Redis serves 100,000 simple commands a second without effort. What the instance count does change is the number of processes that can stampede a key at once, 16 rebuilders instead of 8 when a seat map expires, which is why the rebuild lock of Chapter 9 is per key in Redis and not per process. The seat-map cache is also the thing that keeps the primary out of the hot path at all: without it, four hosts would be 2,600 queries a second against pg-primary, and the fourth instance would have bought a faster way to saturate it. The balancer and the network are the platform's capacity, provisioned by whoever runs the load balancer and measured by the load test rather than reasoned about here.

What Cannot Be Scaled Out

Three things in Stagedoor are singletons by design, and each has a reason and a ceiling that the design states. The scheduler of Chapter 8 runs in every worker but fires once, under a Redis lock, because "expire the holds every minute" run by three workers is three expiries a minute racing each other; its ceiling is the number of scheduled jobs it can enqueue in a tick, which is thousands, against Stagedoor's dozen. The outbox relay of Chapter 7 is one per ordered kind, because two relays on the same rows publish out of order and a refund reaches the worker before the order it refunds; its ceiling is 1,000 rows a second at 100 rows every 100 milliseconds, against a peak of a few hundred orders a second, a margin of 3 that the postmortem of Topic 75 checks after every on-sale. The primary is one writer, because a row lock on seat 14C means one thing only if there is one place that holds it; its ceiling is its write throughput once the pool, the cache and the indexes are done, and raising it is PostgreSQL Deep Dive's Chapter 13 and the last topic of this book.

Naming the singletons is the discipline. A process that is "stateless" but runs a scheduler loop is a process that becomes N schedulers the day it is scaled, and Chapter 8's lock exists because the worker was scaled before anyone noticed it was one. Every singleton in the codebase carries a comment saying that it is one, why it is one, and what its ceiling is, and the scaling change that adds an instance is reviewed against that list.

Autoscaling From the Service's Numbers

An autoscaler is a loop that reads a number and changes N, and the service's job is to publish the right number and to be usable quickly once N changes. For the API the number is CPU per instance or request rate per instance, which Chapter 11 handed to the platform and Chapter 13's metrics endpoint exposes; a target of 60 percent CPU per instance adds a host when the average crosses it, and the knee at 100 percent stays 40 points away. For the worker the number is queue age, the age of the oldest unacknowledged entry per kind from Chapter 8, because a worker's CPU says nothing about whether the buyers' PDFs are 4 seconds or 40 minutes away; 30 seconds of age is a second worker, and a scaler that reads a queue metric rather than CPU is what KEDA exists for. Both numbers are the service's own, published by the code that knows them, and an autoscaler pointed at the platform's generic CPU gauge for the worker would have added workers during the render of a 40-megapixel venue photograph and none during the 40-minute backlog.

The second half is startup. An instance that the autoscaler adds at 19:00:30 because CPU crossed 60 percent at 19:00:15 helps the on-sale only if it is passing readiness and taking traffic by 19:01, and a process that needs five minutes to warm up is a process the autoscaler cannot use: the spike is over, or the service has fallen over, before the help arrives. Stagedoor's api is ready in 5 seconds on an ordinary evening and in 45 before an on-sale, when the warm-up of the hot seat maps that Chapter 9 arranged and Chapter 11 gated readiness on takes 40 of them, and the balancer's 2-second check picks it up on the next probe either way. Start time is a scaling property, and it is measured on every deploy for the same reason the knee is measured on every load test.

Scaling Down

Every scale-up has a scale-down, most evenings at 22:00 when the autoscaler reads 20 percent CPU and removes two hosts, and the removal is only free because of the drain of Chapter 11. The instance receives SIGTERM, fails readiness so the balancer stops sending within 4 seconds, finishes the requests in flight inside the 30-second grace, closes the pool after the last response, and exits. Without that sequence the scale-down is the first production deploy again: 340 checkouts closed mid-request, some after Payrail had charged, every evening, by a mechanism nobody is watching because it is automatic. A drain that works for deploys works for scale-downs, and the test of Chapter 12 that sends a SIGTERM during load is the test of both.

The pool is the detail that makes the order matter. An instance's 96 connections, four loops of 24, are returned to the primary only when each pool closes cleanly; a process killed at second 30 with connections open leaves the server's backends to notice the dead sockets on their own, which takes the TCP keepalive interval, and for those minutes the connections count against max_connections while belonging to nobody. Two hosts scaled down uncleanly is 192 connections that are gone from the pools and still present on the server, and the next scale-up's new instances cannot connect. The drain closes the pool after the last response and before the exit, in that order, and the connections are free the moment the process is.