Connections — Keep-Alive, HTTP/2 and TLS Termination
Before the first byte of a request reaches a handler, a TCP connection was opened and a TLS handshake completed, and a buyer on a bad mobile network spent 100 milliseconds doing it. Keep-alive reuses the connection for the next request. HTTP/2 puts many requests on it at once. TLS termination at the load balancer decides where the encryption ends and where the handshake's CPU is paid. None of this is the service's code, and all of it is the service's latency, which is why the buyer's time-to-first-byte and the handler's time are two different numbers that Topic 02 of Chapter 1 kept apart.
The service still owns three decisions here: how long it keeps an idle connection open, whether it expects to see HTTP/2 at all, and what it does with a client that opens a connection and then does almost nothing with it. Each one has a number, and getting one of them wrong produces the failure nobody can reproduce: a 502 once in a thousand requests with no log line on any instance.
What a Request Costs Before It Starts
A DNS lookup, unless cached. A TCP handshake, one round trip. A TLS 1.3 handshake, one more round trip, or two on TLS 1.2. Then the request. On a 50-millisecond round trip that is 100 milliseconds of handshakes before any byte of GET /events leaves the phone, and on a congested cellular link with a 200-millisecond round trip it is 400. The cost is paid per connection, not per request, which is the argument for keeping the connection open and reusing it.
The seat-map page loads 40 small resources. Over HTTP/1.1 with a fresh connection for each, that is 40 handshake pairs; even with a browser's six parallel connections it is six handshakes and a queue. The numbers are why every part of this topic exists, and they are invisible from inside the handler: the request arrives, the handler runs in 20 milliseconds, and the buyer waited 120.
Keep-Alive
One connection carries many requests in sequence. The load balancer holds a warm pool of connections to api-01 and api-02 and sends each request down one that is already open; the buyer's browser holds a few connections to the balancer and does the same. The service's job is to not close the connection after every response, which every modern server does by default, and to keep it open for longer than the balancer will.
That ordering is the rule that matters. The balancer has an idle timeout, 60 seconds on AWS's application load balancer and a number of its own on every other one, after which it closes a connection with nothing on it. The service has its own, and uvicorn's default is 5 seconds. When the service's is shorter, the balancer holds a connection the service closed at second 5, sends the next request into it at second 7, and gets a reset. It answers the client with a 502, and no instance logged anything because no instance received a request. Set the service's keep-alive timeout above the balancer's idle timeout, 75 seconds against 60, and the balancer is always the one to close, at a moment when nothing is in flight.
HTTP/2 and Multiplexing
HTTP/2 puts many requests in flight on one connection, interleaved as frames, with header compression and no head-of-line blocking at the HTTP layer: a slow response no longer holds up the ones behind it in the queue. The browser opens one connection to the balancer instead of six, and the seat map's 40 resources arrive together over it. HTTP/3 over QUIC removes the TCP-level blocking as well, so one lost packet stalls one stream rather than the whole connection, which matters most on exactly the mobile networks where the handshake cost was highest.
Both are terminated at the edge. The balancer speaks HTTP/2 and HTTP/3 to the browser and HTTP/1.1 to api-01, and Stagedoor's api never sees a frame of either. The multiplexing benefit is real and it is entirely between the browser and the balancer; the service should not be written to expect it, and the balancer's pool of HTTP/1.1 connections to each instance is sized for the concurrency the instance actually handles. A service that tries to speak HTTP/2 directly has taken on TLS termination as well, for a benefit that was already delivered one hop earlier.
TLS Termination
The certificate lives on the load balancer, the handshake happens there, and the traffic from the balancer to api-01 is plain HTTP inside the private network. The instance never loads a certificate, never rotates one, and never spends its CPU on a handshake; that cost lands on the balancer, which exists to absorb it. The service learns that the buyer used HTTPS from X-Forwarded-Proto: https, trusted only from the balancer's address as Topic 08 of this chapter set out, and the redirect from plain HTTP to HTTPS happens at the balancer too, never in a handler.
End-to-end TLS to the instance is the alternative for one situation: the network between the balancer and the instance is not private. A shared cloud network with other tenants, a hop across a region boundary, a compliance rule that names the wire. Then the instance terminates a second TLS session, with an internal certificate the platform issues and rotates, and the cost is accepted because the alternative is plaintext on a wire somebody else can see. Stagedoor's instances sit on a private subnet behind one balancer and terminate nothing.
Connection Limits Are Real
Each open connection is a file descriptor and a small buffer on the instance: a few kilobytes of kernel memory, a slot in the server's accept loop, an entry in the parser's state. 10,000 idle keep-alive connections from mobile clients that each fetched one page and went quiet would be a serious resource on the instance, and the balancer in front is exactly why the instance never holds them: it terminates those client connections itself and carries their requests over its own pool of 200 warm connections to each instance. The instance sizes for 200, the balancer absorbs the rest, and Topic 61 of Chapter 11 does that arithmetic with the outbound pools added in.
Timeouts on the Connection Itself
Three timeouts live below everything in Chapter 7. The read timeout: the client began sending a body and stopped halfway. The write timeout: the client stopped reading the response and the send buffer filled. The idle timeout: a keep-alive connection with nothing on it. Each one, when missing, is a slow-client attack waiting to happen: a client that opens a connection and sends one byte a minute holds a slot forever, and a few hundred of them is an outage with no traffic in the graphs. The server's defaults are the place to look before tuning anything in the application.
# edge: the load balancer idle_timeout = 60s # closes a quiet client or backend connection client_header_timeout = 10s # slow-client limits belong here, in front client_body_timeout = 10s send_timeout = 10s # api-01: uvicorn uvicorn stagedoor.api:app --timeout-keep-alive 75 # must outlive the edge's 60 --limit-concurrency 400 # 503 beyond this, never a queue that grows
The block is two configurations kept in one place on purpose. The edge closes an idle connection at 60 seconds and cuts off a client that takes more than 10 seconds to send its headers, its body, or to read the response. The instance keeps an idle connection for 75 seconds, longer than the edge's 60, so the edge is always the side that closes and the phantom 502 cannot happen. And the instance refuses with a 503 beyond 400 in-flight requests rather than queueing them, which is the first line of the load shedding that Chapter 14 finishes. The slow-client limits sit at the edge because uvicorn exposes the keep-alive timeout and nothing finer; a server with no edge in front would need nginx or the equivalent to hold them.
- A service keep-alive timeout shorter than the load balancer's idle timeout — the balancer reuses a connection the service has already closed, and one request in a thousand gets a 502 with no log line on any instance.
- Terminating TLS at the instance for no reason — the certificate is now on every instance, rotated on every instance, and the handshake's CPU lands on the process that should be running handlers.
- No idle, read or write timeout on the serving path — a client that opens a connection and sends one byte a minute holds a slot forever, and a few hundred of them is an outage with nothing in the traffic graphs.
- Assuming HTTP/2 reaches the code — the framework sees HTTP/1.1 from the balancer; the multiplexing benefit is between the browser and the edge, and a service written to expect it has taken on termination for nothing.
- Redirecting plain HTTP to HTTPS inside a handler — the handler sees plain HTTP from the balancer on every request, and the redirect loops until the balancer's own rule is written instead.
- Sizing the instance's descriptor limit for the balancer's pool and forgetting the worker's outbound connections — the pool to Postgres, the Redis client and the Payrail connections count against the same limit, and the instance runs out under load with the accept loop still idle.
- Set the service's keep-alive timeout above the load balancer's idle timeout, 75 seconds against 60, and write both numbers down next to each other in the deploy configuration.
- Terminate TLS at the edge, trust
X-Forwarded-Protoonly from the edge's address range, and put the HTTP-to-HTTPS redirect on the balancer, never in a handler. - Configure the read, write and idle timeouts on the serving path before tuning anything in the application, and keep the slow-client limits at the edge where the server lacks them.
- Cap in-flight requests per instance with the server's concurrency limit, so overload produces a fast 503 rather than a growing queue.
- Measure the client's time-to-first-byte separately from the handler's time, and know which of the two the connection costs belong to.
Knowledge Check
A buyer on a 200-millisecond cellular round trip opens a new TLS 1.3 connection. What has she paid before the first byte of her request leaves the phone?
- About 400 milliseconds of handshakes, plus a DNS lookup on top if the name is not cached yet
- About 400 milliseconds on every request, since TLS renegotiates before each request on a mobile link
- Nothing beyond the DNS lookup, since TLS 1.3 completes the handshake inside the first request packet
- About 200 milliseconds in total, since the balancer completes the handshakes on the fast private network
The load balancer's idle timeout is 60 seconds and uvicorn's keep-alive timeout is 5. What is the symptom?
- Buyers are disconnected after five seconds of inactivity and have to reload the page to continue
- A 502 for roughly one request in a thousand, with no matching log line on either instance
- The instance runs out of file descriptors, because the balancer keeps opening connections it never closes
- A 503 with Retry-After from the instance whenever the balancer reuses an idle connection
Where does HTTP/2 stop in Stagedoor, and what does that mean for the service's code?
- At the instance, so each handler sees multiplexed streams and must be written to serve several at once
- At the balancer, which means the browser's multiplexing gain is lost because the instance link is still HTTP/1.1
- At the balancer, which speaks HTTP/1.1 to the instances, so the code gets the benefit without ever seeing the protocol
- Nowhere, because HTTP/2 must run end to end or the browser falls back to six HTTP/1.1 connections
A client opens a connection, sends a request line and one header, then sends one byte every 50 seconds. Which timeout ends it?
- The idle timeout, since 50 seconds of silence between bytes counts as a connection with nothing at all on it
- The read timeout, because the request is still arriving and the client is too slow to finish sending it
- The write timeout, because the server has a response ready and the client has stopped reading it
- The request budget from Chapter 7, since the handler's deadline expires before the headers are complete
You got correct