Timeouts Everywhere

A call without a timeout does not fail. It waits, and the request that made it waits, and the connection it holds waits, and the buyer looking at a spinner refreshes and makes another one. A timeout turns "lost" into "failed after N milliseconds," and that is the only form of failure a service can reason about: it has a number, it can be logged, it can be mapped to a response, and it can be handed to the idempotency and reconciliation machinery that the rest of this chapter builds. Every call that leaves the process gets one. The request as a whole gets a budget. Each call's timeout is derived from what is left of that budget, not from a constant in a library.

The night of the spring on-sale, Payrail answered in 4 seconds instead of its usual 400 milliseconds, and Stagedoor's Payrail client had no timeout at all. The double charge that Topic 03 of Chapter 1 read slowly began there: a checkout that had been waiting long enough for the buyer's connection to drop and her browser to try again. This topic is the first of the three repairs, and it is the one that makes the other two possible, because a retry policy and an idempotency key both need a moment at which the first attempt is declared over.

The Call That Never Returns

The Payrail client was an httpx.AsyncClient constructed with timeout=None, because a 3-second timeout had once produced a spurious failure during a test and someone removed it. With no timeout, a call ends when the far side answers or closes the socket, and on the incident night Payrail's edge did neither for a minute at a time. The client's connection pool held 20 connections. Within 12 seconds all 20 were occupied by calls that would not return for 60 seconds, and every checkout after that parked on pool acquisition behind them. Buyers refreshed. Each refresh was a new task waiting on the same pool.

12:00:41.207 WARN  payrail.pool_acquire_wait  waiting=3    in_flight=20  oldest_ms=12040
12:00:52.311 WARN  payrail.pool_acquire_wait  waiting=41   in_flight=20  oldest_ms=23144
12:01:14.980 WARN  payrail.pool_acquire_wait  waiting=188  in_flight=20  oldest_ms=45813
12:01:30.004 INFO  health  status=ok  cpu=3%  loop_lag_ms=1  db_pool_in_use=0  errors_5xx=0
12:01:41.552 WARN  payrail.pool_acquire_wait  waiting=402  in_flight=20  oldest_ms=60003

The log shows 20 calls in flight, the oldest of them a minute old, and a queue of tasks waiting for one of those 20 connections that grows from 3 to 402 in a minute. Between those lines is the health check, and it is the point of the exercise: CPU at 3 percent, event-loop lag of 1 millisecond, no database connections in use, zero errors. By every internal measure the instance was healthy and idle. It was idle because every checkout on it was asleep on a socket, and it reported no errors because nothing had failed; nothing had finished either. A timeout of 3 seconds would have produced 20 failures in the first 3 seconds, a 5xx rate the dashboard would have shown, and a pool that freed itself 20 times a minute instead of never.

Every Boundary Gets a Number

The number is different at each boundary because the boundaries are different. Postgres on the same network answers a hold-path statement in 5 milliseconds and a report in 2 seconds, so the statement timeout is per operation: 5 seconds as the default set for the session, 500 milliseconds set with SET LOCAL statement_timeout inside the hold transaction, both under the 30-second ceiling that Chapter 6 put on the application role. Redis answers in under a millisecond, and a Redis call that has taken 100 milliseconds is a Redis that is gone; waiting 3 seconds for it would stall the seat-map path 30 times longer than it takes to learn the truth. Payrail is on the other side of the internet, answers in 400 milliseconds at P50, and gets 3 seconds. The request as a whole gets 10 seconds at the load balancer, and the service is expected to answer before that.

Seven rows, and they live in the configuration of Topic 23 of Chapter 4, in one place, with their reasons beside them. The reasons are the part a new engineer reads. A Redis timeout of 100 milliseconds looks like a typo next to Payrail's 3 seconds until the row says that a cache which takes 100 milliseconds has already failed at its one job. A hold-path statement timeout of 500 milliseconds looks tight until the row says the transaction takes 5 milliseconds and the only thing that can make it take 500 is a queue of other holds on the same row, which Chapter 6's NOWAIT already refused to wait for. Every number in the table is 3 to 100 times the healthy latency, and none of them is a guess.

The Budget That Propagates

The per-call numbers are ceilings. The actual timeout of any given call is the smaller of its ceiling and what remains of the request's budget, and the budget lives in the request context of Topic 21 of Chapter 4 as a deadline: a timestamp set at the edge, 10 seconds after the checkout request arrived. The seat hold spends 130 milliseconds of it. Payrail's usual answer spends 400. When Payrail is slow and the first attempt has used its 3 seconds, the one retry the idempotency key makes safe is a decision made with about 6.9 seconds left, and it gets its full 3 seconds. A third is made with 3.9 seconds left and gets 3. A fourth, if the policy of Topic 38 allowed one, would be made with 870 milliseconds left and would get 870, not 3,000.

The point of the arithmetic is the last row. A call that is given more time than the request has left is a call whose answer nobody will receive: the load balancer gives up at 10 seconds and sends the buyer its own 504, the service keeps talking to Payrail for 3 more seconds, and whatever Payrail says goes to a closed socket. Capping every call by the remaining budget means the service's own 504 leaves before the balancer's, carries a Problem Details body the client can act on, and leaves the order in a state the service knows about. Every outbound client in Stagedoor reads ctx.remaining(); none of them reads its ceiling from configuration alone.

Connect, Read, and the Whole Exchange

A single timeout number hides three questions. Is anyone there: the connect timeout, which covers the TCP handshake and the TLS handshake, and which should be short, because a far side that takes a second to accept a connection is down or unreachable and a further 2 seconds of waiting learns nothing. Are they still sending: the read timeout, which is the longest the client will wait for the next bytes of the response, and which resets every time bytes arrive. And how long can the whole exchange take: the total, which the other two do not bound. A read timeout of 3 seconds on a response that arrives in 200-byte chunks every 2 seconds never fires, and a 6-kilobyte body takes 60 seconds to read through it.

payrail = httpx.AsyncClient(
    base_url=settings.payrail_url,
    timeout=httpx.Timeout(connect=1.0, read=3.0, write=3.0, pool=1.0),   # never None
    limits=httpx.Limits(max_connections=20),
)

async def charge(order, ctx):
    budget = min(3.0, ctx.remaining())           # the whole exchange, capped by the deadline
    async with asyncio.timeout(budget):        # total: connect + send + every read together
        r = await payrail.post("/v1/charges", json=body,
                               headers={"Idempotency-Key": f"order-{order.public_id}"})
    return parse_charge(r)

The client is built once, at startup, with four numbers that httpx keeps separately: a second to connect, 3 seconds between bytes when reading, 3 seconds to send, and a second to wait for a free connection in its pool of 20. None of them is the total. The total is the asyncio.timeout block around the call, and its number is the smaller of Payrail's 3-second ceiling and the request's remaining budget, so a slow, trickling response is cut off at 3 seconds regardless of how often a byte arrives, and a call made with 400 milliseconds left is cut off at 400. The pool timeout of 1 second is the number that would have ended the incident in its first minute: a task that cannot get one of the 20 connections within a second fails with a pool error the handler can map, instead of joining a queue that grew to 402.

The library default matters, because the incident was a library default. httpx ships with 5 seconds on all four; a client built with timeout=None turns them all off. The older requests library ships with no timeout at all, and a requests.get with no argument waits for as long as the socket lives, which is the classic trap of a whole generation of Python services. Set the numbers explicitly on every client so that the next reader sees a decision rather than a default.

What Timing Out Means

A timeout means unknown. Topic 03 of Chapter 1 drew the four outcomes of one call and the two the caller cannot tell apart, and a timeout is precisely the caller sitting in that ambiguity: Payrail may have never seen the request, or may have charged the card and been writing the response when the 3 seconds ran out. The handler's response is therefore not "failed" and not "try again." It is 504 with the Problem Details type https://stagedoor.example/problems/payment-unresolved, and the order stays pending, with a payments row that records the attempt and no provider_ref. The buyer's page says the payment is being confirmed. The reconciliation job of Chapter 10 asks Payrail tonight whether a charge with idempotency key order-{public_id} exists, and moves the order to paid or failed on Payrail's answer.

The alternative is the double charge, this time committed by the service instead of the browser. A handler that catches the timeout, marks the order failed and tells the buyer to try again has just told a buyer whose card was charged to charge it again, and nothing in the second request knows about the first. Chapter 2 reserved 502, 503 and 504 for the edge and the drain path; this 504 and the breaker's 503 in Topic 40 are the two the service adds on purpose, each with a stable type, so a 5xx from a handler without one of those types is still a bug. Marek's rule from Chapter 1 applies here in its exact form: what if this succeeds and I never hear back? Then the order waits, and reconciliation answers.

Timeouts on the Way In

The same fact runs in the other direction. The service is somebody's far side, and Topic 10 of Chapter 2 gave the edge its numbers: a 60-second idle timeout for a keep-alive connection with nothing on it, 10 seconds for a client to send its headers or its body. What the edge cannot do is stop a handler that has run past the client's patience. If the buyer's browser gives up at 10 seconds and the handler is at 11, still waiting on Payrail, it is doing work whose result will be written to a socket nobody is reading, and holding a connection from the pool and a slot in Payrail's 20 while it does.

The deadline middleware of Topic 21 of Chapter 4 therefore wraps the handler itself in asyncio.timeout with the route's budget. When it expires, the task is cancelled at whatever await it is sitting on: the pooled connection's context manager exits and returns the connection, the Payrail socket is closed, and the handler's response is the 504 above. The server side of a query needs its own guard, because cancelling the client does not by itself stop Postgres executing the statement; that is what statement_timeout in the table is for, and it is why the table has rows on both sides of every boundary. A timeout only on the client's side is a client that has stopped listening to a server that is still working.