Health, Readiness and Graceful Shutdown

The platform asks the process two questions and gives it one instruction. "Are you alive?" decides whether to restart it. "Are you ready?" decides whether to send it traffic. SIGTERM tells it to stop. Stagedoor's first version answered both questions with one endpoint, and got it wrong twice. The endpoint returned 200 unconditionally, so on the evening the database was unreachable the balancer kept sending traffic to two instances that answered every request with a 500. Marek's fix made the same endpoint run SELECT 1, and on the next outage, a 40-second failover of the primary, every instance reported dead, was restarted, came up, reported dead again, and was restarted again, 200 restarts across two hosts for an outage the process had nothing to do with. And the first version ignored the instruction entirely: the deploy script started the new process, waited for its port, and killed the old one, discarding the 340 checkouts that were in flight at that second.

This topic is the two endpoints, what each must check and must not, the six steps between SIGTERM and exit, and the 30-second grace that bounds them. Chapter 7 already relied on the distinction when it wrote the dependency table; here is the mechanism the table assumed.

Liveness

GET /healthz returns 200 if the process can serve HTTP at all: the event loop is running, a handler executes, a response is written. It checks nothing external, on purpose. A failed liveness check means "restart me," and restarting does not fix a dead database, a slow Redis or a Payrail outage; it only adds a restart to the outage. What restarting does fix is a process that is genuinely stuck: a loop blocked by a four-second render that became a forty-second one, a deadlock in a thread pool, an interpreter that stopped scheduling. Those are the cases where the process cannot answer even a three-line handler, and the platform restarting it is the right call. The handler is those three lines.

async def healthz():
    return {"status": "ok"}                      # 200 if the loop can run this line at all

async def readyz(svc):
    if not svc.warm.is_set() or svc.draining:
        return JSONResponse({"ready": False, "reason": "warming or draining"}, status_code=503)
    checks = {
        "postgres": probe(svc.pool.execute, "SELECT 1", timeout=0.5),
        "redis":    probe(svc.redis.ping, timeout=0.5),
        "clock":    probe(svc.clock.offset_ok, timeout=0.5),   # the NTP check Topic 55 added
    }
    results = dict(zip(checks, await asyncio.gather(*checks.values())))   # concurrently: 500 ms worst case
    failed = [name for name, ok in results.items() if not ok]
    return JSONResponse({"ready": not failed, "failed": failed}, status_code=503 if failed else 200)

The first handler does nothing, and doing nothing is its job: if the loop can run it, the process is alive. The second handler runs three dependency checks concurrently, each with a 500-millisecond timeout, and answers with the list of the ones that failed, so that a 503 from readiness says postgres rather than making the on-call engineer guess. The concurrency is not optional: a readiness endpoint that checks three things in sequence at 500 milliseconds each can take 1.5 seconds, and the platform's probe gives up after 1 second by default and counts the slow answer as a failure. The two flags at the top, warm-up and draining, are set by the process itself, and they are the whole of the next three sections.

Readiness

GET /readyz returns 200 if the process can serve real traffic right now. That means the pool can check out a connection and run a trivial statement, Redis answers a PING, the clock is within tolerance of the reference, the warm-up of Topic 52 has finished filling the hot seat maps, and the process is not draining. A failed readiness check means "stop sending me traffic," and nothing else: the process keeps running, the platform does not touch it, and the check runs again 10 seconds later. That is exactly the right response during startup, when the pool is not open yet, and during a database outage, when the instance would only turn requests into 503s that another instance could not have served either.

What readiness does not check is Payrail. Payrail's outage degrades one endpoint, and the dependency table of Topic 42 handles it with a 503 and an extended hold on POST /orders while the seat map, sign-in and the scanner carry on. A readiness check that included Payrail would take every instance out of rotation for a provider's bad afternoon and turn a checkout degrade into a total outage. Readiness names the dependencies without which nothing on the instance works, and for Stagedoor that is the primary, Redis and a sane clock. The replica is not on the list either: reads fall back to the primary when it is gone, as Topic 42 arranged, and the instance is still fully able to serve.

Why They Differ

Liveness restarts; readiness removes from rotation. A dependency outage should do the second and never the first, because the process is fine and the dependency is not. A deadlocked process should do the first, because nothing short of a restart will make it serve again. The two questions have different answers on the same evening, and one endpoint cannot give both: if it checks the database, a database outage becomes a restart loop; if it does not, a hung process stays in rotation returning nothing. Stagedoor's 200 restarts were the first failure, and the fix was not a cleverer check but two checks with two meanings.

The third probe exists for the gap between them at startup. Kubernetes calls it a startup probe: while it has not yet passed, the liveness probe is not consulted at all, so a process that takes 40 seconds to warm its hot seat maps is given those 40 seconds rather than being restarted at 30 for failing to answer. Its budget is a failure count times a period, and Stagedoor sets it to cover the worst measured warm-up with headroom. On a platform without a third probe, the same effect comes from a generous initial delay on liveness and a readiness check that fails until the warm-up flag is set.

SIGTERM and the Drain

On SIGTERM the process does six things in order. It sets the draining flag, so the next readiness probe fails and the balancer stops sending new requests. It keeps accepting for a few seconds, because the balancer's decision to stop and the signal's delivery are not synchronized, and on Kubernetes in particular the endpoint removal and the SIGTERM run in parallel, so a request or two can still arrive after the signal. Then it closes the listening socket, so nothing new arrives. It waits for every request already in flight to write its response, or for the grace period to run out. It closes the pool, the Redis client and the Payrail client, so that Postgres sees clean disconnects rather than 20 abandoned sessions. And it exits with status 0, which tells the platform the stop was orderly.

async def drain(svc, server):
    svc.draining = True                       # 1. readiness now answers 503
    await asyncio.sleep(3)                   # 2. let the balancer's next probe see it; still serving
    server.should_exit = True                # 3. uvicorn closes the listener and waits for in-flight requests
                                             #    (--timeout-graceful-shutdown 20, under the 30 s grace)

async def on_shutdown(svc):                  # 4. runs after the last in-flight response was written
    await svc.pool.close()                   # 5. clean disconnects, not 20 abandoned sessions
    await svc.redis.aclose()
    await svc.payrail.aclose()
    log.info("drained", in_flight_at_signal=svc.in_flight_at_signal)   # 6. exit 0 follows

loop.add_signal_handler(signal.SIGTERM, lambda: asyncio.create_task(drain(svc, server)))

The signal handler starts the drain instead of exiting. The first line flips readiness, the second waits 3 seconds while still serving, and the third tells the server to stop accepting and to wait for what it has, capped at 20 seconds by the server's own graceful timeout so that the total stays inside the platform's 30. The shutdown hook runs once the last response has been written and closes every process-lifetime object that build_service opened, in Chapter 4's terms the other end of the constructor. The log line at the end records how many requests were in flight when the signal arrived, which is the number that was 340 on the first deploy and is usually between 5 and 40 now, and every one of them finishes.

The 340 were the requests a kill-equivalent stop discarded. The old script sent no signal; it killed the process, which closed 340 sockets mid-request. The buyers saw a connection error. Some of those requests had already called Payrail and were between the charge and the order row, which in the spring meant a charged card with no order, and the browser's retry then created the second order that Chapter 1 described. With the drain, the same deploy lets each of those requests finish its work and write its response, and the deploy is invisible.

The Grace Period

The platform waits 30 seconds after SIGTERM and then sends SIGKILL, which cannot be caught and ends the process wherever it is. Kubernetes calls the number terminationGracePeriodSeconds and defaults it to 30; Docker's stop uses 10 by default; systemd's stop timeout defaults to 90. Whatever the platform, every request must be able to finish in less than it, which is why the request deadline of Topic 37 is 10 seconds on checkout and shorter elsewhere: a request that could legitimately take 45 seconds is a request that will be killed at 30 no matter how careful the drain is. The arithmetic is the drain's 3 seconds of continued acceptance, plus the longest request deadline, plus the time to close the pool, and the sum must sit under the grace with room.

The worker's drain has the same shape with a job instead of a request. On SIGTERM it stops reading from the stream, finishes the job it is running, acknowledges it, closes its clients and exits. It does not claim another job, and the reclaim loop of Topic 46 must not see the finished job as pending. A job that takes longer than the grace, the reconciliation of Topic 55 walking 3,000 orders after an outage, will be killed mid-run, its stream entry will sit unacknowledged, and another worker's XAUTOCLAIM will take it over after 60 seconds idle. That is fine if and only if the job is idempotent and resumable, which Chapter 8 required of every job and Chapter 10 gave the reconciliation a cursor for. The grace does not make long jobs safe; idempotency does, and the grace is why it has to.

Startup Order

The process binds its port last. Config is parsed, secrets are read, the pool is opened and a connection checked out once, the Redis client connects and pings, the warm-up fills the hot seat maps, and only then does the server listen. Readiness fails until that last step because there is no port to probe, and passes on the first probe after it, because everything the probe checks was checked during startup. A process that binds first and connects afterwards serves 500s for the seconds in between, and at on-sale those seconds are a few hundred requests that hit a pool with no connections.

The one exception is the warm-up when it is long. Binding at 2 seconds with the warm-up flag unset and readiness failing until the flag is set gives the same result from the balancer's side and lets liveness see a live process during a 40-second warm-up on a platform without a startup probe. Stagedoor does the strict version, bind last, under Kubernetes with the startup probe sized to the warm-up, and the flag version on the VM under systemd. Both hold the invariant that matters: no request reaches a handler before every dependency it needs has been proven to work.