Circuit Breakers and Bulkheads

When Payrail is down, every checkout waits 3 seconds to learn it. At 3,000 requests a second that is a service spending its whole capacity discovering the same fact, and the retries of Topic 38 triple the calls to a provider that is already failing. A circuit breaker counts failures per dependency and, past a threshold, stops calling for a while and fails immediately: 1 millisecond to learn what took 3 seconds a moment ago, and no load at all on the far side while it recovers. A bulkhead is the other half of the same defence: a limit on how much of the service one dependency can occupy, so that a dead Payrail holds at most 20 of the loop's tasks and the seat-map path, which never calls Payrail, keeps serving.

Both are the mechanics of failing fast, and both need a decision that no library makes for you: what the caller gets instead of the call. A breaker that opens and raises into an unhandled 500 has moved the outage from Payrail to Stagedoor. The breaker is the detector; the fallback written beside it is the design, and Topic 42 is the table of every fallback the service has.

The Three States

The breaker starts closed, which is the confusing name for the normal state: calls flow through, and each outcome is counted. When the failure rate over the window crosses the threshold, the breaker opens, and for a cooldown of 30 seconds every call fails immediately with a breaker-open error without touching the network. When the cooldown ends, the breaker goes half-open and lets exactly one trial call through. If the trial succeeds, the breaker closes and traffic resumes; if it fails, the breaker opens again for another 30 seconds. One call every 30 seconds is the price of finding out whether Payrail is back, and it is paid by one buyer, who sees a 3-second wait instead of an instant 503.

The breaker is per dependency and per instance. Stagedoor has one for Payrail, one for Redis and one for the replica on each of api-01 and api-02, six breakers in all, and each one sees only the calls its own process made. That is a deliberate simplicity: a shared breaker would need coordination through Redis, which is one of the things it is protecting against. The cost is that the two instances can disagree, and the last section of this topic is about making that disagreement visible instead of mysterious.

What Counts as Failure

A timeout counts. A 5xx counts. A 4xx does not, and the case that makes the rule concrete is a 402: a declined card is Payrail working exactly as designed, answering in 400 milliseconds with a correct refusal, and a run of declined cards on a night when a fraud ring is testing stolen numbers must not open the breaker and stop checkout for everyone else. Connection refused counts. A breaker-open error from a nested breaker does not, because it was not a call. The predicate is written once, beside the breaker, and it is the same classification Topic 38 wrote for retries with the polarity reversed: what the retry policy calls transient is what the breaker counts as a failure.

The threshold is a rate over a window, not a count. Stagedoor's is 50 percent of the last 20 calls, with a minimum of 20 calls before the rate means anything. A count, five failures and then open, trips on five unrelated timeouts spread across a day, at a random moment, for a dependency that is 99.9 percent healthy. A rate over a sliding window trips only when the recent calls are mostly failing, which is the definition of the dependency being down, and it recovers its meaning as soon as the window fills with successes. Twenty calls at Stagedoor's checkout rate is a window of a few hundred milliseconds at peak and a few minutes at 3 a.m., and the rate is right in both.

What the Caller Gets

An open breaker raises immediately, and the handler above it has to have decided what that means for this endpoint. For checkout, it means 503 with Retry-After set to the seconds left in the cooldown, the Problem Details type https://stagedoor.example/problems/payments-unavailable, and the order left in pending with its holds intact, so that the buyer who retries in a minute finds her seats still hers. For the seat map, an open Redis breaker means reading the map from the database, which Chapter 9 makes safe with a lower rate limit. For an organizer report, an open replica breaker means the primary answers, and a metric says so. Three dependencies, three different fallbacks, and none of them is "raise."

The 503 is the second of the two 5xx codes the service sends on purpose; Topic 37 added the 504. Both carry a type, both leave the order in a state the service knows, and both are answers rather than accidents. Topic 07 of Chapter 2 said a 503 must carry a Retry-After that is true, and the breaker's cooldown makes it true: the header says 24 because the breaker will try Payrail again in 24 seconds, and a client that comes back then has a real chance instead of a guess.

Bulkheads

A breaker limits calls once a dependency is known to be failing. A bulkhead limits how much of the service a dependency can hold at any time, including the 20 seconds before the breaker has seen enough failures to trip. The word is from ship design, where a hull is divided into compartments so that one flood does not sink the whole vessel, and the mechanism is a bounded concurrency limit per dependency: at most 20 Payrail calls in flight per instance, enforced by an asyncio.Semaphore around the call. A slow Payrail can then hold 20 of the loop's tasks and no more. The other tasks on the loop, which are serving 2,600 seat-map requests a second, never wait behind it.

payrail_slots = asyncio.Semaphore(20)                       # at most 20 in flight per instance
payrail_breaker = Breaker(window=20, failure_rate=0.5, cooldown=30.0, counts=is_failure)

async def charge(order, ctx):
    try:
        async with asyncio.timeout(min(1.0, ctx.remaining())):
            await payrail_slots.acquire()                  # no slot within 1 s: bulkhead full
    except TimeoutError:
        raise PaymentsUnavailable(retry_after=5)
    try:
        async def attempt(timeout):
            async with payrail_breaker:                    # open: raises BreakerOpen at once, not retried
                return await payrail.post("/v1/charges", json=body(order), timeout=timeout)
        return await with_retry(attempt, ctx)              # Topic 38's policy, per attempt through the breaker
    except BreakerOpen as e:
        raise PaymentsUnavailable(retry_after=e.seconds_until_half_open)
    finally:
        payrail_slots.release()

The wrapper is three layers, and the order of the layers is the lesson. Outermost is the semaphore, acquired with a timeout of 1 second so that a task which cannot get one of the 20 slots fails fast with a 503 instead of joining a queue. Inside it is the retry policy of Topic 38. Inside each attempt is the breaker, so that every attempt is a call the breaker counts, and an open breaker fails the attempt at once with an error the retry policy does not classify as transient, which stops the retries with it. The exception at the bottom maps both kinds of refusal, no slot and open breaker, to the same 503 the handler already knows how to send, with a Retry-After that is the breaker's remaining cooldown in one case and a flat 5 seconds in the other.

The database pool of Topic 31 of Chapter 6 is a bulkhead by construction: 20 connections per instance is a hard limit on how many tasks can be inside Postgres at once. What the pool does not do is separate the paths that share it, and that is the second rule: the seat-map path and the checkout path must not share a limit. A single semaphore of 20 for "outbound calls" lets a slow Payrail consume all 20 with checkouts, and the seat map, which needed one of them for Redis, is stalled by a provider it never talks to. One limit per dependency, and the dependency that is slow can starve only its own callers.

Breakers and Retries Together

The retry policy and the breaker are two views of the same signal. A retry looks at one request and asks whether another attempt is worth it; the breaker looks at the last 20 calls and asks whether any attempt is worth it. Put together in the order the code above shows, they compose: three attempts against a failing Payrail are three failures toward the threshold, so a burst of retries brings the breaker to its trip point three times faster than single calls would, which is correct, because retries against a dead provider are the storm the breaker exists to stop. When it opens, the next attempt fails at once and the retry policy stops, because breaker-open is not on its list of transient errors.

The retry budget of Topic 38 is the same observation again in a third form: retries over 10 percent of calls means the failures are not transient, which is what a breaker at 50 percent of the last 20 also means, at a different threshold and with a different response. Stagedoor keeps all three in one wrapper per dependency, with their numbers in one configuration block, because a policy spread over three files is a policy nobody can read. The library that implements them matters less than the fact that they share the failure predicate and the metric.

Observing the Breaker

Every state change is a log event with the dependency, the instance, the old state, the new state and the failure rate that caused it, and the current state is a gauge per dependency per instance. The gauge is the alert that arrives first. When the Payrail breaker on api-01 opens, checkout on that instance is degraded from that second, and the alert on the gauge fires before the error rate has risen enough to cross its own threshold. It is also the explanation for the dashboard that would otherwise make no sense: 50 percent of checkouts failing with a 503, no 5xx from Payrail in the logs, and the answer is that api-01's breaker is open and api-02's is not, because the two saw different windows.

A breaker that flaps, opening and closing every 30 seconds for an hour, is a threshold set too tight or a cooldown set too short for the dependency's recovery, and the transition log is where that is diagnosed. A breaker that never opens through a real outage has a failure predicate that is missing the outage's error type, and the same log, empty, is the evidence. Chapter 13 puts the gauge on the service dashboard next to the RED metrics, and it is the one line on it that names a cause rather than a symptom.