Retries, Backoff and Jitter

A retry is a bet that the second attempt will succeed where the first did not. It is a good bet for a dropped packet, a connection refused during a deploy, or a 503 from a provider that is shedding load for a second, and it is a bad bet for a bug, because the same request against the same code fails the same way three times. A retry is also load. When Payrail slows down and every one of Stagedoor's 3,000 requests a second retries at once, the retries are what finish Payrail off, and then they are what finish Stagedoor off, since each retry holds a task and a connection for another 3 seconds.

The policy has four parts: what to retry, how many times, how long to wait between attempts, and how much randomness to add to the wait. The first part is the one that matters, and it is the one most retry decorators get wrong by default, because they retry on any exception. Timeouts in Topic 37 gave every attempt an end; this topic decides whether there is a next attempt at all, and Topic 39 is what makes the next attempt safe.

What Is Safe to Retry

The question is not "did it fail" but "is the far side in a known state, and would the same request fail again." A connection refused or a connect timeout means the request never arrived: the far side's state is unchanged and a retry is safe. A 503 with Retry-After or a 429 is the far side saying "not now," which is a state it has told the caller about, and a retry after the header's number of seconds is what the header asks for. A read timeout on a POST that has no idempotency key is the unknown outcome of Topic 03 of Chapter 1: the work may have happened, and a retry may do it twice. A 400 or a 422 is the caller's own request being wrong, and the same bytes will be rejected the same way on every attempt.

The third and fourth rows are the same error with opposite answers, and the difference is the operation, not the status. Topic 06 of Chapter 2 defined idempotency as a property of the method and the request: a GET reads, a PUT sets a state, and repeating either leaves the far side where one attempt would have. A POST /orders with the Idempotency-Key that Topic 39 introduces joins that group, because the far side will recognize the repeat. A POST without one does not, and no retry policy can be written for it that is both useful and safe. The policy is therefore per operation, encoded once in the client wrapper, and the domain code never decides it.

Exponential Backoff

The wait between attempts doubles: 100 milliseconds, then 200, then 400, then 800, capped at 2 seconds. The doubling is a guess about recovery time that is right at every scale. A provider that needed 150 milliseconds to fail over a node is back by the second attempt; one that needs 5 seconds to restart a process gets 5 seconds of accumulated waiting by the sixth, without the client having to know which kind of failure it is looking at. The cap is the client's own patience: a wait of 51 seconds on the tenth attempt is a client that has stopped serving its user to be polite to a server, and nothing about the doubling says when to stop, which is what the attempt limit and the deadline are for.

Constant backoff is the tight loop. Three attempts 100 milliseconds apart are three requests inside a window shorter than a slow TLS handshake, against a provider that has failed in a way that takes seconds to clear; all three fail, the client reports failure, and the provider absorbed three requests for the price of one. The doubling costs the client nothing on the first attempt and buys it the whole recovery curve of the far side on the later ones.

Jitter

Backoff alone has a flaw that only appears at scale, and Stagedoor's scale is 3,000 requests a second. When Payrail drops a node, every request in flight at that instant fails within the same few milliseconds. Every one of them backs off 200 milliseconds. Every one of them retries at the same instant 200 milliseconds later, as a wave the same size as the one that just failed, and the next node absorbs it or does not. If it does not, the wave backs off 400 milliseconds, in step, and returns in step. Exponential backoff without jitter turns one failure into a series of synchronized waves, each as tall as the first, spaced further apart.

Full jitter replaces the fixed wait with a random one between zero and the backoff: not 200 milliseconds but somewhere in the 200-millisecond window, and on the next round somewhere in a 400-millisecond window. The same 1,000 retries arrive spread across the window instead of stacked at its end, and the provider sees a plateau it can serve instead of a spike it cannot. The AWS paper that named the technique measured it against the alternatives and found full jitter both finished the work sooner and made fewer total calls than backoff alone, which is the rare case of a change that is better on both axes. The wait_random_exponential strategy in tenacity is exactly this formula.

The Retry Budget

A limit of 3 attempts per request bounds what one request can do to Payrail. It does not bound what the service does, because at 100 checkouts a second a limit of 3 per request is up to 300 calls a second against a provider that was already struggling with 100. The retry budget is the service-wide view: retries are allowed to be at most 10 percent of the calls to a dependency over the last 10 seconds, counted per instance, and when they exceed that, the next failure is returned to the caller at once with no retry. The reasoning is that a retry rate above 10 percent means the failures are not transient any more, and the retries are now the load rather than the cure.

The two limits are different shapes and both are needed. The per-request limit is a count, and it protects the request's own deadline. The budget is a ratio, and it protects the dependency and the service's own pool from a coordinated response to a failure that is bigger than any one request can see. Stagedoor keeps a counter of calls and retries per dependency in each process, and the client wrapper checks the ratio before it decides an attempt is worth making. A budget that is spent is also a signal: the breaker in Topic 40 is the same observation made formally, and the two share a metric.

Retries and the Deadline

Three attempts at 3 seconds each is 9 seconds, and the checkout budget from Topic 37 is 10. The arithmetic only works because every attempt's timeout is taken from what remains: the first attempt gets 3 seconds because 9.9 remain, the second gets 3 because 6.7 remain, and the third gets 3 because 3.5 remain, with the backoff waits of 100 and 200 milliseconds spent in between. Had the seat hold taken longer, or had the cap on backoff been 2 seconds and the waits actually reached it, the third attempt would have been made with less than a second and given less than a second, and if nothing remained at all it would not have been made. That is correct. An attempt that cannot finish inside the client's patience is an attempt whose answer nobody will read, and refusing to make it is what the deadline is for.

The stop condition of the policy is therefore two conditions joined with or: stop after 3 attempts, or stop when the deadline has passed. The second one wins on a slow night, and the handler answers 504 with the order pending, the same response as a single timeout, because from the buyer's side it is the same situation. What changed is only how much of the 10 seconds the service spent trying before it said so.

Where the Retry Lives

The policy lives in the outbound client wrapper that Chapter 10 builds around Payrail, in one place, with its numbers in the configuration of Topic 23 of Chapter 4. It does not live at the call site, because there are 6 call sites and they would drift. It does not live in the domain, because place_order has never heard of a connect timeout. And it does not also live in the load balancer, and in the browser, and in Payrail's client library, each with a policy of three attempts, because those layers multiply: 3 attempts in the browser, each of which is 3 attempts at the balancer, each of which is 3 attempts in the service, is 27 requests to Payrail from one click. Stagedoor's rule is that exactly one layer retries an operation, and for outbound calls that layer is the service's wrapper.

def transient(exc) -> bool:
    return isinstance(exc, (httpx.ConnectError, httpx.ConnectTimeout)) \
        or (isinstance(exc, UpstreamStatus) and exc.status in (429, 503))

async def with_retry(op, ctx):
    if not budget.allows("payrail"):                       # retries over 10% of calls: fail fast
        return await op(timeout=min(3.0, ctx.remaining()))
    async for attempt in AsyncRetrying(
        retry=retry_if_exception(transient),                  # never a read timeout on a keyless POST
        stop=stop_after_attempt(3) | stop_after_delay(ctx.remaining()),
        wait=wait_random_exponential(multiplier=0.1, max=2.0),   # random(0, 100 ms · 2^n), capped
        reraise=True,
    ):
        with attempt:
            budget.note("payrail", retry=attempt.retry_state.attempt_number > 1)
            return await op(timeout=min(3.0, ctx.remaining()))

The wrapper does five things in order. It checks the service-wide budget and, if retries are already over 10 percent, makes one attempt and returns whatever happens. It classifies the outcome with one predicate, which names the connect errors and the two "not now" statuses and nothing else, so a read timeout or a 500 falls through to the caller unretried. It stops after 3 attempts or when the deadline passes, whichever is first. It waits a random interval between zero and 100 milliseconds times two to the attempt, capped at 2 seconds, which is full jitter. And every attempt's timeout is the smaller of Payrail's ceiling and the request's remaining budget, so that the third attempt is cut to fit. When the far side sends Retry-After, the wrapper sleeps that many seconds instead of the computed wait, because the far side has said what it needs and guessing over it is rude and useless.