Timeouts, Rate Limits, and Backoff

Carriers time out, model APIs rate-limit, and both happen inside a loop with a customer waiting. Per-call timeouts, exponential backoff with jitter, bounded retries and a queue in front of a rate limit are ordinary backend engineering that works here exactly as it works in any HTTP service — a future Backend Deep Dive course owns that material, and this page assumes it rather than teaching it.

Two things are genuinely different because a model sits in the control flow, and they are what the rest of this page is about. A run has a wall-clock budget that has to travel through a loop nobody wrote as a single request. And there is a third retry layer above the two you already know, because the model can simply ask for the same call again — which arrives as a fresh decision and is counted by nothing.

Standard Mechanics, Briefly

Sundry's settings, stated once so the numbers later have something to stand on. track_parcel has an 8-second timeout and 2 retries with exponential backoff and jitter, retried inside the tool because a carrier dropping one request in fifty is noise your code should absorb rather than narrate (Chapter 3). The read-only tools retry freely at any layer. The write tools retry automatically at no layer, per Chapter 3's classification, because a repeat there is a second refund rather than a second lookup. The model API has its own client, its own timeout, and a queue in front of it.

Every one of those settings is defensible on its own, and none of them is interesting in isolation. What makes them worth a page is that they compose, inside a loop, and the number the customer experiences is the total rather than any individual line.

Deadline Propagation Through the Loop

The budget belongs to the run, not to any call in it. Sundry's target is p95 9 seconds to a first useful message, with a hard 45-second ceiling after which the run must say something true even if it is not finished. That only works if every call knows how much of the 45 seconds is left, which means the deadline is set once and each timeout is derived from it.

MIN_USEFUL = 3.0        # below this, no call can finish in time to matter

def call_budget(run, default):
    remaining = run.deadline - monotonic()      # deadline set once, at ticket start
    if remaining < MIN_USEFUL:
        raise OutOfTime(remaining)              # go degraded, do not start
    return min(default, remaining)

# in the tool client, for every call the loop makes
timeout = call_budget(run, TOOL_TIMEOUTS[call.name])

In words: set one deadline when the ticket starts, from a monotonic clock so that a clock adjustment cannot move it mid-run, and give every call the smaller of its own timeout and the time actually left. When the remainder falls below the shortest call that could still be useful — three seconds at Sundry — stop starting work and take the degraded path, because a call that cannot finish in the time available still costs the whole wait before it fails.

The arithmetic of what this prevents is worth doing once. A single track_parcel call against a sick carrier: 8 seconds, 1 second of backoff, 8 seconds, 2 seconds of backoff, 8 seconds — 27 seconds, and every one of those numbers passed review. Add two model turns at about 2 seconds each and one further tool call, and a ticket with a 9-second target takes 40. No layer misbehaved. Each did precisely what it was configured to do, which is why no layer can detect the problem: the deadline is the only thing in the system that can see the total.

Rate Limits on the Model API

A loop is bursty by construction — one ticket is up to twelve model calls in quick succession — so what the provider sees is turns in flight rather than tickets in flight. At Sundry's steady volume this never binds: 4,200 tickets a week is roughly 42 an hour across a fourteen-hour working day, which is fewer than five concurrent runs. The limits bind on three things instead, and all three are self-inflicted: the nightly eval suite running 120 tickets at concurrency 20, a backlog replay after an incident where a three-hour outage leaves 130 tickets starting at once, and a retry storm feeding itself.

Treat a 429 during turn eight as what it is: a partially completed run, with a return already started (Topic 45). Three rules follow. Queue against the limit with a concurrency cap per model rather than retrying into a wall, because retrying at full concurrency extends the limit window and slows everybody's tickets including the ones already half done. Give in-flight runs priority over new ones, since a run that has committed a write is more expensive to abandon than a run that has not begun. And honour the retry-after value the provider sends instead of a backoff you invented, which is the one number on this page you do not have to guess.

Degraded Modes

Every external dependency needs a written answer to "what do we do without it", decided before the outage rather than during it. A carrier being down does not mean the ticket fails; it means answering from what is known and saying plainly what is not.

The second row is the important one. For some dependencies the honest answer is that there is no degraded mode, and writing that down is a design decision rather than an admission. What is not acceptable is finding out at 03:00 by watching the agent answer a policy question with no policy in context — a grounding failure (Topic 44) produced by an availability problem, which is precisely why this otherwise-routine material sits in a chapter about failure modes.

The Interaction With the Model's Own Retry

Chapter 3 counted three retry layers and cared about correctness. The same three layers multiply latency, and the arithmetic is worse: three HTTP attempts inside two tool-level attempts, with the model free to ask again on any of its twelve turns, turns one slow dependency into a 90-second ticket. Only two of those layers appear in your retry metrics. The third looks like a new decision, because it is one.

The control is a result the model cannot misread. When a tool has exhausted its retries, say so unambiguously, mark it do_not_retry, and name what is still possible: "The carrier did not respond within 8s, after 2 retries. Tracking is unavailable right now. This does not mean the parcel was not sent." A model reading that escalates or answers from the order record. A model reading error: timeout asks again in good faith, and that call is bounded by nothing except the turn limit. The companion rule is structural — retry at exactly one layer per class of failure, and leave a comment at every place where a second layer would be easy to add, because eventually somebody will add one.