The Context Budget

Give the context a budget in tokens and enforce it in code, the same way you would a memory limit on a service that must not be killed by the kernel. Without one, a ticket that happens to call three tools with large payloads triples in cost and degrades in quality, and nobody notices until the invoice arrives or the eval numbers move.

A budget is not the context window. The window is the provider's ceiling and it is much larger than the number you should be sending; the budget is the number you have decided is worth spending on a turn of this kind of work. Sundry's is 8,000 tokens per request against a window many times that, and the gap between those two numbers is deliberate.

Setting the Budget

Start from the tenants and allocate. Sundry's 8,000 splits into 1,100 fixed — the 420-token system prompt and 680 tokens of tool schemas — plus 800 reserved for the answer, leaving 6,100 that history and tool results have to share. The fixed part is fixed because changing it is a deploy; the reserve is untouchable; the 6,100 is where every decision in this topic actually happens.

The number itself comes from measurement rather than from the window size. Sundry set 8,000 because the 120-ticket eval set showed resolution flat between 6,000 and 12,000 tokens per request and falling below 5,000, and because a request budget of 8,000 keeps a typical run — a prefix that grows toward the cap over the eight or nine turns most resolved tickets take — inside the 60,000-token per-ticket ceiling the loop already enforced in Chapter 2. It does not keep the worst case inside: twelve turns sitting at the cap would be 96,000, and on that ticket the ceiling stops the run first, which is what it is for. Two constraints meeting at a round number is a better justification than "the window is big".

Write the allocation down as constants next to the loop, not as a comment in a design document. The reserve especially: a budget without an explicit reserve is a budget that spends the answer's room on one more tool result, and the failure shows up as a reply that stops mid-sentence.

Enforcement Points

There are three places to enforce, and they are not interchangeable. Cap each tool result at the tool boundary, which is the cheapest cut because a field trimmed there is trimmed on every remaining turn (Chapter 3). Cap the history, which is compaction and is Topic 27's subject. And check the total immediately before sending, which is the backstop that catches the combination nobody predicted — two medium results and a long thread, each individually within its own limit.

BUDGET     = 8_000   # tokens in a single request
RESERVE    =   800   # room the answer must have
CAP_RESULT = 1_200   # per tool result, at the tool boundary

def fit(messages, tools):
    while count(messages, tools) + RESERVE > BUDGET:
        if not compactable(messages):
            return escalate_to_human(ticket_id, summary_so_far())
        messages = compact(messages)   # middle of the history first
    return messages

In words: before each call, count what you are about to send plus the room the answer needs. While that exceeds the budget, compact the history. If there is nothing left that may be compacted — the pinned material alone has overflowed the budget — hand the ticket to a person with the work so far attached, rather than sending a request you know is too big and letting the API decide what to lose.

That last clause is the whole point of enforcing in code. If you do not define the behaviour at the cap, the provider defines it for you, and it defines it by truncating at exactly the wrong moment: mid-decision, on the turn where the agent had finally assembled everything it needed. A defined action — compact, drop, escalate — is a designed outcome. Truncation is an outcome that arrives.

Priority Order When Cutting

Some material is pinned and never eligible for removal: the buyer's original ticket text, the hard constraints, the structured state recording what the agent has already done, and the results of the most recent tool calls. Those four are what the current decision is actually made of. Everything else is negotiable, and the first thing to go is the middle of a long history — the turns from the middle of the thread, which are also the region the model attends to least (Topic 28). Cutting where attention is already lowest is the one place where the cheap fix and the correct fix coincide.

The naive alternative is to drop the oldest messages, and it is wrong for a specific reason: the oldest message is the buyer's original ticket, and it is the most important thing in the buffer. Sundry shipped that version for a week. The symptom was an agent that, twenty turns into a damage claim, started answering a question about delivery timing — because the sentence saying the shelving unit arrived cracked had been evicted as the oldest item, and everything still in context was about the parcel.

Measuring the Composition

You cannot set a sensible budget without knowing what fills it. Here are two real Sundry contexts side by side: a typical order-status ticket, which is 55% of the queue, and a disputed damage claim that ran to eleven turns.

The bad ticket is not bad because the conversation was long. Nine extra turns of dialogue added 700 tokens; the whole overflow came from two payloads — a get_order response on an eleven-item order at 4,200 tokens and a track_parcel response carrying the carrier's entire scan history at 3,800. Tool results were 59% of the canonical ticket in Chapter 2 and are 78% here, and they are the tenant nobody was watching. Capping those two results at 1,200 each brings the same ticket to 4,650 tokens without removing a single turn of the conversation the buyer actually had.

The Budget as a Product Decision

A higher budget buys quality on hard tickets and costs money on every easy one. That makes it a routing decision rather than a global setting: the 55% of the queue that asks where a parcel is does not need 8,000 tokens, and the disputed damage claim with a seller in the middle may deserve 16,000. Sundry runs three budgets keyed on the triage classification — 4,000 for status, 8,000 for returns and refunds, 16,000 for disputes — and the blended cost per ticket came down because the largest class got the smallest number.

Set the budget per task type, and revisit it when Chapter 13 puts real cost figures next to real resolution figures. A single global budget is always wrong in both directions at once: generous enough to waste money on the easy majority, and tight enough to lose the hard minority that was worth spending on.