Tokens, Windows, and the Bill

Tokens are the unit of everything: what fits, what it costs, and how long it takes. Input and output are priced differently, the context window is a hard ceiling on both together, and in a loop the input is re-sent with everything that came before it on every single turn.

That last property is why an agent costs several times what a naive estimate predicts. Vera's first budget for the Sundry agent was "about four model calls per ticket, about 1,300 tokens each" — a figure that was wrong by a factor of two on an easy ticket and by a factor of six on a hard one. This page does the arithmetic properly, because every optimization in Chapter 5 is justified by it.

What Counts as a Token

A token is a fragment of text the model treats as one symbol. For ordinary English prose the rule of thumb is about four characters, or roughly three quarters of a word. That ratio gets worse for everything an agent actually handles: JSON keys and punctuation, UUIDs, order ids like SU-88421, timestamps, currency codes, and non-English text all tokenize far less efficiently than a paragraph of prose does.

The practical consequence is that a 2 KB tool result is not "2,000 characters of free context". It is somewhere between 500 and 900 tokens of real spend, charged again on every turn that follows it. Engineers who have only ever counted characters consistently underestimate the cost of structured data, which is exactly the material an agent's context fills up with.

Input, Output and the Price Asymmetry

Input and output are billed separately and are not billed at the same rate. Output costs several times input at every major provider, and that ordering has held for as long as these APIs have existed, because generating a token is far more expensive than reading one. Cached input, where the provider offers it, is cheaper again by roughly an order of magnitude.

Two design consequences follow. Verbose model replies are expensive in a way that long tool results are not — an instruction like "explain your reasoning in detail before every action" is a per-turn charge at the highest rate on the price sheet. And a long tool result, while merely costly on the turn it arrives, becomes an input cost on every remaining turn, which is a slower and quieter kind of expensive. Shaping tool results is Chapter 3's problem; keeping replies terse is a prompt decision you make once.

The Window Is a Ceiling, Not a Budget

Everything must fit inside the context window at once: the system message, the tool schemas, the whole history, every tool result you have appended, and the reply the model is about to write. It is a hard ceiling on the sum, not a per-message limit and not an allowance you are meant to spend.

Treating it as a budget is a mistake with two separate costs. The first is mechanical: fill 95% of the window and there is no room left for a useful answer, so the reply truncates and the loop gets a half-sentence it must handle as a distinct outcome (Topic 10). The second is behavioural and arrives long before truncation does — instruction adherence degrades as the context fills, and the constraint stated at the top stops competing successfully with the 8,000 tokens of tool output stacked below it. That is the drift Chapter 5 opens with, and it happens well under the ceiling.

The Quadratic Bill of a Loop

Turn N re-sends turns 1 through N-1. That single sentence is the whole cost model, and the arithmetic is worth doing once by hand rather than trusting to intuition. Here is a real Sundry ticket — the cracked shelving unit — resolved in four model calls: a search, an order lookup, a policy lookup, and a final answer.

The context tops out at 4,880 tokens including the final 300-token answer. The bill is 11,990 input tokens plus 700 output tokens. So the ticket costs roughly two and a half times its own final context, and more than twice what "four calls at 1,280 tokens" predicted. Nothing was wasted and nothing misbehaved — this is what a correct, efficient four-call ticket costs.

Now take the hard case: an ambiguous ticket that runs to Sundry's twelve-turn limit, each turn adding about 1,100 tokens of reply plus tool result. The final context is 13,380 tokens, which sounds manageable. The billed input is the sum of twelve growing prefixes: 87,960 tokens, about six and a half times the final context and nearly six times the naive twelve-calls-at-1,280 estimate. It is also more than the 60,000-token ceiling the loop below enforces, so on this ticket the spend control would have stopped the run around turn ten, before the turn limit ever mattered — two stops, and on the longest tickets the money one fires first. The cost of a run grows with the square of its length, which is why a turn limit is a spend control and not just a termination guarantee, and why compaction and caching in Chapter 5 pay for themselves at exactly the point where tickets get hard.

Where the Tokens Actually Go

Measured on that same ticket, at the moment of the final call, the context breaks down like this.

Tool results are 59% of the context, and they are the only part that scales with how hard the ticket is. The system prompt — the part everyone spends their afternoons editing — is 420 tokens inside a 1,100-token fixed prefix that never changes and caches almost for free. Trimming it by a third saves under 150 tokens per turn. Trimming the get_order response down to the fourteen fields the model actually reads saves 600 tokens on every turn from that point on, and it is a change at the tool boundary rather than in a prompt.

This is the single most useful measurement to take in an agent's first week, and almost nobody takes it. Instrument the four buckets separately, per turn and per resolved ticket, and the optimization work orders itself: the expensive thing is usually a JSON blob nobody read, not an adjective in the system prompt.

Counting Before Sending

You can count tokens before you send them. Providers expose a counting endpoint, and local tokenizer libraries give you the same figure without a network round trip. Neither is required to be exact for this purpose — an estimate within 10% is enough to enforce a ceiling in code, because you are deciding whether to make another call, not issuing an invoice.

BUDGET = 60_000          # tokens per ticket, input plus output

used = 0
while turn < 12 and used < BUDGET:
    est = count_tokens(messages, TOOLS)     # local tokenizer, within ~10%
    if used + est > BUDGET:
        return escalate_to_human(ticket_id, summary_so_far())
    reply = model(messages, tools=TOOLS)
    used += reply.usage.input_tokens + reply.usage.output_tokens

In words: before each call, estimate what this request will cost and check it against what the ticket has already spent. If the next call would push the run past its ceiling, hand the ticket to a person with the work so far attached rather than making the call and discovering the overspend afterwards. Hitting the ceiling is a defined outcome with a defined action, exactly like hitting the turn limit — Chapter 7 treats both as stopping conditions rather than errors.

The counter comes from the response itself. Every provider returns the input and output token counts it billed for, so the run's true spend is a sum of numbers you already have, not a model of what you think you sent. Log both figures per turn from the very first version; retrofitting cost instrumentation after a surprise invoice means you cannot see which change caused it.