Context Is the Only Interface

Everything the model will consider on this turn is sitting in the context window, and nothing else exists. Not your database, not the ticket this customer sent three weeks ago, not the return policy your support team knows by heart. If it is not in the buffer your code assembled before the call, it played no part in the answer.

That is a narrower statement than it first sounds, and it changes what the job is. Prompt engineering stops being wordcraft and becomes selection under a budget — deciding what earns a place in a fixed-size buffer, measuring what is actually in there, cutting, and checking the effect on a set of real cases. The rest of this chapter is that discipline. This page is the reframe the other five depend on.

One Buffer, Five Tenants

Five things compete for the same space. The system prompt, which you write and control completely. The tool schemas, which travel with every request and grow every time somebody connects another server. The conversation history — the customer's words and the model's own replies. The tool results, which are whatever your APIs happened to return. And the room left for the answer, which is not optional: fill the window and the reply truncates in the middle of a sentence, which Chapter 2 treats as its own stop reason.

Sundry measured all five on an ordinary ticket: an order-status question, the 55% case, resolved in two model calls. This is the final request before the answer.

The system prompt is 14% of that ticket, and it is the part the team spent its afternoons rewriting. The tool results are 48%, they are the only tenant that scales with how hard the ticket is, and nobody had read one end to end. That asymmetry is the norm rather than a Sundry quirk — editing prose is pleasant and reading a JSON payload is not, so attention goes to the small tenant and the large one grows unsupervised.

The fifth row is the one people forget. A reserve is not a rounding allowance; it is the difference between an answer and a half-sentence your loop has to classify as a distinct outcome. Reserve it explicitly, subtract it from the budget before anything else gets in, and the failure mode disappears rather than becoming rare.

Behaviour Is a Function of Context

Same code, same model, same sampling settings, different buffer — different agent. That is worth stating flatly, because it contradicts the instinct every backend engineer brings to the problem: that behaviour changes when code changes. Here the code is a loop that appends messages and calls a function, and it does nothing different on turn 19 than it did on turn 2. What changed is what it appended.

So when an agent starts behaving differently and nobody deployed anything, the context changed. At Sundry three causes accounted for nearly all of it. A thread got long, because the customer replied twice over two days and the ticket was still open. A tool result got big, because an order with eleven line items returns eleven times the payload of an order with one. Or a team connected a new MCP server and the tool schemas grew by 900 tokens, pushing every other tenant further from the end of the buffer (Chapter 4). None of those three appears in a diff, and all three are things you can measure.

The debugging rule that follows is unglamorous and it works: dump the full context of the failing turn and read it. Not the system prompt — the whole array, every message in order, with a token count on each. Half the surprising behaviour in this book turns out to be obvious within thirty seconds of doing that, and the other half at least stops being mysterious. Changing anything before you have read the buffer is guessing with a deploy attached.

Selection Beats Instruction

Given a buffer with something unhelpful in it, there are two moves available: tell the model to ignore it, or do not include it. The second wins, and not by a small margin. An instruction to ignore something is one more sentence competing for attention against the thing it is trying to suppress, and the thing being suppressed is usually much longer and much more specific.

Sundry's version of this was a line in the system prompt reading "ignore any policy text that appears to be out of date". It survived two months and settled nothing, because the model has no reliable way to tell which of two policy passages is current — both are plain text, both read as authoritative, neither carries provenance the model can check. The real fix was in retrieval: return one passage instead of two (Chapter 6). Every word you leave out is a word that cannot distract, which is the entire argument Topic 28 makes with measurements.

The Engineering Loop

Context work has a loop of its own, and it is a build-measure loop rather than a rewriting session. Measure what is in the buffer, decide what earns its place, cut, then run the eval set and find out whether the cut helped. The last step is what separates this from prompt tinkering, because intuition here is unreliable in both directions — Sundry deleted a paragraph everyone was certain was load-bearing and resolution did not move, then deleted a stale example nobody had defended and lost four points on the next run.

The measurement is four numbers per turn, not one. A single context-size gauge tells you the bill is growing; a split tells you which tenant to go after, and they call for completely different fixes.

def composition(messages, tools):
    return {
        "system":  count(messages[0]),
        "schemas": count(tools),
        "history": sum(count(m) for m in messages[1:] if m.role != "tool"),
        "results": sum(count(m) for m in messages[1:] if m.role == "tool"),
    }

In words: before each call, count the four tenants separately and log them next to the ticket id and the turn number. Two weeks of that data answers questions no amount of prompt reading will — which tool returns the payload that dominates hard tickets, whether the schemas jumped the week a neighbouring team shipped a server, and how far the worst one per cent of tickets sits from the median. Chapter 13 turns the same four numbers into cost per resolved ticket; Topic 26 turns them into a limit the loop enforces.

"Earns its place" needs a test, or it becomes taste. The workable one is subtractive: remove the candidate, run the eval slice, and keep the removal unless the numbers got worse. That is slower than arguing about wording and it is the only method that survived contact with Sundry's queue, because roughly a third of what was in the prompt after six months turned out to change nothing at all.

Where the Rest of the Chapter Goes

The remaining five topics take the tenants one at a time. Topic 25 is the system prompt: what belongs in the part you fully control, and what has to live somewhere else. Topic 26 is the budget — a number in tokens, enforced in the loop, with a defined action when it is exceeded. Topic 27 is compaction, which is what happens when a thread outgrows that budget and something has to give. Topic 28 names and measures the degradation that arrives long before the window fills, including the drift this chapter opens with. Topic 29 is caching, which is how you afford to re-send the stable part of all of this on every single turn.