Prompt Caching

In a loop the same prefix goes out over and over: the system prompt, the tool schemas, and every turn that has already happened. Providers will cache that prefix and charge roughly an order of magnitude less to read it back, which turns the quadratic bill of Chapter 2 into something a support queue can actually afford.

The catch is that caching rewards a stable prefix, and most teams destroy theirs by accident. One interpolated timestamp near the top of the system prompt is enough to miss on every single request forever, and nothing in the response says so unless you go looking. Sundry went from $0.41 a ticket to $0.14 on this topic alone, before any of the routing in Chapter 7 or the model-selection work in Chapter 13.

How Prefix Caching Works

The cache is keyed on an exact prefix of the request. The provider hashes the request from the first byte forward and looks for the longest stretch it has already processed; everything up to the first difference can be served from cache, and everything after it is processed normally. There is also a floor: below a per-model minimum — a few hundred to a few thousand tokens, depending on the provider and model — nothing is cached and no error says so, which matters because Sundry's fixed prefix is 1,100 tokens and clears that floor on some models and not others. Byte-identical is the standard — one changed character anywhere means the cache match stops at that character, and everything after it is charged in full.

The first turn pays to write the entry, which costs a premium over ordinary input on most providers, and every later turn pays a fraction to read it. That asymmetry is what makes this a loop optimization rather than a general one: a single-shot call pays the write premium and gets nothing back, while a twelve-turn ticket writes once and reads eleven times against a prefix that is growing the whole way.

Ordering for Cache Hits

One rule covers most of the value: order the context stable-to-volatile. System prompt first, then tool schemas, then the conversation history, then whatever is new on this turn. Anything that changes belongs after everything that does not, because the cache match ends at the first difference and everything downstream of it is lost.

# everything above the first difference can come from cache
request = [
    system_prompt,     # 420 tokens, byte-identical every turn
    TOOLS,             # nine schemas, sorted by name, order never varies
    *history,          # append-only; earlier turns are never rewritten
    current_turn,      # the only part that differs from the last request
]

# the line Sundry shipped for three weeks, which cost every hit:
# system_prompt = BASE.format(now=datetime.utcnow().isoformat())

Read the commented-out line as the lesson. Putting the current time into the system prompt is a reasonable-looking thing to do — the model does not know today's date, and a support agent reasoning about delivery windows needs it. But it changes the first few hundred bytes of every request, so the cache match ends almost immediately and the entire context is charged at the full rate on every turn of every ticket. Sundry's fix was to pass the date in the user turn, at the end of the buffer, where changing it costs nothing.

The same rule applies to the tool list. Schemas assembled from a dictionary, or from MCP servers in whatever order they finished connecting, produce a differently ordered list on some requests and an identical one on others (Chapter 4). Sort them by name and the ordering stops being a source of random cache misses that nobody can reproduce.

What the Savings Actually Are

Take Chapter 2's four-call ticket, which bills 11,990 input tokens. Of that, 7,410 is prefix that calls two, three and four re-sent unchanged — 1,280, then 2,550, then 3,580. Charged at roughly a tenth, that repeated material costs the equivalent of about 740 tokens instead of 7,410, bringing the ticket to around 5,300 billed-equivalent tokens: a 56% cut without changing a single thing the agent does.

The twelve-turn ticket does much better, because the longer the run the larger the share that is pure repetition. Its 87,960 billed input tokens contain about 74,900 of re-sent prefix, so it lands near 21,000 — a 77% cut. That ordering is the important part: caching pays most exactly where the bill is worst, which is the opposite of most optimizations and the reason this one comes before the harder work in Chapter 13.

Blended across Sundry's queue the arithmetic predicts about $0.10 a ticket if every eligible turn hit the cache. The measured figure is $0.14, because the measured hit rate is 86% rather than 100%. Both numbers matter: the first tells you the ceiling, the second is what the invoice says. Budgeting on the first is how a team ends up explaining a 40% overrun to finance.

Cache Invalidation You Cause Yourself

Four things reset the cache, and all four are yours. Editing the system prompt, which changes the prefix for every conversation in flight across the fleet. Adding or removing a tool, which changes the schema block. Changing a schema's description — a one-word improvement to a tool description invalidates as thoroughly as a rewrite. And interpolating anything variable into the prefix: a timestamp, a request id, a ticket number, an A/B bucket, a hostname.

The operational consequence is a release discipline rather than a code change. Batch prompt and schema edits into planned releases instead of trickling them out through the day, because each deploy resets caches everywhere at once and a team that ships six prompt tweaks between nine and five has spent the day paying full price. Sundry moved prompt changes to the same cadence as application deploys, which cost nothing and recovered several points of hit rate.

Lifetime and Locality

Cache entries do not live long by default. As of 2026 the default lifetime is a few minutes; longer ones exist — an hour is common, and some providers go further — but they cost more to write, so the default is what most loops run on. Entries are scoped per provider account and often per region — so a request routed to a different region than the one that wrote the entry misses, and so does a thread that sat idle. That is not a rare edge case for a support agent: a buyer who replies four hours later resumes a thread whose prefix expired long ago, and turn one of the resumed conversation pays to write it all over again.

Design for the miss rather than assuming the hit. Measure the hit rate as a first-class metric next to cost per ticket, alert when it drops, and treat a sudden fall as a deploy or a routing change until proven otherwise — it is usually one of those two. At a 40% hit rate the same Sundry ticket costs about $0.29 instead of $0.14, which is a doubling that never shows up in any latency graph or error rate and can run for weeks unnoticed. How each provider spells cache control, what it charges to write an entry and how long entries survive are the moving details Chapter 2 quarantines on purpose.