Retrieval as a Tool
The policy library cannot go in the prompt. It is five Sundry-wide documents plus roughly 2,000 per-seller supplements, it changes every week, and 99% of it is irrelevant to any single ticket. Chapter 5 proved the point the expensive way: the twenty most-quoted passages pasted into the system prompt still left the agent quoting Sundry's own 30-day window at a buyer whose seller runs the statutory 14 days.
The fix is to stop deciding for the agent. search_policy becomes a tool it may call, at the moment it chooses, with a query it writes itself — after it has read the ticket and looked up the order, which is more than the prompt author ever knew. That is cheaper and more accurate than prefetching, and the two are not obviously different until you count what each one spends and on which tickets.
Search as a Tool Versus Prefetch
Prefetch is the classic shape: take the buyer's message, run one search, paste the top passages into the context, then make a single model call. It is cheap, predictable, adds no model turn, and it is blind. The search runs against the customer's own words before anybody knows whether the ticket is a policy question at all, and "the shelving unit turned up Tuesday with a cracked side panel, and I've been charged twice for delivery" is not a query anyone would type into a policy library.
A tool call costs one extra model turn — about 1.4 seconds at Sundry — and buys three things a prefetch cannot have. The agent searches after it knows the case: marketplace seller, delivered 10 March, damage in transit, buyer wants a refund rather than a replacement. It can search a second time when the first passages do not settle the question. And it can not search at all, which matters more than the other two, because roughly three quarters of Sundry's 4,200 weekly tickets never turn on a policy document: the 55% asking where a parcel is, and most of what is left. No policy document has ever helped one of them.
Count what the blind version spends. A prefetch injects around 850 tokens of passages into every ticket, and those tokens are then re-sent on every remaining turn of the run, so a six-turn ticket pays for them roughly five times over. Across the queue that is on the order of 16 million input tokens a week bought for tickets that never needed a policy document, and every one of those tokens also sits in the context competing for attention with the ticket the agent is actually trying to settle.
What Comes Back
A passage without provenance is a sentence with no authority behind it. The agent is not being asked to summarize a document; it is being asked to decide which of two legitimate rules applies to this order, and that decision runs on metadata as much as on text. Sundry's tool returns, for every passage: the document title, its identifier, its effective date, its scope, the clause id, and the text itself.
{"query": "return window for damaged marketplace order, delivered 10 March",
"passages": [
{"doc": "Sundry Returns Policy",
"doc_id": "POL-RET-004", "clause": "4.2",
"scope": "sundry-wide",
"effective": "2025-11-01",
"text": "Items may be returned within 30 days of delivery…"},
{"doc": "Ashcombe Furniture — Seller Supplement",
"doc_id": "SUP-ASH-011", "clause": "2.1",
"scope": "seller:ashcombe-furniture",
"effective": "2026-01-15",
"text": "Returns accepted within the statutory 14 days of delivery…"}]}
Two documents came back and both are correct. One is Sundry's own rule; the other is the supplement for the seller who actually shipped this order, and it is newer and narrower. With the scope and effective-date fields in front of it, the model has a decidable question. Strip those fields out and the same two passages are a pair of contradictory paragraphs with no tiebreak, and it picks one — which is the drift wound, reproduced exactly, out of a missing field rather than a missing document.
Provenance also constrains what the answer may say. A passage that arrives with an identifier can be quoted by that identifier, and a reply that names SUP-ASH-011 clause 2.1 is a claim a machine can check. A passage that arrives as bare prose can only be paraphrased, and a paraphrase is unfalsifiable.
Query Formulation Is the Model's Job
The obvious implementation writes the query in code: strip stopwords from the ticket, keep the nouns, search on that. It produces "cracked shelving unit charged twice delivery", which retrieves passages about delivery charges and passages about damage, and nothing about who owes what on a marketplace order. The agent's own query, written on turn four after get_order came back, was "return window and refund liability for damage in transit on a marketplace order, delivered 10 March" — a question that could only be asked by something that had already read the order record.
On Sundry's retrieval eval the difference was 74% to 89% for the correct document appearing in the top three. The tool description is what unlocks it, and it has to invite a question rather than a topic: "Search the policy library. Pass a specific question in plain English, including the seller and the order type if you know them. Returns passages with their scope and effective date; it does not decide the case." Two sentences, and they are the difference between a model that passes a noun phrase and one that passes the question it actually needs answered.
Grounding and Citation
Every policy-based answer names the document and clause it came from. Chapter 5 put half of that rule in the system prompt — never state a return window from memory, call search_policy and answer from what it returns — and the citation requirement is the enforcing half. The agent's structured reply carries a policy_ids list alongside the customer-facing text, and the dispatcher rejects a reply that quotes a window with an empty list.
That buys two things which arrive much later. Chapter 9 can check the citation automatically: does the cited clause exist, was it in the passages actually returned this run, is its scope right for this order, and does its text support the claim the reply makes. Those are cheap deterministic checks that need no model judge for three of the four. And an escalated ticket reaches a person with the rule attached rather than with an assertion, which turns a five-minute investigation into a ten-second read.
Keeping the Index Current
The library changes weekly. Sellers edit their supplements, legal edits the damage-in-transit procedure, and the refund matrix picks up a row whenever finance changes who absorbs a carrier claim. Reindex on document change — the policy system emits an event, the indexer consumes it — rather than on a nightly job that quietly skipped Sunday. Monitor the age of the oldest un-reindexed document as an operational metric, and alert on it; Sundry alerts when any document's indexed copy is more than 60 minutes behind its source.
A stale index is the drift wound arriving through a new door, and it is worse than the original because it arrives dressed as evidence. The agent cites a real document with a real clause id and a February effective date, the reply reads as carefully grounded, and it is wrong because the March version of that clause says something else. The only defence is the effective date travelling with the passage, so a human reviewing an escalation and a check in Chapter 9 can both see which version was used.
The Sundry Result
Of the 120 tickets in the eval set, 32 turn on a policy document. Before this work, 20 of those 32 were policy-correct; after it, 26 — 63% to 81%, six tickets, and the drift closed. Vera measured the three changes separately rather than shipping them as one improvement, because a bundle that works teaches nothing about which part did it: moving from prefetch to a tool was worth two tickets, adding provenance and scope was worth three, and the citation requirement was worth one.
The costs moved in both directions and both are worth stating. Prefetch stopped spending 850 tokens a ticket on three quarters of a queue that never needed them, which is the larger number. Against that, the tickets that do search now pay an extra model turn, about 1.4 seconds, which shows up in the p95 for that slice and is invisible in the queue-wide number the 9-second target is written against. That is the trade in one line: a slower quarter of the queue, a cheaper whole queue, and eighteen points on the slice that was getting the law wrong.
Prefetch — one search on the user's message, before the model has seen anything, with the results pasted into the context. Cheap, predictable, one network call and no extra turn. It is also blind to what the ticket needs, and it pays on every request whether or not a document was ever going to help.
Retrieval as a tool — the agent calls search when it decides it needs to, after reading the ticket and looking up the order. It costs one model turn, and it can search with a better query, search again, or skip searching entirely.
Choose prefetch when nearly every request needs the same kind of document and the user's own words make a decent query — a documentation chatbot, for instance. Choose the tool when a minority of requests need retrieval at all, or when the right query depends on something only a lookup can tell you. For a support queue where a quarter of tickets turn on policy, the tool wins on cost and on accuracy at the same time.
- Prefetching on every ticket — three quarters of Sundry's queue never needed a policy document, paid roughly 850 tokens for one on every turn of the run, and had it competing for attention the whole way.
- Returning passages without provenance — the agent cannot tell Sundry's own policy from a seller supplement, which is not a subtle degradation but the exact mechanism of the drift.
- Letting the tool return ten passages by default — recall rises, precision falls, and the wrong document is now in context arguing with the right one (Topic 34).
- Indexing the library once and forgetting it — a clause updated in March is answered from the February version until somebody notices in July, and every wrong answer arrives with a confident citation.
- Expose retrieval as a tool whose description invites a specific question in plain English, names what it returns, and states that it does not decide the case.
- Return provenance with every passage — title, effective date, scope, and an identifier the reply can cite — and make the identifier the thing Chapter 9 checks.
- Default to few, high-precision passages and let the agent ask a second time rather than widening the first search.
- Reindex on document change and alert on index age, so a stale passage cannot masquerade as a checked one.
Knowledge Check
Why does retrieval as a tool beat prefetching on Sundry's queue, on both cost and accuracy?
- A tool call can use vector search, while a prefetch step is limited to plain keyword matching only
- Most tickets need no policy at all, and the good query depends on what the order lookup returned
- The tool removes a model turn from every single ticket, which is where both the cost and the time go
- Passages fetched by a tool are cheaper per token than passages pasted into the context before the call
A search returns Sundry's 30-day return clause and a seller supplement granting 14 days. What lets the agent decide correctly?
- The ranking, since the more relevant of the two documents is returned in the first position
- A system prompt rule instructing it to always prefer the stricter of two conflicting windows
- The length of the passages, because the fuller document carries the governing statement
- The scope and effective date on each passage, which say which document governs this order
Why default search_policy to three passages rather than ten?
- Extra passages raise recall but put competing documents in front of the model at the same time
- Ten passages would overflow the context window before the agent could finish the ticket
- Returning ten results makes the search itself slow enough to breach the latency budget
- The tool can only be called once per run, so the first result set has to be kept small
The policy team edits a clause in March and nobody reindexes. What does the agent do in April?
- It fails the search outright, because the indexed clause no longer matches the source document
- It notices the discrepancy against its own training data and escalates the ticket to a human
- It answers from the February version and cites a real clause id, so the error looks verified
- It reads the effective date, sees the passage is out of date, and searches the library again
Why is the query written by the model rather than extracted from the ticket text in code?
- Only a model-written query can be matched semantically, since extracted keywords are matched literally
- The agent can ask about facts the ticket never stated, such as the seller and the order type
- Model-written queries are shorter, and short queries retrieve more precisely from a large library
- Queries composed by the model bypass a validation step that would otherwise reject the search
You got correct