Why Retrieval Fails
Retrieval fails in three ways, and only one of them is the way people debug. The passage was never retrieved. The passage was retrieved and outranked by a worse one. Or the right passage was sitting in the context and the model answered from something else. They need three different fixes, and applying the wrong one is how a team spends a fortnight tuning an index that was already returning the correct document.
At Sundry the third kind was the most common. When policy-correct resolutions stood at 63%, twelve of the thirty-two policy tickets were wrong; Vera read all twelve traces in an afternoon. Two had never retrieved the governing document, three had retrieved it and ranked it below a worse one, and seven had the correct passage in context and answered from a different one. No retrieval metric moves for those seven. Recall was perfect on every single one.
Three Failure Kinds
Telling them apart takes ten minutes and one trace, provided the trace records the passages the tool returned rather than only the reply. Ask two questions in order: was the governing document in what came back, and did the answer use it. Two questions, three outcomes, three different teams' worth of work.
| Kind | What the trace shows | Where the fix lives |
|---|---|---|
| Never retrieved | The document is absent from the returned passages | Chunking, query, index coverage |
| Retrieved but outranked | It came back, below a worse passage the answer used | Scope filters, merge weights, reranking |
| Retrieved but ignored | It came back first, and the answer says something else | Fewer passages, citation, placement |
The third row is the uncomfortable one, because every retrieval metric you can compute scores it as a success. Recall is 100%. Precision is fine. Mean reciprocal rank is 1.0. The dashboard is green and the buyer was told the wrong return window, which is why the end-task number in Chapter 9 is the one that decides whether this work helped and the component metrics only say where to look.
Chunking Is Where Most of It Starts
A clause split across two chunks retrieves as two half-answers. Each looks like a policy statement, neither states the rule, and the model assembles something reasonable out of the halves — which is the worst possible outcome, because it produces a fluent answer with no visible defect. The failure is created at index time, months before the ticket that trips it.
Sundry's worst case was the marketplace refund matrix, a table mapping own-stock versus marketplace against the reason for the return, with the outcome and who absorbs the cost in the last two columns. A fixed 512-token window cut it mid-row, producing chunks that carried a scenario and no outcome. Re-chunking it one row per chunk, with the header row repeated into each, moved correct-document-in-top-three by nine points on its own — more than swapping the embedding model moved it in either direction, and at a fraction of the effort.
The rule that follows is short. Chunk along the document's own structure — clause, row, procedure step — never on a token count that knows nothing about the document. Never split a table row or a numbered clause. And carry the document title and heading path into each chunk, because a passage retrieved alone has to say what it is part of or the model will guess.
The Overlapping-Document Case
Two documents matching one query is the normal case at Sundry, not the edge case. A buyer asks about returning a damaged shelving unit; Sundry's own returns policy matches, and so does the Ashcombe Furniture supplement. Both are real, both are current, both are on topic, and one of them governs this order while the other does not. Retrieval did nothing wrong by returning both — the question is what decides between them.
Three things decide, in this order. Scope filtering comes first and runs before ranking: on a marketplace order, supplements from other sellers are not candidates at all, and on a Sundry-shipped order, no seller supplement is a candidate. Precision comes second: three passages rather than ten means the two that survive are the two that matter. And the passage's own stated scope comes third, in the text handed to the model, so that when both a Sundry-wide rule and this seller's supplement are legitimately in play the model can see which is which and cite the narrower one.
Filtering after ranking instead of before it is the quiet version of this bug. The scores were computed against a candidate set full of documents that could never apply, so the ranking that survives the filter is the leftovers of a competition the right document was never judged in. Filter, then rank.
Retrieved But Ignored
The passage is in context, at the top of the list, correctly scoped, and the reply quotes 30 days at a buyer whose seller grants 14. Three mechanisms produce it. The model answers from its prior, because "30 days" is what most retailers do and the training data agrees enthusiastically. It answers from an earlier passage still sitting in the context from a search on turn four, which contradicts the one from turn seven. Or the right passage is buried in the middle of a long context, where Chapter 5 showed adherence is weakest.
The mitigations are all about competition rather than about retrieval quality. Return fewer passages, so there is less to be wrong with. Drop superseded passages from the context when a second search replaces them, rather than letting both sit there arguing. Place the passages near the decision, at the end of the context rather than the start. And require a citation, which is the only one of the four that also makes the failure visible: a reply that names SUP-ASH-011 clause 2.1 and then states 30 days is a contradiction a deterministic check catches in Chapter 9, while an uncited wrong answer is indistinguishable from a right one until a buyer complains.
Measuring Retrieval Separately
Retrieval gets its own eval set, run independently of the agent. Sundry's is 150 questions drawn from real tickets, each labelled with the document and clause that should answer it and with the order context it was asked in — marketplace or own stock, which seller. The metrics are correct-document-in-top-three and scope-correct rate, and the whole run takes about 90 seconds with no generation calls at all — one embedding call per question and a nearest-neighbour search — because nothing in it needs the agent.
That independence is the point. Run it on every index change, every chunking change, every merge-weight change, and a drop is attributable to the index by construction rather than by argument. Without it, every retrieval problem arrives disguised as an agent problem and gets debugged where agent problems get debugged: in the prompt. Sundry spent two days adding a sentence to the system prompt about checking the seller's terms before anybody looked at what the tool had actually returned, and the tool had been returning a chunk of a table with no outcome column.
The Recall Trap
The instinct when answers are wrong is to return more passages. It works, in the sense that the metric it targets moves: at Sundry, going from three passages to ten raised correct-document-in-top-k from 89% to 97%. Almost nothing was missing any more. Policy-correct resolutions on the 32-ticket slice fell from 26 to 23 in the same run.
Read those two sentences together, because they are the strongest argument in this book for measuring the end task rather than the component. Raising the passage count fixes the rarest of the three failure kinds and feeds the most common one: seven more documents in context is seven more chances for the model to answer from the wrong one, plus a longer context in which adherence is already degrading. If recall genuinely is your problem, the honest fix is to retrieve twenty candidates and rerank them down to three, which raises the chance that the right passage is among the finalists without putting the losers in front of the model.
- Raising top-k when answers are wrong — it addresses the rarest failure kind, worsens the most common one, and at Sundry cost three tickets on the eval slice while every retrieval metric improved.
- Chunking by fixed token count across structured documents — tables and clause lists come apart, half-rows retrieve as policy statements, and the resulting nonsense reads as a model problem.
- Filtering by scope after ranking instead of before it — the scores were computed in a competition full of documents that could never apply to this order.
- Having no retrieval eval — every index problem then presents as an agent problem, and gets two days of prompt edits before anyone reads what the tool returned.
- Diagnose which of the three failure kinds you have from a trace before changing anything — the trace has to record the passages returned, not only the reply.
- Chunk along the document's own structure, never split a table row or a clause, and carry the title and heading path into every chunk.
- Filter by scope and effective date before ranking, so the ranking only ever orders documents that could actually govern this case.
- Keep a separate retrieval eval set with labelled documents, and run it on every index, chunking or merge-weight change.
Knowledge Check
A trace shows the seller supplement returned in first place, and the reply quotes Sundry's 30-day window. Which failure is this?
- Never retrieved — the governing document did not make it into the returned passages
- Retrieved but ignored — the right passage was in context and the answer came from elsewhere
- Retrieved but outranked — a worse passage was ranked above the one that actually governs the case
- A stale index — the supplement stored in the index no longer matches the current source document
Answers are wrong, so the team raises the passage count from three to ten. What happens?
- Retrieval metrics and answer quality both improve, since more candidates can only help
- Retrieval slows enough that the extra candidates cost more in latency than they return
- Retrieval metrics improve and end-task accuracy drops, because more documents compete in context
- The context window overflows on the longer tickets, so the run truncates before the reply is ever written
What does chunking a refund matrix on a fixed 512-token window destroy?
- The embeddings themselves, since a chunk cut mid-row cannot be embedded coherently
- The row structure, so a chunk carries a scenario without the outcome that resolves it
- The document metadata, which cannot be attached once a chunk spans two sections
- The keyword index, which can no longer match identifiers split across two chunks
Why keep a retrieval eval set separate from the agent's 120-ticket eval?
- It replaces the agent eval for index changes, which halves the labelling work
- It can use synthetic questions, which are cheaper to produce than labelled real tickets
- It measures resolution more precisely, since retrieval is the component that decides it
- It runs with no generation calls, so a drop after an index change is attributable to the index
Sundry's policy and a seller supplement both match one query. What is the first thing that should decide between them?
- A scope filter applied before ranking, so only documents that can govern this order compete
- A prompt instruction telling the model that it should always follow whichever window is the shorter one
- The similarity score, since the more relevant document ranks above the less relevant one
- The effective date, since the more recently published document supersedes the older one
You got correct