Sundry in Production
The whole arc in numbers. Resolution on the 120-ticket eval set went from 61% to 88%. Model cost per ticket went from $0.41 to $0.06. Six per cent of tickets reach a human, p95 to a first useful message is 8.7 seconds against a product target of 9, and the incident log has three entries in the first quarter.
This is the topic that makes the book's claims checkable, so it also reports what did not work. Two of the changes the team was proudest of moved nothing at all, and both survived for weeks because nobody had built the instrument that would have said so. Read the whole page as one queue at one company with one policy library: the method transfers, the numbers do not.
The Table
Four points in the build, each one where a group of chapters landed. Column one is the forty-line agent from Chapter 1, running against the same eval set everything else is measured on. Column two is after context engineering and retrieval. Column three is after the failure work and the eval suite. Column four is today, after the trust boundaries and the production chapters.
| Measure | Forty-line agent | After Ch5–6 | After Ch8–9 | Today |
|---|---|---|---|---|
| Resolution, 120-ticket eval set | 61% | 73% | 86% | 88% |
| Model cost per ticket | $0.41 | $0.14 | $0.11 | $0.06 |
| Tickets reaching a person | 39%, as second contacts | 27% | 14% | 6% |
| p95 to first useful message | Not measured | Not measured | 21 s | 8.7 s |
| Duplicate refunds, per quarter | 5 | 0 | 0 | 0 |
The cost row has five stops rather than four, and the missing one matters. Ordering the context stable-to-volatile so the prefix could be cached took 41 cents to 14 in Chapter 5 — the largest single move in the book, and a layout change. Routing by task type took 14 to 11 in Chapter 7. The triage split in Chapter 10 then put 11 back up to 12, deliberately, buying handle time with money and saying so. Chapter 13 started from twelve cents and took six off, the hard six, because everything obvious had already been done. Not one cent came from a cheaper model.
Two cells in the first column deserve a footnote. The forty-line agent had three tools and no escalate_to_human, so nothing reached a person by design — the 39% it failed arrived at the support desk days later as second contacts from annoyed buyers, which is the most expensive possible way to discover a failure. And nobody measured p95 for the first two columns because Chapter 1's loop had no logging worth reading. The instrument arrives late in this book because it arrived late at Sundry, and that is one of the three things Vera would change.
What Moved the Numbers Most
Ranked by measured effect rather than by how much work each one felt like. Every row was measured on its own against the same replayed traffic, because a bundle that works tells you nothing about which half of it worked.
| Change | Built in | What it moved, measured |
|---|---|---|
| Prefix caching and stable context ordering | Chapter 5 | $0.41 to $0.14 a ticket, at an 86% cache hit rate |
| Retrieval with provenance, scope and effective dates | Chapter 6 | Policy slice 63% to 81%, six tickets, and the drift closed |
| Shaping tool results at the boundary | Chapters 3 and 13 | 2.2 cents a ticket, the largest single lever in Chapter 13 |
| The eval set itself | Chapter 9 | Nothing on the day it shipped, and every decision after it |
The fourth row is the one worth arguing about, because it moved no metric and still belongs at the top of the list. Before the 120 graded tickets existed, improvements were argued in review and shipped on conviction. After they existed, a change either moved a number by more than the noise floor or it did not ship — and the set's own limit is part of the discipline, because 120 cases can detect roughly a seven-point change and cannot see a two-point one. The eval set did not make the agent better. It made it possible to know whether anything else had.
Which brings up the two changes that moved nothing, and they were the two the team was proudest of. The first was the self-review pass: after producing a refund decision, ask the model to check its own work. On a replay of 240 production decisions containing 70 real policy errors, it endorsed 229 and found 6. The second was a planner agent above the returns flow, which moved nothing on the eval set, added a model call and 0.9 seconds, and gave the team a second place to look when a run went sideways. Both were deleted, and both failed the same way: they added model calls to make the agent think harder, and lost to changes that gave it better material or a smaller job.
The Three Incidents
Three entries in the first quarter, each one a wound this book opens in one chapter and closes in another. Costs are what actually left the business, not estimates of reputational damage.
| Incident | What it cost | How it surfaced | The control that closed it |
|---|---|---|---|
| The double refund (Ch3, generalized in Ch8) | $610 across 5 tickets in 11 days, 4 sellers | Finance's weekly seller-balance reconciliation | An idempotency key derived from the intent, plus an intent row written before every side effect |
| The drift (opened Ch5, closed Ch6) | About $3,100 of returns accepted outside a seller's window, over six weeks | A marketplace seller disputing three return authorizations in one week | Retrieval carrying scope and effective dates, and a citation requirement on every policy claim |
| The injection (Ch12) | $186 across 3 tickets in 11 days | A support lead spot-reading resolved tickets | A provenance label on every seller-written field, a dispatcher that refuses a consequential action grounded only in untrusted text, and an isolated read-only pass for seller prose. The ceiling bounded the loss; it did not close the class |
Read down the third column, because that is the finding. Not one of the three was caught by an alert. Each was found by a person outside the agent's own monitoring, on a weekly cadence or by accident, days to weeks after it started. Each of the runs terminated cleanly, produced a fluent reply, and was graded a success by whatever was measuring at the time — which is Chapter 8's uncomfortable number arriving in the real world, where 74% of Sundry's classified failures ended with the run marked answered.
Read down the fourth column for the other half. Every control that closed an incident is ordinary code: a hash, a row committed before a write, a scope field on a retrieved passage, a provenance label the dispatcher branches on. None of them is a change to the prompt, and none of them depends on the model behaving. The incident log lives in the team's own repository, one page each, and every page names the control rather than the lesson — a lesson is a sentence anybody can nod at, and a control is a line somebody can read.
What Still Escalates
Six per cent of tickets reach a human, and that is one queue rather than two. Chapter 12 established it when the approval gate came down from 41%: approvals and escalations arrive in the same place, staffed by the same support leads, and every item there gets read. At 4,200 tickets a week that is about 250 items. Roughly 2.6 points of the six are the agent handing over on purpose — a seller who has stopped answering, a parcel two carriers both deny holding, a request Sundry has no process for — and the remaining 3.4 points are held at Chapter 12's approval gate: 1.7 where the policy is silent or two supplements disagree, 1.1 of refunds above the $150 ceiling, and 0.6 the agent flagged as uncertain on its own. The gate's fourth category, an action justified only by untrusted content, is in the queue and too small to show at this precision: since the provenance check went in it has produced a handful of items a month, each one a refund that would otherwise have gone out.
That shape is the one you want. An agent that never escalates is not better than one that does, only less honest, and the difference shows up as wrong answers instead of as queue items. The gap between 88% resolution and 6% escalation is worth stating out loud too: of the twelve points the eval set marks unresolved, about six are runs that ended in a handover — the right outcome, still counted as unresolved because the agent did not settle the problem — and the other six are answers that failed to settle it and came back as a second contact. Counting only the escalation queue undercounts failure by half.
What Vera Would Do Differently
Three things, in the order they would have paid off. Build the eval set first, before the second improvement rather than in month four. Everything shipped in months two and three was argued rather than measured, and two of those changes turned out to do nothing — which nobody could have known, because the instrument that would have said so did not exist yet. Two people spent three days building the set; it should have been the first three days.
Ship the ceiling, the idempotency key and the provenance check before any of the quality work. All three are small — an integer comparison, a hash of the intent, a label the dispatcher branches on. The ceiling was there from Chapter 3 and bounded what the injection could cost; the other two closed wounds that cost real money in a quarter when the team was busy improving resolution. And resist the multi-agent split for another quarter: it bought handle time, 34 seconds to 19, at 11% more cost, but the single agent still held all of Chapter 13's cost and latency headroom, and that work is cheaper done once on one agent than three times on three.
The Honest Ceiling
Eighty-eight per cent is not 100%, and the remaining 12% is not an engineering backlog waiting for a better model. That 12% is a mix of cases where somebody's judgement is required, cases where the policy does not exist yet and someone at Sundry has to decide what it should be, and cases where the agent is correct to be uncertain. Every point above 88% has cost more than the point before it — the last two points took a quarter — and there is no reading of the data that suggests the curve turns back up.
So the product is designed around the ceiling rather than in denial of it. The support page says a person will pick it up, the handoff carries a summary the colleague can act on, and staffing is planned around 6% rather than around a promise. And the three headline numbers are always reported together: resolution, cost per resolution and escalation rate. Any one alone can be manufactured in an afternoon — refuse to escalate and resolution rises, escalate everything and cost falls — which is why a case study quoting one of them has told you nothing.
- Reading these numbers as targets — one queue, one company, one policy library, and a team starting at 61% in a different domain has learned nothing about its own ceiling.
- Attributing the improvement to the model — the pinned model version did not change across the arc, and every move in the table came from context, retrieval, tools, limits or controls.
- Treating the escalation rate as the failure metric — an agent that never escalates is not better, it is less honest, and its failures arrive as wrong answers rather than as queue items.
- Reporting resolution without cost and escalation — each moves on its own by making another worse, so a single number is a claim nobody can check.
- Report resolution, cost per resolution and escalation rate together, always, and next to the eval set's own detection limit.
- Attribute each improvement to the change that produced it, measured on its own against replayed traffic rather than shipped as a bundle.
- Keep an incident log the whole team can read, one page each, naming the control that closed it rather than the lesson it taught.
- Design the product around the ceiling — staff the handoff, write it into the support copy, and stop promising the last twelve points away.
Knowledge Check
Which change contributed most to Sundry's cost falling from $0.41 to $0.06 a ticket?
- Ordering the context so the prefix could be cached, which took the figure from 41 cents to 14
- Moving the whole queue to a smaller and cheaper model once resolution was high enough
- Splitting the agent into triage plus specialists, which cut the tokens each run had to send
- Enforcing the twelve-turn limit, which stopped the expensive runs before they could accumulate
What did Sundry's three production incidents have in common?
- All three began with content written by somebody outside Sundry and pulled into the context
- All three followed a model upgrade, and each was resolved by pinning the previous version back
- No alert fired for any of them, and each was found by a person outside the agent's monitoring
- Each was closed by a rewritten system prompt once the failing behaviour had been reproduced
Sundry's agent escalates 6% of tickets. Why does the team treat that as a design target rather than a defect rate?
- Because escalations are cheap enough that reducing them further would not repay the engineering
- Because the support leads can read every item at that rate, which keeps the approval queue honest
- Because 6% matches the headcount the support team already had, so nobody had to be redeployed
- Because driving it to zero converts queue items into wrong answers nobody has flagged
A team reports "our agent resolves 91% of tickets". What is missing, and why does it matter?
- The size of the eval set, since a small graded set cannot detect a change worth acting on
- Cost per resolution and escalation rate, because each number can be moved by worsening another
- The model version and prompt version, without which the figure cannot be reproduced later
- The judge's agreement rate with human labels, which sets the error bar on any graded score
You got correct