Meet Sundry
Sundry is an online marketplace for home and outdoor goods — furniture, kitchen, tools, garden. Some of the stock sits in Sundry's own warehouses; the rest belongs to roughly 2,000 third-party sellers listed alongside it. Support gets 4,200 tickets a week, written by people who neither know nor care which of those two shipped their bookshelf.
Every example in this book comes from that queue. This page sets up the company, the tools, the policy library and the definition of "resolved" that everything later is measured against — including the eval rubric in Chapter 9 and the numbers reported in Chapter 14. The marketplace structure is the reason a cracked bookshelf cannot be settled by one lookup.
The Business
Sundry owns the customer relationship and the sellers own the stock. The buyer sees one site, one checkout and one support address, and has no reason to think about the difference. Behind that, an order line is either Sundry's own or a marketplace seller's, and almost everything the support agent must decide branches on which.
The return window is the clearest case. Sundry's own policy gives 30 days from delivery. A marketplace seller may run the statutory 14-day withdrawal right and nothing more, which is legal and which the buyer has never read. The money differs too: a refund on Sundry stock comes out of Sundry's account, while a refund on a marketplace order comes out of that seller's balance — and pushing a small seller's balance negative is a conversation with the finance team, not a database update.
The Queue
4,200 tickets a week arrive as free prose by email and web form. Roughly 55% are order status and delivery, 25% are returns and refunds, 12% are product questions, and 8% are everything else — a category that contains both the trivial and the genuinely strange. Nothing arrives as a form with fields. People write paragraphs, mention two problems, and leave out the order number.
One ticket comes back in almost every chapter of this book, so it is worth reading properly now: "Hi — the shelving unit turned up Tuesday with a cracked side panel, and I've been charged twice for delivery. I don't want a replacement, I just want it gone. Can you sort it out?"
Three intents in one paragraph. Damage in transit, a billing error, and an explicit refusal of the remedy the damage policy would normally offer. One of the three costs money. And the answer depends on a seller Sundry does not control, whose supplement may set a shorter window than the buyer expects. That is a Tuesday ticket, not a contrived one — which is precisely why it is the example.
search_orders, get_order, track_parcel, search_policy. Free to be wrong: a bad lookup produces a bad sentence, and a person can correct a sentence.start_return books a pickup and offer_replacement reserves stock, both reversible. message_seller cannot be recalled at all. issue_refund moves money out of a seller's balance, and stops at $150 without a human.escalate_to_human hands the ticket to a person with a summary. It is the only terminal tool, and the line between the first two columns is where the permission model is built.The Nine Tools
Vera's agent gets nine functions and no others. Four of them only read. Four of them change something in the world. The ninth hands the ticket to a person. That split is the most important line in the whole design, and it is the line Chapter 12 builds its permission model along.
| Tool | What it does | Undoing it |
|---|---|---|
search_orders | Finds orders by email, order id or date range | Nothing to undo |
get_order | Full detail: seller, items, charges, and the seller-written product description | Nothing to undo |
track_parcel | Calls the carrier's API — slow, and occasionally wrong | Nothing to undo |
search_policy | Retrieval over the policy library | Nothing to undo |
start_return | Generates a label and books a pickup | Cancel the pickup, apologize |
offer_replacement | Reserves stock against the order | Release the reservation |
issue_refund | Moves money; capped at $150 without human approval | A phone call and a negative balance |
message_seller | Sends text to a third party outside Sundry | Nothing — it has been read |
escalate_to_human | Hands the ticket to a person with a summary | The escape hatch, not a failure |
Read down the third column rather than the first. The four read-only tools are free to be wrong: a bad lookup produces a bad sentence, and a person can correct a sentence. The four that act are not symmetric with each other — releasing a stock reservation is a database write, while message_seller puts words in front of a business partner and cannot be recalled at all, and issue_refund moves money out of somebody's balance.
The refund tool takes an order id, an amount in cents, and an idempotency key. That third parameter exists for a reason the first version of the agent does not respect, and Chapter 3 opens with the day it mattered: the call timed out, the loop retried it, and a buyer was paid twice out of a seller's balance.
The Policy Library
Five things live in the library. Sundry's own 30-day return policy. The statutory 14-day withdrawal right. A damage-in-transit procedure that decides between replacement, refund and a carrier claim. A refund matrix that maps marketplace-versus-own-stock against the reason for the return. And roughly 2,000 per-seller supplements, each a page or two written by somebody at a furniture company in 2023.
Nobody at Sundry has read it end to end, which is exactly why the agent cannot have it hardcoded. A prompt containing the policy is a fork of a document the business is still editing, and the fork goes stale the first time legal changes a sentence. It also contradicts itself in places — a seller supplement that promises more than the damage procedure allows is not a bug to hide from the reader, it is Tuesday. Chapter 6 is about retrieving the right passage out of that library; two documents matching one query is the normal case, not the edge case.
What "Resolved" Means
Four conditions, all of them, on the same ticket. The customer's problem is settled. The money is right. The policy was followed. And no human touched it. Drop any one and the number stops meaning anything: a full refund on every ticket satisfies the first and fails the second, and a perfect policy citation that leaves the buyer still holding a cracked shelf fails the first.
That four-part definition is not a slogan. It becomes the grading rubric in Chapter 9, applied to 120 real tickets by graders who need to agree with each other, and it is the number reported in Chapter 14 when the book adds up what was actually built. Write your own version before the first prompt, in terms somebody can grade a transcript against — if two reasonable people read the same transcript and disagree about whether it was resolved, the definition is not finished.
Where Vera Starts
Vera is the engineer on this. The queue is growing faster than headcount, average first response time is eleven hours, and the mandate she was handed is "put AI on the queue" with no design attached to it. She has the nine tools because the internal APIs behind them already exist — this is a marketplace with a decade of software behind it, not a startup with a blank repository.
Her first move is measuring the queue she already has, not writing a prompt: volume by type, handle time, escalation rate, and how often a ticket comes back a second time. Without those, Chapter 9 has nothing to compare against and every later improvement is an opinion. With them, the forty lines on the next page have a baseline to beat.
- Defining success as "the customer was happy" — a full refund makes everybody happy and loses money on every ticket, so the definition has to carry the policy and the money as well as the sentiment.
- Treating marketplace and own-stock orders as the same case — the return window, the refund source and the seller's obligations all differ, and an agent that ignores the distinction is wrong on a quarter of the queue.
- Assuming the policy library is authoritative and internally consistent — it contradicts itself in places, which is a retrieval problem in Chapter 6 and a product problem for the business, and pretending otherwise ships the contradiction to customers.
- Starting from the interesting tickets — the boring majority is where the cost lives, where the regressions hide, and where a demo built on the strange 10% tells you nothing useful.
- Write the definition of "resolved" before the first prompt, in terms concrete enough that two people grading the same transcript reach the same verdict.
- Separate tools that only read from tools that change the world, both in code and in the tool list, from day one — it is free now and expensive to retrofit.
- Keep policy in the documents the business already maintains rather than in prompts engineers maintain, which is the whole reason retrieval exists in Chapter 6.
- Measure the queue before the agent touches it: volume by type, handle time, escalation rate and repeat contacts, so Chapters 9 and 13 have a real baseline rather than a remembered one.
Knowledge Check
A buyer wants to return a garden bench 20 days after delivery. Why can the agent not answer from Sundry's own 30-day policy alone?
- If a marketplace seller shipped it, their supplement may grant only the statutory 14 days
- Bulky garden items are excluded from the standard window because of their return shipping cost
- The 30-day window is measured from dispatch rather than delivery, so the deadline has passed
- Returns past the second week always need human approval before a label can be generated
The agent gives a buyer a full refund for a late delivery and a warm apology. The buyer is delighted. Is the ticket resolved?
- No — the definition requires the money to be right and the policy to be followed, not only the customer to be satisfied
- Yes — the customer's problem is settled and no human being was involved, which is exactly what the resolution number measures
- No — every refund of any size has to be reviewed by a person before the ticket is allowed to count as resolved
- Yes — provided that the amount stayed under the $150 ceiling that the agent is allowed to approve on its own
Which of the nine tools is hardest to undo once it has run?
- message_seller, because the words have already reached a business partner outside Sundry
- offer_replacement, because reserving stock removes it from the pool other buyers can see
- start_return, because a courier pickup has been booked against a real address and date
- track_parcel, because the carrier's API is slow and its last scan is sometimes wrong
Why does Sundry's policy stay in the document library instead of being written into the agent's system prompt?
- A prompt copy forks a document the business keeps editing, and 2,000 seller supplements do not fit in a context window
- Models read legal language poorly, so policy text has to be summarized before it can be used at all
- Policy documents contain confidential seller terms that must never be placed in a model's context
- Prompt caching cannot cover a system prompt that includes long policy text, so every single turn of every run costs more
You got correct