Building an Eval Set

The eval set is the most valuable artefact this team will build, and it costs less than any of them expect: two people, three days, 120 tickets lifted out of a queue that already existed. It outlives the prompt, the tool surface, the model and probably the engineer who built it. Every number reported anywhere in this book is a statement about that set.

It comes out of the ticket queue rather than out of anyone's imagination, and that single decision is what makes it worth anything. Cases written from memory test what the team already thought of. Real tickets carry the sentence nobody predicted — the buyer who changes their mind in the third paragraph, the order with two sellers on it, the person who writes "never mind, just tell me who to shout at".

Sampling From Reality

Start from the production mix and then deliberately distort it. Sundry's queue is 55% order status and delivery, 25% returns and refunds, 12% product questions and 8% everything else. Sampled in exact proportion, a 120-case set would spend 66 cases on the class a template answers correctly and 30 on the class where money moves — most of the measurement aimed at the part of the job that was never in doubt.

Stratify inside each class as well as across them. Within returns and refunds, Sundry pulled marketplace and own-stock orders in roughly equal numbers even though marketplace orders are the minority, because the whole difficulty of a refund is which of the two it is. That is also where the slices later chapters lean on came from: 32 of the 120 turn on a policy document, and 19 carry more than one intent. Neither of those would exist in a set drawn proportionally.

One consequence has to be said out loud or somebody will misreport it. A deliberately hard set produces a pessimistic number: the agent that resolves 86% of these 120 tickets does better than that on the live queue, because the live queue is mostly "where is my parcel". The eval figure and the production figure are two different measurements, and quoting one where the other belongs is how a team ends up defending a number nobody can reproduce. Report both, label both, never subtract them.

What a Case Contains

Four parts: the ticket text exactly as the customer wrote it, the account state it runs against, the expected outcome in a form a machine can compare, and a note saying what makes the case hard. The last of those has no automated use and is the reason a case is still comprehensible two years later, when the person who chose it has moved teams.

{"case":     "EV-041",
 "class":    "returns_and_refunds",
 "ticket":   "Hi — the shelving unit turned up Tuesday with a cracked side "
              "panel, and I've been charged twice for delivery…",
 "fixture":  "orders-snapshot-2026-03-14",   # frozen, never live
 "account":  {"order": "SU-88421", "seller": "Ashcombe Furniture"},
 "expected": {
   "required":   ["refund the duplicate delivery charge, 995 cents"],
   "acceptable": [["start_return"], ["issue_refund 11800"]],  # either remedy
   "forbidden":  ["offer_replacement"],       # the buyer refused one
   "policy":     "SUP-ASH-011 clause 2.1",   # the seller's window, not Sundry's
   "escalate":   False},
 "hard_because": "three intents; the remedy the damage procedure offers "
                 "first is the one the buyer explicitly refused"}

The shape of expected is the part worth stealing. It does not pin one correct transcript, because this ticket has two defensible resolutions — return the unit or refund it outright — and a set that marks one of them wrong measures conformity rather than correctness. Instead it states what must happen, what may happen in more than one legitimate way, what must not happen, and which policy document governs. That structure is also what makes the deterministic checks in Topic 49 and the trajectory grade in Topic 51 possible at all.

Grading Rules Written Before Grading

The rubric is the four-part definition of resolved from Chapter 1, turned into something two people can apply and reach the same verdict on: the customer's problem is settled, the money is right, the policy was followed, and no human was needed. Written down before a single case is graded, because a rubric written afterwards is a description of what the grader already decided.

Then measure whether the rubric works, on the cheapest possible experiment. Two people graded the same twenty runs independently. They agreed on fourteen. Six disagreements out of twenty is a rubric that cannot support a five-point claim about anything, and reading those six took an hour: four were about partial resolutions — one of three intents left unanswered — and two were about whether citing Sundry's own policy on a marketplace order counted as "policy followed". Both gaps were in the definition, not in the graders.

The rewrite added two sentences. A run that leaves any customer-visible intent unanswered is not resolved, regardless of how well it handled the others. A policy claim is followed only if the cited document is the one that governs this order. Re-graded on the same twenty runs, the two agreed on nineteen. That number gets published next to every score the set produces, and it is re-measured whenever the rubric is edited — the rubric is versioned code, not a shared understanding.

Fixtures and Isolation

Every case runs against a frozen snapshot of the order system, not against production. Order SU-88421 in the eval fixture has the same two delivery charges, the same seller, the same delivery scan and the same seller balance it had the day the case was captured, and it will still have them in three years. Cases that hit live data stop being comparable the moment somebody in operations closes an order, refunds a charge or edits a seller record — and they will, because that is their job.

Sundry learned this on a case whose expected outcome was a refund and whose live order had been settled by a human the previous week, so the correct behaviour became "explain that this is already refunded". The case failed for two months and nobody could reproduce it locally. Freeze the fixture, version it with the set, and let the retrieval index used by the eval run be a pinned snapshot too — a policy library that changes underneath the set changes the correct answers underneath it.

Including the Failures

Every incident becomes a permanent case. The double refund from Chapter 3, with the timeout in the fixture so the retry path actually executes. The nine track_parcel calls on an unscanned parcel from Chapter 8. The injection strings from Chapter 12, each one a case asserting that the instruction inside the seller's product description was not followed. These do not expire when the bug is fixed; that is the entire point of them.

Keep them in a separate regression file rather than folding them into the 120. The graded set has to stay fixed to remain comparable, and a set that grows every time production hiccups produces a resolution series in which nothing can be compared with anything. The regression file runs on the same trigger, is graded pass or fail rather than scored, and blocks unconditionally. When the 120 does eventually change — new ticket types, retired classes — cut a new version, run the current agent against both, and publish the two numbers side by side so the series has a documented joint rather than a silent step.

Maintenance

A set decays. Sundry retired four cases in the first year: two referred to a returns flow that no longer exists, one was a duplicate of another in everything but wording, and one turned out to have been graded wrong at capture time and had been quietly punishing correct behaviour for months. Retiring is fine. It is written in the version history, the replacement is drawn from the same class, and the count is kept at 120 so the arithmetic stays stable.

One rule has no exceptions. Never edit an expected outcome because a change failed against it. The temptation arrives phrased reasonably — "the agent's answer is arguably better than what we wrote down" — and sometimes that is even true. The correct route is to argue it in review with a second person, change the case as a documented amendment, and re-run everything since the last version. The shortcut turns the set from a measurement into a mirror, and a mirror agrees with every change you ship.