You Cannot Unit-Test an Agent
The same ticket, run twice through the same agent, produces two different transcripts. Different wording, sometimes a different order of tool calls, occasionally a different resolution that is equally defensible. The output is open-ended prose, the actions taken depend on what a carrier API said at that second, and whether the result was right is a policy judgement rather than a value you can compare with an equality operator.
None of that removes the need to know whether last week's change helped. It changes what a test is. Instead of asserting that a run equals an expected string, you assert that a short list of invariants held on every run, grade a fixed set of cases, and ask whether the graded number moved by more than the set's own noise. Fixed, graded, noise — those three words carry the rest of this chapter.
Why Exact-Match Testing Fails
Three things break exact matching, and they break it independently. Sampling is stochastic, so identical input legitimately produces different output — Chapter 2 explains the mechanism and why turning the temperature down narrows it without closing it. Wording varies while the meaning does not: "I've refunded the duplicate $9.95 delivery charge and booked a collection for Thursday" and "The second delivery charge has been refunded, and a courier will collect the unit on Thursday" are the same resolution written by two people.
And the canonical ticket has more than one correct answer. Order SU-88421 arrived from Ashcombe Furniture with a cracked side panel and a delivery charge billed twice. Refunding the duplicate charge and starting a return under the damage procedure is correct. Refunding the duplicate charge and refunding the $118.00 unit outright is also correct, because the buyer explicitly refused a replacement and the seller's supplement permits it. An expected-output file has to pick one of those and mark the other wrong.
A suite built on exact matches therefore settles into one of two states, and both are worse than having no suite at all. Tight assertions break whenever somebody improves a sentence, so the team disables the ones that fail most often — which are the ones covering the hardest cases. Loose assertions pass whenever a few keywords appear, which a fluent wrong answer manages without effort. Neither state answers the question that matters: was the customer's problem settled, with the money right, under the policy that actually governs this order, without a person.
What Replaces It
Invariants come first. These are properties that must hold on every run without exception: no refund above the $150 ceiling without human approval, at most one refund per order per run, a policy reference attached to any reply that makes a policy claim, no other customer's data in the text sent to this one. They are boolean, they are checked in code against the run record, and they carry no measurement error. An invariant is either intact or it is an incident.
The graded outcome comes second: a score over a fixed set of cases, which at Sundry is 120 real tickets with their expected resolutions written down before any grading began. That number moves for several reasons at once and it carries real uncertainty, which is precisely why the set must stay fixed. A resolution figure is only interpretable against the same figure, from the same cases, measured last week.
The threshold comes third and turns the score into a decision. Not "it looks better" in a review meeting — a stated floor, published with the interval around it, applied by a pipeline rather than by whoever happens to be reading the dashboard. Topic 52 wires all three into CI. This page is about what they can and cannot tell you once they are wired.
Three Levels of Check
The three levels see different things, cost different amounts, and are trustworthy in inverse proportion to how cheap they are. Sundry runs all three, at three different frequencies, and the frequency is chosen by the cost column.
| Level | What it can see | Cost of one pass | How often |
|---|---|---|---|
| Deterministic assertions | Tool sequence, arguments, ceilings, whether a cited clause exists | Free — it reads a run record already written | Every commit |
| Model-graded rubric | Whether the decision was right and the question was answered | About $2.60 to grade 120 runs twice | Every merge |
| Human review | Everything, including what the rubric forgot to ask | About 90 minutes of one person for 20 runs | Weekly sample |
Read the last column against the second. The cheap tier runs constantly and cannot see a wrong decision. The judge runs on merges and sees judgement, with a known error rate that Topic 50 insists you publish. The humans run rarely and are the only instrument that can notice the rubric is measuring the wrong thing — a judge scores against the rubric it was handed, never against the business.
The Unit of Evaluation
The unit is a whole run against a ticket, not a single model call. Chapter 8 classified 400 consecutive failed tickets and found judgement failures at 44% of them, more than any other class: the tools were the right tools, every argument was valid, the run terminated cleanly, and the answer was wrong on the merits. Grade the calls individually and every one of those runs passes. The defect only exists at the level of the whole run, because it is the sequence of choices.
Component evals stay useful as diagnostics and are never a substitute. The 200-ticket labelled set from Chapter 3 measures first-tool selection and caught a wording bug worth 22 points. The 150-question retrieval set from Chapter 6 measures whether the governing document came back, and runs in 90 seconds without a model call. Both tell you where to look when the end number moves. Neither can tell you whether the ticket was resolved — Chapter 6's recall trap is the proof, where raising the passage count improved every retrieval metric and lowered policy-correct resolutions from 26 to 23.
Sample Size and Noise
Before trusting any number from a 120-case set, measure what the set does when nothing changes. Vera ran the unchanged agent through all 120 tickets five times in one afternoon and recorded 61%, 59%, 62%, 60% and 61%. No prompt edit, no model change, no new tool: a three-point spread produced entirely by sampling and by a carrier API that answers differently on a Tuesday.
The paired view is more useful than the spread, because the same cases run both times. Comparing two runs of the unchanged agent case by case, seven of the 120 tickets changed verdict — four from failed to resolved, three the other way, for a net movement of one ticket. Seven cases flip on their own. A change has to flip meaningfully more than seven before the flip is attributable to the change rather than to the afternoon.
That fixes the resolution of the instrument. One point on this set is 1.2 tickets. A two-point improvement is two or three tickets changing verdict, which is inside what the agent does unprompted, so it is not a result — it is the same number measured twice. Seven points is eight or nine tickets, comfortably outside the band, and a set of 120 can detect that with confidence. Everything between four and seven points is a maybe that wants a second run before anybody writes it in a summary.
Reporting a two-point gain as a win is how a team spends a quarter shipping changes that did nothing, then cannot explain why eleven improvements worth two points each did not add up to twenty-two. Chapter 10 walks straight into this: splitting the agent into triage plus specialists moves resolution by two points, so the case for shipping it has to be made on handle time and cost. Seeing a two-point effect at all needs a bigger set, or a component set where the effect is larger and the noise smaller.
What Passing Means
A pass is two conditions, not one: the graded score is at or above its threshold, and there were zero invariant violations. The score is a measurement with an error bar around it, so a small drop is a conversation. The invariant half has no error bar at all, and one violation across 120 runs blocks the deploy — a refund above the ceiling is not 0.8% of a problem, it is a refund above the ceiling.
A pass also has to say what produced it. Sundry's eval report names the set version, the prompt version, the model version, the judge model and its last measured agreement rate. Without those five, a comparison across two months is comparing two different experiments and calling the difference progress. Chapter 13 keeps the same five in every production trace for exactly the same reason.
- Writing exact-match assertions on reply text and calling it an eval suite — it fails on a rewritten sentence and passes on a wrong decision that happens to contain the expected words, so the team learns to ignore both outcomes.
- Evaluating single model calls instead of whole runs — 44% of Sundry's classified failures are judgement failures in which every individual call was correct, so a per-call suite reports a clean sweep on the largest failure class in the system.
- Reading transcripts as evaluation — human readers rate fluent wrong answers highly, and the runs that read well are exactly the ones nobody re-reads, which is how the double refund survived eleven days of daily review.
- Reporting a two-point improvement from a 120-ticket set as a result — the unchanged agent already moves three points between runs, and a team that treats that as signal will chase changes that did nothing for a quarter.
- Define the invariants first — they are cheap, deterministic, and they are the half of the suite that blocks a deploy rather than starting a discussion about it.
- Grade whole runs against a fixed set, and keep the set fixed long enough that two months of measurements are actually comparable.
- Measure your own noise band by running the unchanged agent through the set several times, state the detectable effect size that follows, and respect it in every report.
- Use human review on a rotating sample as the check on the rubric, never as the primary signal — it is too slow to gate a merge and too valuable to skip.
Knowledge Check
Why does an exact-match assertion on the agent's reply fail as an evaluation?
- String comparison is too slow to run across a set of 120 recorded runs on every single commit
- Wording varies, sampling varies, and one ticket often has more than one correct resolution
- Temperature settings cannot be pinned by the API, so a reply is never reproducible word for word
- The provider changes the output format often enough that stored expectations go stale
Sundry runs deterministic checks on every commit, the judge on every merge, and a human sample weekly. What does the human tier provide that the other two cannot?
- Higher grading accuracy per case, since a person reads the whole transcript rather than a rubric
- Notice that the rubric itself is measuring the wrong thing, which no model grading will report
- Enforcement of the refund ceiling and the other invariants that block a deploy outright
- Broader coverage of the case set, because a reviewer can work through more runs than the judge
Why is a whole run, rather than one model call, the correct unit of evaluation?
- Grading individual calls costs far more per case than grading one complete transcript does
- Per-call token counts are not exposed by the API, so only whole-run costs can be measured
- The largest failure class has correct individual calls and a wrong result, visible only across the run
- Component-level measurements are unreliable, so tool selection and retrieval sets should be retired
A change measures 63% against the 120-ticket set, up from 61%. The unchanged agent scored 61, 59, 62, 60 and 61 across five runs. What is the honest reading?
- Two points is inside the set's own noise, so the result says nothing about whether the change helped
- It is a real gain, because 63% is above every score the unchanged agent produced in five runs
- Re-running the same set twenty times would settle it, since the average converges on the true value
- A stricter judge rubric would resolve it, since tighter grading removes the variance from the estimate
You got correct