Eleven improvements since the forty-line agent, and the only number anybody can point at is the 61% recorded before any of them. Six topics on building the instrument: why an agent cannot be unit-tested, where 120 real tickets come from and what a case must contain, the assertions that catch two thirds of regressions for nothing, a model judge with its three pathologies and their controls, why the path matters as much as the answer, and the pipeline that runs all of it on every change.
6 topics
Vera has shipped eleven changes since the agent was forty lines long — idempotency keys, a rewritten tool description, an MCP server in front of the order system, prompt caching, a context budget with compaction, retrieval over the policy library, subtask state, routing by task type, a turn limit, repetition guards, compensation on partial failure — and she cannot say whether the agent is better. The transcripts read well. The support team's view is that it feels better. Chapter 8 finished by pointing at the largest failure class in the system, judgement, at 44% of 400 classified failures, and noting that nothing built so far can see it.
This chapter builds the thing that can. A hundred and twenty real tickets pulled from the queue, stratified with the hard classes deliberately over-sampled, each with the account state it runs against and an expected outcome written down before anybody graded anything — the four-part definition of resolved from Chapter 1, turned into a rubric two people can apply and reach the same verdict on. Invariants asserted in code. A model judge for the judgement half, validated against human labels and reported with its agreement rate. Outcome and trajectory scored separately, because a right answer reached through eleven tool calls and forty cents is a different problem from a wrong one. And the whole suite wired to three triggers so that skipping it takes effort.
The agent ends this chapter at 86% on that set, up from 61%, and the 32 policy tickets inside it are where the drift from Chapter 5 is finally shown closed and kept closed. The more useful half of the result sits underneath that headline: fourteen of the runs it counts as successes carry a defect in the path, and nine of those reached the right answer without reading the policy at all. The last thing this chapter hands over is a limit rather than a number: 120 cases can detect roughly a seven-point change and cannot detect a two-point one, so a two-point gain reported from this set is noise — which is exactly the situation Chapter 10 walks into when its multi-agent split moves resolution by two points and has to make its case on handle time instead.
Three instruments, and the cheapest one is the only one with no error bar
Human sample · twenty runs a week
The only tier that can notice the rubric is asking the wrong question, because everything below scores against the rubric it was handed. Ninety minutes of one person, covering twenty of the 120 — too slow to gate a merge and too valuable to skip.
Model judge · on every merge
Sees the judgement half: whether the decision was right and the question was answered. That is 44% of the 400 classified failures and invisible to the layer beneath. About $2.60 a pass, and it carries its own error — 89% agreement with the human labels, published next to every score it produces.
Deterministic assertions · on every commit
Reads a run record that was written anyway, so it costs nothing, never drifts, and gives the same answer next year. Twelve of the eighteen regressions caught in a quarter, and no error bar at all — one violation across 120 runs blocks the deploy outright.