Trajectory vs Outcome

Two runs resolve the same ticket correctly. The first read the order, retrieved the seller's supplement, refunded the duplicate delivery charge and started a return: three tool calls, eight seconds, nine cents. The second called track_parcel twice on a parcel the carrier had already scanned, searched the policy library three times with three different phrasings, proposed a replacement and withdrew it, then landed on the same resolution: eleven calls, thirty-one seconds, forty cents.

Outcome grading says those two runs are identical. Both resolved the ticket, both cited the governing clause, both stayed under the ceiling. Multiply the second one across 4,200 tickets a week and it is a different product with a different bill and a customer who waited four times as long — which is why the path gets its own rubric, its own score, and its own line in the report.

Two Questions, Two Rubrics

The two questions are separable and neither is sufficient. Was the result correct — the four-part definition of resolved, graded by the outcome rubric from Topic 50. Was the path defensible — did the run take the steps the case requires, avoid the ones it forbids, and reach the answer without work that bought nothing. Sundry's trajectory rubric has four criteria: required steps taken, no unnecessary tool calls, no retrieved information contradicted or ignored, no decision reversed in front of the customer.

Report them separately and never average them. A single blended score hides the two cases you most need to distinguish: a wrong answer reached cleanly, which is usually a policy or retrieval problem, and a right answer reached badly, which is usually a cost and latency problem heading for an incident. Those go to different people and get fixed in different weeks.

What Trajectory Grading Catches

Four shapes, all invisible to outcome grading. Unnecessary tool calls — the second search that rephrases the first because nothing told the model the first one succeeded. Wrong-then-corrected decisions, where the run proposes a replacement, reads the ticket again and withdraws it; harmless in the transcript, alarming when the withdrawal happens after offer_replacement has already reserved stock. Information gathered and ignored, which is Chapter 6's third retrieval failure with a grader that can finally see it. And near-misses that resolve correctly by luck.

On one full pass of main at Sundry, 103 of the 120 runs had a correct outcome and 23 runs failed at least one trajectory criterion — and 14 of those 23 were among the 103. Fourteen runs that count as successes in the headline number and contain a defect somebody will be paged about. That gap is the whole argument for the second rubric.

Cost and Latency as First-Class Grades

Tokens and wall clock per case belong in the eval report next to correctness, measured on the same run, not looked up afterwards in a billing dashboard where they cannot be attributed to a change. Chapter 13 builds the production version of both; here they are simply two more numbers the set produces for free, because the run record already holds them.

Once they are in the report, a class of decision stops being an argument. Chapter 7 measured reasoning traces on every turn: correct tool sequence on multi-intent tickets rose from 72% to 87%, cost per ticket on the returns class went from $0.22 to $0.31, and p95 time to first useful message on that class went from 24 seconds to 28. Enabling them everywhere is not an improvement — it is a trade, priced at nine cents and four seconds a ticket, and the version that shipped enabled them only on the turn that produces the decision object. You cannot make that call from a correctness number alone.

The Lucky Correct

A run that reached the right refund without ever calling search_policy is a failure. The outcome is correct, the customer is happy, the money is right — and the decision was made from the model's prior about what retailers usually do, which happened to match this seller's supplement. Nothing in it is reproducible. It is the run that breaks the day the supplement changes, and it breaks silently, because it was never reading the document in the first place.

Sundry counted nine of these among those 103 correct outcomes. In March, one seller shortened its return window from 30 days to the statutory 14, the policy library was updated the same afternoon, and seven of those nine cases started failing while the runs that had actually retrieved the clause carried on being right. Nothing about the agent changed. The set had been reporting nine tickets as resolved on the strength of a coincidence.

So the report gets a third category rather than two. Not resolved and not failed: correct, required step skipped. It is counted separately, it never rolls into the headline as a plain pass, and its size is watched — a rising lucky-correct count is an agent learning to guess faster than it looks things up, which reads as an improvement in every number except this one.

Grading a Path Without Fixing the Sequence

The trap on the other side is asserting an expected sequence. Reading the policy before the order record and reading it after are both legitimate; so is skipping track_parcel when the order record already shows a delivery scan. A rubric that expects one ordering fails correct runs, and a suite that fails correct runs gets switched off within a month.

Grade against the case's required and forbidden sets from Topic 48 instead, plus a bound on the work. Required: a policy retrieval before any policy-dependent decision, an order lookup before any claim about the order. Forbidden: the actions this ticket rules out. Bounded: no more than twice the tool calls of the reference path for that class, which on returns is six. That is a rubric with room for a different route and no room for eleven calls.

Reporting the Pair

Five numbers make a merge decision straightforward, and Sundry's eval report prints exactly these — outcome, trajectory, cost, p95 latency, and invariant violations — for the candidate and for main, side by side, from the same set version. Cost is re-weighted to the production class proportions before it is printed, because the set over-samples the expensive classes on purpose and an unweighted figure would not be comparable with the number Chapter 13 charts.

Read as one row, that candidate merges: resolution went up and nothing blocked. Read as five, it does not go near production. The two points are not distinguishable from noise, the trajectory column says thirteen more runs are wandering, the cost is up by nearly three quarters, and the latency target the product committed to is broken. A merge decision made on the headline alone would have shipped all four of those consequences to buy a number that was never real. The candidate here is the one Chapter 7 warned against — a critique turn after every model call rather than before the irreversible ones — and not the triage split Chapter 10 ships. That split moves resolution by the same two points and is decided on handle time and latency, which it halves, against a cost rise of 11% rather than 73%.