LLM-as-Judge

For the half that assertions cannot reach — was this the right decision under the policy, was the customer's actual question answered, was the tone defensible in front of a buyer who is already annoyed — a model grades the run against a rubric. It works well enough to be the backbone of the eval suite, and it is the component of the suite most likely to lie to you.

The failure is specific rather than general. A judge does not become useless; it becomes agreeable, and the number drifts comfortably upward while the product does not move. Three pathologies produce that, each with a concrete control, and one habit keeps the whole thing honest: every score is published next to the rate at which the judge agrees with human labels. A score without its validation is a number with no error bar.

The Rubric Is the Product

"Rate this resolution 1 to 5 for helpfulness" invites the judge to average its impression of fluency, length and politeness into a number that moves whenever the prose style changes. Sundry's first version was exactly that, and it rated the run that quoted the wrong return window at 4 out of 5, because the reply was clear, warm and wrong.

What works is criterion-level scoring with explicit anchors: named criteria, each with a stated meaning for every point on its scale, written so two people would assign the same score. The version that survived has four criteria — problem settled, money correct, policy correct, escalation appropriate — each scored 0, 1 or 2, with the anchors written out in the rubric file rather than in somebody's head.

{"criterion": "policy_correct",
 "anchors": {
   "2": "names the clause that governs THIS order and the decision matches it",
   "1": "a clause is cited, but it does not govern this order",
   "0": "a policy claim with no clause, or a decision contradicting the one cited"}}

# the judge returns structured output, not prose — Chapter 2
{"case": "EV-041", "criterion": "policy_correct", "score": 1,
 "reason_code": "wrong_scope",
 "evidence": "cited POL-RET-004 (Sundry, 30 days) on a marketplace order",
 "rubric": "outcome-v4", "judge": "family-b/model-id@2026-02"}

Two things in that verdict matter more than the score. The reason code turns 120 verdicts into a distribution you can act on — thirty cases scored 1 for wrong_scope is a retrieval scoping bug with a name, while an average of 1.6 is a mood. And the evidence field forces the judge to point at the thing it judged, which makes a wrong verdict reviewable in seconds instead of requiring somebody to re-read the whole run.

The rubric is versioned with the prompt and the model, and changing it invalidates the series exactly as a model change does. Sundry's is outcome-v4 because it has been rewritten three times, and every rewrite came with a re-validation and a note in the report saying where the joint is.

Validation Against Human Labels

A judge is an instrument, and an instrument that has not been compared with a reference is a number generator. The comparison is cheap: take 40 runs spanning easy and hard cases, have two people grade them by hand under the same rubric, and measure how often the judge lands on the same score.

Sundry's numbers make the case better than an argument does. The "rate helpfulness 1-5" rubric agreed with the human labels on 62% of the 40. The anchored criterion rubric agreed on 89% — same judge model, same runs, same graders, different rubric. Most of what people attribute to judge quality is rubric quality.

Read the agreement rate as the resolution of the instrument. A judge that agrees with your rubric 70% of the time disagrees on twelve of every forty cases, which is far more movement than a five-point change on 120 cases would produce, so it cannot detect one. Report raw agreement if you like, but look at agreement on the cases where the human graders scored below full marks — on a set where 80% of runs are clean passes, a judge that says "fine" to everything already scores 80%.

The Three Pathologies

Position and verbosity bias come first. Show a judge two runs and ask which is better, and the answer depends on which one you showed first: at Sundry, presenting the same pairs in both orders changed the winner in 18% of them. Length pushes the same way — the longer reply won 64% of pairs before the anchors were written, and length is exactly what a customer does not want. The control is to score each run independently against anchors rather than comparing pairs, and, where you do compare, to randomize the order and run both directions.

Self-preference comes second and is the one that flatters you in the direction you want to believe. A judge from the same model family as the worker rated its own family's runs close to a full point higher on the rubric's eight-point total than the human graders did — the same effect Chapter 7 found when an agent reviewed its own answer and endorsed 229 of 240 decisions. The control is structural: judge with a different model family from the one doing the work, and if the product later switches worker models, check that the judge is still the outsider.

Drift on upgrade comes third and is the hardest of the three to notice. Sundry's judge was moved to a newer version of the same model with no change to the agent, and the outcome score rose three points overnight. Three points of nothing, in a report that had been comparing weeks. Pin the judge to an explicit version, treat a judge upgrade as a change that must be re-validated against the human labels before any new number is published, and re-run the previous month's cases through both versions so the joint in the series is measured rather than imagined.

Judge Cost and Cadence

Grading is a model call over a transcript, so it costs what a model call over a transcript costs. Sundry's full pass — 120 runs, graded twice because outcome and trajectory have separate rubrics — comes to about $2.60 and roughly three minutes at eight-way concurrency. That is cheap enough to run often and expensive enough that running it on every commit would triple it into an annoyance nobody funds.

So the cadence follows the cost: deterministic checks on every commit, the judge on every merge to main, a human sample of twenty runs once a week. The weekly humans are not there to grade more cases than the judge; they are the reference the judge gets measured against, and their labels are the calibration set the next re-validation uses. Topic 52 turns this into pipeline configuration.

What to Judge

Outcome and trajectory get separate rubrics and separate scores. A wrong answer reached efficiently and a right answer reached through eleven tool calls and two dead ends are different problems with different fixes, and averaging them into one number destroys exactly the information you needed. Topic 51 is about what the second rubric catches.

Give the judge the same evidence a human grader would need and no more: the ticket, the run record, the passages the run retrieved, and the case's expected outcome. Not the previous score, not which branch produced the run, not whether this is the candidate or the baseline — a judge told which side is the new one is being invited to have an opinion about it.

Reporting Honestly

Every published judge score carries its agreement rate, its rubric version and its judge version. "Outcome 86%, judge agreement 89% on the February labels, rubric outcome-v4" is a claim somebody can check. "Resolution: 86%" is a claim about an instrument nobody has described.

And do not call judge scores accuracy. They are a graded estimate produced by a model whose disagreement with human graders is measured and non-zero, which is a perfectly respectable thing to run a business on as long as everybody says so out loud. The team that starts describing them as ground truth stops asking whether the instrument still works, and the answer changes without notice — that is what a model upgrade is.