Topic 52

Evals in CI

Pipelines

An eval suite that runs when somebody remembers is a document. The version that changes behaviour runs deterministic checks on every push, the graded set on every merge to main, and blocks the merge on an invariant violation or a regression past a stated threshold. Nothing in this topic is new machinery — it is the same three tiers wired to triggers so that skipping them takes deliberate effort.

The difference is not process hygiene. Two of the eighteen regressions Sundry's suite caught in its first quarter were ones no reviewer would have caught by reading a diff, because neither of them was a change to application code. One was a dependency upgrade and the other was a scheduled index rebuild, and both went out on a Tuesday afternoon with nobody feeling nervous about anything.

Three triggers — and prompts, schemas, rubrics and model pins are files, so every one of them is a diff

Weekly · twenty runs, by hand

Ninety minutes of one person against the same rubric. Nothing is gated on it; its output is the calibration set the next judge re-validation compares against, which is why it cannot be dropped when the week gets busy.

Merge to main, and nightly

All 120 cases, both rubrics, the five-number report — eleven minutes at eight-way concurrency and about $16. Blocks four points below main, outside the three-point noise band, so the gate survives its first fortnight. Cost, trajectory and p95 are printed for a human to argue with rather than enforced.

Every push

All the invariants plus the regression file over a 12-case smoke subset: 90 seconds and about $2, cheap enough that nobody batches commits to avoid it. This tier has no band at all — one refund above the ceiling is a defect whether it happens once or forty times.

Three Tiers, Three Triggers

Each tier costs an order of magnitude more than the one below it and sees something the one below it cannot, so each gets a trigger matched to its price.

Tier	Trigger	What runs	Wall clock	Cost
Assertions	Every push	All invariants plus the regression file, over a 12-case smoke subset	90 seconds	About $2
Graded set	Merge to main, and nightly	All 120 cases, both judge rubrics, the five-number report	11 minutes	About $16
Human sample	Weekly	20 runs graded by hand against the same rubric	90 minutes of a person	Irreplaceable

The eleven minutes on the middle tier is eight-way concurrency against the provider, not 120 runs in sequence, and it is worth engineering early: a graded pass that takes an hour gets run on Fridays, and a gate people wait an hour for is a gate somebody will find a way around.

What Counts as a Change

The trigger list is where most pipelines are wrong, because the highest-risk changes to an agent are not application code. A prompt edit changes behaviour on every ticket. A tool description edit moved Sundry's first-call selection by 22 points in Chapter 3 with no code touched at all. A model version change replaces the component that makes every decision. A retrieval index rebuild changes what the agent knows. An MCP server upgrade changes what its tools return and what their descriptions say. So does a rubric edit, which changes the measurement rather than the agent and must be re-validated before it grades anything.

Each of those has the same failure shape: no diff in the service repository, no pipeline run, no eval. The fix is unglamorous — put the prompts, the tool schemas, the rubrics and the pinned model and server versions in the repository as files, and make the index build publish a version identifier that CI keys on. Once every one of them is a diff, the existing trigger covers all of them and nobody has to remember a special case at four in the afternoon.

Handling Noise

A gate that fires on random variation gets disabled within a fortnight, so the threshold has to sit outside the measured noise band rather than at zero. Sundry measured three points of spread on an unchanged agent in Topic 47, so the blocking rule is four points below main's recorded outcome — anything smaller is reported, charted and not blocked.

The gate, in the four rules it actually has

TIERS = {
  "push":    {"cases": SMOKE_12, "judge": False},
  "merge":   {"cases": ALL_120,  "judge": True},
  "nightly": {"cases": ALL_120,  "judge": True}}

def gate(report, main):
    if report.invariant_violations: return "block"   # no band, ever
    if report.regression_failures: return "block"
    if report.outcome < main.outcome - 4: return "block" # noise band is 3
    return "pass"   # trajectory, cost and p95 are reported, not gated

Read the asymmetry in those four rules. Invariants and regression cases have no band at all, because they are not measurements — one refund above the ceiling is a defect whether it happens once or forty times. The graded outcome has a band, because it is an estimate. And the last three of the five numbers are printed on the pull request for a human to argue with rather than enforced, since a cost increase can be the right call and a machine cannot tell which week that is.

Publish the interval with the number, not just the number. "Outcome 86%, band ±3" tells a reviewer what a two-point difference means without them having to remember. Where the provider offers a seed, pin it and pin the temperature — it narrows the spread and does not close it, so measuring your own band is still the only way to know what it is.

Cost Control in CI

Sixteen dollars a merge is affordable at fifteen merges a month and ruinous at forty merges a day. Sundry runs the full set on every merge because its rate is low; a team merging constantly runs a stratified 30-case smoke subset on merges and the full 120 nightly, which is the usual compromise and costs about a fifth as much.

Whichever you pick, label every number with the set it came from. A 30-case subset has a much wider noise band than the full 120 — roughly twice as wide — and quoting a subset figure in a summary next to full-set figures from last month is the same error as comparing eval numbers to production numbers. Sundry's report prints the case count in the header for exactly this reason.

The whole pipeline costs about $960 a month: roughly $240 on push runs, $240 on merges and $480 on the nightly full pass. That is about what two weeks of the production queue costs to run at $0.11 a ticket, and it is the cheapest line item in the chapter — the eval set itself took two people three days, and the incident it prevented in March took one person a week.

Acting on a Regression

A blocked merge has to arrive as evidence, not as a red cross. Sundry's pipeline posts the failing cases with their trace links attached, the reason-code distribution from the judge, and the diff of which cases changed verdict since main — that last one is what turns "resolution fell 6 points" into "these seven tickets, all marketplace returns, all citing the wrong document". Debugging starts at the traces rather than at a guess about the prompt.

Overrides exist and are recorded. The rule is a named owner and a written reason on the merge, and the count is reviewed monthly. Sundry overrode three times in its first quarter, and two of those were the check being wrong rather than the change — which is exactly the information the review is for. An override with no record is how a gate degrades into a formality that everybody has stopped reading.

What the Pipeline Caught

The first regression was an MCP server upgrade. A minor version of the order server renamed a field in the get_order result, the agent started quoting the pre-discount total to buyers, and outcome fell nine points on the merge run. No invariant fired — the amounts were real amounts, on the right order, under the ceiling. It was caught because a dependency bump triggers the graded tier, which is the rule from Chapter 4 written into a pipeline.

The second is the more instructive one. A scheduled index rebuild picked up a new chunker configuration that cut the refund matrix mid-row again, exactly as Chapter 6 described. The headline outcome moved two points — inside the noise band, no block, nothing to see. The policy slice inside the set fell from the 26 correct it has held since Chapter 6 closed the drift to 21, which is a 16-point drop on the 32 cases where policy decides. It was caught because the report breaks out the per-class slices, and it would have shipped if the pipeline had printed one number.

Common Mistakes

Running the full graded set on every push — sixteen dollars and eleven minutes per commit is a real bill and a real wait, and developers respond by batching commits or bypassing the pipeline.
Setting the blocking threshold at zero regression — random variation blocks correct merges, the team learns that red means nothing, and the gate is switched off inside a fortnight.
Not triggering on prompt, schema, index, rubric or model-version changes — those are the highest-risk changes an agent has, and none of them is application code, so the default triggers miss all of them.
Overriding a block without a named owner and a written reason — the count cannot be reviewed, the two overrides that were the check's fault never get fixed, and the gate becomes ceremony.

Best Practices

Run three tiers on three triggers, with the cheap tier on every push blocking unconditionally on invariants and regression cases.
Put prompts, tool schemas, rubrics, index versions and model pins in the repository so every one of them is a diff that triggers the suite.
Set the blocking threshold outside your own measured noise band, publish the interval next to the number, and label every figure with the case count that produced it.
Post failing cases with traces and per-class slices rather than a single score, and require a named override with a written reason that gets reviewed monthly.

Comparable toolsGitHub Actions the triggers and the gateBraintrust eval runs with CI integrationLangSmith dataset runs per commitPromptfoo a suite that runs in a pipelineGit, GitHub & GitHub Actions the pipeline mechanics in depth

Knowledge Check

Which of these changes must trigger the graded eval suite, even though none of it is application code?

A change to the dashboard that charts resolution rate for the support team's weekly review
A prompt edit, a tool description edit, an index rebuild, a model pin or an MCP server upgrade
A change to the log formatter that writes trace records into the observability pipeline
An edit to the eval fixtures, which is the only non-code change capable of moving the score

Why is the blocking threshold four points below main rather than zero regression?

Four points is the judge's measured disagreement rate, so a smaller drop cannot be graded reliably
The unchanged agent already moves three points, so a zero-tolerance gate blocks correct changes
A tighter threshold would force re-runs, and each additional graded pass costs sixteen dollars
Provider rate limits cap how often the suite can run, so the gate must tolerate skipped runs

An index rebuild moved the headline outcome by two points and dropped the 32-case policy slice from 26 correct to 21. Why did the pipeline catch it?

The headline drop crossed the four-point blocking threshold and stopped the merge automatically
A citation invariant fired, because the truncated chunks no longer contained a valid clause reference
The report breaks out per-class slices, so a 16-point drop on policy cases was visible
The nightly retrieval eval flagged it before the graded set ran, which is what index changes trigger

A merge is blocked by the graded tier. What should the pipeline hand the engineer?

The outcome score and the threshold it fell below, so the size of the regression is clear
The failing cases with traces, reason codes, and which cases changed verdict since main
An automatic re-run of the set, since a single failing pass may be run-to-run variation
A one-click override with the reason filled in later, so the branch is not blocked while people investigate

You got correct