Rollout

An agent goes live on 4,200 tickets a week without a controlled rollout exactly once. The sequence that works is shadow, then canary by ticket class, then expansion against criteria written before launch, with a kill switch a support lead can use at two in the morning without a deploy and without calling an engineer.

None of that is unusual for a backend service, and most of it is the progressive-delivery practice any team already has. Two things about an agent make the standard playbook fit badly, and both need saying up front: the failure that matters is a quality regression rather than an outage, so it produces no errors at all, and a random percentage of traffic is not a safe slice when the traffic contains money.

Shadowoutput goes to a comparison log, never to a customer

→

Until it stops surprising youthe exit condition, and it is not a number

→

Canary by classorder status first · 55% of the queue · no money attached

→

10% → 50% → all of itinside one class, then the next class opens

→

CRITERIA WRITTEN FIRSTFull traffic14 days · 500 tickets in the class · zero invariant violations

One flag at dispatchno deploy, no pipeline, no engineer

→

40 SECONDS TO NO NEW RUNSEverything to the human queuea support lead flips it at two in the morning

Shadow Mode

In shadow mode the agent runs on real tickets and its output goes nowhere except a comparison log. Every run records what the agent would have done — the tools it would have called, the remedy it would have offered, the money it would have moved — alongside what the human handling that ticket actually did. Nothing reaches a customer, so the first surprises are free, and they arrive on the real distribution rather than on the 120 cases somebody curated.

Sundry ran three weeks and about 12,600 tickets. Agreement with the human decision was 71% in week one and 84% in week three, and the fixes in between came from reading disagreements rather than from ideas. The exit condition is not a threshold, it is a derivative: run shadow mode until the comparison log stops surprising you. When a week of disagreements contains nothing you had not already seen, the eval set has been validated against reality and the next stage is worth starting.

Canary by Ticket Type

Start with the class where being wrong is cheap and reversible. At Sundry that is order status and delivery — 55% of the queue, no money attached, and the worst outcome is an apology and a corrected message. Returns came second, because a booked pickup can be cancelled. Refunds under the $150 ceiling came third, and refunds above it stayed with the approval gate from Chapter 12 throughout.

A random 5% is the intuitive alternative and it is the wrong shape. A percentage sampled across the whole queue includes refunds on day one, so the first novel failure happens on a ticket that moves money, in front of a customer, before anybody has watched the agent handle a hundred easy ones. Slice by consequence, not by hash. The percentage then rises inside a class rather than across the queue: 10% of order status, then 50%, then all of it, and only afterwards does the next class open.

Graduation Criteria Written First

Write the numbers down before launch, because afterwards they get negotiated. The team that has spent four months building the agent is the worst possible judge of whether a resolution rate four points below the support team's own is close enough, and the criteria exist to take that judgement away from them at the moment it becomes expensive.

Criterion for expanding a class	Threshold
Duration and volume	14 days and at least 500 tickets in that class
Resolution rate on the class	Within 3 points of the support team's own rate
Escalation rate	10% or below
Invariant violations	Zero, with no band and no exceptions
Cost per resolution	At or below the figure in the business case, and flat across the fortnight
Complaints and reopened tickets	No worse than the class baseline

Two of those rows do more work than they look. The volume floor stops a class graduating on a quiet fortnight where thirty tickets all happened to be easy. And measuring resolution against the support team's own rate on the same class, rather than against a fixed number, is what keeps the comparison fair — the returns class is harder for everybody, and holding the agent to a queue-wide average would fail it for being assigned difficult work.

The Kill Switch

One flag, read at dispatch on every ticket, that routes everything to the human queue. No deploy, no pipeline, no engineer: a support lead flips it from a page they already have open. In-flight runs finish the turn they are on and then hand off with their state intact, which is Chapter 8's escalation path doing ordinary work. Sundry measured the time from flip to no new agent runs at 40 seconds, and that measurement is the point — an untested kill switch is a belief.

It ships first, before the agent handles a single real ticket, and it is tested monthly in production during a quiet hour. The runbook names three signals that justify flipping it without asking anybody: an invariant violation seen in the wild, a resolution rate that has fallen below the graduation floor for two consecutive hours, and any incident where the agent is contacting customers about something the team cannot explain. Everything else can wait for a person to wake up.

Monitoring That Would Catch a Bad Day

A quality regression looks nothing like an outage. Error rate flat, latency flat, no alerts, and an agent confidently resolving tickets wrongly at full speed. Sundry saw exactly that shape over one weekend: resolution fell from 88% to 82% and cost rose 18% with zero errors and no deploy on either side of it — the provider had moved the alias the config pointed at, which the next topic is about. It was caught in nine hours by the resolution-rate alert. Nothing in a conventional monitoring stack would have caught it at all.

So the alerts are on ratios, with windows long enough to be stable and short enough to matter. Resolution rate over six hours against the seven-day median. Escalation rate, on the same basis. Approval rate — the share of proposed refunds a lead approves, which sat at 94% and pages below 88%, because a falling approval rate is the agent's judgement degrading while every other number holds. Cost per resolution, from Topic 72. And invariant violations, which have no threshold: one is an incident.

What Sundry's Rollout Found

Two things came out of shadow mode that 120 eval cases had not, and between them they are the argument for shadow mode in a paragraph. The first was format: about 9% of real tickets arrive as forwarded email chains with the entire thread quoted underneath, including replies a human agent sent weeks earlier. The eval set was built from cleaned ticket bodies, so no case had 4,000 tokens of quoted history in it — and on one shadow run the agent read a goodwill gesture a colleague had made in March and repeated it as though it were policy.

The second was correlation. A seller's consignment went missing in the second shadow week and 300 near-identical tickets arrived in one afternoon. Every proposed reply was individually correct and the right action was none of them: one message to the seller, one status banner, and a bulk update. A per-ticket eval set cannot represent that, because the defect is not in any ticket. It also produced the alerting rule in Topic 72 — the same shape recurred in production months later, and the ratio-based alert stayed quiet exactly as it should have, because nothing was wrong that time.