The agent is live on the whole queue: 4,200 tickets a week, Vera on the pager, and a component in the middle of the control flow that answers differently every time and bills for the privilege. Six topics on tracing, logging, cost, latency, rollout and versioning — the familiar operational disciplines, applied to a system where none of the usual assumptions hold.
6 topics
Every problem in this chapter is one that every production service has. Two things make them different here. Nothing is deterministic, so an incident cannot be reproduced by replaying the request, and the record of what happened is the only account of what happened. And every run costs money, so a design decision made in Chapter 3 about the shape of a tool result arrives, nine chapters later, as a line on an invoice with a number attached to it.
The order matters: measurement first, because everything after it depends on a breakdown that does not exist until somebody emits it. Tracing comes first and pays for itself twice — the cost breakdown and the latency breakdown are the same spans read two different ways. Then the two numbers this book has been promising since Chapter 1. Model spend goes from twelve cents a ticket to six, the unit is a resolved ticket with escalations left in the numerator where they belong, and the comparison against what a person costs is made at the fully loaded sixteen and a half cents rather than at six. Latency goes from a p95 of twelve seconds to 8.7, inside the nine the product asked for, and the largest component turns out to belong to a carrier rather than to a model.
The last two topics are about not breaking it. A rollout sequence that puts the agent in front of real tickets in an order where the first surprise is cheap, and monitoring that can see a quality regression — which looks nothing like an outage, produces no errors and will otherwise run for a fortnight. Then the four artefacts that change behaviour without a deploy: the system prompt, the tool schemas, the model version and the retrieval index. Chapter 12 closed the injection wound by bounding what a steered run can reach; this chapter is where the drift from Chapter 5 stops being a mystery, because a trace answers in two minutes what a week of reading transcripts could not.
One run as a span tree — four kinds of node, and the attributes that make each one worth keeping
Run
one ticket, end to end
outcome · cost summed over the tree · run id written onto the ticket
Turn
one pass around the loop
the same span repeats, with a larger context every time
Model call
exactly one per turn
model version · prompt version · schema hash · input tokens split cached and uncached · output tokens · latency · stop reason
Tool call
zero or more, issued together
arguments as sent · result size in tokens · latency · error · document ids on a retrieval call