Chapter Thirteen · Production

Production

The agent is live on the whole queue: 4,200 tickets a week, Vera on the pager, and a component in the middle of the control flow that answers differently every time and bills for the privilege. Six topics on tracing, logging, cost, latency, rollout and versioning — the familiar operational disciplines, applied to a system where none of the usual assumptions hold.

6 topics

Every problem in this chapter is one that every production service has. Two things make them different here. Nothing is deterministic, so an incident cannot be reproduced by replaying the request, and the record of what happened is the only account of what happened. And every run costs money, so a design decision made in Chapter 3 about the shape of a tool result arrives, nine chapters later, as a line on an invoice with a number attached to it.

The order matters: measurement first, because everything after it depends on a breakdown that does not exist until somebody emits it. Tracing comes first and pays for itself twice — the cost breakdown and the latency breakdown are the same spans read two different ways. Then the two numbers this book has been promising since Chapter 1. Model spend goes from twelve cents a ticket to six, the unit is a resolved ticket with escalations left in the numerator where they belong, and the comparison against what a person costs is made at the fully loaded sixteen and a half cents rather than at six. Latency goes from a p95 of twelve seconds to 8.7, inside the nine the product asked for, and the largest component turns out to belong to a carrier rather than to a model.

The last two topics are about not breaking it. A rollout sequence that puts the agent in front of real tickets in an order where the first surprise is cheap, and monitoring that can see a quality regression — which looks nothing like an outage, produces no errors and will otherwise run for a fortnight. Then the four artefacts that change behaviour without a deploy: the system prompt, the tool schemas, the model version and the retrieval index. Chapter 12 closed the injection wound by bounding what a steered run can reach; this chapter is where the drift from Chapter 5 stops being a mystery, because a trace answers in two minutes what a week of reading transcripts could not.

One run as a span tree — four kinds of node, and the attributes that make each one worth keeping
Run
one ticket, end to end
outcome · cost summed over the tree · run id written onto the ticket
Turn
one pass around the loop
the same span repeats, with a larger context every time
Model call
exactly one per turn
model version · prompt version · schema hash · input tokens split cached and uncached · output tokens · latency · stop reason
Tool call
zero or more, issued together
arguments as sent · result size in tokens · latency · error · document ids on a retrieval call
Child run
a handoff to a specialist
nested subtree, not a second trace

Topics in This Chapter

Topic 70
Tracing an Agent Run
One run becomes a tree of turns, model calls and tool calls, each carrying model version, prompt version, tokens, latency and stop reason. What makes an agent trace different from every other trace: the same span repeats with a growing context, and the field you need is usually the input rather than the output.
Tracing
Topic 71
What to Log, What Never to Log
Prompts and completions are the best debugging artefact you will ever have and the most sensitive data your service holds. Redaction at ingestion, three retention windows with three written justifications, a deletion path that reaches the trace store, and an audit trail kept well away from the debug one.
Data Handling
Topic 72
The Cost Model
Twelve cents a ticket to six, with every lever measured rather than asserted, and tool results turning out to be 37% of the bill while the system prompt is 7%. The unit is a resolved ticket with escalations counted, and the honest comparison against a person includes review time and the cost of the agent's mistakes.
Cost
Topic 73
Latency
A p95 of twelve seconds against a nine-second target, decomposed by component: the carrier API is 40% of the wall clock and no model choice touches it. Streaming, parallel tool calls and removed turns are the structural fixes, and the mean is the statistic that hides every complaint.
Latency
Topic 74
Rollout
Shadow mode until the comparison log stops surprising you, then a canary by ticket class rather than by random percentage, against graduation criteria written before launch. A kill switch a support lead flips at two in the morning, and alerts on ratios because a quality regression produces no errors at all.
Rollout
Topic 75
Prompt and Model Versioning
Four artefacts change behaviour with no application code involved: the system prompt, the tool schemas, the model version and the retrieval index. Each is versioned, recorded in every run, revertible on its own without a deploy, and gated by the eval suite — and a moving model alias is how a weekend regression arrives with no commit to blame.
Versioning