Latency

A support widget where the customer waits twelve seconds for the first word is a product problem that no eval score will ever show you. Sundry's resolution rate was 88% and its p95 to a first useful message was 12 seconds against a product target of 9, and those two numbers describe the same agent.

Agent latency is the sum of several model calls and several tool calls, so the fixes are structural rather than a faster model. Removing a turn removes a whole round trip. Issuing two lookups together removes the slower one from the total. Not calling the carrier at all removes the largest single component. Swapping the model, the change everybody reaches for first, moves one term in a sum of five.

Where the Seconds Are

Measure the breakdown before changing anything, because intuition points at the model and the answer is usually the carrier. Sundry's p95 path — a damage claim that needs an order lookup, a delivery check and a policy decision — decomposes like this, before and after this chapter's work. The spans that produce it are the tool-call and model-call nodes from Topic 70, which is the second time in this chapter that tracing turned an argument into a table.

The carrier is 40% of the wall clock, and no model decision touches it. Two changes did: track_parcel is now called only when the order record does not already show a delivery scan, which removes it entirely from about 62% of the tickets that used to make the call, and the first attempt times out at 4 seconds and falls back to the last scan Sundry already holds rather than waiting out the retry ladder from Chapter 8. Neither is clever. Both were invisible for a month because nobody had the breakdown.

Streaming Changes Perception, Not Duration

Streaming removes no work at all. What it removes is the gap between the model starting to compose the reply and the customer seeing evidence of it: the first token lands about 900 ms in, while the complete 220-token reply takes about 3.4 seconds to generate. Same total, entirely different experience — a response appearing word by word reads as a system working, and a blank panel for three and a half seconds reads as a system that has fallen over.

Sundry measured the effect on behaviour rather than on the clock. The share of sessions where the customer sent a second, impatient message before the first reply arrived fell from 18% to 7% when streaming was turned on, and nothing else changed that week. This is the cheapest improvement in the chapter and the most commonly skipped, usually on the grounds that the reply is short — which has it backwards, because a short reply is precisely the one where the wait is all the customer experiences.

Parallel Tool Calls

Independent reads in one turn go out concurrently. Chapter 10 made this argument and measured it: sequentially, an order lookup, a carrier check and a policy search take 2.9 seconds, and concurrently they take 2.0, which is the slowest call plus dispatch. Same tokens, same result, 0.9 seconds returned to the customer. It remains the cheapest structural win available and it is free, in the sense that nothing is traded away for it.

What Chapter 13 found was the half that had been left undone. The two sentences Chapter 10 credits for taking multi-lookup tickets from 34% batched to 81% — a line in the system prompt saying independent lookups should be requested together, and tool descriptions stating which tools do not depend on each other — had been applied to the triage agent and never to the specialists, so every handed-off ticket went back to issuing its calls one at a time on its first turn. Copying the same two sentences into the specialist prompts moved the specialists' own batched-turn share from 44% to 91% and took 0.6 seconds off the p95 path. Check that the batching is actually happening rather than assuming it: the trace says how many tool calls sat under each turn, and one is a suspicious number.

Fewer Turns Is the Real Fix

A removed turn is a whole model round trip — about 1.4 seconds at Sundry — plus whatever tool call hung off it. Nothing else in this topic returns that much per change, and three earlier chapters were each removing turns without describing themselves that way. Task-shaped tools from Chapter 3 collapse a sequence into one call. Retrieval that returns the right passage the first time, from Chapter 6, removes the second search that rephrases the first. A plan that survives contact with the ticket, from Chapter 7, removes the turn spent reconsidering.

Together those moved the median run from six turns to five and the returns class from eight to six. That is the whole reason a faster model is a disappointing lever by comparison: it makes each of six round trips slightly quicker, while removing two of them removes them completely. Look at turn count before looking at the provider's latency page.

Routing for Latency

Not every ticket deserves the same machinery. The 55% of Sundry's queue that is order status and delivery is answered by triage from a template, with no policy retrieval, no specialist prompt and no state machine — a p50 of 3.1 seconds on that class. The damage claims still run out to the 8.7-second p95, because they require three lookups and a judgement, and pretending otherwise would mean guessing at the policy.

The consequence is that a single blended latency number describes no ticket anybody actually sent. Report per class. A queue-wide p95 of 8.7 seconds is composed of a fast majority and a slow tail with different causes, and the improvements available to each are different: the fast path is bounded by one model call, and the slow path is bounded by the carrier.

What to Measure

Three numbers, kept separate. Time to first visible token, which is what streaming moves. Time to the first useful message — the reply that actually addresses the ticket, which is the 9-second product target and the number this page has been tracking. And time to full resolution, which includes the return being booked and the refund landing, and which matters to the customer who is waiting for a label rather than a sentence. A run can be excellent on the second and terrible on the third, and only the third is what "sorted it out" means.

Report p95, not the mean. Sundry's mean time to first useful message is 6.2 seconds, which is comfortably inside target and describes nobody's bad day. The complaints, the abandoned sessions and the customer who emails a second time all live in the tail, and a mean is specifically the statistic that hides a tail. Alert on the percentile against the product target, per class, and treat a rising gap between p50 and p95 as a signal in its own right — it usually means one component has started failing slowly rather than failing.