Tracing an Agent Run

A trace is the only readable account of what an agent did. One run becomes a tree — the run, its turns, every model call and every tool call — with inputs, outputs, token counts and timings hanging off each node. Without one you debug by reading prose and guessing which turn went wrong. With one, most investigations at Sundry take about two minutes.

This is not a page about observability. Spans, metrics, SLOs and the backend that holds them are ordinary platform work, and Observability Deep Dive in this catalogue owns that ground; nothing here replaces it. What is different about an agent is the shape of the data. The same span repeats a dozen times with a larger context on every pass, the field you usually need is what was in the context rather than what came out of it, and a run's cost is a sum over the tree rather than a number attached to one request.

The Span Tree

Four kinds of node cover every run Sundry produces. The run span covers one ticket end to end and owns the outcome. A turn span is one pass around the loop from Chapter 1. Under each turn sits exactly one model call and zero or more tool calls — siblings, not parents, because the calls in a turn are issued together. A handoff opens a child run span owned by the receiving agent, so the triage and specialist agents from Chapter 10 appear as nested subtrees rather than as two unrelated traces nobody can join.

{"span": "turn", "run_id": "R-9f31", "ticket": "T-40219", "turn": 4,
 "agent": "returns-specialist", "parent": "run/R-9f31",
 "children": [
   {"span": "model_call",   # exactly one per turn
    "model": "provider/model-id@2026-02-11",
    "prompt_version": "sys-2026-03-04", "tools_hash": "9c1e",
    "in_tokens": 6120, "cached_in": 5290, "out_tokens": 210,
    "latency_ms": 1840, "stop_reason": "tool_use"},
   {"span": "tool_call", "name": "search_policy",
    "args": {"query": "damage in transit marketplace seller refund"},
    "result_tokens": 830, "latency_ms": 610, "error": null,
    "doc_ids": ["POL-RET-004", "SUP-ASH-011"]}
 ]}

Read the two children rather than the shape. The model call records which model version answered, which system prompt it ran under, a hash of the tool schemas it was shown, its input tokens split into cached and uncached, its output tokens, how long it took and why it stopped. The tool call records the arguments as sent, the size of the result in tokens, its own latency, whether it errored, and — for a retrieval call — which documents came back. Between them those two nodes answer most production questions anyone will ever ask about an agent, and neither is expensive to emit.

Two of those attributes carry more weight than the rest. Splitting input tokens into cached and uncached is what makes Chapter 5's hit rate visible per turn instead of per invoice, and the document ids on a retrieval call turn "it quoted the wrong window" from a theory into a fact. The run id has to reach everywhere else: every application log line, every audit row, and the ticket itself. A support lead reading a complaint should be able to copy an id off the ticket and land on the trace — when that id lives only inside the tracing backend, every investigation begins with somebody correlating timestamps by hand.

What Makes an Agent Trace Different

In a conventional service a span appears once per request. In an agent the same model-call span appears once per turn, and its input grows monotonically, because the model retains nothing and the history is re-sent in full every time. A twelve-turn Sundry run produces twelve model calls whose contexts run from about 1,400 tokens to about 31,000, and reading those as twelve independent events tells you almost nothing. The view that pays is the difference between consecutive turns: what was added to the context, and what the model did with it on the next pass.

So the interesting field is usually the input rather than the output. A wrong answer is rarely wrong given its context — it is a correct answer to a context holding the wrong policy passage, a stale order record, or a seller's own prose arriving through a tool result as Chapter 12 describes. Keep the input or accept that every investigation ends in speculation. The third difference is arithmetic: in a normal system a request's cost is a property of that request, while a run's cost is a sum over a tree, so every model-call node has to roll its tokens upward before anyone can say what a ticket cost.

Sampling and Retention

Sundry traces the structure of every run and samples what the runs contained. Errors, escalations, approval requests and invariant violations keep everything — full prompts, full completions, full tool results. Successful runs keep the tree, the attributes and the tool arguments, plus full content on a 5% sample taken by hashing the run id so the sample is stable rather than re-rolled per query. Skew the sampling that way round: the runs you most need are the rare ones, and a uniform sampler is a policy for discarding them.

Volume is not the constraint most teams expect. At 4,200 tickets a week and roughly seven model calls each, that is about 29,000 calls, and holding every prompt and completion in full comes to a few hundred megabytes a week — affordable by any standard. What makes retention a decision is that those bytes are the customer's own words, the agent's conclusions about them, and whatever a panicking buyer pasted into a web form at midnight. This is a privacy decision wearing a storage decision's clothes, and the next topic is where it gets settled.

Reading a Bad Run

Take the drift from Chapter 5, in the shape it came back in once retrieval was doing the fetching: the agent quoted Sundry's own 30-day return window at a buyer whose marketplace seller runs the statutory 14. The complaint arrives with a ticket number, the ticket carries the run id, and the run span lists its turns with their stop reasons. The reply the customer objected to was composed on turn 7, so there is exactly one question worth asking — which turn first put a 30-day passage in front of the model.

Filter the tree to search_policy calls and read the document ids. Turn 4 returned POL-RET-004 and the seller supplement SUP-ASH-011. Turn 6 returned only POL-RET-004, because the query had been rephrased and the supplement fell below the cutoff. So the turn-7 context held two passages that disagreed with each other and the wrong one twice. That is the whole diagnosis, and it took two lookups: not "the model hallucinated a policy" but "retrieval stopped returning the supplement when the query changed", which is a Chapter 6 problem with a Chapter 6 fix.

Notice what was never required. Nobody read the transcript as prose. Nobody re-ran the ticket hoping it would misbehave again — which it would not do reliably, because sampling is stochastic and Chapter 2 explains why. The trace converts a non-deterministic complaint into a deterministic record of one specific run, and that record is what makes an argument about an agent settleable at all.

Connecting to the Rest of the System

The agent is a service inside a system, and its spans belong in the same trace as everything around them. A ticket arrives over HTTP, lands in a queue, is picked up by a worker, runs the loop, and writes to Postgres. When the run span hangs off the worker's span, a slow ticket can be attributed to a component instead of blamed on a mood. Sundry's first latency investigation ended at the model because the model was the only thing instrumented; the second one, with tool spans sharing a trace with the carrier's HTTP client, ended at the carrier API — where the seconds actually were.

The rule that follows is one backend. An agent-observability product holding the model calls while the platform's tracing holds everything else gives you two truths and a correlation problem between them, and the correlation was the thing you needed. Where a specialist tool earns its place is on top of the same data: prompt diffs, eval integration and per-run cost views are better in a product built for them, and a product built for them can read standard traces.

Standard Instead of Bespoke

OpenTelemetry's GenAI semantic conventions name the attributes this page has been describing — the model, the operation, token counts, the tool — so that spans emitted by your own code and spans emitted by a library agree on what things are called. As of 2026 those conventions are still moving, and they are still the better bet than any vendor's proprietary event format, because they send the agent's spans through the collector you already run into the backend you already pay for.

The cost of the alternative arrives at the second vendor. Instrumentation written against one platform's SDK gets rewritten when the platform changes, and the historical traces do not come with you, so the comparison you wanted to make is the one you cannot. Emit standard spans, add the two or three attributes your own system cares about — ticket id, agent name, prompt version — and the vendor question stays a question about backends rather than about instrumentation.