Reasoning Traces

Asking the model to reason before it acts improves tool selection on hard cases. The visible reasoning is not an explanation of what happened — it is more text the model generated, which may or may not describe the actual basis of the decision. Both halves of that sentence have operational consequences, and teams tend to accept the first and forget the second.

The forgetting is expensive in two different currencies. One is money: deliberation is output tokens, on every turn it is enabled, at output prices. The other is trust: a rationale filed as the audit record for a refund is a story about a decision rather than the decision itself, and the difference becomes visible on the day somebody disputes one.

Reason-Then-Act

The pattern is one short rationale attached to each tool request, in the same assistant message. The model says what it thinks the situation is, then asks for the tool. Nothing about the loop changes — the dispatcher still reads the tool request and ignores the prose.

{
  "role": "assistant",
  "content": [
    {"type": "text",
     "text": "Two delivery charges on one order, and the seller is a "
             "marketplace seller, so the return window is theirs and "
             "not Sundry's. Check the supplement before promising."},
    {"type": "tool_use", "name": "search_policy",
     "input": {"query": "Ashcombe Furniture return window supplement"}}
  ]
}

Read what that turn contains. A few sentences of situation assessment, and one structured request for search_policy with a query the assessment implies. The rationale is a sibling of the tool call, not a wrapper around it, and the two can disagree — which is the whole subject of this page and the reason the section on debugging exists.

The measured effect at Sundry is narrow and real. On the 46 tickets in the labelled set of 200 that carry more than one intent — the set over-weights them deliberately, because they are 9% of the live queue and most of its selection failures — the correct tool sequence rose from 72% to 87% with rationales enabled. On the order-status majority it moved nothing at all: 96% before, 96% after, because there was never a decision to deliberate about. The benefit lands exactly where the work is ambiguous and nowhere else.

Reasoning Modes

Separately from anything you prompt for, providers expose a mode where the model spends tokens thinking before it answers — as of 2026, typically a per-request setting with a token budget attached. Those tokens are billed as output whether or not you are shown them, and they arrive before the first useful character of the response, so they buy accuracy with latency as well as money.

Where that trade is worth making is the same place a written rationale pays: the turns where an actual decision is made. At Sundry that is one turn per ticket — the one that produces the refund decision object — and enabling deliberation there rather than everywhere is the difference between a measurable improvement and a measurable bill. A routine get_order lookup does not become more correct because the model thought about it first.

What a Trace Is Evidence Of

A reasoning trace is evidence about what the model produced. It is not a verified account of the computation that produced the answer, and nothing in the API makes it one. The text is generated by the same process, with the same priors, as the answer it accompanies, and it is entirely capable of being confidently wrong about itself. Post-hoc rationalization — a fluent explanation constructed after the fact for a decision reached some other way — is well documented and should be your default assumption rather than an exotic edge case.

That makes traces useful for one job and dangerous for another. Useful: debugging, where a rationale is a hypothesis you then check against the tool calls in the same run. Dangerous: audit, where a rationale filed as the reason a customer was refunded $118.00 is a piece of prose asserting a justification that may never have been the operative one. The audit record for a money decision is the structured decision object and the tool calls it produced — the amount, the source, the policy id, the approval — because those are things the code did, not things the model said (Chapter 13).

Hold the distinction in the same terms you would apply to any other system. A log line your code wrote when it took a branch is a record of the branch. A paragraph explaining why a branch was taken, written by the component whose behaviour is in question, is testimony. Both belong in the trace; only one of them settles an argument.

Never Show It to the Customer

Reasoning text is written for the model's own next turn, and it reads that way. It speculates — "this might be a non-delivery claim rather than damage". It contradicts itself across turns as new tool results arrive. It names internal systems, policy ids, and seller balances. It occasionally states an amount the agent then decides not to refund, which is the single worst sentence a support system can put in front of a buyer.

The customer-facing reply is a separate generation with its own contract: what was done, what happens next, what is still open, in the company's voice. That separation earns its keep: it is the only way a customer-facing string can be validated against something — length, tone, the absence of internal identifiers, the presence of every unresolved subtask — because you cannot validate an artefact that is also load-bearing for the model's reasoning (Chapter 13).

Using It in Debugging

The place a trace earns its cost is an incident. Sundry's drift — the agent quoting a 30-day return window at buyers whose seller runs 14 — produced no error and no alert, and it took six weeks to find from the outside. Inside a rationale it is a single line: the model writes "the seller's supplement applies here", and the tool call in the same message asks search_policy for Sundry's own returns page. The rationale and the action disagree, and the disagreement is the diagnosis.

Use it that way and only that way. A rationale that matches the tool calls tells you very little, because a model that made the wrong choice will usually explain the wrong choice fluently. A rationale that contradicts the tool calls tells you exactly where to look. Verify against what the run actually did — the calls, the arguments, the results — rather than believing the narration, which is the mistake that turns tracing into a slower way of trusting the model.

Cost Control

Deliberation is output tokens at output prices, on every turn it is enabled, for the whole run. Sundry's rationales run about 180 output tokens a turn. On the returns class that is not a rounding error: enabling them on every turn moved cost per ticket from $0.22 to $0.31 and pushed p95 time to first useful message on that class from 24 seconds to 28. The accuracy gain was real and it was confined to one decision per ticket.

Enabling deliberation only on the turn that produces the decision object costs $0.24 on the same class, keeps the 87% sequence accuracy, and leaves p95 where it was. Roughly 60% of turns in a Sundry run are lookups whose result is not in question, and they pay for nothing when deliberation is on. Turning it on globally because it improved a hard case is one of the most common expensive defaults in this field.