Hallucinated Tools, Arguments, and Facts

On one ticket the model asked for cancel_order, a tool Sundry has never had. On another it sent issue_refund an order id of SU-88412 — one transposed digit from the SU-88421 in the conversation, well-formed, real, and belonging to somebody else. On a third it told a buyer the replacement would arrive Thursday, a date no tool returned and nobody had promised.

Three surfaces, one behaviour: a plausible completion of a pattern. The model is not lying and it is not confused in any sense a person would recognize — it is producing the most likely continuation given what is in front of it, and a tool name that would obviously exist, an id that looks exactly like the ones it has seen, and a delivery date that fits the shape of a support reply are all extremely likely continuations. Arguing with that in the system prompt reduces the rate and never removes it. Catching it at the boundary does, and one of the three checks below is worth more than the other two together.

Invented Tools and Arguments

An unknown tool name and an unknown parameter are the easy case, because the dispatcher sees them before anything runs. The rule from Chapter 3 holds exactly: never raise, never silently no-op. An exception ends a run that was recoverable, and a silent no-op is worse than either — the model believes the action happened and tells the customer so. What goes back is a structured error that lists what actually exists.

{"tool_use_id": "tu_11",
 "is_error": true,
 "code": "unknown_tool",
 "message": "No tool named cancel_order. Available: search_orders, get_order,
   track_parcel, search_policy, start_return, offer_replacement, issue_refund,
   message_seller, escalate_to_human. Cancelling an order is not something this
   agent can do at all; escalate_to_human handles those.",
 "retry": "fix_and_retry"}

Three parts, and the third is the one teams leave out. It names the mistake, it lists the real tools so the next call can be right rather than differently wrong, and it says what to do when the capability genuinely does not exist — because without that sentence the model picks the nearest-looking tool and does something else wrong with complete conviction. Sundry sees an unknown name or parameter on roughly 0.3% of turns, and 94% of those runs correct themselves on the next turn. The invented names are worth logging for the same reason Topic 43 logs repeats: the top three were cancel_order, check_delivery_status and update_address, which is a list of capabilities the support team had been asking for.

Invented Values

The refund that went to a stranger is the expensive class, and it is expensive because every check that existed passed. SU-88412 matched the schema's pattern for an order id. It existed in the orders database. It had a $64.00 charge on it. The only thing wrong with it was that no tool in that run had ever returned it — the run had looked up SU-88421, and two turns later the digits arrived in the other order. Format validation is not existence validation, and existence validation is still not the check that mattered.

The check that mattered is provenance: every identifier and every amount in a write call must trace to something this run actually saw. The dispatcher already handles every tool result, so it can record the ids it hands back and the charges it has read, and refuse anything else before the call goes out.

ID_FIELDS = {"order_id", "item_id", "tracking_number", "seller_id"}

def check_provenance(run, call):
    for field, value in call.args.items():
        if field in ID_FIELDS and value not in run.seen_ids:
            return err("unknown_id",
                       f"{value} did not appear in this run. Seen: "
                       f"{sorted(run.seen_ids)}. Look it up before using it.")

    if call.name == "issue_refund":
        order = run.orders[call.args["order_id"]]
        if order.customer_id != run.ticket.customer_id:
            return err("wrong_customer", "that order belongs to another buyer")
        if call.args["amount_cents"] not in {c.cents for c in order.charges}:
            return err("amount_unmatched",
                       f"no charge of that amount on {order.id}; charges are "
                       f"{[c.cents for c in order.charges]}")
    return None

In words: walk the arguments, and for anything that is an identifier, refuse it unless this run has seen it in a tool result or the customer's own message. Then, for the tool that moves money, add two checks that need one lookup each — that the order belongs to the person who opened the ticket, and that the amount matches an actual charge on it rather than a plausible round number. Eighteen lines, one dictionary and one set, running before every write. It is the highest-value check in this chapter, and it has one clear limit: it cannot catch an id that is in the run but is the wrong one of the two orders in front of the model. That is a judgement failure, and only the eval set in Chapter 9 sees it.

Invented Facts in the Answer

The third surface has no dispatcher in front of it, because the output is prose going to a customer. A delivery date nobody supplied, a policy clause that does not exist, a seller commitment invented on the seller's behalf — none of these passes through a validator by default, and all three are things Sundry has sent. The defence is the citation requirement from Chapter 6 turned into a check: any statement carrying a date, an amount or a policy claim must reference a tool result or a clause id, and the reply is verified against those references before it leaves.

Be precise about what that verification does. It confirms that the cited clause exists, that it was among the passages actually returned in this run, and that its scope fits this order — all mechanical, all cheap. It does not confirm that the clause says what the reply claims it says. The failure that survives is therefore the most convincing one in the whole chapter: a real citation, correctly retrieved, attached to a sentence the passage does not support. Catching that needs a grader with the passage and the reply side by side, which is the judge in Chapter 9.

Why Confidence Is Not a Signal

The wrong answer arrives in exactly the register of the right one. There is no hedging, no wobble in the phrasing, no tell of any kind, because fluency and truth are produced by the same process and only one of them is being optimized. This is the property that makes hallucination different from ordinary bugs: a null-pointer error announces itself, and an invented delivery date reads like customer service.

Vera tested the obvious control. Twenty known-wrong replies were seeded into a set of 100 and three experienced support staff were asked to find them by reading. They averaged six of the twenty, and each of them flagged around eleven correct replies as suspect. Human transcript review is a diagnostic instrument — it is how you understand a failure you already know about — and it is close to worthless as a detector. Detection needs something with an answer key: a deterministic check against a source, or the graded eval set. Any process whose control is "somebody reads the output" does not scale past a few dozen tickets and is wrong about a quarter of the ones it does read.

Reducing the Rate

Four levers move the rate, and they are worth pulling before the checks rather than instead of them. Grounding: never ask the model for a fact a tool can return, because a value it has to produce from memory is a value it will produce whether or not it knows it. Enums instead of free text (Chapter 3): a reason field with six members cannot be invented, where a free string can be anything. Fewer competing documents in context (Chapter 6): the confusion rate climbs with the number of near-identical passages, which is why retrieving ten passages can be worse than retrieving three. And lower temperature on the turns where a decision or a value is chosen (Chapter 2), which narrows sampling exactly where variance is not wanted.

None of them reaches zero, and it is worth being flat about the trajectory: the rate falls with each generation of model, the class does not disappear, and as of 2026 there is no configuration, prompt or provider that makes argument validation unnecessary. Designing as though the next model removes the need for the check is precisely how $64.00 reaches a stranger's payment method — the check costs eighteen lines and the failure it prevents costs a customer.

The Sundry Numbers

Four weeks, 14,800 tickets handled by the agent, 3,100 write-tool calls, and 2,050 replies that made a policy-bearing statement. The measurements before and after the two checks:

The first row does not improve, and it is not supposed to: you cannot stop the model asking for a tool that does not exist, so you make asking harmless and cheap. The second row is the one that justifies the topic — 41 attempts, none executed, and one of them was a $64.00 refund on a stranger's order. The last two rows are the honest part. Citations moved uncited policy claims from 369 replies to 82, and left behind a residue of 61 that carry a real clause id and say something the clause does not — 143 unsupported replies against the 369, and the residue is a more convincing wrong answer than the claim it replaced.