Side Effects, Idempotency, and Retries

issue_refund timed out after the payment provider had already accepted it. The tool reported a failure, the model read the failure, asked again, and a buyer received two refunds while a seller's balance went negative. Nobody wrote a bug. Every component behaved exactly as it was built to behave.

Nothing here is exotic. It is the oldest problem in distributed systems, and the fix has been standard practice in payments for twenty years. What is different is the frequency: an agent hits it far more often, because a probabilistic component is now deciding when to try again, and that decision is invisible to every retry counter you have.

The Wound, Told Properly

Tuesday 17 March, 14:32. The ticket is the cracked shelving unit — order SU-88421, one 4-shelf oak unit at $118.00, sold by Ashcombe Furniture, a marketplace seller with $189 in their balance that morning. The agent has read the order, retrieved the damage-in-transit procedure, established that the buyer refused a replacement, and decided a refund is the correct remedy. It is $118.00, comfortably under the $150 ceiling, so no human is involved. It asks for issue_refund.

Then, in twelve seconds: at 14:32:07 the tool posts the refund and the payment provider accepts it, queuing the transfer. At 14:32:12 Sundry's HTTP client gives up at its 5-second timeout, having received nothing. The tool catches the timeout and returns {"error": "refund failed: upstream timeout"}. On the next turn the model reads the word "failed", concludes the refund did not happen, and asks for it again with the same arguments. The tool generates a fresh idempotency key — it calls uuid4() at the moment of the attempt — so the provider sees a request it has never seen before and accepts it. 14:32:19: the second transfer is queued. The model receives a success, and writes to the buyer: "I've refunded $118.00 to your original payment method — you'll see it within 3–5 working days." One refund described. Two performed.

Nothing errored. No alert fired. The transcript reads like a well-handled ticket, and the eval run scored it as resolved, because the eval graded the reply and the reply was correct. It surfaced on Friday 20 March, in finance's weekly seller-balance reconciliation: Ashcombe Furniture at minus $47, two identical $118 debits twelve seconds apart. The same shape turned up on four more tickets across eleven days, $610 of duplicate payouts, four different sellers — and every one of those tickets had been graded a success.

Take the twelve seconds apart and each layer had a defensible job it did wrongly. The payment call had no key derived from the request, so the provider could not recognize the second attempt as the first one repeated. The tool returned "failed" for a state that was genuinely unknown, which is a different thing. The model, reading "failed", made the only sensible decision available to it and retried. And nobody had written a test in which a write times out, because the write always worked on a laptop.

Idempotency Keys

An idempotency key is a string that identifies the intent, so that a repeated request is recognized as the same request rather than a new one. Everything turns on where it comes from. A key generated inside the call — uuid4() at attempt time — is fresh on every attempt and therefore offers precisely no protection; it is the version Sundry shipped. A key derived from the intent is identical on every attempt, in every process, and after a restart, because the same facts produce the same string.

def issue_refund(ticket_id, order_id, item_id, amount_cents):
    # derived from intent, so every attempt produces the same key
    key = sha256(f"refund:{ticket_id}:{order_id}:{item_id}:{amount_cents}")

    seen = ledger.find(key)
    if seen:
        return {"outcome": "already_performed", "refund_id": seen.id,
                "cents": seen.cents, "at": seen.at}

    try:
        resp = psp.refund(key=key, cents=amount_cents, order=order_id)
    except Timeout:
        state = psp.lookup(key)          # read before writing again
        if state.found:
            ledger.record(key, state.refund_id)
            return {"outcome": "performed", "refund_id": state.refund_id}
        return {"outcome": "not_performed", "retry": "wait_and_retry",
                "note": "no refund recorded for this key; safe to try once more"}

    ledger.record(key, resp.refund_id)
    return {"outcome": "performed", "refund_id": resp.refund_id}

In words: build the key from the four facts that define this refund — which ticket, which order, which item, how much — so that a second attempt at the same refund produces the same string and a separate refund produces a new one. Check the ledger first, and if this key has paid out already, say so rather than paying again. Send the key to the payment provider, which deduplicates on it, so even a request that reaches them twice moves money once. When the call times out, do not guess: ask the provider what happened to that key, and answer with what you find. Every path returns one of three named outcomes, and none of them is a bare error string.

The design question hiding in that first line is what makes two requests the same request. Include too little and a legitimate second refund on the same order — a different damaged item, a week later — is rejected as a duplicate. Include too much, such as a timestamp or a turn number, and every attempt is unique again, which is where uuid4() came from in the first place. Write down the fields that constitute one intent, keep them in the key, and enforce uniqueness at the boundary that moves the money — the provider's key if they support one, a unique index in your own ledger if they do not.

Three Layers of Retry, Only One of Which You Control Well

Three separate mechanisms in a Sundry run can decide to try a call again, and they do not know about each other.

The third row is the one that gets people, and it is specific to agents. The model's retry does not look like a retry: it is a fresh decision, made from the transcript, that happens to produce the same call. No backoff applies to it. No retry counter increments. It does not appear in your metrics as a retry at all, and if you disable retries everywhere in the HTTP stack the model will still do it. The only defences are a key that makes a repeat harmless and a result that does not read as "failed" when the truth is "unknown". Meanwhile the first row is a genuine trap in default configuration: an HTTP client whose retry policy includes POST will repeat a money-moving request before your tool's protection sees the second attempt at all.

At-Most-Once for Money, At-Least-Once for Reads

Classify every tool, in code, and let the classification decide whether a retry is permitted rather than leaving it to whoever is debugging that afternoon.

The middle row is the one teams skip because the damage is small, and small damage at 4,200 tickets a week is a courier company invoicing Sundry for pickups nobody scheduled. message_seller sits in the bottom row for a reason worth stating: it is not money, and it is the least reversible thing on the list, because a message that has been read cannot be unread. Its key is derived from the ticket, the seller and a hash of the text. escalate_to_human is the one exception where a duplicate is merely wasteful — two people opening the same ticket — and it still gets a key, because the waste is a human minute rather than a machine one.

Making the Result Unambiguous

A write tool must return a result that distinguishes three states, because the model's next move depends entirely on which one it is in. Performed means the effect happened on this call: report it to the customer and move on. Already performed means the effect happened earlier, on this ticket or another attempt: say so, do not do it again, and reconcile what the customer was told. Not performed means nothing happened and the world is unchanged: trying again is legitimate.

What destroyed Ashcombe's balance was a fourth state pretending to be the third. "Refund failed: upstream timeout" reads as "not performed" and actually means "unknown". Those are opposite instructions to the component reading them, and the collapse is not the model's error — it is a tool that reported certainty it did not have. If a write tool cannot determine which of the three states it is in, that is itself the answer to return, together with a do-not-retry marker and an escalation, because a human can resolve an unknown and a loop cannot.

Verification Instead of Faith

After an ambiguous write, the correct next move is a read, not a retry. The state exists somewhere authoritative — in the payment provider's records, or in the order — and asking costs one cheap call against a system that is safe to query as often as you like. get_order returns refunded_cents for exactly this reason: before any second issue_refund, read the order and see whether the money already moved. That is the rule that closes this wound.

Put the rule in the tool, not in the prompt. "Always check whether a refund has already been issued before issuing one" in the system message is followed most of the time, and the exceptions cluster on long, confusing threads — precisely the tickets where a duplicate is most likely. The same sentence as a lookup inside issue_refund runs on every call, including the ones where the transcript has been compacted and the model no longer remembers the earlier attempt at all (Chapter 5). Chapter 8 takes this further into partial failure and compensation: what to do when a run dies between two writes, and how to undo the half that landed.