Partial Failure and Rollback

Wednesday, 14:04, a rolling deploy. Nineteen runs were in flight and one of them was working order SU-90317, a garden bench from Ashcombe Furniture that arrived with a split slat and an expedited delivery fee the buyer should never have paid. At 14:03:51 it called start_return: label generated, courier booked for Friday. At 14:03:58 it called offer_replacement: one bench reserved in the Leeds warehouse. At 14:04:02 the pod was replaced. The $14.95 refund never happened, and nothing anywhere recorded that it had been about to.

Three actions, two committed, no transaction anywhere in that sentence. Chapter 3 closed one specific version of this — a retried refund paying twice — with an idempotency key derived from intent, and that key is still doing its job here. It just does not help: nothing was retried and nothing errored. The run stopped existing between two calls, which is the general shape of the problem, and the answer to it is the one distributed systems arrived at decades ago.

Why There Is No Transaction

Count the systems that run touched. The returns service is Sundry's, on Sundry's database. The reservation lives in the warehouse system, owned by another team, behind its own API. The refund goes to a third-party payment provider. The seller notification leaves as an email. Atomicity across those four would require all of them to enlist in one distributed transaction, and two of them have no protocol for it at all — you cannot roll back an email, and the payment provider will not hold a prepared transaction open while a language model thinks about its next turn.

So the design assumption inverts: partial completion is the normal case to be handled, not the edge case to be prevented. At Sundry, six weeks and roughly 25,000 runs produced 84 that died mid-sequence — 0.3%, and 61 of those had exactly one committed write. The causes are boring, which is the point: 47 were deploys, 21 were pod evictions under memory pressure, 9 were an upstream returning a 502 during a write, and 7 were the node being drained by the autoscaler. The single most common reason an agent run dies halfway is the team shipping a change, which means this is not a rare-event problem but a Tuesday-afternoon problem.

Recording Before Acting

Write the intent to durable state before the call, and the outcome after it. A crash between the two leaves a row that says what was about to happen, which is evidence. Without it the crash leaves nothing, and nobody afterwards can distinguish "we never called" from "we called and have no idea what happened" — two situations with opposite correct responses.

def perform(run, call):
    step = run.state.record(               # committed before the call leaves
        step_id=f"{run.id}:{run.turn}:{call.name}",
        tool=call.name, args=call.args,
        idempotency_key=key_for(call),
        status="intended", at=now())

    result = dispatch(call)                # the process dies on this line

    step.update(status="performed" if result.ok else "failed",
                outcome=result.summary, at=now())
    return result

In words: before the call goes out, commit a row naming the tool, its arguments, and the idempotency key it is going to use. Make the call. Commit the outcome. If the process disappears on the middle line — and that is the line it disappears on — what survives is a row marked intended with no outcome, and that row names the exact key to ask the downstream system about. The row has to live in a store that outlives the process; a field on an in-memory object is a comment, not a record.

Resolution is then mechanical. A recovery pass picks up intended rows older than 60 seconds and asks the authoritative system what happened to that key. Three answers are possible and each has a defined next move: it happened, so mark it performed and carry on; it did not, so mark it not-performed and the work is still to do; the system cannot say, so mark it unknown and put it in front of a person. That last branch is the same rule Chapter 3 landed on — a human can resolve an unknown and a loop cannot — and it is why "unknown" must be a status you can store rather than a state of mind.

Compensating Actions

For every write tool, name the action that undoes it, and be honest where there is none. This list gets written before launch or it gets invented during an incident, and a compensation invented during an incident is usually one that does not exist.

Two things fall out of that table. Compensation is not rollback: every row that has one performs a brand-new action with its own side effects and its own failure modes, and the refund reversal can itself time out, at which point you are back in Chapter 3 with a different sign. And two of the five have no compensation at all — which is not a gap to be closed but a constraint to design around. Order the sequence so that the irreversible actions come last, which is why Sundry's returns flow sends the seller message after the money has moved rather than before.

Resumption Versus Compensation

A half-finished run can be continued or unwound, and which one you get is a design decision per task type, made in advance and written down. Resumption reads the state, works out which steps remain, and finishes the job. Compensation undoes what it can, states what it could not, and hands the ticket to a person. Resumption is correct when the remaining actions are still right given only the recorded state — when nothing about the world has moved the decision — and when the run's wall-clock deadline has not already passed.

Sundry's split, with the reasoning attached. Order-status runs resume trivially: read-only, nothing to reconcile, the recovery is a re-run. The returns flow resumes up to the act phase and compensates after it, because a return decision rests on a policy passage and a stock level read minutes earlier and both are still fresh. Damage disputes always compensate and escalate, because the decision involved judgement about a seller's liability, and re-deriving a judgement from a state record is not the same act as making it. The lazy default in both directions is wrong: "always resume" re-executes decisions from information that has since moved, and "always compensate" cancels courier bookings for dozens of runs that would have finished perfectly well.

The Human's View

An escalation after partial completion must state exactly what was done and what was not. A colleague who cannot see that will do the obvious, helpful thing — issue the refund the customer is clearly owed — and the double refund arrives from a third direction, this time with a person's name in the audit log. The handover is a generated object, not a paragraph of prose.

{"ticket_id": "T-40912", "order_id": "SU-90317",
 "reason": "run_died_mid_sequence",
 "completed": [
   {"tool": "start_return", "at": "14:03:51",
    "detail": "label RL-7734, courier pickup booked Fri 20 Mar",
    "reversible": "until collection"},
   {"tool": "offer_replacement", "at": "14:03:58",
    "detail": "1 x oak bench reserved, warehouse LDS-2",
    "reversible": true}],
 "not_done": [
   {"tool": "issue_refund", "detail": "$14.95 expedited delivery fee"}],
 "customer_told": "Return label sent. No refund was mentioned.",
 "next_action": "issue the $14.95 refund, or cancel the pickup if the buyer
   would rather keep the bench"}

Read it as a handover between two people. Three lists: what happened, what did not, and what the customer has already been told — and the third is the one that gets left out, even though the colleague's next message has to be consistent with it. Reversibility is marked per completed action, so the person knows what can still be taken back and what cannot. The whole object is generated from the state rows rather than summarized by the model, because a summary written by the run is a description of work it did not see finish.

Testing It

Kill the process between two write calls, on purpose, in the suite. That is the only way any of the above is known to work, and it is the test nobody writes, because everything above reads as obviously correct on the page. Run scripted tickets through the real dispatcher with a fault injector that hard-aborts after the Nth write, parametrized over every N in the flow, then assert the invariants in a fresh process.

@pytest.mark.parametrize("kill_after", range(0, 4))
def test_survives_process_death(kill_after):
    run = replay_ticket("T-40912", kill_after_writes=kill_after)  # hard abort
    resumed = recover(run.id)                                    # fresh process

    assert no_orphan_writes(run.id)          # every write has an intent row
    assert resumed.unresolved == []          # every intent row was settled
    assert resumed.escalation.completed == committed_writes(run.id)

In words: replay a real ticket, abort the process after zero, one, two and three writes, then let the recovery path run and check three things — that no side effect happened without an intent row in front of it, that every intent row was resolved to performed, not-performed or unknown, and that the escalation names precisely the actions that actually committed. Sundry has 31 kill points across 7 flows. The first full run found four bugs, three of them in the compensations: a cancel that returned success without cancelling anything, a release that needed a reservation id the state row had never stored, and a reversal that had never once been called, not even against the provider's sandbox. A compensating action nobody has executed is a plan, and an incident is a poor place to discover the difference.