Long-Running Agents

The reconciliation run takes 41 minutes. That one fact moves it out of request-response and into ordinary asynchronous engineering: a job record, a queue, a worker, checkpoints at turn boundaries, a progress view somebody can look at, and a stop button that works. None of this is exotic and most of it predates language models by decades.

Almost none of it is agent-specific either, so this is one topic rather than a chapter — Chapter 8 already made the case that everyday backend problems get one sentence and a pointer. Two things here are different. The checkpoint has to interact with the intent record from Chapter 8, or resumption repeats an action that already committed. And the budget has to be enforced by the worker, because the thing choosing the next step is a model and a model cannot be trusted to stop.

Out of the Request Cycle

The shape is a job record, a queue and a worker. Whatever triggers the run — a schedule, an API call, a button in the finance tool — writes a row and returns an id in about 40 milliseconds. A worker picks the row up, builds the loop from Chapter 1 exactly as written, and persists the run's state at every turn boundary rather than holding it in a local variable. Nothing about the loop changes. What changes is that its state lives in a table that survives the process, which Chapter 6 already required for a support thread that spans three days and here becomes non-negotiable.

def work(job_id):
    run = Run.load(job_id)                    # resumes at turn 19 if it died there

    while run.status == "running":
        run.settle_intents()                  # Ch8: resolve anything left "intended"
        if run.cancel_requested:
            return run.finish("cancelled")
        if run.over_budget():                  # turns, seconds, tokens, dollars
            return run.finish("exhausted")

        run.step()                            # one model call + its tool calls
        run.checkpoint()                      # messages, task state, spend, turn

    return run.finish(run.stop_reason)

In words: load the run from storage, and at the top of every turn do three things before deciding anything — settle any action left in the intended state by a previous process, check whether a human has asked for cancellation, and check whether any ceiling has been crossed. Only then take one turn and write the checkpoint. Each of those three checks is two lines, each sits at a turn boundary rather than inside a step, and together they are the difference between a job that can be operated and a job that can only be killed.

Checkpointing and Resumption

A checkpoint is the message array, the task state, the turn number and the budget consumed, committed at the boundary between turns. Get that right and a crash at minute 21 resumes at turn 19 instead of restarting at turn 1 — for a 41-minute job that is the difference between a delay nobody notices and a re-run that misses the 09:00 deadline. Turn boundaries are the natural place because that is where the state is coherent: a model call has completed, its tool results are in, and nothing is half-applied.

Resumption is only safe because of the intent record. Chapter 8 established the discipline — write what you are about to do before you do it, write the outcome after — and here it is what stops a resuming worker from repeating a committed action. Without it, a job that died between writing an artefact and recording that it had written one comes back and writes it again; with a money-moving tool in the sequence, comes back and pays again. The resuming worker settles every intended row against the authoritative system first, and only then asks the model what to do next. Checkpointing without the intent record is not a safety mechanism, it is a faster way to repeat yourself.

Progress and Cancellation

A human must be able to see what the run is doing and stop it. A forty-minute job with no cancel button is a forty-minute incident, and the version of that sentence that gets used in practice is: at minute six somebody notices the job is running against last week's export, and the only available action is asking an engineer with cluster access to kill a pod. Progress is cheap — the turn number, the current step in one line of English, the artefacts produced so far, and the tokens and dollars spent — and it is written by the same checkpoint that already exists.

{"run_id": "recon-2026-08-10", "status": "running",
 "turn": 19, "turn_limit": 60,
 "step": "joining refunds to seller balances (6,412 rows)",
 "elapsed_s": 1284, "budget_s": 3600,
 "tokens": 91420, "spend_usd": 1.12,
 "artefacts": ["scratch/joined.parquet"],
 "cancel_url": "/runs/recon-2026-08-10/cancel"}

Cancellation has to stop at a safe boundary, which is why it is a flag rather than a signal. The cancel endpoint sets cancel_requested; the worker reads it at the top of the next turn and after each write, then finalizes properly — persist the state, extract whatever artefacts exist, mark the run cancelled with a record of what was done and what was not. Killing the pod is not a cancellation path. It lands mid-action, leaves an intended row with no outcome, and hands you the Chapter 8 problem on top of the one you were trying to stop.

Budgets Over Long Horizons

Wall clock, turns, tokens and dollars all need ceilings, and the support agent's numbers do not transfer. Twelve turns is a support-ticket limit; the reconciliation is allowed 60 turns, 60 minutes and 400,000 tokens, whichever it reaches first, and hitting any of them ends the run as exhausted in Chapter 7's vocabulary — with its artefacts extracted and its state intact, because an over-budget run that also loses its work is two failures. Pick the numbers from a measured run and set them at roughly 1.5 times it, so that a job which has genuinely gone wrong is distinguishable from a Monday with more rows than usual.

The ceiling belongs in the worker and nowhere else. A model instructed to stay under budget will report that it stayed under budget, because reporting is the only thing it can do about it. This is the same rule Chapter 12 applies to the $150 refund limit — enforcement lives in the code that executes, not in the prose that asks — and it matters more over a long horizon, because a support ticket that overruns costs cents and a batch job that overruns runs until somebody notices on Tuesday.

Scheduled and Triggered Runs

The weekly reconciliation is a scheduled job with every control above still attached: Monday 06:00, the same queue, the same worker, the same ceilings, the same cancel endpoint. Scheduling adds nothing to the run and one thing to the operations — a trigger nobody is watching. Finance opens the file at 09:00, so a failure at 06:04 has just under three hours of slack, and that slack only helps if something raises a hand inside it.

Alert on two different conditions, because they fail differently. A run that errors produces an exception you can route like any other. A run that never started produces nothing at all — no error, no log line, no page — and is discovered by the finance team at 09:00 asking where the spreadsheet is. So the second alert fires on absence: no verified artefact for this week by 07:00, page the on-call. That is standard production practice from Chapter 13, and scheduled agents need it more than most jobs because their output is a file somebody trusts rather than a service somebody uses.