When a Loop Is Worth It
Open-endedness has a price: variable cost, multi-turn latency, non-deterministic behaviour, and a much larger surface to secure. A loop earns that price when the steps cannot be enumerated in advance. When they can, it burns money and reliability to solve a problem that a function already solved.
Most production systems that work well are workflows with one agentic pocket. That is not a compromise between two ideologies — it is what the arithmetic on a real queue keeps producing, and it is the shape Sundry ends this book in.
The Test
One question decides it. Can you write down the sequence of steps before the request arrives? If you can, write it in code and use the model inside a step, for the part that needs judgement over language: classify this ticket, extract these fields, draft this reply. The model is doing real work there, and your code still owns the order in which things happen.
If the sequence depends on what earlier steps return — if you cannot say whether the second lookup is needed until you have read the first — then a loop is doing something a pipeline cannot fake. A Sundry ticket about a shelving unit that arrived damaged is the example: whether the agent needs the carrier's last scan depends on what the order record says about who shipped it, and whether it needs a policy lookup at all depends on the delivery date it has not read yet.
What Open-Endedness Buys
It buys the long tail. The ticket with three intents in one paragraph. The case the policy does not quite cover, where the buyer's seller runs a shorter return window than Sundry's own. The question that needs two lookups nobody predicted, in an order nobody wrote down. That tail is 30 to 40% of Sundry's queue, and it is exactly the part that a decade of rules-based automation never reached — not because the rules were badly written, but because each new rule collided with three existing ones.
It also buys graceful degradation on inputs nobody anticipated. A pipeline meeting a ticket that mentions a delivery charge and a cracked panel and a refusal to accept a replacement has one behaviour available: pick a class, act on it, and be wrong about the other two. A loop reads all three, decides what to check, and at worst produces a partial answer with the rest escalated. Partial is a much better failure than confidently wrong.
What It Costs, Priced
A workflow answer is one model call over a context whose size you know before you deploy. The same ticket in a loop is six to twelve calls over a context that grows on each one, and because every call re-sends the whole transcript, the input bill is a sum over a growing prefix rather than a multiple of the first prompt. Put the two side by side on one real ticket.
| Deterministic pipeline | Agentic loop | |
|---|---|---|
| Model calls per ticket | 1 | 6–12, nine on this one |
| Input tokens billed | ~1,400, known in advance | ~50,000, known afterwards |
| Wall clock to a reply | 900 ms | 8 seconds |
| How you test it | Fixtures and exact assertions | A graded eval set (Chapter 9) |
| How it fails | Refuses anything with two intents | Occasionally does something unexpected |
Read the middle row first. The loop bills roughly thirty-five times the input tokens of the pipeline for the same customer question, and that ratio is not a constant — it varies by an order of magnitude between the easiest ticket and the hardest, because the number of turns is decided at runtime by a component you do not control. Sundry's first version costs $0.41 a ticket against a pipeline that costs a fraction of a cent, and Chapter 13 gets it down to six cents without giving up the tail.
Latency deserves its own line rather than a footnote. Eight seconds of silence in a support widget is a product problem no eval score will ever show you, because the eval set does not contain a customer closing the tab. Chapter 13 treats time-to-first-useful-message as a number to be defended, not as a by-product of however many turns the model felt like taking.
The Middle Ground That Usually Wins
Classify deterministically, then let a loop handle only the classes that need it. Sundry's 55% that are pure order status go down a fixed path: look up the order, read the tracking status, fill a template. The returns, the multi-intent tickets and the ones the classifier is unsure about go to the loop. The split is measured on real traffic rather than assumed, and it is how the system actually ships in Chapter 13.
This shape also contains the blast radius, which is the part teams notice second. The fixed path touches read-only tools and a reply template, so the surface an attacker or a bad turn can reach is small by construction. The loop gets the tools that change things, a turn limit, a spend ceiling and an approval gate, because it is the only path where a wrong decision has anywhere expensive to go.
Reversibility as the Deciding Factor
An agent that drafts a reply for a human to send is cheap to be wrong. An agent that issues refunds is not. The useful question is almost never "can the model do this" — by 2026 the answer is usually yes on a good day — but "what happens the times it is wrong, how fast does anybody notice, and what does it cost to undo".
Answer that per tool, in writing, before granting it. Sundry's four lookup tools are free to be wrong, because a bad lookup produces a bad sentence that a human can correct. start_return books a courier and is annoying to undo. issue_refund moves money out of a seller's balance and is a phone call and an apology to undo, which is why it stops at $150 without a human and why the approval gate in Chapter 12 exists at all. An eval score of 94% says nothing about the other 6% until you know which tools they were holding.
The pipeline classifies the ticket, fetches the order, applies the return policy in code and sends a templated reply. 900 ms, one model call, fully testable with fixtures. It fails outright on anything carrying two intents, because it picked one class at the top and everything after that assumed it.
The loop reads the ticket, decides what to look up, notices the duplicate delivery charge nobody wrote a classifier for, and settles both problems in one reply. Eight seconds, nine model calls, and every so often it does something you did not expect.
Sundry ships both, routed by a classifier: the pipeline for the 55% that are pure order status, the loop for the rest. Choosing the shape per class of work beats choosing either extreme for all of it.
- Choosing a loop because the demo was impressive — the demo used the interesting 10% of tickets, and the boring 55% now cost about thirty-five times the tokens they should and take nearly nine times as long to answer.
- Choosing a pipeline for genuinely open work and patching it with conditionals — after the fifteenth special case you have an unmaintainable agent with no loop, and nobody can say what it does on a new input.
- Ignoring latency in the decision — a support widget where the customer waits eight seconds for the first sentence is a product failure that no resolution-rate number will ever surface.
- Deciding once and never revisiting — the balance moves as models get cheaper and as your eval set grows, so the choice belongs in a document with a review date rather than in somebody's memory of a meeting.
- Route by task type: a cheap deterministic path for the known majority, the loop for the tail, with the split measured on real traffic rather than assumed from a sample of interesting tickets.
- Price both options against real traffic before choosing, using the growing-prefix arithmetic from Chapter 2 rather than an intuition about how many model calls are involved.
- Constrain the loop to the smallest scope that still solves the tail — fewer tools, fewer turns, one domain (Chapter 3).
- Write down the reversibility of every action the loop can take before granting it, and put the irreversible ones behind the approval gate from Chapter 12 regardless of how good the eval numbers look.
Knowledge Check
Which single question best decides whether a task needs an agentic loop?
- Can the sequence of steps be written down before the request arrives, or does it depend on what earlier steps return?
- Is the model capable enough to handle this entire category of task without a human being asked to review its output?
- Does this task arrive often enough to justify the engineering effort of building and then maintaining something automated?
- Does the task need more than two or three separate tools before it reaches an answer the customer can actually act on?
Sundry's queue is 55% plain order-status questions. Why is putting all of them through the loop the expensive mistake?
- The majority carries the volume, so paying a loop's premium there costs far more than the interesting tail ever does
- The loop answers simple status questions less accurately than a template, because it has too many tools available
- The sheer volume of simple tickets exhausts the provider rate limit, leaving no capacity at all for the harder tickets
- The loop cannot use the reply templates that the support team already wrote, so every single answer is generated from scratch
What does the hybrid split — classifier in front, loop behind — actually buy beyond cost?
- A smaller blast radius, because the tools that change things are reachable only from the loop path
- Deterministic behaviour overall, since the classifier fixes the sequence for every ticket that arrives
- A simpler evaluation story, because only the classifier's accuracy has to be measured on real traffic
- Immunity to the loop's failure modes, which now apply only to tickets nobody classified
Two designs score the same on the eval set. One can only draft replies; the other can also issue refunds. Why does that difference outweigh the score?
- The score says nothing about what the failures cost, and one design's failures are irreversible while the other's are edits
- The design with money-moving tools will drift toward using them, which lowers its accuracy over time
- Eval sets cannot measure actions that move money, so the second design's real score is unknown
- The drafting design answers faster, and latency matters more to customers than resolution does
You got correct