Reflection and Self-Correction

Asking the model to check its own work sometimes catches a real error and frequently produces a confident endorsement of what it already did. What separates the two outcomes has nothing to do with the wording of the prompt: it is whether the critique has something external to check against — a schema, a policy passage, a tool result — or is being asked to introspect on an answer it just wrote.

Sundry measured both on the same material, and the gap is not subtle: 61% of would-be policy errors caught by code validation plus one grounded critique, against 9% caught by ungrounded self-review. Everything on this page follows from that gap.

Grounded Critique Works

"Does this refund amount match the passage it cites, and is it inside the ceiling?" is a checkable question. There is a document, there is a number, and a wrong answer is contradicted by material sitting in front of the critique. "Is this a good answer?" is not a checkable question at all — there is nothing to compare against, so the model generates the most plausible continuation, and the most plausible continuation after a well-written answer is agreement.

Grounding is therefore a property of the inputs, not of the instruction. A critique that receives the decision object and the specific policy passage is doing comparison. The same prompt with the passage left out is doing recall with the same priors that produced the original answer, and it will reproduce the original error with the same confidence. If you cannot name the artefact the critique is checking against, you do not have a critique. The test takes ten seconds. Run it against any review step already in your system: point at the thing the verdict is compared with. If the answer is "the model's understanding of the policy", the step is decoration.

Where Code Beats a Critique Turn

Most of what a critique is usually asked to check is a rule you can write down, and every rule you can write down belongs in validation code instead — it runs in microseconds, costs nothing, never endorses anything out of politeness, and can be unit-tested.

Read the last row as the boundary. Four of these are comparisons between structured values and none of them needs a model. The fifth is a judgement about whether a paragraph of policy prose supports a decision — a marketplace supplement that says returns are accepted "for manufacturing defects reported within 14 days" against a decision citing it for transit damage on day 19. No amount of validation code reads that; that is the residue a critique turn is for, and it is a much smaller residue than most designs assume.

The Endorsement Failure

Sundry's first attempt was the obvious one: after producing a refund decision, ask the model to review it. Across a replay of 240 production decisions it rejected 11 and endorsed 229 — a 95% endorsement rate — and only 6 of those 11 rejections were real errors. A control that fires on 5% of cases and is right about half the time is not a control. It is a model call added to every ticket that reaches a decision, with a number attached that makes the system look supervised.

The mechanism is not mysterious and it is not fixed by sterner wording. The answer under review is in the context, written in the model's own register, with whatever reasoning produced it still present. Agreement is the high-probability continuation, and asking more forcefully for scrutiny mostly produces more forcefully worded agreement. This is the same failure as an LLM-as-judge rating its own model's output a full point higher, which Chapter 9 measures properly.

Critique From a Different Vantage

What worked was changing what the critique can see. A separate call, a different system prompt, the decision object and the cited passage as its entire input — and no transcript, no rationale, no memory of having written the thing it is judging.

verdict = model(
    system=CRITIQUE_PROMPT,            # check the claim against the passage
    messages=[{"role": "user", "content": [
        decision.as_json(),            # the object, not the conversation
        passage(decision.policy_ref),  # the exact text it cited
    ]}],
    tools=[],                          # nothing to call, one thing to answer
)

Three omissions do the work, and all three are on purpose. The transcript is absent, so the critique cannot inherit the reasoning that produced the error. The rationale is absent, so it is not being asked to agree with an argument. And no tools are offered, so the only available output is a verdict on the material provided. It is the same argument the subagent boundary makes in Chapter 10 — a fresh context is what makes a second opinion a second opinion — applied to one decision instead of one task.

When to Reflect

Two places, both narrow. Before an irreversible action: the critique runs between decide and act, on the roughly 25% of the queue that reaches a refund or a return, and it costs about 1.4 seconds and a fraction of a cent on those tickets. And at the final answer for policy-bearing replies, where the risk is a customer being told the wrong return window in a message that will be quoted back at Sundry.

Not on routine lookups. Adding a reflection turn to the order-status majority buys nothing measurable and adds latency to the path 55% of customers are on, which is the trade this book keeps making the same way. And reflection after the action is not a control at all — a critique that runs once the refund has gone out is a report, and reports do not stop money. The placement is easy to get wrong because a post-action critique is easier to build: it needs no rollback path, no rejection branch and no decision about what happens when the verdict is negative. Those three are exactly the work that makes it a control, and skipping them buys a dashboard instead of a defence.

The Measured Result

The replay set was 240 refund decisions taken from production, 70 of which were wrong on policy — wrong amount, wrong source, no valid citation, or an item that was never eligible. Code validation alone caught 31 of the 70. Adding one grounded critique before money movement took it to 43. Ungrounded self-review, asked on the same 70, found 6.

Those are the three numbers to carry out of this chapter: 44%, 61%, 9%. The jump from 44 to 61 is what one well-grounded model call is worth, and the distance from 61 to 100 is why Chapter 12's approval gate exists and why nobody at Sundry describes this as solved.

The critique's own cost shows up as false rejections. Of the 170 correct decisions in the replay, the grounded critique rejected 7 — a 4% false-rejection rate — and three of those became escalations a person had to close by agreeing with the original answer. That number belongs on the dashboard beside the catch rate, because a critique whose false rejections climb is quietly converting resolved tickets into human work, and the fix for that is deletion rather than tuning.