What Is Still Unsolved
Some of the problems in this book are engineering problems with known answers, and some are open. Being clear about which is which is the most useful thing a course like this can do, and it is also the section most likely to be dated — so every claim here is stated as of 2026 and comes with a way for you to check it yourself.
The distinction is operational rather than academic. A hard engineering problem gets a ticket, an estimate and a person. An open problem gets a design that degrades into a human handoff, because there is no estimate to give. Presenting a mitigation as a solution is the failure this topic exists to prevent, and it is the failure that costs a team a quarter: a product built on the wrong side of that line looks fine in a demo and cannot be fixed by working harder.
The Reliability Ceiling
Per-step accuracy compounds, and Chapter 11 did the arithmetic on it. A step that is right 95% of the time is a good step; twenty of them in a row produce a clean run 36% of the time. At 99% per step, which no agent achieves on messy real-world tasks, a fifty-step task still fails four times in ten. Sundry's own eleven-step carrier claim flow completed unattended in 54% of 300 runs, below the arithmetic, because step difficulty is not uniform and the hard steps cluster at the end. No amount of prompting changes multiplication.
The current answers are shorter horizons, verification through a different channel, and checkpoints so a failure resumes rather than restarts. Each of them either raises the effective per-step rate or reduces the exponent, and none of them removes it. That is the boundary in one example: turning a 24-step task into three 8-step tasks with verified handoffs is hard engineering with known answers, and Chapter 7 taught it. Making a 24-step task reliable end to end without decomposing it is open, and a vendor demo that shows one working is showing you a sample of one.
Evaluating Open-Ended Work
Grading a support resolution is hard, and Sundry's rubric took three rewrites to get there: four criteria, each scored zero to two, with the anchors written down so two people assign the same score. Even then the judge is an instrument with an error bar. The same judge model agreed with human labels 62% of the time on a vague helpfulness scale and 89% on the anchored criteria — same runs, same graders, different rubric — and 89% agreement is a proxy with a known error rate, never ground truth.
Grading a research report, a refactor or a plan is harder, and this is where the open part sits: there is no general method for scoring work whose quality criteria you cannot write down before you see the output. Human preference is the fallback, and it is slow, expensive and inconsistent between graders. What stays firmly in the hard-engineering column is the deterministic half — did the money move, was a clause cited, did the run finish, did it stay inside the ceiling — and Chapter 9 measured that half catching twelve of eighteen real regressions in its first quarter, for a rounding error of the cost. Build that half first, and treat the judge as the part with an error bar published next to it.
Prompt Injection Has No Fix
As of 2026 there is no general defence against prompt injection, and the reason is structural rather than incidental: instructions and data arrive in the same channel, and separating them requires understanding what the text means. Filters help and are partial by construction. Sundry's rule, tuned aggressively enough to catch four of the five cases that survived the prompt instruction, flagged 2.3% of ordinary tickets — around 97 a week, mostly buyers pasting Sundry's own automated emails into a complaint. Tuned back to a tolerable false-positive rate, it caught one case in five. Detection models score better and are subject to exactly the same arithmetic, because a score has a threshold and a threshold has two error rates.
What exists is containment and least privilege, which Chapter 12 built in full: scoped credentials per tool, authorization against the ticket's verified customer rather than an argument the model supplied, a ceiling that is an integer comparison, volume limits at three scopes, untrusted content read in a pass that holds no write tools, and a person on what remains. Treat any claim of a solved defence with suspicion, and read its threat model before its benchmark: which attacker, through which channel, with what coverage, measured how. A defence with no stated threat model is a marketing claim, and a defence evaluated only against a published attack corpus tells you it beat the attacks somebody already wrote down.
Long-Horizon Memory
Storage is trivial and the decisions around it are not. What is worth writing, on what evidence, when it stops counting, and what happens when a stored fact turns out to be wrong. Chapter 6's answer was deliberately narrow: dated observations drawn from a closed set of fourteen predicates, each carrying the ticket it came from and an expiry date, extracted after the ticket closes rather than during it. Sundry shipped that last of everything in the chapter and still had two incidents in the first month.
The open part is correction and contradiction at scale. A store holding tens of thousands of facts about 2,000 sellers has no general method for noticing that one of them stopped being true in March, and no principled way to decide which of two contradicting records governs a decision today. The current answer — store little, scope it tightly, expire it aggressively — is a workaround that trades capability for safety, and it is the right trade at the moment. A product that assumes an agent will accumulate durable understanding of a person or a domain over months is assuming the unsolved part.
Cost and Latency at Depth
More turns and deeper reasoning buy quality and cost money and time. That is a tradeoff, not a problem waiting for a solution, and the numbers in this book make the shape concrete. Adding a reflection turn to every decision would have put an extra model call on 4,200 tickets a week for value that exists only on the quarter of them where something irreversible happens. Chapter 13's greedier routing saved 0.9 cents a ticket and dropped resolution by seven points — $38 a week saved against roughly $700 a week of extra human handling. Depth is worth buying exactly where the consequence justifies it, and nowhere else.
Token prices have fallen by an order of magnitude over several years and will fall again, which changes the arithmetic without changing the shape. Cheaper tokens move the point at which depth pays for itself; they do not make depth free, and they do nothing at all about latency, because a customer waiting on turn nine is waiting on wall-clock time rather than on money. When a price changes, re-measure your own cost per resolution rather than reasoning about the new rate card — that is a twenty-minute job with the eval set you already have.
What Changed Recently, and How to Track It
This page was written in 2026, and the five claims above will not age at the same rate. The cost and latency one moves constantly and does not need re-reading, only re-measuring. Per-step accuracy improves steadily, and the arithmetic underneath it does not care. The injection claim is the one least likely to have moved, because it is a property of the interface rather than of any model. Here is the whole page as a table you can re-check, with no links in it, because a URL that rots is worse than no citation at all.
| Claim, as of 2026 | Kind | What would change it, and where you would see that |
|---|---|---|
| Per-step accuracy compounds, so long horizons fail | Arithmetic | Nothing changes the multiplication; what moves is per-step accuracy and step count. Check both on your own runs. |
| Open-ended work needs a judge with a measured agreement rate | Open | A grader whose agreement holds across domains without re-validation. Check your own agreement rate on your own labels. |
| Prompt injection has no general defence | Open | A defence with a stated threat model, a coverage claim and independent reproduction. Read security literature and system cards, not launch posts. |
| Long-horizon memory is unsolved in the general case | Open | A method that corrects a stale fact and resolves contradictions, measured. Look for benchmarks that test staleness rather than recall. |
| Depth costs money and latency | Tradeoff | Pricing and serving change often. Re-measure cost per resolution rather than re-reading a rate card. |
Three sources are worth the time, and one habit beats all of them. Vendor changelogs and system cards are primary, dated by the people who shipped the thing, and they state failure modes that marketing pages do not. Benchmark leaderboards are useful only when you can read the task definition and the scoring rules — a leaderboard tells you how a model performs on somebody else's work, and the ones that do not publish their tasks tell you nothing at all. Conference talks and launch posts are not sources for a capability claim in your design document.
The habit is running your own eval set against the new model on your own tickets, which settles in an afternoon what a month of reading cannot. Date every capability claim you write internally, exactly as this page does, so a stale line is visibly stale rather than quietly wrong — and put the date in the sentence, not in the file's modification time, because the sentence is what somebody quotes back to you in a planning meeting eighteen months later.
- Presenting a mitigation as a solution — a filter that catches four cases in five and a ceiling that bounds the fifth are both worth having, and neither is a fix.
- Assuming the next model release changes the structural problems — compounding accuracy is multiplication and injection is a property of the interface, and neither is a capability gap.
- Building a product on a capability that demos at a 60% success rate — the arithmetic of Chapter 11 then applies to your whole product, and the gap between demo and pilot is a quarter.
- Reading a two-year-old claim about agent capability as current, in either direction — the optimistic ones and the dismissive ones both go stale, and the dismissive ones age worse.
- Separate "hard engineering" from "open problem" explicitly in your design documents, and give the first an estimate and the second a handoff.
- Date every capability claim you make internally, in the sentence itself, exactly as this page does.
- Design products so the unsolved parts degrade into a human handoff with a summary, rather than into a confident answer nobody flagged.
- Re-check these claims at each audit cycle against changelogs, system cards and your own eval set, rather than against whatever was posted this week.
Knowledge Check
Why does this book call the reliability ceiling on long tasks structural rather than a capability gap?
- Long tasks exhaust the context window, and no model ships a window large enough for fifty full steps
- Training data contains few examples of very long tasks, so models have not learned to finish them
- Per-step accuracy multiplies across steps, and multiplication does not care how good the model is
- Errors early in a run stay in the context and bias every later step toward the same mistake
A vendor announces a defence that "eliminates prompt injection". What does this topic say to do?
- Read its threat model first: which attacker, which channel, what coverage, measured how
- Adopt it and retire the ceiling and the scoped credentials, which then become redundant cost
- Run it against a published attack corpus, and treat a clean pass as coverage of the problem
- Ignore it, since no defence claim in this field has ever survived independent scrutiny
Which of these is an open problem rather than hard engineering with known answers?
- Stopping a retried write from paying a customer twice when the first attempt timed out
- Getting a twenty-four-step task to finish by decomposing it into verified shorter stages
- Bringing cost per resolution down once the trace carries token counts per model call
- Correcting a stored fact that quietly stopped being true, across tens of thousands of records
Why does this page date every capability claim inside the sentence rather than relying on the document's own date?
- Because the content audit cannot locate undated claims when it rewrites a page wholesale
- Because sentences get quoted away from their document, and an undated one reads as current
- Because a page's publication date is not visible to readers who arrive from a search engine
- Because every claim about model capability expires on a known schedule of roughly a year
You got correct