Hallucination: Confidently Made Up

Tessa is three hours into a client itinerary — a week in the lake country for a couple celebrating their anniversary — and she still needs somewhere to put them on the second night. She asks the chat box for small lakeside hotels with a quiet terrace. Four come back, each with a line of description. The third is the Harbourview Annex: a restored boathouse, twelve rooms, recently renovated, a terrace that sits out over the water. It sounds exactly right. It also does not exist, has never existed, and is about to teach the most important lesson in this book.

What the model did there has a name: hallucination — inventing a specific and presenting it as fact. Not a crash, not an error message, not a rough edge that the next version will file off. It is a routine consequence of how the machine works, and it is the reason this chapter exists. The other three hotels on that list were real. Nothing in the text told Tessa which one was not.

What Actually Happened

Chapter 1 established the mechanism: the model builds text one piece at a time, each piece being the one that fits what came before. Now watch that mechanism produce the Annex. Tessa's request set up a pattern — a list of small lakeside hotels, one descriptive line each — and the model filled it in. A hotel name is easy to fit, because hotel names are made of familiar parts: harbour, view, annex, each of which appears in thousands of real ones. A twelve-room restored boathouse with a terrace is a description that fits a small lakeside hotel beautifully. Every piece was the right shape.

The part worth sitting with is what did not happen. No register of hotels was opened. No booking site was checked. There is no step inside the machine where a claim gets compared against the world, which means there is no step that failed — the machine did what it always does, and this time the result was false. Chapter 1's summary was that the model produces text that fits. Fitting and existing are two different tests, and only the first one is ever run.

The Word and Its Edges

Say what hallucination covers, precisely, because the word gets thrown at everything an AI does badly. It means invented specifics delivered in the same voice as verified ones: names, dates, numbers, prices, direct quotes, sources, clauses in a contract, features a product does not have. The word itself is borrowed and imperfect — it suggests the model is seeing something that is not there, when in fact it is doing its ordinary job and the output happens not to match reality. It is the standard term, so this book uses it. Just do not read a malfunction into it.

Two neighbouring problems are not hallucination, and mixing them up leads to the wrong fix. The first is being out of date: the model describes a real hotel as it was three years ago, because that is when its knowledge stops (Chapter 1). That is staleness, and the cure is to supply current information, which Chapter 4 is entirely about.

The second is misreading your request: you asked for hotels near the north shore and got hotels near the lake generally. That is a miss, and it has one saving grace — you can see it. You read the answer, notice it answered a different question, and steer. Hallucination is the failure you cannot see by reading, and that is what makes it the dangerous one.

Where It Strikes Hardest

The risk is not spread evenly across everything you might ask, and knowing the shape of it is most of the practical skill. The pattern: invention rises as an answer comes to depend on one particular fact being true out in the world, and rises further as that fact gets smaller, rarer, or more recent.

So the danger zones are specific and learnable. Small entities — a twelve-room hotel is far riskier than a national capital. Exact numbers — a refund window, a distance, a headcount. Direct quotes and sources, which come out complete with a plausible author and year and are among the most convincing inventions the machine makes. Anything recent. And any question that quietly assumes an exhaustive list exists: which of these three ferries runs on Sundays presumes there are three, and that they run.

General explanation is the safe ground. Ask how trip-cancellation cover usually works on a package holiday and the answer rests on a broad, heavily repeated pattern rather than one fragile fact. Ask which clause of Waymark's 120-page supplier contract covers late cancellation and you are deep in invention territory, because the model has never seen that contract. The rule of thumb is short: the more an answer hangs on one specific thing existing, the more carefully you check it.

There Is No Warning Light

Read the four hotels again and there is nothing to find. Same sentence length, same warmth, same level of concrete detail. The invented one is, if anything, slightly better written, because a name with no reality behind it puts no awkward facts in the way. The model is not concealing anything. Nothing in the writing marks which parts of an answer stand on solid ground and which were assembled to fit — that difference never reaches the page.

Picture a charming acquaintance who cannot bring themselves to say "I don't know." Ask them for a restaurant in a city they know well and the tip is gold. Ask them for one in a city they have never visited and a restaurant arrives anyway, in exactly the same warm, specific voice. You would not stop asking them for tips. You would learn which of their tips to check before booking — and that is precisely the relationship this chapter is teaching you to have with the model.

Detection, then, is your job, and the next four pages are the toolkit: why good writing fooled you in the first place, the cheap habits that catch most of it, and two more leans in the machine that work the same way. The Harbourview Annex is not a story that gets resolved. It comes back in Chapter 5, sitting inside a table Tessa generated, and once more in Chapter 10. It is this book's memento, and it never gets fixed.