Talking to Models

Plateful is piloting a support chatbot, and Iris is testing it. She types a customer's question: can I get a refund if my food arrives cold? The answer comes back instantly — warm, well-organized, quoting a "48-hour full-refund guarantee" with the confidence of a lawyer reading from the contract. One problem. Plateful has no 48-hour guarantee. The policy, the deadline, the reassuring clause — the model composed all of it. Fluent, confident, wrong.

Every user of these systems has to metabolize that moment, and this page is where you do it. Two behaviors define working with an LLM: why what you type in changes the answer so much, and why fluent falsehood is not a rare glitch but the default failure mode. Both fall straight out of the machinery you already own — the loop from Topic 40 and the recipe from Topic 41. No new parts arrive; this page is where the two you have collide with reality.

The Prompt Is the Context, Not a Command

What you type into the chat is called a prompt, and the first thing to unlearn is that it works like a command. A command is interpreted and obeyed. A prompt is something humbler and stranger: it becomes the context — the "everything written so far" that the loop re-reads before predicting each next word. Your words, the earlier turns of the conversation, any document you paste in: all of it sits in the model's window, and the answer is simply the most likely continuation of that whole pile.

This explains a fact that surprises everyone at first: the quality of the answer tracks the quality of the input, mechanically. Iris's bare question gave the loop nothing to continue from except "a refund question, in a support chat" — so it predicted what refund policies plausibly sound like. When she instead pastes Plateful's actual policy document above the question, the likeliest continuation of that context is an answer grounded in the real rules. Pasting the policy beats typing "please be accurate" every time — one adds predictive signal, the other adds a polite phrase. Production systems automate exactly this move, fetching the relevant documents and placing them in the context before the model answers; that general grounding pattern is most of what separates a toy chatbot from a usable one.

Why It Makes Things Up

Now the wrong answer itself. The invented guarantee has a name — a hallucination, the standard term for a model stating things that were never true — and the mechanism is nothing more than Topic 40's loop doing its job. The loop generates the likely, not the verified. A crisp refund clause with a concrete deadline is a superb prediction: it is exactly the kind of sentence that follows a refund question in the world's support pages. It is also false. Fluency and truth simply come apart, because nothing in Topic 41's recipe ever welded them together — likely text, helpful shape, preferred tone, and not one stage that checked a claim against reality.

Picture an improv actor handed a doctor's coat. They will stay in character — take the pulse, furrow the brow, deliver a diagnosis with perfect bedside manner — because continuing the scene is their entire training, and walking off stage fights everything they know. The model is that performer: given a refund scene, it plays the refund expert. Saying "I don't know" is a way of stopping the continuation, and continuation is the only thing it was ever built to do. That is hallucination — not lying, which requires knowing the truth and hiding it, but performing, with no backstage to check.

Confidence Is Part of the Style

The cruelest detail of Iris's wrong answer was its tone: assured, specific, organized. Here is why. The training text of the world is written in confident prose — support pages do not hedge, encyclopedias do not stammer, and few authors write "I might be wrong about all of this" into every paragraph. Confident prose is what the model read, so confident prose is what it predicts. The tone is part of the learned style, and it is applied evenly to true statements and invented ones alike.

This is Topic 04's "confidently wrong" in its final form. Back then, a delivery estimate could be confidently wrong because confidence measured resemblance to training patterns, not reality. Now the model produces whole paragraphs whose confidence is itself a predicted pattern. So carry the rule in one line: tone carries zero evidence. A specific number, a firm deadline, a lawyerly clause — none of it tells you anything about truth, because the style was learned from text, not earned from facts. And the model rarely knows when it doesn't know: the loop produces an answer either way, so the doubt has to come from you.

Working with It Honestly

None of this makes the tool useless — it makes the tool a specific kind of colleague. Treat the model's output as a draft from a brilliant, unreliable intern: enormously well-read, tireless, fast, and entirely capable of inventing a policy to keep the scene going. You would not fire that intern; you would also never send their draft to a customer unread.

The working habits follow directly from the mechanism. Give it the relevant facts — the document, the data, the constraints — because context is the one input you control. Ask for sources, then actually check them; a hallucinated answer often comes with hallucinated citations, and one click exposes them. Verify anything that matters before it leaves your hands. And when the stakes are real — Plateful's chatbot answering actual customers — route the system through grounding in the real documents, the pattern from the figure above. Where the stakes get highest, and who is accountable when the intern's draft ships anyway, is a Chapter 10 conversation; the habits here are yours to start today.