How LLMs Are Made
The last page left a gap. The loop predicts likely next words — yet the assistant Iris talks to answers questions, follows instructions, and stays unfailingly polite while doing it. Your phone's autocomplete does none of that. Something happens between "raw next-word predictor" and "helpful assistant", and this page is that something: a recipe with three training stages. Knowing them turns every "how can it possibly do that?!" into "ah — stage two."
The real term first: a large language model (LLM) is a next-word predictor of enormous size — the Topic 40 machine with billions of the learned numbers you met in Topic 04. The recipe below is told version-free on purpose: it was true of the first modern assistants and it will be true of the next ones, whatever they are named. And it maps, stage for stage, onto how an actor becomes an actor: years absorbing how people talk, then drama school, then live audiences. Keep that in your pocket — each stage cashes part of it.
Stage 1: Pretraining — Read Everything
Stage one is Topic 40's observation, executed at full scale. Any text is its own answer sheet, so the model trains on next-word prediction over a huge slice of the internet — books, articles, forums, manuals, code. This is where Topic 36's three ingredients earn their chapter: data at civilization scale, warehouses of GPUs, and the accumulated craft of training giant networks, all multiplied together. The actor's first years work the same way — long before any stage, they absorb how people actually speak, from everyone around them. This stage is called pretraining, and it is where almost all of the model's size and cost lives.
What comes out is called a base model, and it is a strange creature: brilliant and unhelpful at once. It speaks fluently, because fluency is exactly what next-word prediction rewards. It carries a compressed image of everything it read — facts, styles, arguments — baked into its numbers. But ask it "What are your delivery hours?" and it may reply with three more questions, because on the internet, questions often travel in lists. It continues text; it does not serve people. Nobody would ship it as a product.
Stage 2: Instruction Tuning — Learn the Job
Stage two bends the predictor's habits from "continue the text" to "answer the person". The method is one you already own: supervised learning, Chapter 4 in a cameo role. People write curated example dialogues — "given this request, this is a good response" — thousands upon thousands of them, covering questions, summaries, rewrites, refusals. The model is then trained further on these examples, the same nudge-the-numbers process as ever, until the likeliest continuation of a request is a response in that helpful shape. This stage is called instruction tuning.
This is the actor's drama school: scripted scenes, played over and over, until the raw talent takes direction. Notice what the stage runs on — examples written by people. Topic 08's labeling workforce returns here in force; behind every polished assistant sits a real global industry of writers and reviewers producing the dialogues it learned its job from.
Stage 3: Human Feedback — Learn the Manners
Stage three is the promise Topic 24 made you: reward-based learning, cashed in. The model generates several candidate answers to the same request, and people rank them — this one is more helpful, that one is too curt, this one refuses when it should not. Those rankings become a reward signal, and the model is trained to earn more of it. The technique is called RLHF — reinforcement learning from human feedback — and that is the whole meaning of the phrase: rewards, from people ranking answers, shaping behavior.
Helpfulness, tone, tact, and the model's refusals live largely in this stage — its live audiences, whose reactions shape the performance night after night. Put the three stages together and you get a superb performer: absorbed the language of the world, trained on scripted scenes, polished in front of crowds. And notice the question that was never asked of this performer at any stage: did they verify their lines?
What the Recipe Implies
Three consequences follow directly, and they set up the rest of the book. First: the model learned from us, so it inherits us. Pretraining text is human text, with human brilliance and human bias baked in together — Topic 10's garbage-in lesson at planetary scale, and Chapter 10 takes up what that means when these systems make decisions about people. Second: the model does not browse the internet when it answers. Pretraining bakes its text into the learned numbers, and Topic 04 told you what a trained model is — a frozen file. By default it recalls a compressed past, not today; tools that bolt on live search exist, but they are additions to the recipe, not part of it.
Third, and most important for the next page: nothing in the recipe installed a fact-checker. Stage one rewarded likely text. Stage two rewarded helpfully shaped text. Stage three rewarded text people preferred. At no point did any stage check the text against reality — there is no step for it, no verified-facts ingredient, nothing. What happens when a machine optimized for fluent, helpful, well-mannered text meets a question it has no grounded answer to? That is Topic 42, and it is the page every user of these systems needs most.
- "LLMs are programmed with knowledge and rules." Nothing was typed in. Grammar, facts, and manners were all learned — fluency from pretraining, task-following from instruction tuning, manners from human feedback. There is no rulebook inside, only the learned numbers.
- "The model browses the internet when answering." By default, no. Pretraining text is baked into the frozen file (Topic 04); the model recalls its compressed past, which is why it can be blank on last month's news. Tools that add live search exist — as attachments, not as the model.
- "One training stage did everything." Three distinct stages contribute three distinct things: fluency, task-following, manners. When a model behaves oddly, the odd behavior usually traces to one of them — too agreeable is a stage-three story, not a stage-one one.
- "Trained on the internet, tuned on example dialogues, shaped by human feedback" — you can now explain the assistant on your phone at a dinner party, correctly, in one sentence per stage.
- Stage three is where the model's personality comes from — real people ranked answers, and the model learned their preferences. Chapter 10 asks the obvious next question: whose preferences?
- Knowing that no stage installed verification is the single best preparation for the next page — and for every confident answer an LLM will ever give you.
Knowledge Check
What comes out of pretraining, before the other two stages run?
- A helpful assistant that already answers questions politely and follows instructions
- A base model — fluent, full of absorbed text, unhelpful
- A searchable archive of every page the model read
- A quick warm-up pass that the real training then replaces
What turns a text-continuing base model into something that answers requests?
- Engineers program in rules for how to respond to questions
- A second, much larger pass over the same internet text, repeated until helpfulness emerges
- People rank its answers until it learns to answer at all
- More training, on example dialogues pairing a request with a good response
Where do the assistant's manners — tone, helpfulness, when to decline — mostly come from?
- Human feedback — people rank candidate answers, creating a reward signal
- Pretraining — polite text is simply what the internet contains
- A filter bolted on after training that deletes rude sentences whenever they are generated
- A hand-written etiquette rulebook that ships inside the model
Which of these was never installed at any of the three stages?
- Fluency in human language, across every style and topic it read
- The habit of following instructions
- A step that checks generated text against reality
- A preference for a helpful, polite tone
You got correct