Overfitting and Generalization

Monday's numbers arrive for a new, more elaborate delivery model. On the training data it is astonishing — barely a minute off, order after order. On the test set it is off by eleven. Iris braces for alarm around the table, and instead the data team just nods, the way you nod at an old acquaintance. They have seen this exact face before.

The model has memorized its homework instead of learning the subject. That failure has a name — overfitting — and it is the central disease of machine learning: the reason the last page's split exists, the trap behind most spectacular demos that die in production, the answer to "why not just make the model bigger?". If you keep one page of this book, keep this one.

Two Ways to Score Well on Homework

Go back to the loop from Topic 11. It has one instruction: shrink the loss on the training rows. Here is the uncomfortable fact this page turns on — there are two entirely different ways to obey that instruction. A model can find the real pattern: distance and rush hour genuinely drive delivery time, so trusting them shrinks the bill honestly. Or it can memorize the answers: absorb the quirks of these particular ten thousand rows so exactly that each guess lands near its label for reasons that hold nowhere else. Both strategies shrink the loss. The loop cannot tell them apart, because the loop only reads the bill.

It is the two students from the last page, one step darker. One understood the method. The other got hold of last year's answer sheet and memorized it — flawless on those exact questions, lost the moment a single word changes. On homework, the two are indistinguishable; both turn in perfect pages. Only new problems expose the difference. Memorizing the answer sheet is overfitting; understanding the method is what we are actually after.

Overfitting

Now the mechanism, precisely. Any real table is full of accidents. One rainy Tuesday, every order from the pier district ran late. A new courier had a bad first week. Topic 12's noise columns hide a dozen small flukes. None of these are patterns of the world — they are the paperwork of this particular table, things that happened once and left a mark in the rows. A model with enough flexibility, given enough laps, will absorb them anyway, because absorbing them shrinks the training loss a little more. Fitting the training examples so closely that their accidents get learned as rules — that is overfitting.

Notice what overfitting is not: it is not a bug, not a crash, not the loop malfunctioning. The loop did exactly what it was asked — it shrank the loss on what it saw, by any means available. The objective itself creates the trap. That is why the data team nodded instead of gasping: an overfit model is not broken machinery, it is training working as specified on a goal that was always slightly wrong. The goal was never really "do well on these rows." It was the next section.

Generalization

What the team actually wants is a model that performs well on cases it has never seen — because what it learned is the pattern of the world, not the paperwork of the table. That ability has a name too: generalization. It is the true goal of all machine learning, and the awkward part is that training cannot pursue it directly. The loop can only shrink loss on rows it has; "do well on rows that don't exist yet" is not something arithmetic can chase.

What you can do is measure it — and you already know how. This is the train/test split earning its keep. The training score says how well the model absorbed its homework; the test score estimates how well it does on the genuinely new; and the gap between them is the tell. Model A: 99 on training, 71 on test — a 28-point gap, screaming memorization. Model B: 91 and 89 — a model whose learning mostly survives contact with the unseen. Without the split, both models would have shown Iris one number each, and the worse one would have shown the prettier number.

What Feeds It, What Helps

Two ingredients feed overfitting, and they usually arrive together. The first is flexibility: a model with many parameters can bend into very intricate shapes, and an intricate shape can wrap itself around every fluke in the table. The second is scarcity: the fewer examples there are, the more each accident looks like a pattern — one rainy Tuesday out of forty Tuesdays is a fluke, one out of three is a "rule". A very flexible model plus a small table is the classic recipe, which is why "just make the model bigger" is not free: flexibility is also the capacity to memorize.

The cures are the same ingredients, reversed. More examples — honest, varied ones — are often the whole cure: in a table with two hundred rainy Tuesdays, the coincidences average out and stop looking like law. Choosing a deliberately simpler model is the other lever: less flexibility means less room to store paperwork, so the loop has to spend its capacity on the patterns that repeat. Practitioners also make training itself charge a fee for complexity, a family of techniques called regularization — deep-dive material; one day a hands-on course will show you the knobs, and it is enough for now to know the knobs exist. Matched-to-data beats maximal, every time.