The Train/Test Split

At the next review, the data team puts up a slide about the delivery model: "off by about four minutes on data it has never seen." Iris stops on that phrase. Never seen? The model was built from the orders table — every number in it was tuned against those rows. How could anyone know what it does on orders it never saw, short of waiting weeks for new deliveries to happen?

The answer is simple, almost anticlimactic, and it is the most important honesty ritual in machine learning. Before training ever started, the team locked part of the table away. The model trained on one share of the rows and then sat an exam on the hidden rest. That ritual is the train/test split, and everything this book will ever say about how good a model is stands on it.

The full table10,000 labeled orders

→

Split before trainingseal about a fifth away

→

Training set~8,000 rows · the loop runs here

→

The exampredict the sealed ~2,000 rows

→

Honest scoreskill on the truly new

Why Not Grade on the Training Data?

The obvious way to measure a model is the one thing you must never do: score it on the rows it trained on. Think about what training was. The loop spent thousands of laps nudging the parameters to shrink the misses on exactly those examples. Of course the model does well there — doing well there is the literal thing the loop optimized. A score on the training data measures how thoroughly the model absorbed rows it has already seen. It says nothing about the only thing anyone actually wants: how it handles an order it has never met.

Here is the everyday version. A student re-solves the semester's homework problems and gets every one right. Impressive? Not yet — they might understand the method, or they might just remember the answers, and re-solving old problems cannot tell those apart. A real exam uses problems the student has never seen, because only new problems measure skill rather than memory. The training set is the homework; the test set is the exam. That one distinction — practiced problems versus fresh ones — is what the next page is built on.

The Split

So, before training begins, the team divides the table. A slice — often around a fifth of the rows — is held out and set aside: the test set. Everything else becomes the training set, and the loop from Topic 11 runs on it alone. The held-out rows take no part in training. No guess is made on them, no loss is measured on them, no parameter is nudged because of them. As far as the model is concerned, they do not exist.

Two small details make the ritual work. The split comes first, before the model does — so no information from the test rows can seep into the numbers. And the held-out rows are picked at random, not the last month or one district, so the exam is a fair sample of the world the model will face rather than one unusual corner of it.

The Exam

Training ends; the model is frozen, as Topic 04 promised it would be. Now the sealed rows come out. The model predicts each one — orders it has genuinely never seen — and each prediction is compared to the label that reality already wrote down. The result is the honest score: an estimate of how the model will do out in the world, available weeks before the world weighs in. That is the whole trick behind the slide. "Off by about four minutes on data it has never seen" is not a forecast or a boast; it is an exam grade, and now you know how the exam was run.

Notice what the exam does not do: it does not change the model. Not one parameter moves. The test set teaches nothing, improves nothing, fixes nothing — it exists only to measure. And a score is only as honest as the way it is read: "off by four minutes" is one blunt summary, and Chapter 6 opens the proper toolbox for scoring models — including the ways a summary number can flatter.

The Rule That Makes It Work

The ritual has one standing rule: the test set is used rarely, and it is never, ever trained on. The reason is subtler than it looks. Suppose the team tries ten model variants, checks each one against the test set, and keeps whichever scored best. No variant trained on the sealed rows — yet the choice of model was steered by them, ten times over. Peek by peek, the team fits its decisions to the exam questions, and the "honest" score drifts back toward flattery. It is Topic 12's leakage showing up in a new place: the exam leaking into the choosing. So data teams treat the test set like a sealed envelope — opened seldom, and never while deciding.

One flag before moving on: this two-way split is the simplified picture. Real teams slice finer — they carve out additional held-back sets so they can compare and tune variants without ever touching the final exam. The extra machinery belongs to a hands-on course; the principle you now have is the honest heart of all of it. Score a model only on data that had no part — direct or sneaky — in making it.