Features and Signals

At a sprint review, Iris watches two versions of the delivery model compared side by side. Same ten thousand orders, same team, same training loop from the last page — yet one predicts within a few minutes and the other might as well roll dice. The difference is not effort, and it is not a cleverer algorithm. Someone gave the two models different columns to look at.

The columns a model sees — its features, from Chapter 2 — decide what it can possibly learn. What the model sees is what it can know; everything else does not exist for it. And features come in three kinds. Some carry signal — a genuine connection to the answer. Some carry only noise. And one kind carries a lie. This page is about choosing the model's eyes, and about telling the three kinds apart.

Signal: Features That Genuinely Relate

Distance is a signal feature for delivery time. Longer trips genuinely take longer — in every district, every season, for every restaurant. The evening rush is another: 7 p.m. orders genuinely wait longer than 3 p.m. ones, because kitchens and roads are fuller. These connections exist in the world, so they exist in the table, row after row after row.

Watch what the training loop does with such a column. Any nudge that makes the model trust distance more also shrinks the loss — not just on a few rows, but across thousands. So, lap by lap, the parameters attached to distance accumulate weight, and the weight is earned honestly: it reflects something real. That is all "signal" means — a feature the loop can convert into genuine predictive skill.

Noise: Features That Don't

Now add a column with no connection to the answer — say, the customer's favorite app color theme. It cannot help. The danger is that it can still seem to help. In any table of ten thousand real orders, pure chance guarantees small flukes: perhaps dark-theme users happened to get their food ninety seconds faster this spring. A coincidence — but the loop has no way to know that. It only sees that trusting the column shrank the loss on these particular rows, and the loop rewards whatever shrinks the loss.

One junk column is survivable. Pile on dozens, and you have handed the model a haystack of coincidences to mine, and mine it will. Learning coincidence as if it were law is a failure important enough to get its own name two pages from now, on the overfitting page. For now, the working rule: a feature should be in the table because a human believes it relates to the answer — not because it happened to be lying around.

The Feature the Model Can't Have

The third kind is the strangest, because it looks like the best feature in the table. Consider the courier's actual route — which streets they really took. For every past delivery, that column is filled in, and it practically gives away the answer: know the true route and you nearly know the minutes. Train on it and the loss falls through the floor. The team celebrates.

Then the model goes live, and the column is empty — because when a customer places an order, the route has not happened yet. The model learned to lean on information that only exists after the thing it is predicting. This is called leakage: a feature from the future leaking into training. A leaked feature makes a model brilliant in the lab and useless in production, and the test for it is one question — will we have this column at prediction time?

Humans Pick the Menu

Here is a way to hold all three kinds at once. Think of features as witnesses to an incident. One witness saw it happen. One is repeating third-hand gossip. And one confidently recites the verdict — because they heard it announced after the trial. You want the first, you must ignore the second, and you must never let the third testify: knowing the outcome proves nothing about predicting it. Signal, noise, leakage — three names for three fates a column can meet.

Who decides which witnesses get called? In classical machine learning: people. Choosing features is human judgment about the world — knowing that rain slows couriers, that a stadium match backs up every kitchen nearby, that the route column arrives too late to use. There is craft here, and it has a name and its own page in Chapter 7: feature engineering. Deep learning partially automates the choosing for photos and text — Chapter 8 will say exactly how, and how much, honestly. But for the tables that run most businesses, the model's eyes are picked by hand. Which gives you your second reflex question about any ML idea, right after Chapter 2's "where would the labels come from?" — what features would we even use?