The Limits of Prediction
A data scientist shows Iris a dashboard curiosity: across the summer, Plateful's ice-cream orders and the city's pool-accident reports rise in near-perfect sync. Week by week, the two lines move together like partners. Nobody in the room proposes banning dessert to save swimmers — the correct explanation is obvious once said aloud. Hot weather drives both. But hold the moment, because a model looking at those two lines sees exactly what the room saw before anyone spoke: two things that move together, and nothing more.
That is the subject of this page: the two limits baked into every prediction, which no accuracy score can fix. Models are correlation engines — Topic 11 showed that training never asks why, only what fits — so a model knows what moves together but not what causes what. And second: the moment a model's predictions start triggering actions, those actions change the world the model was predicting. These are the last big ideas of the book, and they are the ones most often missed in rooms where dashboards move budgets.
Correlation Machines
Two things that reliably move together are said to be correlated — ice cream and pool accidents are a textbook case. What links them is a third thing standing behind both: summer. Statisticians call this the third-variable pattern, and it is everywhere once you look — swimsuit sales correlate with sunburn, umbrella sales with traffic jams. The pairs are honest signals; they just are not cause and effect.
Here is the uncomfortable part: nothing in training distinguishes the two. A model rewarded for predicting pool accidents will happily lean on ice-cream sales, because in the training data the lean pays off. The model learned "predicts"; a human reading the dashboard hears "causes". That gap — between what the machinery found and what the sentence sounds like — is where the trouble on this page lives.
Why the Difference Bites
For pure prediction, correlation is fine — that is the everyday business of ML, and it works. The churn model from Chapter 7 leans on behavioral features to flag who is at risk of leaving, and flagging someone does not cause anything; the model is a smoke detector, not a fire starter. Trouble begins with intervention — acting on a feature as if it were a lever. Suppose the churn model shows that customers who use discounts tend to stay. "So give everyone discounts!" says the meeting. That plan quietly assumes discounts cause staying — a claim the model never established. Maybe loyal customers simply hunt for deals; then the discounts change nothing and burn margin, and the correlation was real the whole time.
"Will acting on this change the outcome?" is a different question from "does this predict the outcome?", and it needs different evidence. The gold standard is a controlled experiment — in product work, an A/B test: give the discount to a random half and compare. One sentence, but load-bearing: randomness breaks the third variable's grip, which no amount of dashboard staring can do. Think of the rooster who notices his crowing perfectly precedes every sunrise — flawless correlation, spotless prediction record, wrong causal story. The farm can laugh at him harmlessly right up until the day it acts on his theory and keeps him up all night to hurry the dawn. Acting is when a wrong causal story starts sending bills.
Predictions That Eat Their World
The second limit is stranger. Deploy a model and its outputs become actions — rankings, alerts, patrol routes — and those actions change people's behavior, which changes the data, which is what the next model trains on. Topic 45 drew this loop for bias; it turns out to be general. Every deployed model is quietly rewriting its own future textbook.
Three live examples. A recommender suggests dishes, people order what is suggested, and next month's data "confirms" those dishes are what people want — the model is partly predicting taste and partly manufacturing it. A fraud model blocks known tricks, so fraudsters change tricks — the drift of Topic 34, except here the model itself caused the world to shift. And predictive-policing systems direct patrols to certain areas; more patrols find more incidents there, and the statistics come back confirming the prediction that sent them. In each case the model's report card is contaminated by the model's own influence.
The Working Posture
None of this says predictions are worthless — it says they have a job description. Use them to allocate attention: which customers to call, which transactions to review, which orders to double-check. Attention is cheap to misallocate and predictions are good at pointing it. Become skeptical exactly at the moment a prediction is used to justify an intervention — that is when "predicts" is being quietly upgraded to "causes", and the upgrade needs an experiment, not a dashboard.
And for any deployed model, add the question Iris now carries into every review: how does this model's own output feed back into its future data? It sounds paranoid the first time and turns out to be plain literacy — every recommender, fraud filter, and risk score runs a feedback loop; the only variable is whether anyone is watching it.
- "The model predicts well, so it understands what drives the outcome." It found co-movement. "Drives" is a causal claim that training never tested — the rooster predicts the sunrise flawlessly.
- "A strong enough correlation is close enough to causation to act on." Acting is an intervention, and interventions need experimental evidence. Otherwise the hidden third variable sends the bill — discounts for deal-hunting loyalists.
- "A model's accuracy has nothing to do with it being deployed." Deployed models trigger actions that change behavior and thus their own future data. The act of predicting changes the predicted.
- "Feedback loops are exotic edge cases." Every recommender, fraud filter, and risk score runs one. The question is never whether the loop exists — only whether anyone is watching it.
- "The model found what causes churn" is the most seductive misread of a good dashboard, and it gets said in rooms where it moves real budgets. You can now catch it — and name the experiment that would settle it.
- The feedback-loop lens explains systems you live inside every day — feeds shaping taste, prices chasing demand, patrols confirming forecasts — and completes this book's honest picture of what deployed ML is.
Knowledge Check
Ice-cream orders reliably predict pool accidents. What is the correct reading?
- Ice cream itself contributes to the accidents, since the prediction keeps working
- A hidden third factor drives both, so the pair correlates without causing
- The pattern is a coincidence and will vanish next summer
- A model should refuse to use a feature like this
The churn model shows discount users stay longer, and the team proposes discounts for everyone. What is the flaw?
- The churn model must be inaccurate
- The correlation needs more training data before acting
- The plan treats a predictive feature as a cause, which was never tested
- Discounts can never influence whether customers stay
What does a controlled experiment (an A/B test) add that even a huge pile of observational data cannot?
- Random assignment cuts hidden third variables out of the comparison
- Far more data points than the dashboard has
- A more accurate prediction of the outcome
- Complete certainty about what causes what
Patrols are sent where a model predicts incidents; more patrols record more incidents there; the next model trains on those records. What is happening?
- The model is being genuinely proven right by each round of new incident statistics
- Ordinary data drift, solved by routine retraining
- A sign the original model was poorly trained
- A feedback loop — the model is manufacturing the data that confirms it
You got correct