Baselines and Sanity Checks

The fastest way the smartest teams protect themselves from self-deception is almost insultingly simple: before celebrating any model, build the dumbest possible rival — a rule so simple it fits in one sentence — and demand the model beat it convincingly. This chapter has been circling that idea from the start: the cardboard sign that matched a 99%-accurate detector, the always-predict-40 guesser that shadowed the delivery model. This page turns the trick into a discipline, and then collects the rest of the chapter's lessons into one checklist.

Medicine solved this problem long ago. A new drug is never judged by "patients improved" — patients improve on their own, on hope, on rest. It is judged against a group given a sugar pill, because "improved" means nothing without "…more than the do-nothing group did". The dumb rival plays exactly that role for a model: it is machine learning's placebo control, the do-nothing group every claimed improvement must out-perform. That is the whole borrowing — from here on, models and their rivals.

The Dumb Rival, Formalized

The rival has a real name, and it is one of the most load-bearing words in the field: the baseline — the simplest strategy that could possibly be used, built in minutes, scored on the same held-out data as the model. Each kind of problem has its classic. For yes/no models, it is the majority-class guesser — always say the common thing, Topic 25's cardboard sign. For number-predicting models, it is the historical-average guesser from the last page. For forecasting tomorrow from today, it is "same as yesterday" — tomorrow's weather is today's weather, next week's orders are this week's.

What makes these floors useful is that they are surprisingly hard to beat. Zero intelligence, zero cost — and "same as yesterday" routinely embarrasses expensive forecasting projects, because most days really are like yesterday. A baseline is not a straw man set up to lose. It is an honest measurement of how much of the job was already easy, so the model gets credit only for the part that was hard.

The Margin Is the Real Result

That reframes what a score even is. A fake-review detector announcing "94% accurate" has told you nothing yet; the same detector announcing "94% against a 91% baseline" has told you everything — three points of that number were earned by the model, and ninety-one came free with the data. The margin over the baseline, not the headline, is the model's actual contribution, and mature teams report both numbers in the same breath, every time.

And sometimes the margin is zero. The model scores 91, the majority-guesser scores 91, weeks of work added nothing. Painful — but that is a finding, not a failure: it says the signal the team hoped for is not in this data, and it says so before the model ships, before customers depend on it, before another quarter is spent tuning. A zero margin discovered in week two is one of the cheapest results a data team can deliver. The expensive version is discovering it in production.

The Self-Deception Checklist

The baseline is one guard among five, and you have already earned the other four. Collected in one place, here are the classic ways teams accidentally lie to themselves — every one of which produces beautiful numbers and a useless model. Scored on training data: the model is graded on examples it memorized, Topic 13's exam-with-the-answer-sheet. Leaked features: a column that will not exist at prediction time smuggles the answer into the inputs, Topic 12. Accuracy on imbalanced classes: the headline measures the imbalance, not the skill, Topic 25. Peeking at the test set: re-checking against the held-out data over and over, tweaking until it agrees — which quietly turns the final exam into practice material, Topic 13's rule broken in slow motion. And no baseline: an impressive number floating free of any floor — this page.

Notice what the five have in common: none of them is lying, exactly. Nobody fakes a number; the number is real and the conclusion is false. That is what makes self-deception more dangerous than fraud — everyone in the room is sincere. So the smartest teams do not rely on sincerity. They walk every claim through the same gauntlet, in the open, before believing it:

Does it beat the dumbest rival — and by how much?→Baseline cleared

Was the score measured on held-out data — checked once, not until it agreed?→No self-grading

Would every feature actually exist at prediction time?→No leakage

If one class is rare — what is the score on the rare class alone?→Imbalance-aware

All four gates passed?→Now believe the number

Sanity Questions Anyone May Ask

Read the gauntlet again and notice something about its language. "What's the baseline?" "Was that score on held-out data?" "Would this feature exist at prediction time?" "What's the score on the rare class alone?" There is no mathematics in any of them. They are plain-English questions about the setup of a claim — and the setup, not the number, is where every item on the checklist hides. Low-tech questions catch high-tech mistakes, which is why the best data scientists ask them of their own work compulsively.

This is Iris's licence, in writing. When she asks the fraud team "what does always-say-fine score on this data?", she is not auditing anyone or trespassing on technical ground — she is contributing the exact rigor the field prizes, in the form it actually takes: attitude, not equations. A team that welcomes these questions from anyone at the table is a team whose numbers you can trust. A team that bristles at them has usually skipped a gate — and the chapter you have just finished tells you which one to check first.