Training and Tuning

The features are ready, and Iris expects the team to finally train the churn model — singular, definite, the model. What happens instead surprises her: over the next two weeks they train dozens. Different model types from Chapter 4, different settings, different subsets of the features, all racing on the same track. When she asks which one is the real model, the answer is cheerful and unsentimental: "whichever wins."

This page is what "we're experimenting" actually means when a data team says it. Iris was braced for a eureka moment — the brilliant insight that cracks churn. She finds a well-run tournament instead: patient, systematic, and slightly boring on purpose. Training itself you already know from Chapter 3 — the guess-measure-nudge loop that sets the model's numbers. The new word is tuning, and it is the tournament's engine.

The Knobs on the Machine

Here is a distinction the whole page turns on. The numbers inside a model — the learned parameters from Topic 04 — are set by training; no human touches them. But every model also has settings chosen before training starts: how deep a decision tree may grow, how many trees a forest gets, how big each training nudge is. These are called hyperparameters — an intimidating word for a plain thing. They are dials on the machine, not knowledge in it. Training fills the machine with knowledge; the hyperparameters decided what kind of machine you switched on.

Think of perfecting a bread recipe. Oven heat and flour ratio are decided before anything bakes — those are the settings. What the dough actually does in the oven — the rise, the crumb — unfolds inside the settings you chose. And nobody derives the perfect oven temperature from theory: you bake many small loaves, taste them blind, keep notes, and adjust. That repeated bake-and-taste is exactly what tuning is: the settings are hyperparameters, and the blind tasting is honest scoring.

Tuning Is Systematic Search

Tuning, then: pick a combination of settings, train, score the result fairly, write it down, adjust, repeat. No step in that sentence is clever, and that is a feature — teams even run automated sweeps that try whole grids of setting combinations overnight. What separates a professional tournament from weeks of aimless fiddling is the writing-down: every attempt, its settings, and its score go into one record. Intuition proposes the next thing to try; the leaderboard disposes.

Read the board the way Iris learns to. The baseline is Topic 29's dumb rule — the floor that gives every other number meaning. Each contender lists its settings next to its score, because the settings are the experiment: the winner is not a different idea from the small forest, just different positions of the same dials, found by search. And every score comes from the same exam, which brings us to the rules.

A Fair Race

A tournament is only as good as its rules, and this one has three. First: every contender is scored on the same held-out data — Topic 13's exam — never on rows it trained on. Second: every score is read against the same baseline, so "0.69" means "0.28 above the dumb rule", not just a number floating in space. Third, and easiest to miss: the final exam stays sealed until the very end. Compare contenders on one held-out set often enough and the team slowly learns that set's quirks — choosing whatever happens to shine on those particular rows. So the race is scored on one held-out slice, and a second, truly untouched slice is opened exactly once, to confirm the winner before it ships.

That last rule means the honest picture has three parts, not two — training data, a comparison slice for the tournament (practitioners call it a validation set), and the sealed final test. This book has worked with the simpler two-way split until now, and that simplification ends here: the two-set picture is the honest core, and the third set is what keeps it honest when dozens of contenders compete. Discipline, not brilliance, is what makes tournament results real.

Knowing When to Stop

The first week of tuning gains six points over the baseline. The next gains one. The one after that gains 0.2, and costs three days of compute and attention. Improvements shrink — reliably, everywhere, always — and past a certain point the tournament stops finding better models and starts finding lucky ones: squeeze a leaderboard long enough and you are learning the comparison slice's quirks, the very thing the sealed final exam exists to catch. There is no single correct configuration waiting to be discovered; there is a good-enough region, and then there is noise.

Which is why the stop rule was written before the tournament began. Back in Topic 30, the framing card fixed the bar — the score that makes this feature worth shipping. The tuned forest clears it, so the tournament ends, not because nobody could squeeze out another 0.3, but because shipping a model that clears the bar beats squeezing one that gleams. And notice whose call that is: the bar came from the product side. Iris does not tune models, but she owns the number that ends the tournament.