Models Age
Three months after launch, the churn model quietly gets worse. No bug. No code change. Nothing broke, and nobody touched anything — yet the weekly win-back list catches fewer and fewer real churners. What moved was the world: Plateful launched a new menu, a competitor opened in three districts, and the holiday season rewrote everyone's ordering rhythm. The model, meanwhile, still lives in the spring its training data came from.
Topic 10 made a promise it is time to keep: a model is a photograph of the past, and the world walks out of frame. Nothing stops it, no matter how well the model was built. What professionals do about it is a pair of disciplines — monitoring to catch the decay, retraining to answer it — and together they are the reason ML is an operation, not a project. This page closes the chapter's lifecycle, loop and all.
Drift, Named
The decay has a name, and it comes in two kinds. The first: the data arriving today stops resembling the data the model trained on. New districts, a new customer mix, dishes that didn't exist in spring — the inputs themselves have shifted shape. This is data drift. The second kind is sneakier: the inputs look the same, but the relationship between inputs and outcomes has changed — the same quiet two weeks that meant "leaving" in spring might mean "on vacation" in August. The pandemic did this to every forecasting model on Earth in a single month: same features, suddenly wrong answers. Practitioners bundle both kinds under one umbrella word, drift.
A model cannot feel either kind happening. It keeps producing confident numbers from its frozen spring-time patterns — remember, trained then frozen — and the numbers simply mean less and less. Think of navigating with a city map from five years ago. Nothing is wrong with the map; the city grew. New neighborhoods are blank space, closed bridges still look open, and your navigation quality decays without the map changing at all. Map the analogy and keep it for one more section: the fix will not be a better map — it will be a process of re-mapping.
Watching for It
Since drift is silent, someone has to listen on purpose. The most direct check uses a gift the churn problem gives for free: truth arrives on its own. Every prediction is graded by reality thirty days later — the customers flagged as churners either ordered again or didn't. Compare predictions against arriving truth, week after week, and the model's real quality draws itself as a line on a dashboard. When the line sags, that is drift made visible.
The second check doesn't wait for truth: watch the inputs themselves. If the average days-since-last-order across all customers suddenly jumps, or a new district floods the table with customers unlike any the model trained on, the data has changed shape — a warning that arrives weeks before the quality line sags. Dashboards for both, and alerts when either crosses a line: this is monitoring, the same discipline ordinary software gets — servers watched for overload, apps watched for crashes — aimed at a target ordinary software doesn't have: accuracy.
Retraining
When the alarm fires — or, in disciplined teams, on a calendar regardless — the answer is retraining: fold the freshest months of examples into the table, and run the pipeline this chapter already built. The features from Topic 31 are recomputed, the tournament from Topic 32 re-runs with its baselines and its sealed exam, and the refreshed winner rolls out the safe 5%-first way from Topic 33. Nothing on that list is new; that is the point. Retraining is not heroics — it is the existing machinery, turned one more time with fresher ore.
Seen from far enough away, something satisfying comes into focus. Chapter 3's training loop — guess, measure, adjust, repeat — never actually ended. It just got slower: instead of a million laps in one night, one lap every few months, with deployment and monitoring inside the lap. The re-mapping process our city map needed turns out to be the training loop itself, running at the speed of the world. What the map called going stale, watching for it, and redrawing are, in plain terms, drift, monitoring, and retraining.
Budgeting for Forever
Back in Topic 05, this book made a claim it asked you to hold onto: an ML feature is a pet, not a stone. You have now met every item on the feeding schedule, so let's write the line item out properly. Monitoring: dashboards watched and alerts answered, every week the feature is alive. Retraining: the pipeline re-run on schedule or on alarm, gates and all. And occasionally, re-framing: when Plateful's product changes enough, the framing card itself — what counts as churn, what action follows — has to be renegotiated, and the loop restarts from Topic 30.
That cost is permanent. It should appear, explicitly, in the plan for every ML feature anyone ever pitches you — and it usually doesn't, because launch day feels like a finish line and the budget conversation stops there. Be the person in the room who adds the line item. The chapter's opening figure — frame, data, features, train, evaluate, deploy, monitor — has now been walked end to end, and its last arrow bends back to the start. That bend is not a decoration. It is the shape of every ML feature that stays alive.
- "A deployed model keeps its launch quality." Its quality is tied to how much the world still resembles its training data — and that resemblance decays. The model doesn't change; its relevance does.
- "If the model degrades, the team built it badly." Drift hits well-built models exactly as hard; no amount of craftsmanship freezes the world. The only real failure is not watching for it.
- "Retraining means starting the project over." It is the routine maintenance cycle: fresh examples through the existing pipeline — same features, same tournament, same gates, same safe rollout. Maintenance, not demolition.
- "With enough effort up front, you can build a set-and-forget model." No up-front quality stops the world from changing. Only operations — monitoring and retraining, running for the feature's whole life — keep a model good.
- Ongoing cost is the single most-forgotten item in ML planning. You now budget for monitoring and retraining by reflex — which puts you ahead of a surprising share of real product plans.
- The lifecycle is complete: the chapter's opening figure has been walked end to end, and you know why its last arrow bends back to the start. That loop is what "we have an ML feature" actually commits a team to.
Knowledge Check
Three months after launch, the churn model's predictions get worse — with no bug and no code change. What is the most likely explanation?
- The model's parameters slowly corrupt as the file is used in production
- Drift — the world stopped resembling the training data
- Customers learned to fool the model on purpose
- The tournament must have been scored dishonestly
During an unusual summer, the same quiet two weeks that used to mean "customer is leaving" now usually means "customer is on vacation." Which kind of drift is this?
- The relationship changed — same inputs now mean different outcomes
- Data drift — the incoming data itself has changed its shape and mix
- Leakage — a feature is quoting the answer
- Overfitting — the model memorized its training rows
How does the team actually notice the churn model is decaying?
- The model reports falling confidence in its own predictions
- Customer complaints eventually reveal the problem to the support team
- Grade predictions against arriving truth; watch inputs for shape-shifts
- By re-running the original held-out exam from the tournament every few months
What does retraining the churn model involve?
- Renegotiating the framing card and rebuilding the project from scratch
- Nothing — the model absorbs fresh data automatically while serving
- Fresh examples through the existing pipeline — same gauntlet and rollout
- Manually adjusting the model's parameters to match the new season
A pitch for a new ML feature ends its budget at launch day. What is missing?
- A bigger tuning budget up front, so the model launches too good to ever decay
- A backup copy of the model in case the original is lost
- The permanent line item — monitoring, retraining, occasional re-framing
- A decision about whether the model serves in batch or real time
You got correct