Messy Data

Iris drops by the data team expecting to watch models being built. Instead, a data scientist is spending the morning on two spellings of the same district — "Riverside" and "River Side" — and hunting down orders that appear in the table twice. Iris asks, carefully, when the real work starts. The answer: this is the real work.

The training table from Topic 06 looked tidy on the screen, but data fresh from the logs never arrives that way. Real recorded data is full of holes, duplicates, and outright nonsense — and since the model learns whatever is in the table, every flaw left in becomes a flaw learned. Cleaning isn't housekeeping before the important part. It is the important part.

What Mess Looks Like

Mess comes in a few recurring shapes. Missing values: the courier's phone lost GPS in a tunnel, so that delivery has no distance — a hole in the row. Duplicates: a customer double-tapped the order button, and one real order became two identical rows. Impossible entries: a delivery recorded as crossing the whole city in 4 minutes — some bug wrote a wrong timestamp, and the log faithfully kept it. And inconsistent formats: one district entered as "Riverside" here and "River Side" there, which to a computer are two different places.

None of this comes from carelessness. Remember where the data comes from (Topic 08): apps logging themselves as a side effect of working. Phones lose signal, buttons get double-tapped, software has bugs, and two versions of an app spell things two ways. The logs record it all without judgment — that's their job. Judging is the next job, and it belongs to people.

Why the Model Can't Shrug It Off

You just read "4 minutes across town" and snorted. That snort is common sense — a lifetime of knowing how cities, traffic, and scooters work. A model has none of that. Topic 04 showed you what a model is: numbers plus a fixed calculation. There is no place in the file where skepticism could live. During training, every row counts as truth.

So garbage rows don't get ignored; they get learned, with full confidence. The impossible 4-minute delivery quietly teaches "crossings can be that fast", and every prediction shifts a little toward it. The duplicated order counts twice, making its pattern look twice as common as it was. Small lies in the table become small lies in the model — and nobody wrote a bug, because the model did exactly what training asks: learn what the examples say.

What Cleaning Actually Means

Fixing this is called data cleaning: going through the table and deciding, case by case, what each flaw deserves. Drop the impossible row entirely — it describes no real delivery. Merge the duplicates into one. Fill the missing distance with a sensible stand-in, such as the typical distance for that route — or drop that row too, if guessing would do more harm. Rewrite "River Side" as "Riverside" everywhere. Thousands of small judgment calls, each one shaping what the model will believe.

Think of cooking from a market basket rather than a meal kit. Before anything touches the pan, you wash the vegetables, trim the stems, and throw out the bruised ones — and skipping that step doesn't save time, it just moves every flaw into the dish. Cleaning data is the same pass over the ingredients, and just like in the kitchen, it takes taste: which rows are bruised, and which are merely unusual, is a judgment, not a lookup. A 74-minute delivery in a snowstorm is unusual and true — drop it and the model never learns what snow does.

The Honest Ratio

Here is a number worth carrying into meetings: practitioners routinely report spending more of their time preparing data than building models with it. Not because they're slow — because that's the actual shape of the work. The glamorous part, training, often runs overnight without supervision. The human hours go into the table.

And it never fully ends. A new data source arrives with its own quirks. An app update changes how a field is written. A new phone model logs GPS differently. Each change mints fresh mess, so cleaning runs as a standing pipeline — a permanent part of the machinery, not a one-time scrub before the project starts.