Fairness and Bias

Plateful ships a new courier-assignment model, and it works: deliveries get faster. Then someone notices what "works" is doing. The model gives the best dinner-rush slots — the high-tip, high-volume hours — to couriers with long order histories. In this city, who has a long history skews sharply along neighborhood lines, so the good shifts flow to some neighborhoods and away from others. Iris pulls up the feature list expecting to find the culprit, and finds nothing: the data never mentions neighborhoods, ethnicity, or anything of the kind.

Nobody wrote a biased rule, and nobody had to. Topic 10 made this promise about bad data in general — no villain needed — and this page is where the promise lands on actual people. Bias, in the ML sense, is a systematic tilt in a model's outputs against some group of people. It gets in through examples, not intentions: the model distilled yesterday's patterns and shipped them as tomorrow's decisions. This page covers how that happens, what it has already done out in the world, and why "make it fair" turns out to be a harder instruction than it sounds.

How Does Bias Get In Without Anyone Typing It?

Three routes, all of them familiar machinery from earlier chapters, now with people on the receiving end. First, unrepresentative examples: if some group appears rarely in the training data, the model learns them poorly — and is simply worse, more often wrong, for them. Second, labels that encode past judgment: when the "right answers" in the training table are old human decisions — who got hired, who got a loan — the model learns to imitate those decisions, including every prejudice baked into them. Training treats the labels as ground truth; history gets graded as if it were correct.

Third, and this is what happened to the couriers: proxy features. A proxy is an innocent-looking variable that moves together with a sensitive one. Order history stands in for neighborhood; neighborhood stands in for demographics; a name or a shopping pattern can stand in for either. Topic 12 taught that models grab any feature that carries signal — and they cannot tell a legitimate signal from a proxy for something you never meant to use. Deleting the sensitive column does not delete the pattern.

A model, in other words, is a mirror polished on yesterday. It reflects the world that produced its data with perfect diligence — and if that world was tilted, the reflection is tilted too, except now it is automated, fast, and applied to everyone. The rest of this page is about what that reflection has already done, and what teams do about it.

Historical decisionswith their tilt

→

Training tablehistory as 'truth'

→

Modellearns the pattern

→

Automated decisionsthe tilt, at scale

↻

Tomorrow's "history"feeds the next model

Real Cases, Told Plainly

This is not a hypothetical risk. A large company built a résumé-screening tool, trained it on a decade of its own hiring decisions, and found it had learned to downgrade résumés from women — it had faithfully learned what yesterday's workforce looked like, and the tool was scrapped. Face-recognition systems were measured with error rates many times higher on darker-skinned faces than lighter ones, because darker faces were scarce in the training photos. Credit and risk scores have repeatedly been found echoing decades of unequal lending and policing — the labels encoded the history, and the models passed it forward. Each of these is a reported case, and each is one of the three routes above playing out on real people.

Notice what the cases have in common: no one set out to build a discriminatory system. That is precisely the lesson. Given training data made of human history, bias is not the rare malfunction — it is the default outcome. The absence of bias is the thing that takes deliberate work.

Why "Fair" Is Not One Number

So the team says: fine, we will make the courier model fair. Immediately they hit a question that math cannot answer. Fair how? One reasonable definition: the model should be wrong about people at the same rate in every group — equal error rates. Another reasonable definition: every group should get the good dinner slots at the same rate — equal approval rates. Both sound like "fair". And when groups start from different historical positions, the two definitions genuinely conflict: fixing one measure can worsen the other. Mathematicians have shown you often cannot satisfy both at once.

Which means fairness involves choosing what to equalize — and that is a values-and-product decision that only looks like a math problem. Topic 26 asked "which error hurts more?" when tuning a threshold; this is the same question, elevated: whose mistakes are we willing to accept, and whose are we not? A team that claims its model is "fair" without saying fair by which definition has not finished the sentence.

What Responsible Teams Do

Mitigation is real work, not a disclaimer paragraph. It starts with measurement: take the confusion matrix from Topic 27 and slice it per group — error rates for each neighborhood, each demographic, side by side. A model can post a beautiful overall accuracy while being twice as wrong for one group; only the sliced view shows it. Then teams audit features for proxies (what does order history actually move with?), collect data where groups are underrepresented, and re-check after every retraining, because Topic 45's own loop above means the tilt can creep back in.

And one thing stays constant: a human owns the decision. Not "the model decided" — someone chose the training data, someone chose the fairness definition, someone signed off on the threshold. For Iris, this is the seat she is already sitting in: features, labels, and thresholds are product decisions, and they are exactly where bias gets caught or gets shipped.