A model's score is the easiest thing in machine learning to get wrong — not by miscalculating it, but by believing it. This chapter builds the small set of tools that honest teams use to check their own numbers: why the most quoted metric quietly lies, the two sharper questions that replace it, the four-cell table every claim traces back to, how to judge models that predict numbers instead of categories, and the almost insultingly simple discipline that protects smart people from fooling themselves.
5 topics
By now you know how models learn (Chapter 3), what the two supervised shapes look like (Chapter 4), and what happens when there are no answer sheets at all (Chapter 5). What you don't yet have is a way to tell a good model from a well-dressed disappointment. Topic 15 planted the reflex — a score means nothing without a floor and a bar — and promised that the actual tools would arrive later. This chapter is later.
The working material is Plateful's payment-fraud detector from Chapter 5, and it was chosen on purpose: roughly one transaction in a hundred is fraud, and that imbalance is exactly where naive scoring falls apart. Five topics build the toolkit in layers. First, the case against accuracy — the number everyone quotes and the number that flatters worst when stakes are highest. Then precision and recall, the two questions that split "is it right?" into something you can act on. Then the confusion matrix, the four honest counts underneath every metric. Then a detour into regression, where "right or wrong" doesn't even apply. And finally the chapter's point: baselines and sanity checks, the discipline that turns all of it into protection.
The honesty toolkit — each layer fixes the blind spot of the one below
The confusion matrix — the full ledger
all four outcomes counted, with a price tag on each kind of error
Precision & recall — the two error kinds
splits "is it right?" into false alarms vs quiet misses
Accuracy — the first look
share of predictions right; blind to which class the errors land on
The baseline — the floor
what a zero-intelligence strategy scores; every number is judged against it