Why Did the Model Say No?
Plateful launches a "pay later" option, guarded by a risk model. A loyal customer — three years, hundreds of orders — applies and is declined. She writes to support, reasonably, asking why. Support asks the data team. The data team looks at the model's output and finds… a number below a threshold. No reason, no factors, no sentence anyone could send her. The verdict arrived scoreless and reasonless, and "the computer said no" is what support ends up not-quite-saying.
For a dinner recommendation, that opacity costs nothing — if the suggestion is bad, you scroll past. For credit, hiring, housing, or parole, "the computer said no" is not an acceptable sentence, and in a growing number of places it is not a legal one either. This page is about explainability — whether a model's decision can be traced to reasons a human can state — what it costs, who needs it, and when it should win over raw accuracy.
Readable by Design vs Black Box
Topic 19 introduced the split; now it gets consequential. Some models show their reasoning natively. A small decision tree is its own explanation — follow the branches and you can read the verdict as a sentence: declined because account age is under six months and order frequency dropped. A simple linear model is similar: each feature has a visible weight, pushing the score up or down. These models are readable by design.
Then there are the models that earn their accuracy by combining hundreds of trees (Topic 20) or millions of learned numbers in layers (Topic 35). Nothing in them corresponds to a reason; the answer emerges from arithmetic no human can follow. These are black boxes — not because anyone hides them, but because there is nothing shaped like an explanation inside. Chapter 4 named this trade honestly: accuracy is often bought with opacity. This page is where the price gets real, because Plateful's pay-later model is a black box, and the customer is still waiting for her sentence.
Explaining the Unexplainable
Black boxes are not hopeless. A family of explanation tools has grown up around them, and what they do is best said plainly: for one specific decision, they estimate which features pushed it, and in which direction. Run one against the customer's decline and you get something like the right-hand column above — payment history pushed down hard, account age pushed down a little, order volume pushed up but not enough. That is a usable answer where before there was a wall.
One honest asterisk, and it matters: these tools produce approximations of the model's behavior, not a transcript of its reasoning. They probe the box from outside — nudge the inputs, watch the output — and summarize what seems to drive it. The summaries are genuinely useful and occasionally misleading, and a team that treats them as gospel has replaced one kind of overconfidence with another. Useful, not gospel: that is the calibration.
Who Needs the Explanation?
Picture two rejected mortgage applications. One says: "Declined." The other says: "Declined — your income-to-payment ratio is below our bar; here is the number, and here is what it would take to clear it." Same verdict. The first is a wall; the second is actionable and appealable. That difference — not politeness, actionability — is what explanations are for, and different audiences need different depths of it.
The customer needs the second mortgage letter: what mattered, and what could I change? The team needs something deeper: explanations are how you audit a model for the leaks of Topic 12 and the biases of Topic 45 — you cannot check why a model favors one group if the model cannot show its why. And the regulator increasingly needs it on paper: in more and more jurisdictions, people have a right to an explanation of consequential automated decisions about credit and employment. Three audiences, one lesson: an explanation is not a courtesy feature. For decisions that matter, it is part of the product.
The Stakes Dial
So must every model explain itself? No — and pretending so would be its own kind of dishonesty. Opacity is priced by stakes. A dish recommender can stay a black box forever; a wrong guess costs a scroll. But as decisions climb toward money, livelihood, and liberty, the accuracy-for-transparency trade tilts, and Topic 39's discipline — match the tool to the task — gets an ethical edition: sometimes the right engineering call is the less accurate, readable-by-design model, because a decision you can explain, audit, and appeal is worth a point of accuracy.
That is what Plateful does. The data team rebuilds the pay-later gate as a simpler, readable model — slightly worse on the benchmark, fully explainable in a sentence — and keeps the black box for ranking restaurant photos, where nobody needs a why. "We can't explain it" has quietly flipped from an excuse into an argument: for high-stakes calls, it now argues for changing the model, not for skipping the explanation.
- "Accuracy is what matters; explanations are a nice-to-have." For high-stakes decisions, unexplainable means unauditable and unappealable. The explanation is part of the product — sometimes the legally required part.
- "Deep models simply can't be examined." Approximate explanation tools exist and genuinely help — they show which features pushed a specific decision, and which way. The honest claim is harder and imperfect, not impossible.
- "Explainability is a preference engineers argue about." Its main audiences are customers (what can I change?), auditors (is it biased or leaking?), and regulators (explanation rights). The technical-sounding name hides what it really is: a trust and legal requirement.
- "Can we explain this decision to the person affected?" joins your framing checklist for any model touching money, work, or access — and sometimes it decides the model choice itself, in favor of the simpler one.
- As a citizen, you will be scored by systems like this. Knowing that an explanation can exist — and increasingly can be demanded — is practical power the next time a computer says no to you.
Knowledge Check
What makes a small decision tree "readable by design", unlike a deep network?
- It has fewer parameters, so there is less to read through
- Its verdict can be read off as a chain of plain conditions
- It is more accurate, so its answers need no justification
- Engineers write its rules by hand, so the rules are documented
An explanation tool reports which features pushed the customer's pay-later decline. How should the team treat that report?
- As the model's actual reasoning, read out exactly
- As meaningless, since black boxes cannot be examined at all
- As a useful approximation of the model's behavior, not gospel
- As proof the model is broken and must be retired
Why does the data team itself need model explanations — not just the declined customer?
- Explanations make the model compute its answers faster
- They save the team from writing documentation by hand
- Without explanations the model cannot be retrained
- Audits for bias and leakage need to see which features drive decisions
Plateful's readable pay-later model scores slightly worse than the black box it replaced. Was the swap defensible?
- Yes — for high-stakes decisions, explainability can be worth some accuracy
- No — the most accurate model is always the right choice
- Yes, and every model at Plateful should now be replaced the same way
- No — they should have kept the black box and relied on explanation tools instead
You got correct