Explainability (XAI)
A neural network gives you an answer but not a reason — millions of weights, no argument. Explainability (XAI) techniques reconstruct one: attribution methods (saliency maps, SHAP, integrated gradients) estimate how much each input — each pixel, word, or feature — pushed the prediction up or down. The payoff is not just comfort: explanations expose shortcut learning, where a model gets the right answers for indefensible reasons, like rejecting loans by postal code or detecting pneumonia from the hospital's scanner watermark. Regulators increasingly require them, and they are estimates of the model's reasoning, not transcripts — useful evidence, not gospel.
Inspect three loan decisions. The first two attributions read like sane reasoning; the third is a confident rejection driven almost entirely by postal code — a shortcut the raw prediction alone would never reveal.
Model says: APPROVED
Attribution: how much each input pushed the decision
The attributions read like a sane loan officer's reasoning — each factor pushes the expected direction. Explanations that match domain sense build justified trust.
Check yourself
A model detects pneumonia with 95% accuracy, and its saliency maps highlight a text marker printed on the X-ray corner rather than the lungs. What actually happened during training?
Go deeper (free): Christoph Molnar — Interpretable ML (free book) ↗