Explainability — How Much Is Enough?
Explainability is one of the most debated topics in AI governance — and one of the least well-understood. The question is not simply "can we explain this model?" but "what level of explanation is required, by whom, for what purpose, and is it technically achievable?"
The Three Audiences for Explainability
- ◆Individuals affected by decisions — Under GDPR and the EU AI Act, they are entitled to "meaningful information about the logic involved" in automated decisions. This does not require a full technical explanation, but it must genuinely explain why they received the outcome they did.
- ◆Internal governance and oversight — Business DRIs and review committees need to understand how a model makes decisions to provide effective oversight. This requires more depth than individual explanations.
- ◆Regulators and auditors — Full technical transparency including model architecture, training data, performance metrics, and decision logic may be required.
Explainability Techniques
- ◆LIME (Local Interpretable Model-agnostic Explanations) — Approximates a complex model's behaviour locally around a specific prediction to generate a simple explanation
- ◆SHAP (SHapley Additive exPlanations) — Assigns each feature a contribution value for a specific prediction based on game theory
- ◆Counterfactual explanations — "Your application was declined. If your income had been £5,000 higher, it would have been approved." Intuitively understandable and legally robust.
- ◆Attention maps — For vision and NLP models, highlights which parts of the input most influenced the output
There is a genuine tension between model accuracy and explainability. Simpler models (logistic regression, decision trees) are highly explainable but often less accurate. Complex models (deep neural networks, gradient boosting ensembles) are often more accurate but less explainable. For high-risk decisions, regulators are increasingly clear: explainability requirements may override accuracy optimisation.
Many organisations use post-hoc explanation tools (LIME, SHAP) to generate explanations for models that were never designed with explainability in mind. Regulators and courts are increasingly scrutinising whether these explanations are genuinely faithful to the model's actual decision process — or just plausible-sounding approximations. Design for explainability from the start, don't bolt it on.
For any AI system making significant decisions about individuals: the explanation should be specific to their case, expressed in plain language, identify the main factors that determined the outcome, and indicate what would need to change for a different outcome. If your current system cannot produce this, it needs redesigning before it can be considered compliant.
