Most teams treat "bias testing" as checking one aggregate fairness metric after training. That catches almost nothing. If you're responsible for shipping a model into anything high-impact (hiring, credit, healthcare, criminal justice), here are three interpretability techniques worth actually building into your pipeline before deployment, not after a complaint.
1. Saliency maps: what is the model actually looking at?
Saliency maps highlight which input features (pixels, tokens, columns) most influenced a given prediction. For image models, this can reveal a model classifying "doctor" based on background objects rather than the person. For tabular models, gradient-based saliency on a batch of edge-case inputs often exposes a proxy variable (zip code standing in for race, for instance) that a raw feature-importance chart missed.
2. Concept Activation Vectors (TCAV): testing for concepts, not just features
TCAV lets you test whether a human-defined concept (e.g., "gender presentation," "accent," "socioeconomic markers") is linearly separable inside the model's internal representations, and how much that concept's presence shifts a prediction. It's more useful than raw feature importance because the concept doesn't need to be an explicit input column — it can be latent in unstructured data like images or text embeddings.
3. Perturbation stress testing: does the decision flip when it shouldn't?
Take real cases, alter only a protected-adjacent attribute (name, dialect, photo background) while holding the substantive facts constant, and check whether the model's output changes. A resume screener that flips a hiring recommendation because a name signals a different ethnicity — with identical qualifications — has failed a perturbation test, even if its aggregate fairness metrics look fine.
Why this matters operationally
None of these techniques are exotic — TCAV and saliency tooling exist in open-source libraries (Captum, tf-explain, etc.) — but they're rarely run before deployment because there's no forcing function requiring it. Regulatory frameworks like the EU AI Act's Article 10 (data governance) and NIST's AI RMF gesture at this kind of testing without mandating a specific methodology, which is part of why it gets skipped in practice. Building even one of these three checks into a pre-launch checklist catches failure modes that standard accuracy/precision metrics simply can't see.
Disclosure: This article was drafted with AI assistance (Claude) and reviewed/edited by me before publishing.
I write about AI ethics, governance, and interpretability at darioamadorperez.com.
Top comments (0)