Originally published at han-co.com · Part of the "Basics" strand of my Credit & Finance Data Science series. (The original has hand-drawn diagrams; the text below is identical.)
Up through Part 7, we've talked about building a good model (Part 4), evaluating it honestly (Part 5), setting policy with causation (Part 6), and validating that model and keeping it alive (Part 7). Now the last axis of trust remains: you have to be able to explain why it decided the way it did, and to answer whether that decision is fair.
In Part 0, among the reasons this field differs from ordinary ML, I named "interpretability isn't a choice, it's a duty" and "regulatory and governance overhead is always underneath" (points 6 and 7). This piece takes those two head-on. In other fields, being able to answer "why?" is a nice bonus; in credit, not having it is often illegal, or stops the release outright.
Explanation isn't a bonus, it's a duty
First, why interpretability is a duty. In credit, a model turns down someone's loan, trims their limit, raises their rate. A model that makes these decisions has to be able to explain why it judged the way it did. But whom you owe the explanation to differs by country.
At one end is the US. There's a requirement to notify a rejected applicant of the main reasons for denial, so the duty to explain points at the consumer. At the other end is Japan. A card company's underwriting criteria are private, and there's no obligation to tell a rejected applicant why. About all an applicant can do is request disclosure of their own credit information from a bureau like CIC and check that.
But not notifying the consumer doesn't mean no explanation is needed. The direction is just different. In Japan too, a model has to be explainable to the regulator; where the model becomes the legal basis itself, as with the payment-capacity estimate (支払可能見込額) under the Installment Sales Act, you have to be able to defend that judgment; and internal validation and audit (Part 7) also ask "what's the basis for this score?" So in credit a black box isn't a cool piece of cutting-edge tech, it's a risk in itself. However good the performance, if you can't say why it came out that way, you can't use it.
Transparent models and post-hoc explanations
There are broadly two ways to make a model explainable.
One is to use a transparent model from the start. Scorecards, logistic regression, shallow decision rules — their structure is the explanation. The WOE and scorecards from Parts 4 and 5 are the classic case: which feature moves the score up or down, and by how many points, is right there in a table. Models like this are often called glass boxes.
The other is to use a complex model for performance but attach a post-hoc explanation. The standard is SHAP. Grounded in game theory, it decomposes, in a consistent way, how much each feature contributed to this prediction. There are similar tools like LIME, partial dependence, and permutation importance.
It helps to distinguish two kinds of explanation here. Global interpretation looks at what the model as a whole weighs; local interpretation looks at why this particular person got this score. In credit, local interpretation matters especially. To send an adverse-action notice, you ultimately have to be able to say, one individual at a time, "you were declined for these reasons."
One honest caveat: a post-hoc explanation doesn't really open up the black box, it approximates it from the outside. SHAP values aren't the perfect truth either. So where possible you favor an interpretable structure from the start, and even when you use a complex model, you don't bolt the explanation on in a rush afterward — you design it in from the beginning.
A device often layered on top of this is the monotonicity constraint. It forces the model to keep to a direction that fits common sense and regulation, such as "a higher income must not lower the score" (in Part 0 I mentioned monotonicity as a condition for stability). Giving up a little performance to keep a sensible direction is often the better trade for explanation and trust.
Not using them doesn't make you fair
Now fairness. The starting point is clear. You must not discriminate on protected attributes like sex, race, or religion. This is prohibited by law, and putting those variables into the model directly is plain discrimination.
The trap is that dropping those variables doesn't make you fair. If other variables like ZIP code, occupation, or spending pattern are correlated with a protected attribute, the model learns the same discrimination by a detour, through that proxy. For example, in a society where ZIP code is strongly tied to race, using ZIP code alone without race still ends up as racial discrimination. This is called proxy discrimination.
At its root it connects back to Part 0. The data we hold already has the past approval policy and society's biases soaked into it. The model learns that data, so if you train it thoughtlessly, it inherits and reproduces the past discrimination as is. So "just drop the protected variable and it's fair" is naive; you have to look at the proxies too.
Fairness has no single definition
So how do you measure whether it's fair? The problem is that fairness has more than one definition.
Just to name a few: there's a definition that says equalize the approval rate across groups (demographic parity); one that says equalize the approval rate among the people who actually repay (equal opportunity); and one that says equalize the calibration of predicted probabilities across groups (calibration within groups — the calibration from Part 5). All plausible, yet they say different things.
Take the first, the instinct to compare approval rates across groups. There's a rule of thumb long used in the US: the four-fifths rule, often called the 80% rule. If one group's approval rate falls short of 80% of the highest group's rate, that's treated as a sign of adverse impact. It originally came out of hiring and selection, but it's frequently referenced in lending-discrimination judgments too. That said, it isn't a yardstick that settles discrimination; it's closer to a first sign that you should look harder. You can clear 80% and still be discriminating in substance, and fall short of it yet be explainable by a legitimate reason.
And there's an uncomfortable fact known mathematically. When the base rate — the actual default rate — differs across groups, you can't satisfy all these definitions at once. Line one up and another goes off. So "what counts as fair" is a value judgment technology can't make for you; it's something people have to choose to fit the context and the jurisdiction.
It can conflict with accuracy, too. Reaching for fairness often means giving up some performance. Mitigation can act on the data before training, on the objective during training, or on the output after training, but none of it is free.
So here too the honest stance is the same as in Part 6: state which fairness criterion you chose and why, and be aware of what that choice gives up. And this area must be handled together with legal and compliance. Being technically fair doesn't mean it's legally lawful.
Regulation holds all of this up
The explanation requirements and fairness demands so far aren't just well-meaning advice. A good deal of it is enforced by law and regulation.
The form differs by region. The US, through fair-lending law, requires adverse-action notices and prohibits discrimination; Europe, through its data-protection rules, addresses explanation of automated decisions, and its new AI regulation classifies credit scoring as high-risk. In Japan, the duty to compute a payment-capacity estimate (支払可能見込額) under the Installment Sales Act attaches to card issuance and limits, so the model becomes a legal basis directly (I flagged this in Part 0), and the Personal Information Protection Act governs data use. In Korea, the Credit Information Act and the Personal Information Protection Act cover explanation of, and objection to, automated decisions.
Privacy overlaps here. Which variables you put into the model is, directly, the question of how you use personal data. Purpose limitation, consent, data minimization, retention periods, access control — all of it is tangled up with feature selection. So choosing a single feature becomes a data-governance decision.
In the end, the "you can't deploy a model freely" from Part 0 meets the governance of Part 7 right here. Regulation and governance are always laid underneath all of this.
So, it's trust
With that, we've reached the last piece in the Basics series of why credit data science is different. Predicting well is the entry ticket (Part 5); handling causality is skill (Part 6); honestly validating the model you built and keeping it alive for the long haul is one axis of trust (Part 7); and explaining why it decided the way it did, and taking responsibility for whether that decision is fair, is the other axis of trust.
In Part 0, to the question of whether generative AI would just do all of this for us, I answered that you need it more, not less. Because in the face of requirements to be explainable, to be fair, and to pass regulation, a black box is structurally blocked. So the person who knows why explanation is required, what fairness is, and how you validate stays in the seat that judges what the automation produced.
That's the end of the conceptual part. From the next piece, we move to the side that actually makes these concepts run. A model only lives on top of a pipeline, and reproducibility and consistency come before performance — on to data engineering and tooling.
I write han-co.com, a blog on credit and finance data science, in the language of practice. New posts by email: https://han-co.com/en/blog/
Top comments (0)