DEV Community: Josephine Amponsah Baah

Deep Learning Explainability : A Credit Risk Scoring Case Study

Josephine Amponsah Baah — Tue, 23 Jun 2026 12:41:08 +0000

Content

Introduction
Data Brief and Model Training
XAI Frameworks and Examples
Risk Score Calibration
End-to-End Pipeline

Introduction

Predictive models have come a long way from basic statistical methods. For classification, logistic regression is often used as a baseline model, while increasingly complex models from SVM, Tree-based models to Deep Learning models are used in production as final models.

This increased complexity in build, correlates with increased complexity in explainability.

Logistics Regression Architecture

The above architecture shows logistic regression requires little effort to understand how each feature impacts the predicted class of data input. For deep learning models, the architecture is far more complex. It therefore requires more complex frameworks to arrive at the explanation we need to understand the models we have built.

In the following sections, we will explore a few frameworks that enable the explanation of Deep Learning models, a credit risk prediction use case and a proposed presentation of predicted probabilities.

Data Brief and Model Training

The data used is a feature engineered version of a kaggle dataset on multiple credit history records data of anonymized users (raw_data). The features were generated based on the value to credit behaviour per user and data readiness for Deep Learning models. You can find the notebook where I implemented the feature engineering and full model training here.

Data Features :

data = pd.read_csv('data/data_dl.csv')
features = data.columns[:-1] 
# 44 features with each record representing a users current credit info and some features from the preceding 2 months for historical context
target = data.columns[-1]

Output/Target Label: Labels that interpret the credit health of each user at the given time of the record. [Good, Standard, Poor]
Intended Output: Risk score bands for each target label and a more fluid classification of creditworthiness of users with individual risk scores

Model

We will be using the following Multi-Layer Perceptron model for the implementation examples of the XAI frameworks

Defining the Model Class

# While several models were trained in my full project, we will focus on an MLP- Deep Learning model for the purposes of this article

class MLPClassifier(nn.Module):
    def __init__(self, cat_dims, num_cols, hidden = [128, 64, 32], dropout = 0.3):
        super().__init__()
        self.embs = nn.ModuleList([nn.Embedding(v, d) for v, d in cat_dims]) 
# embedding categorical values into meaning dimensional vectors
        in_d = sum(d for _, d in cat_dims) + len(num_cols)
        layers = []
# looping through the various params of hidden instead of repeating the layers multiple times
        for h in hidden:
            layers += [nn.Linear(in_d, h), nn.BatchNorm1d(h), nn.ReLU(), nn.Dropout(dropout)]
            in_d = h
        layers.append(nn.Linear(in_d, 3))
        self.net = nn.Sequential(*layers) 
# runs the layers in order as defined

    def forward(self, cx, nx):
        x = torch.cat([e(cx[:, i]) for i, e in enumerate(self.embs)] + [nx], dim=1) 
# applying necessary transforms defined in __init__ function

        return self.net(x) #returning prediction from the NN

Training the model

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

mlp_clf = MLPClassifier(cat_dims=cat_dims, num_cols=num_cols)
mlp_clf.to(device)

#set criterion, optimizer and epochs 

# train model and save params of best performing version of model

model.load_state_dict(best_state)

#assuming we are satisfied with the evaluation results of model, we will proceed to explore the explainability of the model

# Data prep for XAI implementation
import shap

X_train_cat = torch.tensor(X_train[cat_cols].values, dtype=torch.float32)
X_train_num = torch.tensor(X_train[num_cols].values, dtype=torch.float32)
X_train_combined = torch.cat([X_train_cat, X_train_num], dim=1)
# you can just convert all of X_train if all features are only categorical or only numerical

AI Explainability (XAI) Frameworks & Examples

SHAP Framework

SHAP ( SHapley Additive exPlanations) is an Explainable AI (XAI) framework that bases the assignments of predictive contributions per feature on game theory. Under this framework are classes purposely built for deep learning models.

# instantiate explainer and return results
import shap

bg_idx = np.random.choice(len(X_train_combined), size=200, replace=False)
# size param splits data into sizeable batches for inference
data = X_train_combined[bg_idx].to(device)

Deep Explainer

This class is based on the DeepLIFT algorithm, approximating per node attribution into Shapley values for each feature of the model. It requires access to the layer structure of the neural network, returning a close approximation to the true values of predictive contribution by each feature. This makes it incompatible with unsupported layer types like normalization and embedding layers as it cannot appropriately work with such structures.


#data is a tensor form of your dataset
explainer = shap.DeepExplainer(model, data)

# write code for plotting results
shap_vals = explainer.shap_values(X_train[:100].to(device)) #only taking a sample

"""
It returns a matrix of a matrix of dimensions: number_of_rows, no_of_features, number_of_labels
"""

Gradient Explainer

This class is based on the logic of expected gradients, an extension of the integrated gradients method. It uses a neutral baseline to assess how much the slope of the output changes with respect to each input. Unlike the Deep Explainer, it does not require to process the specific layers of the neural network and hence can be used for unsupported layer types. It can however be misleading when the activations of the layers saturate, making Deep Explainer the preferred class for deep neural network models.

# instantiate explainer and return results 

explainer = shap.DeepExplainer(model, data)
shap_vals = explainer.shap_values(X_train[:100].to(device)) 
# write code for plotting results

Since our MLP model has a normalization layer, BatchNorm1d and Embedding of the categorical features, the Gradient explainer is most suitable. To use the DeepExplainer successfully, you will have to strip the MLP model off of the Embedding and BatchNorm1d layers. SHAP Documentation

LIME Framework

Locally Interpretable Model-agnostic Explanations (LIME), is an XAI framework that targets the explanation of feature contributions for specific inputs. It executes this by sampling various combinations of features, leaving out some at each instance. Disturbed samples closer to the original input are assigned higher weights, ensuring the explanation reflects local behaviour rather than global patterns. It then fits a weighted regression or simple tree model on these samples, reading off the coefficients as feature attribution. This framework works for all types of model complexities, hence model-agnostic.

# Implement LIME
from lime import lime_tabular

categorical_names = {
    i: list(le_dict[col].classes_)
    for i, col in enumerate(cat_cols)
} # remapping for textual names of categories

lime_explainer = lime_tabular.LimeTabularExplainer(
    training_data        = X_train_combined.numpy(), 
    feature_names        = cat_cols + num_cols,
    class_names          = ['Poor', 'Standard', 'Good'],
    categorical_features = list(range(len(cat_cols))), 
    categorical_names    = categorical_names,
    mode                 = 'classification',
    discretize_continuous= True,   
    random_state         = 42,
) # it makes inference on the training data to understand the contributions of the features to predictions

"""
mlp_predict_fn: takes in array of input (cat_cols + num_cols to returns calibrated probabilities of predicted classes
"""

# explaining the first row of data
instance = X_train_combined[0].numpy()
lime_exp = lime_explainer.explain_instance(
    data_row       = instance,
    predict_fn     = mlp_predict_fn,
    labels         = (0, 1, 2),    # explain all three classes
    num_features   = 10,
    num_samples    = 1000,
)

probs = mlp_predict_fn(instance.reshape(1, -1))[0]
pred_class = int(np.argmax(probs))

# visualizing results
import matplotlib.pyplot as plt

fig = lime_exp.as_pyplot_figure(label=pred_class)
fig.suptitle(
    f"LIME — Predicted: {['Poor','Standard','Good'][pred_class]}  "
    f"(Poor={probs[0]:.2f}  Std={probs[1]:.2f}  Good={probs[2]:.2f})",
    fontsize=11
)
plt.tight_layout()
plt.savefig("lime_explanation.png", dpi=150, bbox_inches="tight")
plt.show()

LIME Documentation

Risk Score Calibration

Predicted Probability Calibration

Probabilities predicted by Machine Learning models are often not properly scaled and hence cannot be interpreted as the likelihood of the item/user belonging to an assigned class. This makes calibration of resulting class probabilities essential , as they are often used for further steps/analysis in most risk scoring pipelines.

The commonly used calibration methods are:

Isotonic Regression,
Sigmoid /Platt Calibration
Temperature Scaling:

We will implement temperature scaling for our use case. If interested in a detailed explanation, read here to understand how it works.


temp_scaler = TemperatureScaler()
"""
TemperatureScaler is to be defined as an nn.Module class that scales the logits from you NN model with a temperature param.
"""
temp_scaler.fit(mlp_logits, mlp_labels)

Calibrating Probabilities to Human Readable Risk Score

While labels are often used for risk classification, they band a variety of credit users together, who would otherwise not be, simply because of the maximum predicted probability class. Establishing a risk score gauge gives both business and users ( if applicable) context on individual credit health, improvements over time and actions that negatively impact it.

The standard convention used to implement this is the Points to Double Odds (PDO) method.

Implementation


PDO = 50 # set parameter by your log transform moves from the base score . you can iterate over a list of values and find the best value for your implementation
BASE_SCORE = 600
FACTOR     = PDO / np.log(2) # ≈ 72.13 score points per log-unit of odds
EPSILON    = 1e-7   # error factor 

p_good    = cal_probs[:, 2] + EPSILON #adding error margins to probabilities
p_poor    = cal_probs[:, 0] + EPSILON

#PDO formula
raw_score = BASE_SCORE + FACTOR * np.log(p_good / p_poor)
raw_score = np.clip(raw_score, 300, 850) # recalibrating results to fit with desired range

You apply to this to you calibrated probabilities and return as a result of your input X, in a dataframe showing the risk score of each processed user.

Build End-to-End Pipeline

Models are meant to exist in a production ready state, allowing for implementation in applications or software services. To achieve this goal, you either:

Define a Pipeline class with each transformation from preprocessing to score calibration.

"""
1. Create a .py file and define a basic Pipeline Class

2. Define named classes for each step (Scaling, Tensorisation,Explanation, Calibration, Scoring-PDO)

3. Save as artefact to be called and used in an API or Local / Cloud CI/CD pipelines
""" 
# Can be also called in a notebook file for validation / or batch transformations

pipeline = CreditScoringPipeline(
    lstm_clf    = lstm_clf,
    temp_scaler = lstm_temp_scaler,
    scaler      = scaler,
    le_dict     = le_dict,
    background  = background,
    cat_cols    = cat_cols,
    num_cols    = num_cols,
)

results = pipeline.run(X_test)   # raw DataFrame in
print(results[["risk_score", "risk_band", "top_positive_factors", "top_negative_factors"]].head())

Build an API with endpoints that transform individual

In some cases you might have the need of implementing an entire API to deploy the pipeline. The implementation has some overlap with the previous option.

# Directory
artefacts/ 
"""
this has the saved models and instance of classes to be maintained
"""
api/ 
    inference.py  
"""
defines classes to be used for inference of raw data input
"""
main.py  
""" define FastAPI here to call classes in inference.py and return outputs"""

Conclusion

In the beginning of the article, we set out to build a full credit scoring pipeline using deep learning models , explainability frameworks , and score calibrations.

We have discuss the necessity of explainability of deep learning models due to their complex structure, touched on three XAI frameworks relevant to Deep Learning models (shap.DeepExplainer, shap.GradientExplainer, LIME) and their various use cases. We proceeded to explore implementations that makes use of our output probabilities into human readable scores along with insights of the significant contributing factors to score health.

If this article has peaked in your interest in implementing a similar end-to-end project or you wish to apply any of the in your current work , check out the full project on my Github repo

References

Explaining Deep Learning Models for Credit Scoring with SHAP by Lars Ole Hjelkrem and Petter Eilif de Lange

Mastering Explainable AI - by Siddhartha Pramanik

Implementing PDO - Youtube Tutorial