Plotting Precision Recall Curve for Neural Networks With SciKitLearn

Plotting precision recall curves for deep learning models can be tricky since the models themselves are not as clearly represented as other machine learning algorithms like decision trees, linear regression etc. There are several layers that stacks on the other, compiled and then eventually trained on a dataset. After a few searches, I was able to uncover an easier way to speedily get the precision recall curves for any deep learning model. Let's discuss the precision recall curve for a bit, shall we?

For classification tasks, we may decide to predict the class values directly. Alternatively, it may be more flexible to estimate the probability for each class. The purpose for this is to allow the user to select and even tune the threshold for interpreting the anticipated probability.

For example, a threshold of 0.5 may be used as a default, implying that a probability in [0.0, 0.49] is a negative outcome (0) while a probability in [0.5, 1.0] is a positive outcome (1).

We can construct a precision-recall curve by calculating and plotting the precision against recall for each classifier we are using at varying thresholds. For example, if our classifier of choice is logistic regression, the threshold would be the positive class of a predicted probability. In logistic regression, if a sample is predicted to belong to the positive class at probability > 0.5, it is labeled as positive. Ideally, it is possible to choose any probability between 0 and 1 depending on the expected behaviour of our classifier. A precision-recall curve however helps to visualize how the choice of threshold affects the performance of a classifier, and can furthermore help us select the best threshold for a specific problem.

Plotting the precision recall curve for logistic regression is as simple as

from sklearn.linear_model import LogisticRegression`
classifier = LogisticRegression(random_state = 0)
classifier.fit(xtrain, ytrain)

from sklearn.metrics import plot_precision_recall_curve
#Use sklearn to plot precision-recall curves
plot_precision_recall_curve(classifier, X_test, y_test, name = 'Logistic Regression Classifier')

SciKitLearn makes everything easier right?

Plotting the precision recall curve for neural networks is a little more complicated, given that most activation functions does not output class probabilities and instead output class predictions directly, this makes it difficult to plot the precision recall curve which works with probabilities. Let's attempt to plot our precision recall curve with the sklearn method

from sklearn.datasets import make_moons
from tensorflow import keras # for building Neural Networks
from keras.models import Sequential # for creating a linear stack of layers for our Neural Network
from keras import Input # for instantiating a keras tensor
from keras.layers import Dense # for creating regular densely-connected NN layer.

# Data manipulation
import pandas as pd # for data manipulation
import numpy as np # for data manipulation

# Sklearn
import sklearn # for model evaluation
from sklearn.model_selection import train_test_split # for splitting the data into train and test samples
from sklearn.metrics import classification_report # for model evaluation metrics

data = make_moons(n_samples=10000, shuffle=True, noise=None, random_state=None)
print("Data Shape", data[0].shape)
X_train, X_test, y_train, y_test = train_test_split(data[0], data[1], test_size=0.2, random_state=10)
model = Sequential(name="Make-Moons-Model") # Model
model.add(Input(shape=(X_train.shape[1],), name='Input-Layer'))
model.add(Dense(8, activation='relu', name='Hidden-Layer'))
model.add(Dense(1, activation='sigmoid', name='Output-Layer'))

##### Step 4 - Compile keras model
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy', 'Precision', 'Recall'])
history = model.fit(X_train, y_train, batch_size=10, epochs=5, verbose=1, validation_split=0.2, shuffle=True)


Data Shape (10000, 2)
Epoch 1/5
640/640 [==============================] - 2s 2ms/step - loss: 0.4281 - accuracy: 0.8233 - precision: 0.8037 - recall: 0.8561 - val_loss: 0.3042 - val_accuracy: 0.8712 - val_precision: 0.8698 - val_recall: 0.8654
Epoch 2/5
640/640 [==============================] - 1s 2ms/step - loss: 0.2596 - accuracy: 0.8795 - precision: 0.8827 - recall: 0.8758 - val_loss: 0.2345 - val_accuracy: 0.8913 - val_precision: 0.8855 - val_recall: 0.8923
Epoch 3/5
640/640 [==============================] - 1s 2ms/step - loss: 0.2126 - accuracy: 0.8992 - precision: 0.8970 - recall: 0.9023 - val_loss: 0.2036 - val_accuracy: 0.9006 - val_precision: 0.8886 - val_recall: 0.9103
Epoch 4/5
640/640 [==============================] - 1s 2ms/step - loss: 0.1847 - accuracy: 0.9147 - precision: 0.9057 - recall: 0.9260 - val_loss: 0.1789 - val_accuracy: 0.9094 - val_precision: 0.9014 - val_recall: 0.9141
Epoch 5/5
640/640 [==============================] - 1s 1ms/step - loss: 0.1617 - accuracy: 0.9259 - precision: 0.9154 - recall: 0.9388 - val_loss: 0.1566 - val_accuracy: 0.9219 - val_precision: 0.9109 - val_recall: 0.9308

Attempting to plot the curve gives an error like this

from sklearn.metrics import plot_precision_recall_curve
plot_precision_recall_curve(model, X_test, y_test, name = 'Feedforward Neural Network')

~\anaconda3\lib\site-packages\sklearn\metrics\_plot\base.py in _get_response(X, estimator, response_method, pos_label)
     87 
     88     if not is_classifier(estimator):
---> 89         raise ValueError(classification_error)
     90 
     91     prediction_method = _check_classifier_response_method(estimator, response_method)

ValueError: Expected 'estimator' to be a binary classifier, but got Sequential

The solution however is simple, we need to use the KerasClassifier method in scikitlearn to build the model and then pass the classifier option to it as an estimator. Here is the code for that

from keras.wrappers.scikit_learn import KerasClassifier

def create_model():
    # create model
    model = Sequential(name="Model-with-All-Features")
    model.add(Input(shape=(X_train.shape[1],), name='Input-Layer'))
    model.add(Dense(8, activation='relu', name='Hidden-Layer'))
    model.add(Dense(1, activation='sigmoid', name='Output-Layer'))
    # Compile model
    model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy', 'Precision', 'Recall'])
    return model

import numpy
from sklearn.model_selection import GridSearchCV
model = KerasClassifier(build_fn=create_model, batch_size=10, epochs=10, validation_split=0.2, shuffle=True, class_weight={0 : 0.3, 1 : 0.70})
model._estimator_type = "classifier"
trained = model.fit(X_train,y_train)

Epoch 1/10
640/640 [==============================] - 2s 2ms/step - loss: 0.2147 - accuracy: 0.6402 - precision: 0.5843 - recall: 0.9636 - val_loss: 0.4870 - val_accuracy: 0.8213 - val_precision: 0.7785 - val_recall: 0.9036
Epoch 2/10
640/640 [==============================] - 1s 2ms/step - loss: 0.1598 - accuracy: 0.8622 - precision: 0.8232 - recall: 0.9213 - val_loss: 0.3981 - val_accuracy: 0.8694 - val_precision: 0.8448 - val_recall: 0.9085
Epoch 3/10
640/640 [==============================] - 1s 2ms/step - loss: 0.1356 - accuracy: 0.8880 - precision: 0.8593 - recall: 0.9270 - val_loss: 0.3490 - val_accuracy: 0.8813 - val_precision: 0.8537 - val_recall: 0.9234
Epoch 4/10
640/640 [==============================] - 1s 2ms/step - loss: 0.1206 - accuracy: 0.8888 - precision: 0.8560 - recall: 0.9339 - val_loss: 0.3051 - val_accuracy: 0.8850 - val_precision: 0.8588 - val_recall: 0.9246
Epoch 5/10
640/640 [==============================] - 1s 2ms/step - loss: 0.1127 - accuracy: 0.8903 - precision: 0.8564 - recall: 0.9370 - val_loss: 0.2852 - val_accuracy: 0.8900 - val_precision: 0.8634 - val_recall: 0.9295
Epoch 6/10
640/640 [==============================] - 1s 2ms/step - loss: 0.1084 - accuracy: 0.8908 - precision: 0.8543 - recall: 0.9414 - val_loss: 0.2785 - val_accuracy: 0.8894 - val_precision: 0.8575 - val_recall: 0.9370
Epoch 7/10
640/640 [==============================] - 1s 2ms/step - loss: 0.1041 - accuracy: 0.8945 - precision: 0.8565 - recall: 0.9470 - val_loss: 0.2792 - val_accuracy: 0.8919 - val_precision: 0.8510 - val_recall: 0.9530
Epoch 8/10
640/640 [==============================] - 1s 2ms/step - loss: 0.1009 - accuracy: 0.8972 - precision: 0.8566 - recall: 0.9533 - val_loss: 0.2663 - val_accuracy: 0.8994 - val_precision: 0.8632 - val_recall: 0.9518
Epoch 9/10
640/640 [==============================] - 1s 2ms/step - loss: 0.0984 - accuracy: 0.9020 - precision: 0.8623 - recall: 0.9561 - val_loss: 0.2598 - val_accuracy: 0.9013 - val_precision: 0.8645 - val_recall: 0.9543
Epoch 10/10
640/640 [==============================] - 1s 2ms/step - loss: 0.0965 - accuracy: 0.9052 - precision: 0.8668 - recall: 0.9567 - val_loss: 0.2611 - val_accuracy: 0.9006 - val_precision: 0.8611 - val_recall: 0.9580

All we need to do next is
from sklearn.metrics import plot_precision_recall_curve
plot_precision_recall_curve(model, X_test, y_test, name = 'BFNN')

And here is our curve

Plus other performance graphs if interested

Of course, you might not need to plot precision recall curves for your deep learning model right now, but whenever the need arises, you can always refer back to this article 😉