DEV Community: Ty Mick

Natural Language Processing for Loan Risk

Ty Mick — Sat, 24 Oct 2020 19:00:31 +0000

The story so far
Exploratory data analysis
Imputing missing values
Optimizing data types
Creating document vectors
Building the pipeline
Evaluating the model
Next steps

The story so far

A few months ago, I built a neural network regression model to predict loan risk, training it with a public dataset from LendingClub. Then I built a public API with Flask to serve the model's predictions.

Then last month, I decided to put my model to the test and found out that my model can pick grade A loans better than LendingClub!

But I'm not done. Now that I've learned the fundamentals of natural language processing (I highly recommend Kaggle's course on the subject), I'm going to see if I can eke out a bit more predictive power using a couple of freeform text fields in the dataset: title and desc (description).

import joblib

prev_notebook_folder = "../input/building-a-neural-network-to-predict-loan-risk/"
loans = joblib.load(prev_notebook_folder + "loans_for_nlp.joblib")
num_loans = loans.shape[0]
print(f"This dataset includes {num_loans:,} loans.")

This dataset includes 1,110,171 loans.

loans.head()

	loan_amnt	term	emp_length	home_ownership	annual_inc	purpose	dti	delinq_2yrs	cr_hist_age_mths	fico_range_low	...	tot_hi_cred_lim	total_bal_ex_mort	total_bc_limit	total_il_high_credit_limit	fraction_recovered	issue_d	title	desc
0	3600.0	36 months	10+ years	MORTGAGE	55000.0	debt_consolidation	5.91	0.0	148	675.0	...	178050.0	7746.0	2400.0	13734.0	1.0	Dec-2015	Debt consolidation	NaN
1	24700.0	36 months	10+ years	MORTGAGE	65000.0	small_business	16.06	1.0	192	715.0	...	314017.0	39475.0	79300.0	24667.0	1.0	Dec-2015	Business	NaN
2	20000.0	60 months	10+ years	MORTGAGE	63000.0	home_improvement	10.78	0.0	184	695.0	...	218418.0	18696.0	6200.0	14877.0	1.0	Dec-2015	NaN	NaN
4	10400.0	60 months	3 years	MORTGAGE	104433.0	major_purchase	25.37	1.0	210	695.0	...	439570.0	95768.0	20300.0	88097.0	1.0	Dec-2015	Major purchase	NaN
5	11950.0	36 months	4 years	RENT	34000.0	debt_consolidation	10.20	0.0	338	690.0	...	16900.0	12798.0	9400.0	4000.0	1.0	Dec-2015	Debt consolidation	NaN

5 rows × 69 columns

This post, like its predecessors, was adapted from a Jupyter Notebook, so feel free to fork my notebook on Kaggle or GitHub if you'd like to follow along.

Exploratory data analysis

There isn't too much exploratory data analysis left to do after how thoroughly I cleaned the data in my first post, but I do have a few quick questions about the title and desc fields I'd like to answer before I move on.

How many loans use each field?
Have these fields always been included in the loan application?
What is the typical length of each field (in number of words)?

nlp_cols = ["title", "desc"]

loans[nlp_cols].describe()

	title	desc
count	1097288	71967
unique	35863	70927
top	Debt consolidation
freq	573992	23

If the most frequent desc value is empty (or maybe just whitespace), perhaps I need to convert all empty or whitespace-only values to NaN before continuing.

import re
import numpy as np

for col in nlp_cols:
    replace_empties = lambda x: x if re.search("\S", x) else np.NaN
    loans[col] = loans[col].map(replace_empties, na_action="ignore")

description = loans[nlp_cols].describe()
description

	title	desc
count	1097288	71943
unique	35863	70925
top	Debt consolidation	Borrower added on 03/17/14 > Debt consolidat...
freq	573992	9

Thankfully that didn't remove too many values, but this "Borrower added on [date]" deal worries me now. I'll deal with that a little later.

for col in nlp_cols:
    percentage = int(description.at["count", col] / num_loans * 100)
    print(f"`{col}` is used in {percentage}% of loans in the dataset.")

percentage = int(description.at["freq", "title"] / num_loans * 100)
print(f'The title "Debt consolidation" is used in {percentage}% of loans.')

`title` is used in 98% of loans in the dataset.
`desc` is used in 6% of loans in the dataset.
The title "Debt consolidation" is used in 51% of loans.

These fields may not be as useful as I had previously thought. Even though there are 35,860 unique titles used across the dataset, 51% of them just use "Debt consolidation". Maybe the titles are more descriptive in the other 49%?

And the desc field is only used with 6% of loans.

Now to check and see when these fields were introduced.

# `issue_d` is just the month and year the loan was issued, by the way.
loans["issue_d"] = loans["issue_d"].astype("datetime64[ns]")

print("Total date range:")
print(loans["issue_d"].agg(["min", "max"]))
print("\n`title` date range:")
print(loans[["title", "issue_d"]].dropna(axis="index")["issue_d"].agg(["min", "max"]))
print("\n`desc` date range:")
print(loans[["desc", "issue_d"]].dropna(axis="index")["issue_d"].agg(["min", "max"]))

Total date range:
min   2012-08-01
max   2018-12-01
Name: issue_d, dtype: datetime64[ns]

`title` date range:
min   2012-08-01
max   2018-12-01
Name: issue_d, dtype: datetime64[ns]

`desc` date range:
min   2012-08-01
max   2016-07-01
Name: issue_d, dtype: datetime64[ns]

Neither of these fields were introduced late, but they may have stopped using the desc field for the last two years of the database.

Now I'll take a closer look at values in these fields.

import pandas as pd

with pd.option_context("display.min_rows", 50):
    print(loans["title"].value_counts())

Debt consolidation                       573992
Credit card refinancing                  214423
Home improvement                          64028
Other                                     56166
Major purchase                            20734
Medical expenses                          11454
Debt Consolidation                        10638
Business                                  10142
Car financing                              9660
Moving and relocation                      6806
Vacation                                   6707
Home buying                                5097
Consolidation                              4069
debt consolidation                         3310
Credit Card Consolidation                  1607
consolidation                              1538
Debt Consolidation Loan                    1265
Consolidation Loan                         1260
Personal Loan                              1040
Credit Card Refinance                      1020
Home Improvement                           1016
Credit Card Payoff                          991
Consolidate                                 947
Green loan                                  626
Loan                                        621
                                          ...
House Buying Consolidation                    1
Credit Card Deby                              1
Crdit cards                                   1
"CCC"                                         1
Loan to Moving & Relocation Expense           1
BILL PAYMENT                                  1
creit card pay off                            1
Auto Repair & Debt Consolidation              1
BMW 2004                                      1
Moving Expenses - STL to PHX                  1
 Pay off Bills                                1
Room addition                                 1
Optimistic                                    1
Consolid_loan2                                1
ASSISTANCE NEEDED                             1
My bail out                                   1
myfirstloan                                   1
second home                                   1
Just consolidating credit cards               1
Financially Sound Loan                        1
refinance loans and home improvements         1
credit cart refincition                       1
Managable Repayment Plan                      1
ccdebit                                       1
Project Pay Off Debt                          1
Name: title, Length: 35863, dtype: int64

Interesting. It seems like there's plenty of variety in loan titles in the other 49%. A lot of them seem to directly correspond to the purpose categorical field, but not so many as to make this field useless, I think.

Side note: I discovered at one point when perusing this column that someone entered the Konami Code as the title of their loan application, and their inclusion in this dataset means that the code apparently worked for them—they got the loan.

loans[loans["title"] == "up up down down left right left right ba"][
    ["loan_amnt", "title", "issue_d"]
]

	loan_amnt	title	issue_d
1856340	12000.0	up up down down left right left right ba	2013-04-01

loans["desc"].value_counts()

  Borrower added on 03/17/14 > Debt consolidation<br>                                                                                                                                                                                                                                        9
  Borrower added on 01/15/14 > Debt consolidation<br>                                                                                                                                                                                                                                        7
  Borrower added on 02/19/14 > Debt consolidation<br>                                                                                                                                                                                                                                        7
  Borrower added on 03/10/14 > Debt consolidation<br>                                                                                                                                                                                                                                        7
  Borrower added on 01/29/14 > Debt consolidation<br>                                                                                                                                                                                                                                        7
                                                                                                                                                                                                                                                                                            ..
  Borrower added on 01/14/13 > Credit Card consolidation<br>                                                                                                                                                                                                                                 1
  Borrower added on 03/14/14 > Debts consolidation and cash for minor improvements on condominium<br>                                                                                                                                                                                        1
  Borrower added on 03/02/14 > I lost a house and need to pay taxes nd have credit card debt thatI already pay $350 a month on and it goes nowhere.<br>                                                                                                                                      1
  Borrower added on 04/09/13 > I want to put in a conscious effort in eliminating my debt by converting high interest cards to a fixed payment that can be effectively managed by me.<br>                                                                                                    1
  Borrower added on 09/18/12 > Want to become debt free, because of several circumstances and going back to school I got into debt. I want to pay for what I have purchased without it having an effect on my credit. That is why I want to consolidate my debt and become debt free!<br>    1
Name: desc, Length: 70925, dtype: int64

Do all these descriptions start with "Borrower added on [date]"?

pattern = "^\s*Borrower added on \d\d/\d\d/\d\d > "
prefix_count = (
    loans["desc"]
    .map(lambda x: True if re.search(pattern, x, re.I) else None, na_action="ignore")
    .count()
)
print(
    f"{prefix_count:,} loan descriptions begin with that pattern.",
    f"({description.loc['count', 'desc'] - prefix_count:,} do not.)",
)

71,858 loan descriptions begin with that pattern. (85 do not.)

Well now I need to check those other 85.

other_desc_map = loans["desc"].map(
    lambda x: False if pd.isna(x) or re.search(pattern, x, re.I) else True
)
other_descs = loans["desc"][other_desc_map]
other_descs.value_counts()

Debt Consolidation                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 2
I would like to pay off 3 different credit cards, at 12%, 17% and 22% (after initial 0% period is up).  It would be great to have everything under one loan, making it easier to pay off.  Also, once I've paid off or down the loan, I can start looking into buying a house.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     1
loan will be used for paying off credit card.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      1
This loan will be used to consolidate high interest credit card debt.    Over the course of this past year my wife and I had our first child, purchase a home and received a large bonus from work.  With the new home and the child on the way I chose to spread my tax withholdings on the bonus to all checks received in 2008 this caused my monthly income to fall by $1500.  This in combination with an unexpected additional down payment for our home of $17,000 with only a weeks notice we were force to dip into our Credit Cards for the past several months.    Starting January 1, 2009 I will be able to readjust my tax withholding and start to pay off the Credit Card debt we have racked up.  This loan will help lower the interest rate during the repayment period and give one central place for payment.  My wife and I have not missed a payment or been late for the past 5 years.  My fico score is 670 mainly due to several low limit credit cards near their max.  I manage the international devision of a software company and my wife is a kindergarten teacher, combined we make 140K a year.    Thank you for your consideration and I look forward to working with you.      1
to pay off different credit cards to consolidate my debt, so I can have just one monthly payment.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  1
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  ..
Hello, I would like to consolidate my debt into a lower more convenient payment. I have a very stable career of more than 20 years with the same company. My community is in a part of the country that made it through the last few years basically unscathed and has a very promising future.<br>Thank You<br>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   1
consolidate my debt                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                1
I am looking to pay off my credit card debts.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      1
This loan is to help me payoff my credit card debt. I've done what I can to negotiate lower rates, but the interest is killing me and my monthly payments are basically just taking care of interest. Paying them off will give me the fresh start I need on my way to financial independence. Thank you.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          1
I have been in business for a year and want to eliminate some personal debt and use the remainder of the loan to take care of business expenses. Also lessening the number of trade lines I have open puts me in a better position to pursue business loans since it will  be based on my personal credit. A detailed report can be created to show where exactly the funds will go and this can be provided at any time during the course of the loan.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            1
Name: desc, Length: 84, dtype: int64

It looks like the borrower may be able to add information to the description at different points in time. I should check and see if any of those dates come after the actual issue date of the loan.

from datetime import datetime, date

for row in loans[["desc", "issue_d"]].itertuples():
    if not pd.isna(row.desc):
        month_after_issue = date(
            day=row.issue_d.day,
            month=row.issue_d.month % 12 + 1,
            year=row.issue_d.year + row.issue_d.month // 12,
        )

        date_strings = re.findall("\d\d/\d\d/\d\d", row.desc)
        dates = []
        for string in date_strings:
            try:
                dates.append(datetime.strptime(string, "%m/%d/%y").date())
            except:
                continue

        for d in dates:
            if d >= month_after_issue:
                print(f"{row.issue_d} – {row.desc}")
                break

2014-01-01 00:00:00 –   Borrower added on 01/08/14 > I am tired of making monthly payments and getting nowhere.  With your help, except for my mortgage, I intend to be completely debt free by 12/31/2016.<br>
2014-01-01 00:00:00 –   Borrower added on 01/08/14 > We have been engaged for  2 1/2yrs and wanted to bring out blended family together as one. We are set to get married on 03/22/14 and we are paying for it on our own. We saved the majority of the budget unfortunately there were a few unexpected cost that we still need help with.<br>
2014-01-01 00:00:00 –   Borrower added on 01/06/14 > I am getting married 04/05/2014 and I want to have a cushion for expenses just in case.<br>
2014-01-01 00:00:00 – BR called in to push payment date to 09/19/14 because of not having the exact amount of funds in their bank account.  Payment was processing. Was able to cancel. It is within grace period.
2014-01-01 00:00:00 –   Borrower added on 01/01/14 > This loan is to consolidate my credit cards debt. I made one year this past  11/28/2013 at my current job. I considered to have job security because I'm a good employee. I make all may credit cards payments on time.<br>
2013-05-01 00:00:00 –   Borrower added on 04/27/13 > My father passed away 05/12/2012 and I had to pay for the funeral.  My mother could not afford it.  He was not ill so I could not have planned it.  I paid with what I had in my savings and the rest I had to pay with my credit cards.  I would like to pay off the CC &amp; pay one monthly payment.<br><br>  Borrower added on 04/27/13 > My paerents own the house so I do not pay rent.    The utilities, insurance and taxes, etc my mother pays.  She can afford that.  I help when needed.<br>
2013-02-01 00:00:00 –   Borrower added on 02/10/13 > I am getting married in a week (02/17/2013) and have made some large purchases across my credit cards.  I would like to consolidate all of my debt with this low rate loan.<br><br> Borrower added on 02/10/13 > I will be getting married in a week (02/17/13) and have had to make some large purchases on my CC. I am financially sound otherwise with low debt obligations.<br>
2012-12-01 00:00:00 –   Borrower added on 12/10/12 > Approximately 1 year ago I had a highefficency furnace /AC installed.  The installing Co. used GECRB to get me a loan.  If I payoff the loan within one year, I pay no interest.  The interest rate if not payed by 12/23/2012 is 26.99%.  A 6.62% rate sounds a lot better.<br>
2012-11-01 00:00:00 –   Borrower added on 11/19/12 > Looking to finish off consolidating the rest of my bills and lower my payments on my exsisting loan. Thanks!!!<br><br>  Borrower added on 11/20/12 > Thanks again for everyone who has invested thus far. With this loan it will give me the ability to have only one payment monthly besides utilities and I will be almost debt free by my wedding date of 12/13/14!! Thanks again everyone!<br>
2012-10-01 00:00:00 –   Borrower added on 10/22/12 > Need money by 10/26/2012 to purchase property on discounted APR.<br>

Good, all of the dates that come after the month the loan is issued only come up because the borrower is talking about a future event.

Now to clean these desc values up a bit I'm going to remove the Borrower added on [date] >s and the <br>s, since those don't add value to the description content.

def clean_desc(desc):
    if pd.isna(desc):
        return desc
    else:
        return re.sub(
            "\s*Borrower added on \d\d/\d\d/\d\d > |<br>", lambda x: " ", desc
        ).strip()


loans["desc"] = loans["desc"].map(clean_desc)

Imputing missing values

Since only 2% of loans in this set are missing a title, and since most titles simply copy the loan's purpose, I'm going to impute missing titles with their loan's purpose.

loans["title"].fillna(
    loans["purpose"].map(lambda x: x.replace("_", " ").capitalize()), inplace=True
)

Since only 6% of loans use a description, I'll just impute missing descriptions with an empty string. I'm going to wait and include that as a pipeline step a little later, though.

Optimizing data types

I'd really love to get right to the fun part, converting these text fields into document vectors, but I ran into a problem the first several times I tried doing so. Manually adding two sets of 300-dimensional vectors to this 1,110,171-row DataFrame caused its size in memory to skyrocket, exhausting the 16GB Kaggle gives me.

My first attempt to fix this was optimizing my data types, which still didn't solve the problem on its own, but it's a worthwhile step to take anyway.

After removing the issue_d column, which is no longer needed, the dataset contains five types of data: float, integer, ordinal, (unordered) categorical, and text.

from pandas.api.types import CategoricalDtype


loans = loans.drop(columns=["issue_d"])

float_cols = ["annual_inc", "dti", "inv_mths_since_last_delinq",
    "inv_mths_since_last_record", "revol_util", "inv_mths_since_last_major_derog",
    "annual_inc_joint", "dti_joint", "bc_util", "inv_mo_sin_rcnt_rev_tl_op",
    "inv_mo_sin_rcnt_tl", "inv_mths_since_recent_bc", "inv_mths_since_recent_bc_dlq",
    "inv_mths_since_recent_inq", "inv_mths_since_recent_revol_delinq", "pct_tl_nvr_dlq",
    "percent_bc_gt_75", "fraction_recovered"]
int_cols = ["loan_amnt", "delinq_2yrs", "cr_hist_age_mths", "fico_range_low",
    "fico_range_high", "inq_last_6mths", "open_acc", "pub_rec", "revol_bal",
    "total_acc", "collections_12_mths_ex_med", "acc_now_delinq", "tot_coll_amt",
    "tot_cur_bal", "total_rev_hi_lim", "acc_open_past_24mths", "avg_cur_bal",
    "bc_open_to_buy", "chargeoff_within_12_mths", "delinq_amnt", "mo_sin_old_il_acct",
    "mo_sin_old_rev_tl_op", "mort_acc", "num_accts_ever_120_pd", "num_actv_bc_tl",
    "num_actv_rev_tl", "num_bc_sats", "num_bc_tl", "num_il_tl", "num_op_rev_tl",
    "num_rev_accts", "num_rev_tl_bal_gt_0", "num_sats", "num_tl_120dpd_2m",
    "num_tl_30dpd", "num_tl_90g_dpd_24m", "num_tl_op_past_12m", "pub_rec_bankruptcies",
    "tax_liens", "tot_hi_cred_lim", "total_bal_ex_mort", "total_bc_limit",
    "total_il_high_credit_limit"]
ordinal_cols = ["emp_length"]
category_cols = ["term", "home_ownership", "purpose", "application_type"]
text_cols = nlp_cols

size_metrics = pd.DataFrame(
    {
        "previous_dtype": loans.dtypes,
        "previous_size": loans.memory_usage(index=False, deep=True),
    }
)
previous_size = loans.memory_usage(deep=True).sum()


for col_name in float_cols:
    loans[col_name] = pd.to_numeric(loans[col_name], downcast="float")

for col_name in int_cols:
    loans[col_name] = pd.to_numeric(loans[col_name], downcast="unsigned")

emp_length_categories = ["< 1 year", "1 year", "2 years", "3 years", "4 years",
    "5 years", "6 years", "7 years", "8 years", "9 years", "10+ years"]
emp_length_type = CategoricalDtype(categories=emp_length_categories, ordered=True)
loans["emp_length"] = loans["emp_length"].astype(emp_length_type)

for col_name in category_cols:
    loans[col_name] = loans[col_name].astype("category")


current_size = loans.memory_usage(deep=True).sum()
reduction = (previous_size - current_size) / previous_size
print(f"Reduced DataFrame size in memory by {int(reduction * 100)}%.")

size_metrics["current_dtype"] = loans.dtypes
size_metrics["current_size"] = loans.memory_usage(index=False, deep=True)
pd.options.display.max_rows = 100
size_metrics

Reduced DataFrame size in memory by 69%.

	previous_dtype	previous_size	current_dtype	current_size
loan_amnt	float64	8881368	uint16	2220342
term	object	73271286	category	1110383
emp_length	object	71853397	category	1111197
home_ownership	object	69841784	category	1110700
annual_inc	float64	8881368	float32	4440684
purpose	object	79927721	category	1111750
dti	float64	8881368	float32	4440684
delinq_2yrs	float64	8881368	uint8	1110171
cr_hist_age_mths	int64	8881368	uint16	2220342
fico_range_low	float64	8881368	uint16	2220342
fico_range_high	float64	8881368	uint16	2220342
inq_last_6mths	float64	8881368	uint8	1110171
inv_mths_since_last_delinq	float64	8881368	float32	4440684
inv_mths_since_last_record	float64	8881368	float32	4440684
open_acc	float64	8881368	uint8	1110171
pub_rec	float64	8881368	uint8	1110171
revol_bal	float64	8881368	uint32	4440684
revol_util	float64	8881368	float32	4440684
total_acc	float64	8881368	uint8	1110171
collections_12_mths_ex_med	float64	8881368	uint8	1110171
inv_mths_since_last_major_derog	float64	8881368	float32	4440684
application_type	object	74360578	category	1110384
annual_inc_joint	float64	8881368	float32	4440684
dti_joint	float64	8881368	float32	4440684
acc_now_delinq	float64	8881368	uint8	1110171
tot_coll_amt	float64	8881368	uint32	4440684
tot_cur_bal	float64	8881368	uint32	4440684
total_rev_hi_lim	float64	8881368	uint32	4440684
acc_open_past_24mths	float64	8881368	uint8	1110171
avg_cur_bal	float64	8881368	uint32	4440684
bc_open_to_buy	float64	8881368	uint32	4440684
bc_util	float64	8881368	float32	4440684
chargeoff_within_12_mths	float64	8881368	uint8	1110171
delinq_amnt	float64	8881368	uint32	4440684
mo_sin_old_il_acct	float64	8881368	uint16	2220342
mo_sin_old_rev_tl_op	float64	8881368	uint16	2220342
inv_mo_sin_rcnt_rev_tl_op	float64	8881368	float32	4440684
inv_mo_sin_rcnt_tl	float64	8881368	float32	4440684
mort_acc	float64	8881368	uint8	1110171
inv_mths_since_recent_bc	float64	8881368	float32	4440684
inv_mths_since_recent_bc_dlq	float64	8881368	float32	4440684
inv_mths_since_recent_inq	float64	8881368	float32	4440684
inv_mths_since_recent_revol_delinq	float64	8881368	float32	4440684
num_accts_ever_120_pd	float64	8881368	uint8	1110171
num_actv_bc_tl	float64	8881368	uint8	1110171
num_actv_rev_tl	float64	8881368	uint8	1110171
num_bc_sats	float64	8881368	uint8	1110171
num_bc_tl	float64	8881368	uint8	1110171
num_il_tl	float64	8881368	uint8	1110171
num_op_rev_tl	float64	8881368	uint8	1110171
num_rev_accts	float64	8881368	uint8	1110171
num_rev_tl_bal_gt_0	float64	8881368	uint8	1110171
num_sats	float64	8881368	uint8	1110171
num_tl_120dpd_2m	float64	8881368	uint8	1110171
num_tl_30dpd	float64	8881368	uint8	1110171
num_tl_90g_dpd_24m	float64	8881368	uint8	1110171
num_tl_op_past_12m	float64	8881368	uint8	1110171
pct_tl_nvr_dlq	float64	8881368	float32	4440684
percent_bc_gt_75	float64	8881368	float32	4440684
pub_rec_bankruptcies	float64	8881368	uint8	1110171
tax_liens	float64	8881368	uint8	1110171
tot_hi_cred_lim	float64	8881368	uint32	4440684
total_bal_ex_mort	float64	8881368	uint32	4440684
total_bc_limit	float64	8881368	uint32	4440684
total_il_high_credit_limit	float64	8881368	uint32	4440684
fraction_recovered	float64	8881368	float32	4440684
title	object	82840461	object	82840461
desc	object	46918516	object	46918516

Creating document vectors

Now the fun part. Wrapping my spaCy document vector function in a scikit-learn FunctionTransformer turned out to be the secret that kept this process within memory limits. Scikit-learn must just be way better optimized than whatever manual process I was using (go figure).

import spacy
from sklearn.preprocessing import FunctionTransformer


def get_doc_vectors(X):
    n_cols = X.shape[1]
    nlp = spacy.load("en_core_web_lg", disable=["tagger", "parser", "ner"])

    result = []
    for row in X:
        result_row = []
        for i in range(n_cols):
            result_row.append(nlp(row[i]).vector)

        result.append(np.concatenate(result_row))

    return np.array(result)


vectorizer = FunctionTransformer(get_doc_vectors)

Building the pipeline

First, the transformer. I'll use scikit-learn's ColumnTransformer to apply different transformations to different kinds of data.

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer, make_column_selector
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OrdinalEncoder, OneHotEncoder, StandardScaler
from pathlib import Path


def generate_cat_encoder(col_name):
    categories = list(loans[col_name].cat.categories)
    if loans[col_name].cat.ordered:
        return (
            col_name,
            OrdinalEncoder(categories=[categories], dtype=np.uint8),
            [col_name],
        )
    else:
        return (
            col_name,
            OneHotEncoder(categories=[categories], drop="if_binary", dtype=np.bool_),
            [col_name],
        )


Path("../tmp/transformer_cache").mkdir(parents=True, exist_ok=True)
transformer = ColumnTransformer(
    [
        (
            "nlp_cols",
            Pipeline(
                [
                    (
                        "nlp_imputer",
                        SimpleImputer(strategy="constant", fill_value=""),
                    ),
                    ("nlp_vectorizer", vectorizer),
                    ("nlp_scaler", StandardScaler(with_mean=False)),
                ],
                verbose=True,
            ),
            make_column_selector("^(title|desc)$"),
        ),
    ]
    + [generate_cat_encoder(col_name) for col_name in ordinal_cols + category_cols],
    remainder=StandardScaler(),
    verbose=True,
)

This model itself will be identical to my previous model, but I'll use Keras callbacks and a tqdm progress bar to make the training logs much more concise.

import tensorflow as tf
from tensorflow.keras import Sequential, Input
from tensorflow.keras.layers import Dense, Dropout
from sklearn.model_selection import train_test_split
from tqdm import tqdm

np.random.seed(0)
tf.random.set_seed(1)


class ProgressBar(tf.keras.callbacks.Callback):
    def __init__(self, epochs=100):
        self.epochs = epochs

    def on_train_begin(self, logs=None):
        self.progress_bar = tqdm(desc="Training model", total=self.epochs, unit="epoch")

    def on_epoch_end(self, epoch, logs=None):
        self.progress_bar.update()

    def on_train_end(self, logs=None):
        self.progress_bar.close()


class FinalMetrics(tf.keras.callbacks.Callback):
    def on_train_end(self, logs=None):
        metrics_msg = "Final metrics:"
        for metric, value in logs.items():
            metrics_msg += f" {metric}: {value:.5f} -"
        metrics_msg = metrics_msg[:-2]
        print(metrics_msg)


def run_pipeline(X, y, transformer, validate=True):
    X_train, X_val, y_train, y_val = (
        train_test_split(X, y, test_size=0.2, random_state=2)
        if validate
        else (X, None, y, None)
    )

    X_train_t = transformer.fit_transform(X_train)
    X_val_t = transformer.transform(X_val) if validate else None

    model = Sequential()
    model.add(Input((X_train_t.shape[1],)))
    model.add(Dense(64, activation="relu"))
    model.add(Dropout(0.3))
    model.add(Dense(32, activation="relu"))
    model.add(Dropout(0.3))
    model.add(Dense(16, activation="relu"))
    model.add(Dropout(0.3))
    model.add(Dense(1))
    model.compile(optimizer="adam", loss="mean_squared_logarithmic_error")

    history = model.fit(
        X_train_t,
        y_train,
        validation_data=(X_val_t, y_val) if validate else None,
        batch_size=128,
        epochs=100,
        verbose=0,
        callbacks=[ProgressBar(), FinalMetrics()],
    )

    return history.history, model, transformer

Evaluating the model

import dill

history_1, _, _ = run_pipeline(
    loans.drop(columns="fraction_recovered").copy(),
    loans["fraction_recovered"],
    transformer,
)

Path("save_points").mkdir(exist_ok=True)
dill.dump_session("save_points/model_1.pkl")

/opt/conda/lib/python3.7/site-packages/pandas/core/strings.py:2001: UserWarning: This pattern has match groups. To actually get the groups, use str.extract.
  return func(self, *args, **kwargs)


[Pipeline] ....... (step 1 of 3) Processing nlp_imputer, total=   0.4s
[Pipeline] .... (step 2 of 3) Processing nlp_vectorizer, total= 1.2min
[Pipeline] ........ (step 3 of 3) Processing nlp_scaler, total=   8.6s
[ColumnTransformer] ...... (1 of 7) Processing nlp_cols, total= 1.3min
[ColumnTransformer] .... (2 of 7) Processing emp_length, total=   0.2s
[ColumnTransformer] .......... (3 of 7) Processing term, total=   0.3s
[ColumnTransformer]  (4 of 7) Processing home_ownership, total=   0.3s
[ColumnTransformer] ....... (5 of 7) Processing purpose, total=   0.3s
[ColumnTransformer]  (6 of 7) Processing application_type, total=   0.3s
[ColumnTransformer] ..... (7 of 7) Processing remainder, total=   1.3s


Training model: 100%|██████████| 100/100 [23:41<00:00, 14.22s/epoch]


Final metrics: loss: 0.02365 - val_loss: 0.02360

# Restore save point if needed
import dill

try:
    history_1
except NameError:
    dill.load_session("save_points/model_1.pkl")

import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline


def plot_loss_metrics(history, model_num=None):
    for metric, values in history.items():
        sns.lineplot(x=range(len(values)), y=values, label=metric)
    plt.xlabel("epoch")
    plt.title(
        f"Model {f'{model_num} ' if model_num else ''} loss metrics during training"
    )
    plt.show()


plot_loss_metrics(history_1, "1")

Well, it didn't overfit, but this model performed a bit worse than my original, which had settled around a loss of 0.0231. I bet the desc feature is getting in the way—zeroes spanning 300 columns of the input data on 94% of the rows is probably quite confusing to the model. I'll see what happens if I repeat the process while leaving desc out (making the title vectors the only new feature of this model compared to my original).

history_2, _, _ = run_pipeline(
    loans.drop(columns=["fraction_recovered", "desc"]).copy(),
    loans["fraction_recovered"],
    transformer,
)

Path("save_points").mkdir(exist_ok=True)
dill.dump_session("save_points/model_2.pkl")

/opt/conda/lib/python3.7/site-packages/pandas/core/strings.py:2001: UserWarning: This pattern has match groups. To actually get the groups, use str.extract.
  return func(self, *args, **kwargs)


[Pipeline] ....... (step 1 of 3) Processing nlp_imputer, total=   0.1s
[Pipeline] .... (step 2 of 3) Processing nlp_vectorizer, total=  41.3s
[Pipeline] ........ (step 3 of 3) Processing nlp_scaler, total=   4.6s
[ColumnTransformer] ...... (1 of 7) Processing nlp_cols, total=  45.9s
[ColumnTransformer] .... (2 of 7) Processing emp_length, total=   0.2s
[ColumnTransformer] .......... (3 of 7) Processing term, total=   0.3s
[ColumnTransformer]  (4 of 7) Processing home_ownership, total=   0.3s
[ColumnTransformer] ....... (5 of 7) Processing purpose, total=   0.3s
[ColumnTransformer]  (6 of 7) Processing application_type, total=   0.3s
[ColumnTransformer] ..... (7 of 7) Processing remainder, total=   1.1s


Training model: 100%|██████████| 100/100 [22:26<00:00, 13.46s/epoch]


Final metrics: loss: 0.02396 - val_loss: 0.02451

# Restore save point if needed
import dill

try:
    history_2
except NameError:
    dill.load_session("save_points/model_2.pkl")

plot_loss_metrics(history_2, "2")

Wow, still not good enough to beat my original model. Just for kicks, I also tried additional runs where I trained for 1,000 epochs, and others where I increased the numbers of nodes in the first two dense layers to 128 and 64. And I tried decreasing the batch size to 64. But still none of these beat my original model. I suppose these text features have no predictive quality to them in terms of loan outcomes. Interesting.

Next steps

If adding these two features decreased predictive capability, then perhaps some of the other variables I was already using are doing the same thing. I should try using some of scikit-learn's feature selection methods to reduce the dimensionality of the input data.

A more efficient method of hyperparameter optimization would be pretty useful as well. I should give AutoKeras a shot.

Well that was fun! Have any thoughts on how to better integrate language data into the model? I'd love to hear them in the discussion below.

Can I Grade Loans Better Than LendingClub?

Ty Mick — Sat, 24 Oct 2020 18:34:55 +0000

Pitting My Neural Network Against a Corporate Benchmark

Introduction
Ground rules
Test metric
LendingClub's turn
My turn
Victory!

Introduction

In case you missed it, I built a neural network to predict loan risk using a public dataset from LendingClub. Then I built a public API to serve the model's predictions. That's nice and all, but… how good is my model?

Today I'm going to put it to the test, pitting it against the risk models of the very institution who issued those loans. That's right, LendingClub included their own calculated loan grades (and sub-grades) in the dataset, so all the pieces are in place for the most thrilling risk modeling smackdown of the ~~century~~ week. May the best algorithm win!

import joblib

prev_notebook_folder = "../input/building-a-neural-network-to-predict-loan-risk/"
loans = joblib.load(prev_notebook_folder + "loans_for_eval.joblib")
loans.shape

(1110171, 70)

loans.head()

	loan_amnt	term	emp_length	home_ownership	annual_inc	purpose	dti	delinq_2yrs	cr_hist_age_mths	fico_range_low	...	tot_hi_cred_lim	total_bal_ex_mort	total_bc_limit	total_il_high_credit_limit	fraction_recovered	issue_d	grade	sub_grade	expected_return
0	3600.0	36 months	10+ years	MORTGAGE	55000.0	debt_consolidation	5.91	0.0	148.0	675.0	...	178050.0	7746.0	2400.0	13734.0	1.0	Dec-2015	C	C4	4429.08
1	24700.0	36 months	10+ years	MORTGAGE	65000.0	small_business	16.06	1.0	192.0	715.0	...	314017.0	39475.0	79300.0	24667.0	1.0	Dec-2015	C	C1	29530.08
2	20000.0	60 months	10+ years	MORTGAGE	63000.0	home_improvement	10.78	0.0	184.0	695.0	...	218418.0	18696.0	6200.0	14877.0	1.0	Dec-2015	B	B4	25959.60
4	10400.0	60 months	3 years	MORTGAGE	104433.0	major_purchase	25.37	1.0	210.0	695.0	...	439570.0	95768.0	20300.0	88097.0	1.0	Dec-2015	F	F1	17394.60
5	11950.0	36 months	4 years	RENT	34000.0	debt_consolidation	10.20	0.0	338.0	690.0	...	16900.0	12798.0	9400.0	4000.0	1.0	Dec-2015	C	C3	14586.48

5 rows × 70 columns

This post was adapted from a Jupyter Notebook, by the way, so if you'd like to follow along in your own notebook, go ahead and fork mine Kaggle or GitHub!

Ground rules

This is going to be a clean fight—my model won't use any data LendingClub wouldn't have access to at the point they calculate a loan's grade (including the grade itself).

I'm going to sort the dataset chronologically (using the issue_d column, the month and year the loan was issued) and split it into two parts. The first 80% I'll use for training my competition model, and I'll compare performance on the last 20%.

from sklearn.model_selection import train_test_split

loans["date"] = loans["issue_d"].astype("datetime64[ns]")
loans.sort_values("date", axis="index", inplace=True, kind="mergesort")

train, test = train_test_split(loans, test_size=0.2, shuffle=False)
train, test = train.copy(), test.copy()
print(f"The test set contains {len(test):,} loans.")

The test set contains 222,035 loans.

At the earlier end of the test set my model may have a slight informational advantage, having been trained on a few loans that may not have closed yet at the point LendingClub was grading those ones. On the other hand, LendingClub may have a slight informational advantage on the later end of the test set, since they would have known the outcomes of some loans on the earlier end of the test set by that time.

I have to give credit to Michael Wurm, by the way, for the idea of comparing my model's performance to LendingClub's loan grades, but my approach is pretty different. I'm not trying to simulate the performance of an investment portfolio; I'm just evaluating how well my predictions of simple risk compare.

Test metric

The test: who can pick the best set of grade A loans, judged on the basis of the independent variable from my last notebook, the fraction of an expected loan return that a prospective borrower will pay back (which I engineered as fraction_recovered).

LendingClub will take the plate first. I'll gather all their grade A loans from the test set, count them, and calculate their average fraction_recovered. That average will be the metric my model has to beat.

Then I'll train my model on the training set using the same pipeline and parameters I settled on in my last notebook. Once it's trained, I'll use it to make predictions on the test set, then gather the number of top predictions equal to the number of LendingClub's grade A loans. Finally, I'll calculate the same average of fraction_recovered on that subset, and we'll have ourselves a winner!

LendingClub's turn

from statistics import mean

lc_grade_a = test[test["grade"] == "A"]
print(f"LendingClub gave {len(lc_grade_a):,} loans in the test set an A grade.")

print("\nAverage `fraction_recovered` on LendingClub's grade A loans:")
print(round(mean(lc_grade_a["fraction_recovered"]), 5))

LendingClub gave 38,779 loans in the test set an A grade.

Average `fraction_recovered` on LendingClub's grade A loans:
0.96021

That's a pretty high percentage. I'm a bit nervous.

My turn

First, I'll copy over my run_pipeline function from my previous notebook:

from sklearn.model_selection import train_test_split
from sklearn_pandas import DataFrameMapper
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler
from tensorflow.keras import Sequential, Input
from tensorflow.keras.layers import Dense, Dropout


def run_pipeline(
    data,
    onehot_cols,
    ordinal_cols,
    batch_size,
    validate=True,
):
    X = data.drop(columns=["fraction_recovered"])
    y = data["fraction_recovered"]
    X_train, X_valid, y_train, y_valid = (
        train_test_split(X, y, test_size=0.2, random_state=0)
        if validate
        else (X, None, y, None)
    )

    transformer = DataFrameMapper(
        [
            (onehot_cols, OneHotEncoder(drop="if_binary")),
            (
                list(ordinal_cols.keys()),
                OrdinalEncoder(categories=list(ordinal_cols.values())),
            ),
        ],
        default=StandardScaler(),
    )

    X_train = transformer.fit_transform(X_train)
    X_valid = transformer.transform(X_valid) if validate else None

    input_nodes = X_train.shape[1]
    output_nodes = 1

    model = Sequential()
    model.add(Input((input_nodes,)))
    model.add(Dense(64, activation="relu"))
    model.add(Dropout(0.3, seed=0))
    model.add(Dense(32, activation="relu"))
    model.add(Dropout(0.3, seed=1))
    model.add(Dense(16, activation="relu"))
    model.add(Dropout(0.3, seed=2))
    model.add(Dense(output_nodes))
    model.compile(optimizer="adam", loss="mean_squared_logarithmic_error")

    history = model.fit(
        X_train,
        y_train,
        batch_size=batch_size,
        epochs=100,
        validation_data=(X_valid, y_valid) if validate else None,
        verbose=2,
    )

    return history.history, model, transformer


onehot_cols = ["term", "application_type", "home_ownership", "purpose"]
ordinal_cols = {
    "emp_length": [
        "< 1 year",
        "1 year",
        "2 years",
        "3 years",
        "4 years",
        "5 years",
        "6 years",
        "7 years",
        "8 years",
        "9 years",
        "10+ years",
    ]
}

Now for the moment of truth:

# Train the model
_, model, transformer = run_pipeline(
    train.drop(columns=["issue_d", "date", "grade", "sub_grade", "expected_return"]),
    onehot_cols,
    ordinal_cols,
    batch_size=128,
    validate=False,
)

# Make predictions
X_test = transformer.transform(
    test.drop(
        columns=[
            "fraction_recovered",
            "issue_d",
            "date",
            "grade",
            "sub_grade",
            "expected_return",
        ]
    )
)
test["model_predictions"] = model.predict(X_test)

# Gather top predictions
test_sorted = test.sort_values("model_predictions", axis="index", ascending=False)
ty_grade_a = test_sorted.iloc[0:len(lc_grade_a)]

# Display results
print("\nAverage `fraction_recovered` on Ty's grade A loans:")
print(format(mean(ty_grade_a["fraction_recovered"]), ".5f"))

Epoch 1/100
6939/6939 - 13s - loss: 0.0249
Epoch 2/100
6939/6939 - 13s - loss: 0.0204
Epoch 3/100
6939/6939 - 13s - loss: 0.0202
Epoch 4/100
6939/6939 - 13s - loss: 0.0202
Epoch 5/100
6939/6939 - 13s - loss: 0.0202
Epoch 6/100
6939/6939 - 14s - loss: 0.0201
Epoch 7/100
6939/6939 - 14s - loss: 0.0201
Epoch 8/100
6939/6939 - 14s - loss: 0.0201
Epoch 9/100
6939/6939 - 13s - loss: 0.0201
Epoch 10/100
6939/6939 - 12s - loss: 0.0201
Epoch 11/100
6939/6939 - 13s - loss: 0.0201
Epoch 12/100
6939/6939 - 13s - loss: 0.0201
Epoch 13/100
6939/6939 - 13s - loss: 0.0201
Epoch 14/100
6939/6939 - 13s - loss: 0.0201
Epoch 15/100
6939/6939 - 12s - loss: 0.0201
Epoch 16/100
6939/6939 - 12s - loss: 0.0201
Epoch 17/100
6939/6939 - 13s - loss: 0.0200
Epoch 18/100
6939/6939 - 13s - loss: 0.0200
Epoch 19/100
6939/6939 - 13s - loss: 0.0200
Epoch 20/100
6939/6939 - 14s - loss: 0.0200
Epoch 21/100
6939/6939 - 13s - loss: 0.0200
Epoch 22/100
6939/6939 - 13s - loss: 0.0200
Epoch 23/100
6939/6939 - 12s - loss: 0.0200
Epoch 24/100
6939/6939 - 12s - loss: 0.0200
Epoch 25/100
6939/6939 - 12s - loss: 0.0200
Epoch 26/100
6939/6939 - 13s - loss: 0.0200
Epoch 27/100
6939/6939 - 13s - loss: 0.0200
Epoch 28/100
6939/6939 - 13s - loss: 0.0200
Epoch 29/100
6939/6939 - 13s - loss: 0.0200
Epoch 30/100
6939/6939 - 13s - loss: 0.0200
Epoch 31/100
6939/6939 - 15s - loss: 0.0200
Epoch 32/100
6939/6939 - 13s - loss: 0.0200
Epoch 33/100
6939/6939 - 12s - loss: 0.0200
Epoch 34/100
6939/6939 - 13s - loss: 0.0200
Epoch 35/100
6939/6939 - 13s - loss: 0.0200
Epoch 36/100
6939/6939 - 13s - loss: 0.0200
Epoch 37/100
6939/6939 - 13s - loss: 0.0200
Epoch 38/100
6939/6939 - 13s - loss: 0.0200
Epoch 39/100
6939/6939 - 13s - loss: 0.0200
Epoch 40/100
6939/6939 - 13s - loss: 0.0200
Epoch 41/100
6939/6939 - 13s - loss: 0.0200
Epoch 42/100
6939/6939 - 13s - loss: 0.0200
Epoch 43/100
6939/6939 - 14s - loss: 0.0200
Epoch 44/100
6939/6939 - 13s - loss: 0.0200
Epoch 45/100
6939/6939 - 13s - loss: 0.0200
Epoch 46/100
6939/6939 - 13s - loss: 0.0200
Epoch 47/100
6939/6939 - 13s - loss: 0.0200
Epoch 48/100
6939/6939 - 13s - loss: 0.0200
Epoch 49/100
6939/6939 - 13s - loss: 0.0200
Epoch 50/100
6939/6939 - 13s - loss: 0.0200
Epoch 51/100
6939/6939 - 13s - loss: 0.0200
Epoch 52/100
6939/6939 - 13s - loss: 0.0200
Epoch 53/100
6939/6939 - 13s - loss: 0.0200
Epoch 54/100
6939/6939 - 14s - loss: 0.0200
Epoch 55/100
6939/6939 - 14s - loss: 0.0200
Epoch 56/100
6939/6939 - 13s - loss: 0.0200
Epoch 57/100
6939/6939 - 13s - loss: 0.0200
Epoch 58/100
6939/6939 - 13s - loss: 0.0200
Epoch 59/100
6939/6939 - 13s - loss: 0.0200
Epoch 60/100
6939/6939 - 13s - loss: 0.0200
Epoch 61/100
6939/6939 - 13s - loss: 0.0200
Epoch 62/100
6939/6939 - 13s - loss: 0.0200
Epoch 63/100
6939/6939 - 13s - loss: 0.0200
Epoch 64/100
6939/6939 - 13s - loss: 0.0200
Epoch 65/100
6939/6939 - 12s - loss: 0.0200
Epoch 66/100
6939/6939 - 13s - loss: 0.0200
Epoch 67/100
6939/6939 - 14s - loss: 0.0200
Epoch 68/100
6939/6939 - 13s - loss: 0.0200
Epoch 69/100
6939/6939 - 13s - loss: 0.0200
Epoch 70/100
6939/6939 - 13s - loss: 0.0200
Epoch 71/100
6939/6939 - 13s - loss: 0.0200
Epoch 72/100
6939/6939 - 13s - loss: 0.0200
Epoch 73/100
6939/6939 - 13s - loss: 0.0200
Epoch 74/100
6939/6939 - 13s - loss: 0.0200
Epoch 75/100
6939/6939 - 13s - loss: 0.0200
Epoch 76/100
6939/6939 - 13s - loss: 0.0200
Epoch 77/100
6939/6939 - 13s - loss: 0.0200
Epoch 78/100
6939/6939 - 13s - loss: 0.0200
Epoch 79/100
6939/6939 - 14s - loss: 0.0200
Epoch 80/100
6939/6939 - 13s - loss: 0.0200
Epoch 81/100
6939/6939 - 13s - loss: 0.0200
Epoch 82/100
6939/6939 - 13s - loss: 0.0200
Epoch 83/100
6939/6939 - 13s - loss: 0.0200
Epoch 84/100
6939/6939 - 12s - loss: 0.0200
Epoch 85/100
6939/6939 - 13s - loss: 0.0200
Epoch 86/100
6939/6939 - 13s - loss: 0.0200
Epoch 87/100
6939/6939 - 13s - loss: 0.0200
Epoch 88/100
6939/6939 - 13s - loss: 0.0200
Epoch 89/100
6939/6939 - 13s - loss: 0.0200
Epoch 90/100
6939/6939 - 13s - loss: 0.0200
Epoch 91/100
6939/6939 - 14s - loss: 0.0200
Epoch 92/100
6939/6939 - 13s - loss: 0.0200
Epoch 93/100
6939/6939 - 13s - loss: 0.0200
Epoch 94/100
6939/6939 - 13s - loss: 0.0200
Epoch 95/100
6939/6939 - 13s - loss: 0.0200
Epoch 96/100
6939/6939 - 13s - loss: 0.0200
Epoch 97/100
6939/6939 - 13s - loss: 0.0200
Epoch 98/100
6939/6939 - 13s - loss: 0.0200
Epoch 99/100
6939/6939 - 13s - loss: 0.0200
Epoch 100/100
6939/6939 - 13s - loss: 0.0200

Average `fraction_recovered` on Ty's grade A loans:
0.96166

Victory!

Phew, that was a close one! My win might be too small to be statistically significant, but hey, it's cool seeing that I can keep up with LendingClub's best and brightest.

What I'd really like to know now is what quantitative range of estimated risk each LendingClub grade and sub-grade corresponds to, but it looks like that's proprietary. Does anyone know if loans grades generally correspond to certain percentage ranges like letter grades in academic classes? If not, have any ideas for better benchmarks I could use to evaluate my model's performance? Go ahead and chime in in the discussion below.

Building a Neural Network to Predict Loan Risk

Ty Mick — Mon, 03 Aug 2020 16:10:24 +0000

or, Ty Goes Into Far Too Much Detail About Cleaning Data

Introduction
Data cleaning
Building the neural networks
Saving the final model
Building the API
Further reading

Introduction

LendingClub is the world's largest peer-to-peer lending platform. Until recently (through the end of 2018), LendingClub published a public dataset of all loans issued since the company's launch in 2007. I'm accessing the dataset via Kaggle.

import pandas as pd

loans_raw = pd.read_csv(
    "../input/lending-club/accepted_2007_to_2018q4.csv/accepted_2007_to_2018Q4.csv",
    low_memory=False,
)

loans_raw.shape

(2260701, 151)

With 2,260,701 loans to look at and 151 potential variables, my goal is to create a neural network model to predict the fraction of an expected loan return that a prospective borrower will pay back. (Spoiler alert: my model can pick grade A loans better than LendingClub).

Afterward, I'll create a public API to serve that model.

Also, as you may have guessed from the preceding code block, this post is adapted from a Jupyter Notebook. If you'd like to follow along in your own notebook, go ahead and fork mine on Kaggle or GitHub.

Data cleaning

I'll first look at the data dictionary (downloaded directly from LendingClub's website) to get an idea of how to create the desired output variable and which remaining features are available at the point of loan application (to avoid data leakage).

dictionary_df = pd.read_excel("https://resources.lendingclub.com/LCDataDictionary.xlsx")

# Drop blank rows, strip white space, convert to Python dictionary, fix one key name
dictionary_df.dropna(axis="index", inplace=True)
dictionary_df = dictionary_df.applymap(lambda x: x.strip())
dictionary_df.set_index("LoanStatNew", inplace=True)
dictionary = dictionary_df["Description"].to_dict()
dictionary["verification_status_joint"] = dictionary.pop("verified_status_joint")

# Print in order of dataset columns (which makes more sense than dictionary's order)
for col in loans_raw.columns:
    print(f"•{col}: {dictionary[col]}")

•id: A unique LC assigned ID for the loan listing.
•member_id: A unique LC assigned Id for the borrower member.
•loan_amnt: The listed amount of the loan applied for by the borrower. If at some point in time, the credit department reduces the loan amount, then it will be reflected in this value.
•funded_amnt: The total amount committed to that loan at that point in time.
•funded_amnt_inv: The total amount committed by investors for that loan at that point in time.
•term: The number of payments on the loan. Values are in months and can be either 36 or 60.
•int_rate: Interest Rate on the loan
•installment: The monthly payment owed by the borrower if the loan originates.
•grade: LC assigned loan grade
•sub_grade: LC assigned loan subgrade
•emp_title: The job title supplied by the Borrower when applying for the loan.*
•emp_length: Employment length in years. Possible values are between 0 and 10 where 0 means less than one year and 10 means ten or more years.
•home_ownership: The home ownership status provided by the borrower during registration or obtained from the credit report. Our values are: RENT, OWN, MORTGAGE, OTHER
•annual_inc: The self-reported annual income provided by the borrower during registration.
•verification_status: Indicates if income was verified by LC, not verified, or if the income source was verified
•issue_d: The month which the loan was funded
•loan_status: Current status of the loan
•pymnt_plan: Indicates if a payment plan has been put in place for the loan
•url: URL for the LC page with listing data.
•desc: Loan description provided by the borrower
•purpose: A category provided by the borrower for the loan request.
•title: The loan title provided by the borrower
•zip_code: The first 3 numbers of the zip code provided by the borrower in the loan application.
•addr_state: The state provided by the borrower in the loan application
•dti: A ratio calculated using the borrower’s total monthly debt payments on the total debt obligations, excluding mortgage and the requested LC loan, divided by the borrower’s self-reported monthly income.
•delinq_2yrs: The number of 30+ days past-due incidences of delinquency in the borrower's credit file for the past 2 years
•earliest_cr_line: The month the borrower's earliest reported credit line was opened
•fico_range_low: The lower boundary range the borrower’s FICO at loan origination belongs to.
•fico_range_high: The upper boundary range the borrower’s FICO at loan origination belongs to.
•inq_last_6mths: The number of inquiries in past 6 months (excluding auto and mortgage inquiries)
•mths_since_last_delinq: The number of months since the borrower's last delinquency.
•mths_since_last_record: The number of months since the last public record.
•open_acc: The number of open credit lines in the borrower's credit file.
•pub_rec: Number of derogatory public records
•revol_bal: Total credit revolving balance
•revol_util: Revolving line utilization rate, or the amount of credit the borrower is using relative to all available revolving credit.
•total_acc: The total number of credit lines currently in the borrower's credit file
•initial_list_status: The initial listing status of the loan. Possible values are – W, F
•out_prncp: Remaining outstanding principal for total amount funded
•out_prncp_inv: Remaining outstanding principal for portion of total amount funded by investors
•total_pymnt: Payments received to date for total amount funded
•total_pymnt_inv: Payments received to date for portion of total amount funded by investors
•total_rec_prncp: Principal received to date
•total_rec_int: Interest received to date
•total_rec_late_fee: Late fees received to date
•recoveries: post charge off gross recovery
•collection_recovery_fee: post charge off collection fee
•last_pymnt_d: Last month payment was received
•last_pymnt_amnt: Last total payment amount received
•next_pymnt_d: Next scheduled payment date
•last_credit_pull_d: The most recent month LC pulled credit for this loan
•last_fico_range_high: The upper boundary range the borrower’s last FICO pulled belongs to.
•last_fico_range_low: The lower boundary range the borrower’s last FICO pulled belongs to.
•collections_12_mths_ex_med: Number of collections in 12 months excluding medical collections
•mths_since_last_major_derog: Months since most recent 90-day or worse rating
•policy_code: publicly available policy_code=1
new products not publicly available policy_code=2
•application_type: Indicates whether the loan is an individual application or a joint application with two co-borrowers
•annual_inc_joint: The combined self-reported annual income provided by the co-borrowers during registration
•dti_joint: A ratio calculated using the co-borrowers' total monthly payments on the total debt obligations, excluding mortgages and the requested LC loan, divided by the co-borrowers' combined self-reported monthly income
•verification_status_joint: Indicates if the co-borrowers' joint income was verified by LC, not verified, or if the income source was verified
•acc_now_delinq: The number of accounts on which the borrower is now delinquent.
•tot_coll_amt: Total collection amounts ever owed
•tot_cur_bal: Total current balance of all accounts
•open_acc_6m: Number of open trades in last 6 months
•open_act_il: Number of currently active installment trades
•open_il_12m: Number of installment accounts opened in past 12 months
•open_il_24m: Number of installment accounts opened in past 24 months
•mths_since_rcnt_il: Months since most recent installment accounts opened
•total_bal_il: Total current balance of all installment accounts
•il_util: Ratio of total current balance to high credit/credit limit on all install acct
•open_rv_12m: Number of revolving trades opened in past 12 months
•open_rv_24m: Number of revolving trades opened in past 24 months
•max_bal_bc: Maximum current balance owed on all revolving accounts
•all_util: Balance to credit limit on all trades
•total_rev_hi_lim: Total revolving high credit/credit limit
•inq_fi: Number of personal finance inquiries
•total_cu_tl: Number of finance trades
•inq_last_12m: Number of credit inquiries in past 12 months
•acc_open_past_24mths: Number of trades opened in past 24 months.
•avg_cur_bal: Average current balance of all accounts
•bc_open_to_buy: Total open to buy on revolving bankcards.
•bc_util: Ratio of total current balance to high credit/credit limit for all bankcard accounts.
•chargeoff_within_12_mths: Number of charge-offs within 12 months
•delinq_amnt: The past-due amount owed for the accounts on which the borrower is now delinquent.
•mo_sin_old_il_acct: Months since oldest bank installment account opened
•mo_sin_old_rev_tl_op: Months since oldest revolving account opened
•mo_sin_rcnt_rev_tl_op: Months since most recent revolving account opened
•mo_sin_rcnt_tl: Months since most recent account opened
•mort_acc: Number of mortgage accounts.
•mths_since_recent_bc: Months since most recent bankcard account opened.
•mths_since_recent_bc_dlq: Months since most recent bankcard delinquency
•mths_since_recent_inq: Months since most recent inquiry.
•mths_since_recent_revol_delinq: Months since most recent revolving delinquency.
•num_accts_ever_120_pd: Number of accounts ever 120 or more days past due
•num_actv_bc_tl: Number of currently active bankcard accounts
•num_actv_rev_tl: Number of currently active revolving trades
•num_bc_sats: Number of satisfactory bankcard accounts
•num_bc_tl: Number of bankcard accounts
•num_il_tl: Number of installment accounts
•num_op_rev_tl: Number of open revolving accounts
•num_rev_accts: Number of revolving accounts
•num_rev_tl_bal_gt_0: Number of revolving trades with balance >0
•num_sats: Number of satisfactory accounts
•num_tl_120dpd_2m: Number of accounts currently 120 days past due (updated in past 2 months)
•num_tl_30dpd: Number of accounts currently 30 days past due (updated in past 2 months)
•num_tl_90g_dpd_24m: Number of accounts 90 or more days past due in last 24 months
•num_tl_op_past_12m: Number of accounts opened in past 12 months
•pct_tl_nvr_dlq: Percent of trades never delinquent
•percent_bc_gt_75: Percentage of all bankcard accounts > 75% of limit.
•pub_rec_bankruptcies: Number of public record bankruptcies
•tax_liens: Number of tax liens
•tot_hi_cred_lim: Total high credit/credit limit
•total_bal_ex_mort: Total credit balance excluding mortgage
•total_bc_limit: Total bankcard high credit/credit limit
•total_il_high_credit_limit: Total installment high credit/credit limit
•revol_bal_joint: Sum of revolving credit balance of the co-borrowers, net of duplicate balances
•sec_app_fico_range_low: FICO range (high) for the secondary applicant
•sec_app_fico_range_high: FICO range (low) for the secondary applicant
•sec_app_earliest_cr_line: Earliest credit line at time of application for the secondary applicant
•sec_app_inq_last_6mths: Credit inquiries in the last 6 months at time of application for the secondary applicant
•sec_app_mort_acc: Number of mortgage accounts at time of application for the secondary applicant
•sec_app_open_acc: Number of open trades at time of application for the secondary applicant
•sec_app_revol_util: Ratio of total current balance to high credit/credit limit for all revolving accounts
•sec_app_open_act_il: Number of currently active installment trades at time of application for the secondary applicant
•sec_app_num_rev_accts: Number of revolving accounts at time of application for the secondary applicant
•sec_app_chargeoff_within_12_mths: Number of charge-offs within last 12 months at time of application for the secondary applicant
•sec_app_collections_12_mths_ex_med: Number of collections within last 12 months excluding medical collections at time of application for the secondary applicant
•sec_app_mths_since_last_major_derog: Months since most recent 90-day or worse rating at time of application for the secondary applicant
•hardship_flag: Flags whether or not the borrower is on a hardship plan
•hardship_type: Describes the hardship plan offering
•hardship_reason: Describes the reason the hardship plan was offered
•hardship_status: Describes if the hardship plan is active, pending, canceled, completed, or broken
•deferral_term: Amount of months that the borrower is expected to pay less than the contractual monthly payment amount due to a hardship plan
•hardship_amount: The interest payment that the borrower has committed to make each month while they are on a hardship plan
•hardship_start_date: The start date of the hardship plan period
•hardship_end_date: The end date of the hardship plan period
•payment_plan_start_date: The day the first hardship plan payment is due. For example, if a borrower has a hardship plan period of 3 months, the start date is the start of the three-month period in which the borrower is allowed to make interest-only payments.
•hardship_length: The number of months the borrower will make smaller payments than normally obligated due to a hardship plan
•hardship_dpd: Account days past due as of the hardship plan start date
•hardship_loan_status: Loan Status as of the hardship plan start date
•orig_projected_additional_accrued_interest: The original projected additional interest amount that will accrue for the given hardship payment plan as of the Hardship Start Date. This field will be null if the borrower has broken their hardship payment plan.
•hardship_payoff_balance_amount: The payoff balance amount as of the hardship plan start date
•hardship_last_payment_amount: The last payment amount as of the hardship plan start date
•disbursement_method: The method by which the borrower receives their loan. Possible values are: CASH, DIRECT_PAY
•debt_settlement_flag: Flags whether or not the borrower, who has charged-off, is working with a debt-settlement company.
•debt_settlement_flag_date: The most recent date that the Debt_Settlement_Flag has been set
•settlement_status: The status of the borrower’s settlement plan. Possible values are: COMPLETE, ACTIVE, BROKEN, CANCELLED, DENIED, DRAFT
•settlement_date: The date that the borrower agrees to the settlement plan
•settlement_amount: The loan amount that the borrower has agreed to settle for
•settlement_percentage: The settlement amount as a percentage of the payoff balance amount on the loan
•settlement_term: The number of months that the borrower will be on the settlement plan

For the output variable (the fraction of expected return that was recovered), I'll calculated the expected return by multiplying the monthly payment amount (installment) by the number of payments on the loan (term), and I'll calculate the amount actually received by summing the total principle, interest, late fees, and post-chargeoff gross recovery received (total_rec_prncp, total_rec_int, total_rec_late_fee, recoveries) and subtracting any collection fee (collection_recovery_fee).

cols_for_output = ["term", "installment", "total_rec_prncp", "total_rec_int", "total_rec_late_fee", "recoveries", "collection_recovery_fee"]

Several other columns contain either irrelevant demographic data or data not created until after a loan is accepted, so those will need to be removed. I'll hold onto issue_d (the month and year the loan was funded) for now, though, in case I want to compare variables to the date of the loan.

emp_title (the applicant's job title) does seem relevant in the context of a loan, but it may have too many unique values to be useful.

loans_raw["emp_title"].nunique()

Too many unique values indeed. In a future version of this model I could perhaps try to generate a feature from this column by aggregating job titles into categories, but that effort may have a low return on investment, since there are already columns for annual income and length of employment.

Two other interesting columns that I'll also remove are title and desc ("description"), which are both freeform text entries written by the borrower. These could be fascinating subjects for natural language processing, but that's outside the scope of the current project. Perhaps in the future I could generate additional features from these fields using measures like syntactic complexity, word count, or keyword inclusion. (I ended up exploring this line of thinking in a later post using document vectors.)

cols_to_drop = ["id", "member_id", "funded_amnt", "funded_amnt_inv", "int_rate", "grade", "sub_grade", "emp_title", "pymnt_plan", "url", "desc", "title", "zip_code", "addr_state", "initial_list_status", "out_prncp", "out_prncp_inv", "total_pymnt", "total_pymnt_inv", "last_pymnt_d", "last_pymnt_amnt", "next_pymnt_d", "last_credit_pull_d", "last_fico_range_high", "last_fico_range_low", "policy_code", "hardship_flag", "hardship_type", "hardship_reason", "hardship_status", "deferral_term", "hardship_amount", "hardship_start_date", "hardship_end_date", "payment_plan_start_date", "hardship_length", "hardship_dpd", "hardship_loan_status", "orig_projected_additional_accrued_interest", "hardship_payoff_balance_amount", "hardship_last_payment_amount", "disbursement_method", "debt_settlement_flag", "debt_settlement_flag_date", "settlement_status", "settlement_date", "settlement_amount", "settlement_percentage", "settlement_term"]
loans = loans_raw.drop(columns=cols_to_drop)

Before creating the output variable, however, I must take a closer look at loan_status, to see if any loans in the dataset are still open.

loans.groupby("loan_status")["loan_status"].count()

loan_status
Charged Off                                             268559
Current                                                 878317
Default                                                     40
Does not meet the credit policy. Status:Charged Off        761
Does not meet the credit policy. Status:Fully Paid        1988
Fully Paid                                             1076751
In Grace Period                                           8436
Late (16-30 days)                                         4349
Late (31-120 days)                                       21467
Name: loan_status, dtype: int64

For practical purposes, I'll consider loans with statuses that don't contain "Fully Paid" or "Charged Off" to still be open, so I'll remove those from the dataset. I'll also merge the "credit policy" columns with their matching status.

credit_policy = "Does not meet the credit policy. Status:"
len_credit_policy = len(credit_policy)
remove_credit_policy = (
    lambda status: status[len_credit_policy:]
    if credit_policy in str(status)
    else status
)
loans["loan_status"] = loans["loan_status"].map(remove_credit_policy)

rows_to_drop = loans[
    (loans["loan_status"] != "Charged Off") & (loans["loan_status"] != "Fully Paid")
].index
loans.drop(index=rows_to_drop, inplace=True)

loans.groupby("loan_status")["loan_status"].count()

loan_status
Charged Off     269320
Fully Paid     1078739
Name: loan_status, dtype: int64

Now to create the output variable. I'll start by checking the null counts of the variables involved.

loans[cols_for_output].info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 1348059 entries, 0 to 2260697
Data columns (total 7 columns):
 #   Column                   Non-Null Count    Dtype
---  ------                   --------------    -----
 0   term                     1348059 non-null  object
 1   installment              1348059 non-null  float64
 2   total_rec_prncp          1348059 non-null  float64
 3   total_rec_int            1348059 non-null  float64
 4   total_rec_late_fee       1348059 non-null  float64
 5   recoveries               1348059 non-null  float64
 6   collection_recovery_fee  1348059 non-null  float64
dtypes: float64(6), object(1)
memory usage: 82.3+ MB

Every remaining row has each of these seven variables, but term's data type is object, so that needs to be fixed first.

loans.groupby("term")["term"].count()

term
 36 months    1023181
 60 months     324878
Name: term, dtype: int64

Ah, so term is a categorical feature with two options. I'll treat it as such when I use it as an input to the model, but to calculate the output variable I'll create a numerical column from it.

Also, I need to trim the whitespace from the beginning of those values—that's no good.

onehot_cols = ["term"]

loans["term"] = loans["term"].map(lambda term_str: term_str.strip())

extract_num = lambda term_str: float(term_str[:2])
loans["term_num"] = loans["term"].map(extract_num)
cols_for_output.remove("term")
cols_for_output.append("term_num")

Now I can create the output variable.

received = (
    loans["total_rec_prncp"]
    + loans["total_rec_int"]
    + loans["total_rec_late_fee"]
    + loans["recoveries"]
    - loans["collection_recovery_fee"]
)
expected = loans["installment"] * loans["term_num"]
loans["fraction_recovered"] = received / expected

loans.groupby("loan_status")["fraction_recovered"].describe()

	count	mean	std	min	25%	50%	75%	max
loan_status
Charged Off	269320.0	0.400162	0.219020	0.000000	0.224463	0.367554	0.550924	2.410680
Fully Paid	1078739.0	0.932705	0.100455	0.506053	0.897912	0.960100	0.997612	60.932353

There is at least one odd outlier on the right in both categories. But also, many of the "fully paid" loans do not quite reach 1. One potential explanation is that when the last payment comes in, the system just flips loan_status to "Fully Paid" without adding the payment amount to the system itself, or perhaps simply multiplying installation by the term number leaves off a few cents in the actual total. If I were performing this analysis for Lending Club themselves, I'd ask them, but this is just a personal project. I'll consider every loan marked "Fully Paid" to have fully recovered the expected return.

For that matter, I'll cap my fraction_recovered values for charged off loans at 1.0 as well, since at least one value is above that for some reason.

import numpy as np

loans["fraction_recovered"] = np.where(
    (loans["loan_status"] == "Fully Paid") | (loans["fraction_recovered"] > 1.0),
    1.0,
    loans["fraction_recovered"],
)
loans.groupby("loan_status")["fraction_recovered"].describe()

	count	mean	std	min	25%	50%	75%	max
loan_status
Charged Off	269320.0	0.400152	0.218971	0.0	0.224463	0.367554	0.550924	1.0
Fully Paid	1078739.0	1.000000	0.000000	1.0	1.000000	1.000000	1.000000	1.0

For the sake of curiosity, I'll plot the distribution of fraction recovered for charged-off loans.

import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns

sns.kdeplot(
    data=loans["fraction_recovered"][loans["loan_status"] == "Charged Off"],
    label="Charged Off",
    shade=True,
)
plt.axis(xmin=0, xmax=1)
plt.title('Distribution of "fraction recovered"')
plt.show()

Now that the output is formatted, it's time to clean up the inputs. I'll check the null counts of each variable.

loans.drop(columns=cols_for_output, inplace=True)
loans.info(verbose=True, null_counts=True)

<class 'pandas.core.frame.DataFrame'>
Int64Index: 1348059 entries, 0 to 2260697
Data columns (total 97 columns):
 #   Column                               Non-Null Count    Dtype
---  ------                               --------------    -----
 0   loan_amnt                            1348059 non-null  float64
 1   term                                 1348059 non-null  object
 2   emp_length                           1269514 non-null  object
 3   home_ownership                       1348059 non-null  object
 4   annual_inc                           1348055 non-null  float64
 5   verification_status                  1348059 non-null  object
 6   issue_d                              1348059 non-null  object
 7   loan_status                          1348059 non-null  object
 8   purpose                              1348059 non-null  object
 9   dti                                  1347685 non-null  float64
 10  delinq_2yrs                          1348030 non-null  float64
 11  earliest_cr_line                     1348030 non-null  object
 12  fico_range_low                       1348059 non-null  float64
 13  fico_range_high                      1348059 non-null  float64
 14  inq_last_6mths                       1348029 non-null  float64
 15  mths_since_last_delinq               668117 non-null   float64
 16  mths_since_last_record               229415 non-null   float64
 17  open_acc                             1348030 non-null  float64
 18  pub_rec                              1348030 non-null  float64
 19  revol_bal                            1348059 non-null  float64
 20  revol_util                           1347162 non-null  float64
 21  total_acc                            1348030 non-null  float64
 22  collections_12_mths_ex_med           1347914 non-null  float64
 23  mths_since_last_major_derog          353750 non-null   float64
 24  application_type                     1348059 non-null  object
 25  annual_inc_joint                     25800 non-null    float64
 26  dti_joint                            25797 non-null    float64
 27  verification_status_joint            25595 non-null    object
 28  acc_now_delinq                       1348030 non-null  float64
 29  tot_coll_amt                         1277783 non-null  float64
 30  tot_cur_bal                          1277783 non-null  float64
 31  open_acc_6m                          537597 non-null   float64
 32  open_act_il                          537598 non-null   float64
 33  open_il_12m                          537598 non-null   float64
 34  open_il_24m                          537598 non-null   float64
 35  mths_since_rcnt_il                   523382 non-null   float64
 36  total_bal_il                         537598 non-null   float64
 37  il_util                              465016 non-null   float64
 38  open_rv_12m                          537598 non-null   float64
 39  open_rv_24m                          537598 non-null   float64
 40  max_bal_bc                           537598 non-null   float64
 41  all_util                             537545 non-null   float64
 42  total_rev_hi_lim                     1277783 non-null  float64
 43  inq_fi                               537598 non-null   float64
 44  total_cu_tl                          537597 non-null   float64
 45  inq_last_12m                         537597 non-null   float64
 46  acc_open_past_24mths                 1298029 non-null  float64
 47  avg_cur_bal                          1277761 non-null  float64
 48  bc_open_to_buy                       1284167 non-null  float64
 49  bc_util                              1283398 non-null  float64
 50  chargeoff_within_12_mths             1347914 non-null  float64
 51  delinq_amnt                          1348030 non-null  float64
 52  mo_sin_old_il_acct                   1239735 non-null  float64
 53  mo_sin_old_rev_tl_op                 1277782 non-null  float64
 54  mo_sin_rcnt_rev_tl_op                1277782 non-null  float64
 55  mo_sin_rcnt_tl                       1277783 non-null  float64
 56  mort_acc                             1298029 non-null  float64
 57  mths_since_recent_bc                 1285089 non-null  float64
 58  mths_since_recent_bc_dlq             319020 non-null   float64
 59  mths_since_recent_inq                1171239 non-null  float64
 60  mths_since_recent_revol_delinq       449962 non-null   float64
 61  num_accts_ever_120_pd                1277783 non-null  float64
 62  num_actv_bc_tl                       1277783 non-null  float64
 63  num_actv_rev_tl                      1277783 non-null  float64
 64  num_bc_sats                          1289469 non-null  float64
 65  num_bc_tl                            1277783 non-null  float64
 66  num_il_tl                            1277783 non-null  float64
 67  num_op_rev_tl                        1277783 non-null  float64
 68  num_rev_accts                        1277782 non-null  float64
 69  num_rev_tl_bal_gt_0                  1277783 non-null  float64
 70  num_sats                             1289469 non-null  float64
 71  num_tl_120dpd_2m                     1227909 non-null  float64
 72  num_tl_30dpd                         1277783 non-null  float64
 73  num_tl_90g_dpd_24m                   1277783 non-null  float64
 74  num_tl_op_past_12m                   1277783 non-null  float64
 75  pct_tl_nvr_dlq                       1277629 non-null  float64
 76  percent_bc_gt_75                     1283755 non-null  float64
 77  pub_rec_bankruptcies                 1346694 non-null  float64
 78  tax_liens                            1347954 non-null  float64
 79  tot_hi_cred_lim                      1277783 non-null  float64
 80  total_bal_ex_mort                    1298029 non-null  float64
 81  total_bc_limit                       1298029 non-null  float64
 82  total_il_high_credit_limit           1277783 non-null  float64
 83  revol_bal_joint                      18629 non-null    float64
 84  sec_app_fico_range_low               18630 non-null    float64
 85  sec_app_fico_range_high              18630 non-null    float64
 86  sec_app_earliest_cr_line             18630 non-null    object
 87  sec_app_inq_last_6mths               18630 non-null    float64
 88  sec_app_mort_acc                     18630 non-null    float64
 89  sec_app_open_acc                     18630 non-null    float64
 90  sec_app_revol_util                   18302 non-null    float64
 91  sec_app_open_act_il                  18630 non-null    float64
 92  sec_app_num_rev_accts                18630 non-null    float64
 93  sec_app_chargeoff_within_12_mths     18630 non-null    float64
 94  sec_app_collections_12_mths_ex_med   18630 non-null    float64
 95  sec_app_mths_since_last_major_derog  6645 non-null     float64
 96  fraction_recovered                   1348059 non-null  float64
dtypes: float64(86), object(11)
memory usage: 1007.9+ MB

Remaining columns with lots of null values seem to fall into three categories:

Derogatory/delinquency metrics (where null means the borrower doesn't have any such marks)
- I'll also add mths_since_recent_inq to this list, since its non-null count is below what seems to be the threshold for complete data, which is around 1,277,783. I'll assume a null value here means no recent inquiries.
Metrics that only apply to joint applications (where null means it was a single application)
An inexplicable series of 14 credit history–related columns that only have around 537,000 entries. Are these newer metrics?

negative_mark_cols = ["mths_since_last_delinq", "mths_since_last_record", "mths_since_last_major_derog", "mths_since_recent_bc_dlq", "mths_since_recent_inq", "mths_since_recent_revol_delinq", "mths_since_recent_revol_delinq", "sec_app_mths_since_last_major_derog"]
joint_cols = ["annual_inc_joint", "dti_joint", "verification_status_joint", "revol_bal_joint", "sec_app_fico_range_low", "sec_app_fico_range_high", "sec_app_earliest_cr_line", "sec_app_inq_last_6mths", "sec_app_mort_acc", "sec_app_open_acc", "sec_app_revol_util", "sec_app_open_act_il", "sec_app_num_rev_accts", "sec_app_chargeoff_within_12_mths", "sec_app_collections_12_mths_ex_med", "sec_app_mths_since_last_major_derog"]
confusing_cols = ["open_acc_6m", "open_act_il", "open_il_12m", "open_il_24m", "mths_since_rcnt_il", "total_bal_il", "il_util", "open_rv_12m", "open_rv_24m", "max_bal_bc", "all_util", "inq_fi", "total_cu_tl", "inq_last_12m"]

I'll first look at those more confusing columns to find out whether or not they're a newer set of metrics. That'll require converting issue_d to date format first.

loans["issue_d"] = loans["issue_d"].astype("datetime64[ns]")

# Check date range of confusing columns
loans[confusing_cols + ["issue_d"]].dropna(axis="index")["issue_d"].agg(
    ["count", "min", "max"]
)

count                 464325
min      2015-12-01 00:00:00
max      2018-12-01 00:00:00
Name: issue_d, dtype: object

# Compare to all entries from Dec 2015 onward
loans["issue_d"][loans["issue_d"] >= np.datetime64("2015-12-01")].agg(
    ["count", "min", "max"]
)

count                 557708
min      2015-12-01 00:00:00
max      2018-12-01 00:00:00
Name: issue_d, dtype: object

It appears that these are indeed newer metrics, their use only beginning in December 2015, but even after that point usage is spotty. I'm curious to see if these additional metrics would make a model more accurate, though, so once I'm done cleaning the data I'll copy the rows with these new metrics into a new dataset and create another model using the new metrics.

new_metric_cols = confusing_cols

As for the derogatory/delinquency metrics, taking a cue from Michael Wurm, I'm going to take the inverse of all the "months since recent/last" fields, which will turn each into a proxy for the frequency of the event and also let me set all the null values (when an event has never happened) to 0. For the "months since oldest" fields, I'll just set the null values to 0 and leave the rest untouched.

mths_since_last_cols = [
    col_name
    for col_name in loans.columns
    if "mths_since" in col_name or "mo_sin_rcnt" in col_name
]
mths_since_old_cols = [
    col_name for col_name in loans.columns if "mo_sin_old" in col_name
]

for col_name in mths_since_last_cols:
    loans[col_name] = [
        0.0 if pd.isna(months) else 1 / 1 if months == 0 else 1 / months
        for months in loans[col_name]
    ]
loans.loc[:, mths_since_old_cols].fillna(0, inplace=True)

# Rename inverse columns
rename_mapper = {}
for col_name in mths_since_last_cols:
    rename_mapper[col_name] = col_name.replace("mths_since", "inv_mths_since").replace(
        "mo_sin_rcnt", "inv_mo_sin_rcnt"
    )
loans.rename(columns=rename_mapper, inplace=True)


def replace_list_value(l, old_value, new_value):
    i = l.index(old_value)
    l.pop(i)
    l.insert(i, new_value)


replace_list_value(new_metric_cols, "mths_since_rcnt_il", "inv_mths_since_rcnt_il")
replace_list_value(
    joint_cols,
    "sec_app_mths_since_last_major_derog",
    "sec_app_inv_mths_since_last_major_derog",
)

Now to look closer at joint loans.

loans.groupby("application_type")["application_type"].count()

application_type
Individual    1322259
Joint App       25800
Name: application_type, dtype: int64

joint_loans = loans[:][loans["application_type"] == "Joint App"]
joint_loans[joint_cols].info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 25800 entries, 2 to 2260663
Data columns (total 16 columns):
 #   Column                                   Non-Null Count  Dtype
---  ------                                   --------------  -----
 0   annual_inc_joint                         25800 non-null  float64
 1   dti_joint                                25797 non-null  float64
 2   verification_status_joint                25595 non-null  object
 3   revol_bal_joint                          18629 non-null  float64
 4   sec_app_fico_range_low                   18630 non-null  float64
 5   sec_app_fico_range_high                  18630 non-null  float64
 6   sec_app_earliest_cr_line                 18630 non-null  object
 7   sec_app_inq_last_6mths                   18630 non-null  float64
 8   sec_app_mort_acc                         18630 non-null  float64
 9   sec_app_open_acc                         18630 non-null  float64
 10  sec_app_revol_util                       18302 non-null  float64
 11  sec_app_open_act_il                      18630 non-null  float64
 12  sec_app_num_rev_accts                    18630 non-null  float64
 13  sec_app_chargeoff_within_12_mths         18630 non-null  float64
 14  sec_app_collections_12_mths_ex_med       18630 non-null  float64
 15  sec_app_inv_mths_since_last_major_derog  25800 non-null  float64
dtypes: float64(14), object(2)
memory usage: 3.3+ MB

It seems there may be a case of newer metrics for joint applications as well. I'll investigate.

joint_new_metric_cols = ["revol_bal_joint", "sec_app_fico_range_low", "sec_app_fico_range_high", "sec_app_earliest_cr_line", "sec_app_inq_last_6mths", "sec_app_mort_acc", "sec_app_open_acc", "sec_app_revol_util", "sec_app_open_act_il", "sec_app_num_rev_accts", "sec_app_chargeoff_within_12_mths", "sec_app_collections_12_mths_ex_med", "sec_app_inv_mths_since_last_major_derog"]
joint_loans[joint_new_metric_cols + ["issue_d"]].dropna(axis="index")["issue_d"].agg(
    ["count", "min", "max"]
)

count                  18301
min      2017-03-01 00:00:00
max      2018-12-01 00:00:00
Name: issue_d, dtype: object

# Check without `sec_app_revol_util` column
joint_new_metric_cols_2 = ["revol_bal_joint", "sec_app_fico_range_low", "sec_app_fico_range_high", "sec_app_earliest_cr_line", "sec_app_inq_last_6mths", "sec_app_mort_acc", "sec_app_open_acc", "sec_app_open_act_il", "sec_app_num_rev_accts", "sec_app_chargeoff_within_12_mths", "sec_app_collections_12_mths_ex_med", "sec_app_inv_mths_since_last_major_derog"]
joint_loans[joint_new_metric_cols_2 + ["issue_d"]].dropna(axis="index")["issue_d"].agg(
    ["count", "min", "max"]
)

count                  18629
min      2017-03-01 00:00:00
max      2018-12-01 00:00:00
Name: issue_d, dtype: object

Newer than the previous set of new metrics, even—these didn't start getting used till March 2017. Now I wonder when joint loans were first introduced.

joint_loans["issue_d"].agg(["count", "min", "max"])

count                  25800
min      2015-10-01 00:00:00
max      2018-12-01 00:00:00
Name: issue_d, dtype: object

2015. I think I'll save the newer joint metrics for perhaps a third model, but I believe I can include annual_inc_joint, dti_joint, and verification_status_joint in the main model—I'll just binary-encode application_type, and for individual applications I'll set annual_inc_joint, dti_joint, and verification_status_joint equal to their non-joint counterparts.

onehot_cols.append("application_type")

# Fill joint columns in individual applications
for joint_col, indiv_col in zip(
    ["annual_inc_joint", "dti_joint", "verification_status_joint"],
    ["annual_inc", "dti", "verification_status"],
):
    loans[joint_col] = [
        joint_val if app_type == "Joint App" else indiv_val
        for app_type, joint_val, indiv_val in zip(
            loans["application_type"], loans[joint_col], loans[indiv_col]
        )
    ]

loans.info(verbose=True, null_counts=True)

<class 'pandas.core.frame.DataFrame'>
Int64Index: 1348059 entries, 0 to 2260697
Data columns (total 97 columns):
 #   Column                                   Non-Null Count    Dtype
---  ------                                   --------------    -----
 0   loan_amnt                                1348059 non-null  float64
 1   term                                     1348059 non-null  object
 2   emp_length                               1269514 non-null  object
 3   home_ownership                           1348059 non-null  object
 4   annual_inc                               1348055 non-null  float64
 5   verification_status                      1348059 non-null  object
 6   issue_d                                  1348059 non-null  datetime64[ns]
 7   loan_status                              1348059 non-null  object
 8   purpose                                  1348059 non-null  object
 9   dti                                      1347685 non-null  float64
 10  delinq_2yrs                              1348030 non-null  float64
 11  earliest_cr_line                         1348030 non-null  object
 12  fico_range_low                           1348059 non-null  float64
 13  fico_range_high                          1348059 non-null  float64
 14  inq_last_6mths                           1348029 non-null  float64
 15  inv_mths_since_last_delinq               1348059 non-null  float64
 16  inv_mths_since_last_record               1348059 non-null  float64
 17  open_acc                                 1348030 non-null  float64
 18  pub_rec                                  1348030 non-null  float64
 19  revol_bal                                1348059 non-null  float64
 20  revol_util                               1347162 non-null  float64
 21  total_acc                                1348030 non-null  float64
 22  collections_12_mths_ex_med               1347914 non-null  float64
 23  inv_mths_since_last_major_derog          1348059 non-null  float64
 24  application_type                         1348059 non-null  object
 25  annual_inc_joint                         1348055 non-null  float64
 26  dti_joint                                1348056 non-null  float64
 27  verification_status_joint                1347854 non-null  object
 28  acc_now_delinq                           1348030 non-null  float64
 29  tot_coll_amt                             1277783 non-null  float64
 30  tot_cur_bal                              1277783 non-null  float64
 31  open_acc_6m                              537597 non-null   float64
 32  open_act_il                              537598 non-null   float64
 33  open_il_12m                              537598 non-null   float64
 34  open_il_24m                              537598 non-null   float64
 35  inv_mths_since_rcnt_il                   1348059 non-null  float64
 36  total_bal_il                             537598 non-null   float64
 37  il_util                                  465016 non-null   float64
 38  open_rv_12m                              537598 non-null   float64
 39  open_rv_24m                              537598 non-null   float64
 40  max_bal_bc                               537598 non-null   float64
 41  all_util                                 537545 non-null   float64
 42  total_rev_hi_lim                         1277783 non-null  float64
 43  inq_fi                                   537598 non-null   float64
 44  total_cu_tl                              537597 non-null   float64
 45  inq_last_12m                             537597 non-null   float64
 46  acc_open_past_24mths                     1298029 non-null  float64
 47  avg_cur_bal                              1277761 non-null  float64
 48  bc_open_to_buy                           1284167 non-null  float64
 49  bc_util                                  1283398 non-null  float64
 50  chargeoff_within_12_mths                 1347914 non-null  float64
 51  delinq_amnt                              1348030 non-null  float64
 52  mo_sin_old_il_acct                       1239735 non-null  float64
 53  mo_sin_old_rev_tl_op                     1277782 non-null  float64
 54  inv_mo_sin_rcnt_rev_tl_op                1348059 non-null  float64
 55  inv_mo_sin_rcnt_tl                       1348059 non-null  float64
 56  mort_acc                                 1298029 non-null  float64
 57  inv_mths_since_recent_bc                 1348059 non-null  float64
 58  inv_mths_since_recent_bc_dlq             1348059 non-null  float64
 59  inv_mths_since_recent_inq                1348059 non-null  float64
 60  inv_mths_since_recent_revol_delinq       1348059 non-null  float64
 61  num_accts_ever_120_pd                    1277783 non-null  float64
 62  num_actv_bc_tl                           1277783 non-null  float64
 63  num_actv_rev_tl                          1277783 non-null  float64
 64  num_bc_sats                              1289469 non-null  float64
 65  num_bc_tl                                1277783 non-null  float64
 66  num_il_tl                                1277783 non-null  float64
 67  num_op_rev_tl                            1277783 non-null  float64
 68  num_rev_accts                            1277782 non-null  float64
 69  num_rev_tl_bal_gt_0                      1277783 non-null  float64
 70  num_sats                                 1289469 non-null  float64
 71  num_tl_120dpd_2m                         1227909 non-null  float64
 72  num_tl_30dpd                             1277783 non-null  float64
 73  num_tl_90g_dpd_24m                       1277783 non-null  float64
 74  num_tl_op_past_12m                       1277783 non-null  float64
 75  pct_tl_nvr_dlq                           1277629 non-null  float64
 76  percent_bc_gt_75                         1283755 non-null  float64
 77  pub_rec_bankruptcies                     1346694 non-null  float64
 78  tax_liens                                1347954 non-null  float64
 79  tot_hi_cred_lim                          1277783 non-null  float64
 80  total_bal_ex_mort                        1298029 non-null  float64
 81  total_bc_limit                           1298029 non-null  float64
 82  total_il_high_credit_limit               1277783 non-null  float64
 83  revol_bal_joint                          18629 non-null    float64
 84  sec_app_fico_range_low                   18630 non-null    float64
 85  sec_app_fico_range_high                  18630 non-null    float64
 86  sec_app_earliest_cr_line                 18630 non-null    object
 87  sec_app_inq_last_6mths                   18630 non-null    float64
 88  sec_app_mort_acc                         18630 non-null    float64
 89  sec_app_open_acc                         18630 non-null    float64
 90  sec_app_revol_util                       18302 non-null    float64
 91  sec_app_open_act_il                      18630 non-null    float64
 92  sec_app_num_rev_accts                    18630 non-null    float64
 93  sec_app_chargeoff_within_12_mths         18630 non-null    float64
 94  sec_app_collections_12_mths_ex_med       18630 non-null    float64
 95  sec_app_inv_mths_since_last_major_derog  1348059 non-null  float64
 96  fraction_recovered                       1348059 non-null  float64
dtypes: datetime64[ns](1), float64(86), object(10)
memory usage: 1007.9+ MB

Now the only remaining steps should be removing rows with null values (in columns that aren't new metrics) and encoding categorical features.

I'm removing rows with null values in those columns because that should still leave the vast majority of rows intact, over 1 million, which is still plenty of data. But I guess I should make sure before I overwrite loans.

cols_to_search = [
    col for col in loans.columns if col not in new_metric_cols + joint_new_metric_cols
]
loans.dropna(axis="index", subset=cols_to_search).shape

(1110171, 97)

Yes, still 1,110,171. That'll do.

loans.dropna(axis="index", subset=cols_to_search, inplace=True)

Then actually I'll tackle earliest_cr_line and its joint counterpart first before looking at the categorical features.

loans[["earliest_cr_line", "sec_app_earliest_cr_line"]]

	earliest_cr_line	sec_app_earliest_cr_line
0	Aug-2003	NaN
1	Dec-1999	NaN
2	Aug-2000	NaN
4	Jun-1998	NaN
5	Oct-1987	NaN
...	...	...
2260688	Jul-2004	NaN
2260690	Mar-2002	NaN
2260691	Jun-2011	NaN
2260692	Aug-1997	NaN
2260697	Jul-1999	NaN

1110171 rows × 2 columns

I should convert that to the age of the credit line at the time of application (or the time of loan issuing, more precisely).

def get_credit_history_age(col_name):
    earliest_cr_line_date = loans[col_name].astype("datetime64[ns]")
    cr_hist_age_delta = loans["issue_d"] - earliest_cr_line_date
    MINUTES_PER_MONTH = int(365.25 / 12 * 24 * 60)
    cr_hist_age_months = cr_hist_age_delta / np.timedelta64(MINUTES_PER_MONTH, "m")
    return cr_hist_age_months.map(
        lambda value: np.nan if pd.isna(value) else round(value)
    )


cr_hist_age_months = get_credit_history_age("earliest_cr_line")
cr_hist_age_months

0          148
1          192
2          184
4          210
5          338
          ...
2260688    147
2260690    175
2260691     64
2260692    230
2260697    207
Length: 1110171, dtype: int64

loans["earliest_cr_line"] = cr_hist_age_months
loans["sec_app_earliest_cr_line"] = get_credit_history_age(
    "sec_app_earliest_cr_line"
).astype("Int64")
loans.rename(
    columns={
        "earliest_cr_line": "cr_hist_age_mths",
        "sec_app_earliest_cr_line": "sec_app_cr_hist_age_mths",
    },
    inplace=True,
)
replace_list_value(
    joint_new_metric_cols, "sec_app_earliest_cr_line", "sec_app_cr_hist_age_mths"
)

Now a look at those categorical features.

categorical_cols = ["term", "emp_length", "home_ownership", "verification_status", "purpose", "verification_status_joint"]
for i, col_name in enumerate(categorical_cols):
    print(
        loans.groupby(col_name)[col_name].count(),
        "\n" if i < len(categorical_cols) - 1 else "",
    )

term
36 months    831601
60 months    278570
Name: term, dtype: int64

emp_length
1 year        76868
10+ years    392883
2 years      106124
3 years       93784
4 years       69031
5 years       72421
6 years       54240
7 years       52229
8 years       53826
9 years       45210
< 1 year      93555
Name: emp_length, dtype: int64

home_ownership
ANY            250
MORTGAGE    559035
NONE            39
OTHER           40
OWN         114577
RENT        436230
Name: home_ownership, dtype: int64

verification_status
Not Verified       335350
Source Verified    463153
Verified           311668
Name: verification_status, dtype: int64

purpose
car                    10754
credit_card           245942
debt_consolidation    653222
educational                1
home_improvement       71089
house                   5720
major_purchase         22901
medical                12302
moving                  7464
other                  60986
renewable_energy         691
small_business         11137
vacation                7169
wedding                  793
Name: purpose, dtype: int64

verification_status_joint
Not Verified       341073
Source Verified    461941
Verified           307157
Name: verification_status_joint, dtype: int64

First, in researching income verification, I learned that LendingClub only tries to verify income on a subset of loan applications based on the content of the application, so this feature is a source of target leakage. I'll remove the two offending columns (and a couple more I don't need anymore).

loans.drop(
    columns=[
        "verification_status",
        "verification_status_joint",
        "issue_d",
        "loan_status",
    ],
    inplace=True,
)

Once I create my pipeline, I'll binary encode term, one-hot encode home_ownership and purpose, and since emp_length is an ordinal variable, I'll convert it to the integers 0–10.

onehot_cols += ["home_ownership", "purpose"]
ordinal_cols = {
    "emp_length": [
        "< 1 year",
        "1 year",
        "2 years",
        "3 years",
        "4 years",
        "5 years",
        "6 years",
        "7 years",
        "8 years",
        "9 years",
        "10+ years",
    ]
}

That should cover all the cleaning necessary for the first model's data. I'll save the columns that'll be used in the first model to a new DataFrame, and while I'm at it, I'll start formatting the DataFrames for the two additional models adding the two sets of new metrics.

loans_1 = loans.drop(columns=new_metric_cols + joint_new_metric_cols)
loans_2 = loans.drop(columns=joint_new_metric_cols)
loans_2.info(verbose=True, null_counts=True)

<class 'pandas.core.frame.DataFrame'>
Int64Index: 1110171 entries, 0 to 2260697
Data columns (total 80 columns):
 #   Column                              Non-Null Count    Dtype
---  ------                              --------------    -----
 0   loan_amnt                           1110171 non-null  float64
 1   term                                1110171 non-null  object
 2   emp_length                          1110171 non-null  object
 3   home_ownership                      1110171 non-null  object
 4   annual_inc                          1110171 non-null  float64
 5   purpose                             1110171 non-null  object
 6   dti                                 1110171 non-null  float64
 7   delinq_2yrs                         1110171 non-null  float64
 8   cr_hist_age_mths                    1110171 non-null  int64
 9   fico_range_low                      1110171 non-null  float64
 10  fico_range_high                     1110171 non-null  float64
 11  inq_last_6mths                      1110171 non-null  float64
 12  inv_mths_since_last_delinq          1110171 non-null  float64
 13  inv_mths_since_last_record          1110171 non-null  float64
 14  open_acc                            1110171 non-null  float64
 15  pub_rec                             1110171 non-null  float64
 16  revol_bal                           1110171 non-null  float64
 17  revol_util                          1110171 non-null  float64
 18  total_acc                           1110171 non-null  float64
 19  collections_12_mths_ex_med          1110171 non-null  float64
 20  inv_mths_since_last_major_derog     1110171 non-null  float64
 21  application_type                    1110171 non-null  object
 22  annual_inc_joint                    1110171 non-null  float64
 23  dti_joint                           1110171 non-null  float64
 24  acc_now_delinq                      1110171 non-null  float64
 25  tot_coll_amt                        1110171 non-null  float64
 26  tot_cur_bal                         1110171 non-null  float64
 27  open_acc_6m                         459541 non-null   float64
 28  open_act_il                         459541 non-null   float64
 29  open_il_12m                         459541 non-null   float64
 30  open_il_24m                         459541 non-null   float64
 31  inv_mths_since_rcnt_il              1110171 non-null  float64
 32  total_bal_il                        459541 non-null   float64
 33  il_util                             408722 non-null   float64
 34  open_rv_12m                         459541 non-null   float64
 35  open_rv_24m                         459541 non-null   float64
 36  max_bal_bc                          459541 non-null   float64
 37  all_util                            459541 non-null   float64
 38  total_rev_hi_lim                    1110171 non-null  float64
 39  inq_fi                              459541 non-null   float64
 40  total_cu_tl                         459541 non-null   float64
 41  inq_last_12m                        459541 non-null   float64
 42  acc_open_past_24mths                1110171 non-null  float64
 43  avg_cur_bal                         1110171 non-null  float64
 44  bc_open_to_buy                      1110171 non-null  float64
 45  bc_util                             1110171 non-null  float64
 46  chargeoff_within_12_mths            1110171 non-null  float64
 47  delinq_amnt                         1110171 non-null  float64
 48  mo_sin_old_il_acct                  1110171 non-null  float64
 49  mo_sin_old_rev_tl_op                1110171 non-null  float64
 50  inv_mo_sin_rcnt_rev_tl_op           1110171 non-null  float64
 51  inv_mo_sin_rcnt_tl                  1110171 non-null  float64
 52  mort_acc                            1110171 non-null  float64
 53  inv_mths_since_recent_bc            1110171 non-null  float64
 54  inv_mths_since_recent_bc_dlq        1110171 non-null  float64
 55  inv_mths_since_recent_inq           1110171 non-null  float64
 56  inv_mths_since_recent_revol_delinq  1110171 non-null  float64
 57  num_accts_ever_120_pd               1110171 non-null  float64
 58  num_actv_bc_tl                      1110171 non-null  float64
 59  num_actv_rev_tl                     1110171 non-null  float64
 60  num_bc_sats                         1110171 non-null  float64
 61  num_bc_tl                           1110171 non-null  float64
 62  num_il_tl                           1110171 non-null  float64
 63  num_op_rev_tl                       1110171 non-null  float64
 64  num_rev_accts                       1110171 non-null  float64
 65  num_rev_tl_bal_gt_0                 1110171 non-null  float64
 66  num_sats                            1110171 non-null  float64
 67  num_tl_120dpd_2m                    1110171 non-null  float64
 68  num_tl_30dpd                        1110171 non-null  float64
 69  num_tl_90g_dpd_24m                  1110171 non-null  float64
 70  num_tl_op_past_12m                  1110171 non-null  float64
 71  pct_tl_nvr_dlq                      1110171 non-null  float64
 72  percent_bc_gt_75                    1110171 non-null  float64
 73  pub_rec_bankruptcies                1110171 non-null  float64
 74  tax_liens                           1110171 non-null  float64
 75  tot_hi_cred_lim                     1110171 non-null  float64
 76  total_bal_ex_mort                   1110171 non-null  float64
 77  total_bc_limit                      1110171 non-null  float64
 78  total_il_high_credit_limit          1110171 non-null  float64
 79  fraction_recovered                  1110171 non-null  float64
dtypes: float64(74), int64(1), object(5)
memory usage: 686.1+ MB

Before I drop a bunch of rows with nulls from loans_2, I'm concerned about il_util, as it's missing values in about 50,000 more rows than the rest of the new metric columns. Why would that be?

loans_2["il_util"][loans_2["il_util"].notna()].describe()

count    408722.000000
mean         71.832894
std          22.311439
min           0.000000
25%          59.000000
50%          75.000000
75%          87.000000
max         464.000000
Name: il_util, dtype: float64

Peeking back up to the data dictionary, il_util is the "ratio of total current balance to high credit/credit limit on all install acct". The relevant balance (total_bal_il) and credit limit (total_il_high_credit_limit) metrics appear to already be in the data, so perhaps this utilization metric doesn't contain any new information. I'll compare il_util (where it's present) to the ratio of the other two variables.

query_df = loans[["il_util", "total_bal_il", "total_il_high_credit_limit"]].dropna(
    axis="index", subset=["il_util"]
)
query_df["il_util_compute"] = (
    query_df["total_bal_il"] / query_df["total_il_high_credit_limit"]
).map(lambda x: float(round(x * 100)))
query_df[["il_util", "il_util_compute"]]

	il_util	il_util_compute
0	36.0	36.0
1	73.0	73.0
2	73.0	73.0
4	84.0	84.0
5	99.0	99.0
...	...	...
2260688	52.0	39.0
2260690	50.0	74.0
2260691	47.0	47.0
2260692	79.0	79.0
2260697	78.0	76.0

408722 rows × 2 columns

(query_df["il_util"] == query_df["il_util_compute"]).describe()

count     408722
unique         2
top         True
freq      307589
dtype: object

query_df["compute_diff"] = abs(query_df["il_util"] - query_df["il_util_compute"])
query_df["compute_diff"][query_df["compute_diff"] != 0].describe()

count    101133.000000
mean         14.638684
std          16.409913
min           1.000000
25%           3.000000
50%          10.000000
75%          21.000000
max        1108.000000
Name: compute_diff, dtype: float64

That's weird. il_util is equal to the computed ratio three-quarters of the time, but when it's off, the median difference is 10 points off. Perhaps there's new information there sometimes after all. Maybe whatever credit bureau is reporting the utilization rate uses a different formula than just a simple ratio? Again, something I could ask if I were performing this analysis for a client, but that's not the case. I'll assume that this variable is still valuable, and where il_util is null I'll impute the value to make it equal to the ratio of total_bal_il to total_il_high_credit_limit (or 0 if the limit is 0). And I'll add one more boolean field to mark the imputed entries.

Also, that 1,108 is a doozy of an outlier, but I think I'll just leave it be, as it appears that outliers aren't too big a deal if the neural network architecture is sufficiently deep.

loans["il_util_imputed"] = [
    True if pd.isna(util) & pd.notna(bal) & pd.notna(limit) else False
    for util, bal, limit in zip(
        loans["il_util"], loans["total_bal_il"], loans["total_il_high_credit_limit"]
    )
]
new_metric_onehot_cols = ["il_util_imputed"]
loans["il_util"] = [
    0.0
    if pd.isna(util) & pd.notna(bal) & (limit == 0)
    else float(round(bal / limit * 100))
    if pd.isna(util) & pd.notna(bal) & pd.notna(limit)
    else util
    for util, bal, limit in zip(
        loans["il_util"], loans["total_bal_il"], loans["total_il_high_credit_limit"]
    )
]

loans_2 = loans.drop(columns=joint_new_metric_cols)
loans_2.info(verbose=True, null_counts=True)

<class 'pandas.core.frame.DataFrame'>
Int64Index: 1110171 entries, 0 to 2260697
Data columns (total 81 columns):
 #   Column                              Non-Null Count    Dtype
---  ------                              --------------    -----
 0   loan_amnt                           1110171 non-null  float64
 1   term                                1110171 non-null  object
 2   emp_length                          1110171 non-null  object
 3   home_ownership                      1110171 non-null  object
 4   annual_inc                          1110171 non-null  float64
 5   purpose                             1110171 non-null  object
 6   dti                                 1110171 non-null  float64
 7   delinq_2yrs                         1110171 non-null  float64
 8   cr_hist_age_mths                    1110171 non-null  int64
 9   fico_range_low                      1110171 non-null  float64
 10  fico_range_high                     1110171 non-null  float64
 11  inq_last_6mths                      1110171 non-null  float64
 12  inv_mths_since_last_delinq          1110171 non-null  float64
 13  inv_mths_since_last_record          1110171 non-null  float64
 14  open_acc                            1110171 non-null  float64
 15  pub_rec                             1110171 non-null  float64
 16  revol_bal                           1110171 non-null  float64
 17  revol_util                          1110171 non-null  float64
 18  total_acc                           1110171 non-null  float64
 19  collections_12_mths_ex_med          1110171 non-null  float64
 20  inv_mths_since_last_major_derog     1110171 non-null  float64
 21  application_type                    1110171 non-null  object
 22  annual_inc_joint                    1110171 non-null  float64
 23  dti_joint                           1110171 non-null  float64
 24  acc_now_delinq                      1110171 non-null  float64
 25  tot_coll_amt                        1110171 non-null  float64
 26  tot_cur_bal                         1110171 non-null  float64
 27  open_acc_6m                         459541 non-null   float64
 28  open_act_il                         459541 non-null   float64
 29  open_il_12m                         459541 non-null   float64
 30  open_il_24m                         459541 non-null   float64
 31  inv_mths_since_rcnt_il              1110171 non-null  float64
 32  total_bal_il                        459541 non-null   float64
 33  il_util                             459541 non-null   float64
 34  open_rv_12m                         459541 non-null   float64
 35  open_rv_24m                         459541 non-null   float64
 36  max_bal_bc                          459541 non-null   float64
 37  all_util                            459541 non-null   float64
 38  total_rev_hi_lim                    1110171 non-null  float64
 39  inq_fi                              459541 non-null   float64
 40  total_cu_tl                         459541 non-null   float64
 41  inq_last_12m                        459541 non-null   float64
 42  acc_open_past_24mths                1110171 non-null  float64
 43  avg_cur_bal                         1110171 non-null  float64
 44  bc_open_to_buy                      1110171 non-null  float64
 45  bc_util                             1110171 non-null  float64
 46  chargeoff_within_12_mths            1110171 non-null  float64
 47  delinq_amnt                         1110171 non-null  float64
 48  mo_sin_old_il_acct                  1110171 non-null  float64
 49  mo_sin_old_rev_tl_op                1110171 non-null  float64
 50  inv_mo_sin_rcnt_rev_tl_op           1110171 non-null  float64
 51  inv_mo_sin_rcnt_tl                  1110171 non-null  float64
 52  mort_acc                            1110171 non-null  float64
 53  inv_mths_since_recent_bc            1110171 non-null  float64
 54  inv_mths_since_recent_bc_dlq        1110171 non-null  float64
 55  inv_mths_since_recent_inq           1110171 non-null  float64
 56  inv_mths_since_recent_revol_delinq  1110171 non-null  float64
 57  num_accts_ever_120_pd               1110171 non-null  float64
 58  num_actv_bc_tl                      1110171 non-null  float64
 59  num_actv_rev_tl                     1110171 non-null  float64
 60  num_bc_sats                         1110171 non-null  float64
 61  num_bc_tl                           1110171 non-null  float64
 62  num_il_tl                           1110171 non-null  float64
 63  num_op_rev_tl                       1110171 non-null  float64
 64  num_rev_accts                       1110171 non-null  float64
 65  num_rev_tl_bal_gt_0                 1110171 non-null  float64
 66  num_sats                            1110171 non-null  float64
 67  num_tl_120dpd_2m                    1110171 non-null  float64
 68  num_tl_30dpd                        1110171 non-null  float64
 69  num_tl_90g_dpd_24m                  1110171 non-null  float64
 70  num_tl_op_past_12m                  1110171 non-null  float64
 71  pct_tl_nvr_dlq                      1110171 non-null  float64
 72  percent_bc_gt_75                    1110171 non-null  float64
 73  pub_rec_bankruptcies                1110171 non-null  float64
 74  tax_liens                           1110171 non-null  float64
 75  tot_hi_cred_lim                     1110171 non-null  float64
 76  total_bal_ex_mort                   1110171 non-null  float64
 77  total_bc_limit                      1110171 non-null  float64
 78  total_il_high_credit_limit          1110171 non-null  float64
 79  fraction_recovered                  1110171 non-null  float64
 80  il_util_imputed                     1110171 non-null  bool
dtypes: bool(1), float64(74), int64(1), object(5)
memory usage: 687.1+ MB

Good. Ready to drop rows with nulls in loans_2 and move on to the DataFrame for the model that adds the new metrics for joint applications.

loans_2.dropna(axis="index", inplace=True)

loans_3 = loans.dropna(axis="index")
loans_3.info(verbose=True, null_counts=True)

<class 'pandas.core.frame.DataFrame'>
Int64Index: 14453 entries, 421222 to 2157147
Data columns (total 94 columns):
 #   Column                                   Non-Null Count  Dtype
---  ------                                   --------------  -----
 0   loan_amnt                                14453 non-null  float64
 1   term                                     14453 non-null  object
 2   emp_length                               14453 non-null  object
 3   home_ownership                           14453 non-null  object
 4   annual_inc                               14453 non-null  float64
 5   purpose                                  14453 non-null  object
 6   dti                                      14453 non-null  float64
 7   delinq_2yrs                              14453 non-null  float64
 8   cr_hist_age_mths                         14453 non-null  int64
 9   fico_range_low                           14453 non-null  float64
 10  fico_range_high                          14453 non-null  float64
 11  inq_last_6mths                           14453 non-null  float64
 12  inv_mths_since_last_delinq               14453 non-null  float64
 13  inv_mths_since_last_record               14453 non-null  float64
 14  open_acc                                 14453 non-null  float64
 15  pub_rec                                  14453 non-null  float64
 16  revol_bal                                14453 non-null  float64
 17  revol_util                               14453 non-null  float64
 18  total_acc                                14453 non-null  float64
 19  collections_12_mths_ex_med               14453 non-null  float64
 20  inv_mths_since_last_major_derog          14453 non-null  float64
 21  application_type                         14453 non-null  object
 22  annual_inc_joint                         14453 non-null  float64
 23  dti_joint                                14453 non-null  float64
 24  acc_now_delinq                           14453 non-null  float64
 25  tot_coll_amt                             14453 non-null  float64
 26  tot_cur_bal                              14453 non-null  float64
 27  open_acc_6m                              14453 non-null  float64
 28  open_act_il                              14453 non-null  float64
 29  open_il_12m                              14453 non-null  float64
 30  open_il_24m                              14453 non-null  float64
 31  inv_mths_since_rcnt_il                   14453 non-null  float64
 32  total_bal_il                             14453 non-null  float64
 33  il_util                                  14453 non-null  float64
 34  open_rv_12m                              14453 non-null  float64
 35  open_rv_24m                              14453 non-null  float64
 36  max_bal_bc                               14453 non-null  float64
 37  all_util                                 14453 non-null  float64
 38  total_rev_hi_lim                         14453 non-null  float64
 39  inq_fi                                   14453 non-null  float64
 40  total_cu_tl                              14453 non-null  float64
 41  inq_last_12m                             14453 non-null  float64
 42  acc_open_past_24mths                     14453 non-null  float64
 43  avg_cur_bal                              14453 non-null  float64
 44  bc_open_to_buy                           14453 non-null  float64
 45  bc_util                                  14453 non-null  float64
 46  chargeoff_within_12_mths                 14453 non-null  float64
 47  delinq_amnt                              14453 non-null  float64
 48  mo_sin_old_il_acct                       14453 non-null  float64
 49  mo_sin_old_rev_tl_op                     14453 non-null  float64
 50  inv_mo_sin_rcnt_rev_tl_op                14453 non-null  float64
 51  inv_mo_sin_rcnt_tl                       14453 non-null  float64
 52  mort_acc                                 14453 non-null  float64
 53  inv_mths_since_recent_bc                 14453 non-null  float64
 54  inv_mths_since_recent_bc_dlq             14453 non-null  float64
 55  inv_mths_since_recent_inq                14453 non-null  float64
 56  inv_mths_since_recent_revol_delinq       14453 non-null  float64
 57  num_accts_ever_120_pd                    14453 non-null  float64
 58  num_actv_bc_tl                           14453 non-null  float64
 59  num_actv_rev_tl                          14453 non-null  float64
 60  num_bc_sats                              14453 non-null  float64
 61  num_bc_tl                                14453 non-null  float64
 62  num_il_tl                                14453 non-null  float64
 63  num_op_rev_tl                            14453 non-null  float64
 64  num_rev_accts                            14453 non-null  float64
 65  num_rev_tl_bal_gt_0                      14453 non-null  float64
 66  num_sats                                 14453 non-null  float64
 67  num_tl_120dpd_2m                         14453 non-null  float64
 68  num_tl_30dpd                             14453 non-null  float64
 69  num_tl_90g_dpd_24m                       14453 non-null  float64
 70  num_tl_op_past_12m                       14453 non-null  float64
 71  pct_tl_nvr_dlq                           14453 non-null  float64
 72  percent_bc_gt_75                         14453 non-null  float64
 73  pub_rec_bankruptcies                     14453 non-null  float64
 74  tax_liens                                14453 non-null  float64
 75  tot_hi_cred_lim                          14453 non-null  float64
 76  total_bal_ex_mort                        14453 non-null  float64
 77  total_bc_limit                           14453 non-null  float64
 78  total_il_high_credit_limit               14453 non-null  float64
 79  revol_bal_joint                          14453 non-null  float64
 80  sec_app_fico_range_low                   14453 non-null  float64
 81  sec_app_fico_range_high                  14453 non-null  float64
 82  sec_app_cr_hist_age_mths                 14453 non-null  Int64
 83  sec_app_inq_last_6mths                   14453 non-null  float64
 84  sec_app_mort_acc                         14453 non-null  float64
 85  sec_app_open_acc                         14453 non-null  float64
 86  sec_app_revol_util                       14453 non-null  float64
 87  sec_app_open_act_il                      14453 non-null  float64
 88  sec_app_num_rev_accts                    14453 non-null  float64
 89  sec_app_chargeoff_within_12_mths         14453 non-null  float64
 90  sec_app_collections_12_mths_ex_med       14453 non-null  float64
 91  sec_app_inv_mths_since_last_major_derog  14453 non-null  float64
 92  fraction_recovered                       14453 non-null  float64
 93  il_util_imputed                          14453 non-null  bool
dtypes: Int64(1), bool(1), float64(86), int64(1), object(5)
memory usage: 10.4+ MB

Phew, the data's all clean now! Time for the fun part.

Building the neural networks

After a good deal of trial and error, I found that a network architecture with three hidden layers, each followed by a dropout layer of rate 0.3, was as good as I could find. I used ReLU activation in those hidden layers, and adam optimization and a loss metric of mean squared error in the model as a whole. I also settled on a mean squared logarithmic error loss function, since it performed better than mean absolute error, mean squared error, and mean absolute percentage error.

The dataset being so large, I had great results increasing the batch size for the first couple models.

from sklearn.model_selection import train_test_split
from sklearn_pandas import DataFrameMapper
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler
from tensorflow.keras import Sequential, Input
from tensorflow.keras.layers import Dense, Dropout


def run_pipeline(
    data, onehot_cols, ordinal_cols, batch_size, validate=True,
):
    X = data.drop(columns=["fraction_recovered"])
    y = data["fraction_recovered"]
    X_train, X_valid, y_train, y_valid = (
        train_test_split(X, y, test_size=0.2, random_state=0)
        if validate
        else (X, None, y, None)
    )

    transformer = DataFrameMapper(
        [
            (onehot_cols, OneHotEncoder(drop="if_binary")),
            (
                list(ordinal_cols.keys()),
                OrdinalEncoder(categories=list(ordinal_cols.values())),
            ),
        ],
        default=StandardScaler(),
    )

    X_train = transformer.fit_transform(X_train)
    X_valid = transformer.transform(X_valid) if validate else None

    input_nodes = X_train.shape[1]
    output_nodes = 1

    model = Sequential()
    model.add(Input((input_nodes,)))
    model.add(Dense(64, activation="relu"))
    model.add(Dropout(0.3, seed=0))
    model.add(Dense(32, activation="relu"))
    model.add(Dropout(0.3, seed=1))
    model.add(Dense(16, activation="relu"))
    model.add(Dropout(0.3, seed=2))
    model.add(Dense(output_nodes))
    model.compile(optimizer="adam", loss="mean_squared_logarithmic_error")

    history = model.fit(
        X_train,
        y_train,
        batch_size=batch_size,
        epochs=100,
        validation_data=(X_valid, y_valid) if validate else None,
        verbose=1,
    )

    return history.history, model, transformer


print("Model 1:")
history_1, _, _ = run_pipeline(loans_1, onehot_cols, ordinal_cols, batch_size=128,)
print("\nModel 2:")
history_2, _, _ = run_pipeline(
    loans_2, onehot_cols + new_metric_onehot_cols, ordinal_cols, batch_size=64,
)
print("\nModel 3:")
history_3, _, _ = run_pipeline(
    loans_3, onehot_cols + new_metric_onehot_cols, ordinal_cols, batch_size=32,
)

Model 1:
Epoch 1/100
6939/6939 - 15s - loss: 0.0267 - val_loss: 0.0233
Epoch 2/100
6939/6939 - 15s - loss: 0.0235 - val_loss: 0.0232
Epoch 3/100
6939/6939 - 15s - loss: 0.0234 - val_loss: 0.0233
Epoch 4/100
6939/6939 - 15s - loss: 0.0233 - val_loss: 0.0234
Epoch 5/100
6939/6939 - 15s - loss: 0.0233 - val_loss: 0.0233
Epoch 6/100
6939/6939 - 15s - loss: 0.0233 - val_loss: 0.0233
Epoch 7/100
6939/6939 - 15s - loss: 0.0232 - val_loss: 0.0232
Epoch 8/100
6939/6939 - 15s - loss: 0.0232 - val_loss: 0.0230
Epoch 9/100
6939/6939 - 15s - loss: 0.0232 - val_loss: 0.0230
Epoch 10/100
6939/6939 - 15s - loss: 0.0232 - val_loss: 0.0231
Epoch 11/100
6939/6939 - 16s - loss: 0.0232 - val_loss: 0.0232
Epoch 12/100
6939/6939 - 15s - loss: 0.0232 - val_loss: 0.0232
Epoch 13/100
6939/6939 - 15s - loss: 0.0232 - val_loss: 0.0232
Epoch 14/100
6939/6939 - 16s - loss: 0.0232 - val_loss: 0.0230
Epoch 15/100
6939/6939 - 18s - loss: 0.0232 - val_loss: 0.0231
Epoch 16/100
6939/6939 - 15s - loss: 0.0232 - val_loss: 0.0232
Epoch 17/100
6939/6939 - 15s - loss: 0.0232 - val_loss: 0.0231
Epoch 18/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 19/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 20/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0230
Epoch 21/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0232
Epoch 22/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0233
Epoch 23/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0230
Epoch 24/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0233
Epoch 25/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 26/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0233
Epoch 27/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 28/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0232
Epoch 29/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0230
Epoch 30/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0232
Epoch 31/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0232
Epoch 32/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 33/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0230
Epoch 34/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0233
Epoch 35/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0230
Epoch 36/100
6939/6939 - 18s - loss: 0.0231 - val_loss: 0.0231
Epoch 37/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0232
Epoch 38/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0234
Epoch 39/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0232
Epoch 40/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 41/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0232
Epoch 42/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0230
Epoch 43/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0229
Epoch 44/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0230
Epoch 45/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0231
Epoch 46/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0232
Epoch 47/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0231
Epoch 48/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0232
Epoch 49/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0232
Epoch 50/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0233
Epoch 51/100
6939/6939 - 17s - loss: 0.0231 - val_loss: 0.0230
Epoch 52/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 53/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0232
Epoch 54/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0232
Epoch 55/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0230
Epoch 56/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 57/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0230
Epoch 58/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 59/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 60/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 61/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0232
Epoch 62/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 63/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 64/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0230
Epoch 65/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 66/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0230
Epoch 67/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0232
Epoch 68/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0232
Epoch 69/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0233
Epoch 70/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0232
Epoch 71/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0231
Epoch 72/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0234
Epoch 73/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 74/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 75/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 76/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 77/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0230
Epoch 78/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0232
Epoch 79/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0232
Epoch 80/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0232
Epoch 81/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0230
Epoch 82/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 83/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0233
Epoch 84/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 85/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0232
Epoch 86/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0230
Epoch 87/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0233
Epoch 88/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 89/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 90/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0231
Epoch 91/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0232
Epoch 92/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0232
Epoch 93/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0230
Epoch 94/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0230
Epoch 95/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0232
Epoch 96/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0231
Epoch 97/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0231
Epoch 98/100
6939/6939 - 20s - loss: 0.0231 - val_loss: 0.0232
Epoch 99/100
6939/6939 - 16s - loss: 0.0231 - val_loss: 0.0231
Epoch 100/100
6939/6939 - 15s - loss: 0.0231 - val_loss: 0.0232

Model 2:
Epoch 1/100
5745/5745 - 11s - loss: 0.0361 - val_loss: 0.0298
Epoch 2/100
5745/5745 - 11s - loss: 0.0302 - val_loss: 0.0295
Epoch 3/100
5745/5745 - 11s - loss: 0.0299 - val_loss: 0.0295
Epoch 4/100
5745/5745 - 11s - loss: 0.0298 - val_loss: 0.0296
Epoch 5/100
5745/5745 - 11s - loss: 0.0298 - val_loss: 0.0293
Epoch 6/100
5745/5745 - 11s - loss: 0.0297 - val_loss: 0.0292
Epoch 7/100
5745/5745 - 11s - loss: 0.0297 - val_loss: 0.0293
Epoch 8/100
5745/5745 - 11s - loss: 0.0296 - val_loss: 0.0292
Epoch 9/100
5745/5745 - 11s - loss: 0.0296 - val_loss: 0.0293
Epoch 10/100
5745/5745 - 11s - loss: 0.0296 - val_loss: 0.0292
Epoch 11/100
5745/5745 - 12s - loss: 0.0296 - val_loss: 0.0292
Epoch 12/100
5745/5745 - 11s - loss: 0.0296 - val_loss: 0.0292
Epoch 13/100
5745/5745 - 11s - loss: 0.0295 - val_loss: 0.0292
Epoch 14/100
5745/5745 - 11s - loss: 0.0295 - val_loss: 0.0292
Epoch 15/100
5745/5745 - 13s - loss: 0.0295 - val_loss: 0.0293
Epoch 16/100
5745/5745 - 12s - loss: 0.0295 - val_loss: 0.0291
Epoch 17/100
5745/5745 - 12s - loss: 0.0295 - val_loss: 0.0291
Epoch 18/100
5745/5745 - 11s - loss: 0.0295 - val_loss: 0.0290
Epoch 19/100
5745/5745 - 11s - loss: 0.0295 - val_loss: 0.0293
Epoch 20/100
5745/5745 - 11s - loss: 0.0295 - val_loss: 0.0292
Epoch 21/100
5745/5745 - 11s - loss: 0.0294 - val_loss: 0.0291
Epoch 22/100
5745/5745 - 12s - loss: 0.0294 - val_loss: 0.0291
Epoch 23/100
5745/5745 - 11s - loss: 0.0294 - val_loss: 0.0293
Epoch 24/100
5745/5745 - 13s - loss: 0.0294 - val_loss: 0.0293
Epoch 25/100
5745/5745 - 14s - loss: 0.0294 - val_loss: 0.0291
Epoch 26/100
5745/5745 - 11s - loss: 0.0294 - val_loss: 0.0292
Epoch 27/100
5745/5745 - 11s - loss: 0.0294 - val_loss: 0.0291
Epoch 28/100
5745/5745 - 11s - loss: 0.0294 - val_loss: 0.0292
Epoch 29/100
5745/5745 - 11s - loss: 0.0294 - val_loss: 0.0292
Epoch 30/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0291
Epoch 31/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0291
Epoch 32/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0292
Epoch 33/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0291
Epoch 34/100
5745/5745 - 11s - loss: 0.0294 - val_loss: 0.0291
Epoch 35/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0291
Epoch 36/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0291
Epoch 37/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0292
Epoch 38/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0291
Epoch 39/100
5745/5745 - 12s - loss: 0.0293 - val_loss: 0.0291
Epoch 40/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0291
Epoch 41/100
5745/5745 - 12s - loss: 0.0294 - val_loss: 0.0292
Epoch 42/100
5745/5745 - 13s - loss: 0.0293 - val_loss: 0.0293
Epoch 43/100
5745/5745 - 13s - loss: 0.0293 - val_loss: 0.0291
Epoch 44/100
5745/5745 - 12s - loss: 0.0293 - val_loss: 0.0291
Epoch 45/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0292
Epoch 46/100
5745/5745 - 12s - loss: 0.0293 - val_loss: 0.0290
Epoch 47/100
5745/5745 - 12s - loss: 0.0293 - val_loss: 0.0291
Epoch 48/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0292
Epoch 49/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0291
Epoch 50/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0294
Epoch 51/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0292
Epoch 52/100
5745/5745 - 14s - loss: 0.0293 - val_loss: 0.0291
Epoch 53/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0290
Epoch 54/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0291
Epoch 55/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0291
Epoch 56/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0293
Epoch 57/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0291
Epoch 58/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0293
Epoch 59/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0292
Epoch 60/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0291
Epoch 61/100
5745/5745 - 11s - loss: 0.0293 - val_loss: 0.0290
Epoch 62/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0292
Epoch 63/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0291
Epoch 64/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0291
Epoch 65/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0294
Epoch 66/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0292
Epoch 67/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0292
Epoch 68/100
5745/5745 - 12s - loss: 0.0292 - val_loss: 0.0292
Epoch 69/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0293
Epoch 70/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0292
Epoch 71/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0291
Epoch 72/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0291
Epoch 73/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0291
Epoch 74/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0291
Epoch 75/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0293
Epoch 76/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0294
Epoch 77/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0291
Epoch 78/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0292
Epoch 79/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0292
Epoch 80/100
5745/5745 - 13s - loss: 0.0292 - val_loss: 0.0295
Epoch 81/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0290
Epoch 82/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0291
Epoch 83/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0291
Epoch 84/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0293
Epoch 85/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0292
Epoch 86/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0292
Epoch 87/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0293
Epoch 88/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0292
Epoch 89/100
5745/5745 - 12s - loss: 0.0292 - val_loss: 0.0291
Epoch 90/100
5745/5745 - 12s - loss: 0.0292 - val_loss: 0.0293
Epoch 91/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0292
Epoch 92/100
5745/5745 - 11s - loss: 0.0292 - val_loss: 0.0291
Epoch 93/100
5745/5745 - 11s - loss: 0.0291 - val_loss: 0.0291
Epoch 94/100
5745/5745 - 12s - loss: 0.0292 - val_loss: 0.0291
Epoch 95/100
5745/5745 - 12s - loss: 0.0292 - val_loss: 0.0293
Epoch 96/100
5745/5745 - 12s - loss: 0.0292 - val_loss: 0.0292
Epoch 97/100
5745/5745 - 13s - loss: 0.0291 - val_loss: 0.0295
Epoch 98/100
5745/5745 - 11s - loss: 0.0291 - val_loss: 0.0291
Epoch 99/100
5745/5745 - 12s - loss: 0.0291 - val_loss: 0.0292
Epoch 100/100
5745/5745 - 12s - loss: 0.0292 - val_loss: 0.0292

Model 3:
Epoch 1/100
362/362 - 1s - loss: 0.1141 - val_loss: 0.0628
Epoch 2/100
362/362 - 1s - loss: 0.0575 - val_loss: 0.0527
Epoch 3/100
362/362 - 1s - loss: 0.0506 - val_loss: 0.0502
Epoch 4/100
362/362 - 1s - loss: 0.0486 - val_loss: 0.0486
Epoch 5/100
362/362 - 1s - loss: 0.0468 - val_loss: 0.0477
Epoch 6/100
362/362 - 1s - loss: 0.0463 - val_loss: 0.0474
Epoch 7/100
362/362 - 1s - loss: 0.0455 - val_loss: 0.0471
Epoch 8/100
362/362 - 1s - loss: 0.0445 - val_loss: 0.0466
Epoch 9/100
362/362 - 1s - loss: 0.0439 - val_loss: 0.0465
Epoch 10/100
362/362 - 1s - loss: 0.0438 - val_loss: 0.0465
Epoch 11/100
362/362 - 1s - loss: 0.0433 - val_loss: 0.0463
Epoch 12/100
362/362 - 1s - loss: 0.0430 - val_loss: 0.0462
Epoch 13/100
362/362 - 1s - loss: 0.0428 - val_loss: 0.0463
Epoch 14/100
362/362 - 1s - loss: 0.0427 - val_loss: 0.0460
Epoch 15/100
362/362 - 1s - loss: 0.0425 - val_loss: 0.0462
Epoch 16/100
362/362 - 1s - loss: 0.0422 - val_loss: 0.0461
Epoch 17/100
362/362 - 1s - loss: 0.0416 - val_loss: 0.0459
Epoch 18/100
362/362 - 1s - loss: 0.0416 - val_loss: 0.0460
Epoch 19/100
362/362 - 1s - loss: 0.0412 - val_loss: 0.0460
Epoch 20/100
362/362 - 1s - loss: 0.0414 - val_loss: 0.0461
Epoch 21/100
362/362 - 1s - loss: 0.0410 - val_loss: 0.0459
Epoch 22/100
362/362 - 1s - loss: 0.0409 - val_loss: 0.0459
Epoch 23/100
362/362 - 1s - loss: 0.0401 - val_loss: 0.0456
Epoch 24/100
362/362 - 1s - loss: 0.0400 - val_loss: 0.0460
Epoch 25/100
362/362 - 1s - loss: 0.0399 - val_loss: 0.0462
Epoch 26/100
362/362 - 1s - loss: 0.0394 - val_loss: 0.0461
Epoch 27/100
362/362 - 1s - loss: 0.0396 - val_loss: 0.0466
Epoch 28/100
362/362 - 1s - loss: 0.0394 - val_loss: 0.0462
Epoch 29/100
362/362 - 1s - loss: 0.0388 - val_loss: 0.0468
Epoch 30/100
362/362 - 1s - loss: 0.0384 - val_loss: 0.0465
Epoch 31/100
362/362 - 1s - loss: 0.0386 - val_loss: 0.0462
Epoch 32/100
362/362 - 1s - loss: 0.0379 - val_loss: 0.0467
Epoch 33/100
362/362 - 1s - loss: 0.0375 - val_loss: 0.0469
Epoch 34/100
362/362 - 1s - loss: 0.0382 - val_loss: 0.0468
Epoch 35/100
362/362 - 1s - loss: 0.0375 - val_loss: 0.0475
Epoch 36/100
362/362 - 1s - loss: 0.0374 - val_loss: 0.0468
Epoch 37/100
362/362 - 1s - loss: 0.0371 - val_loss: 0.0471
Epoch 38/100
362/362 - 1s - loss: 0.0370 - val_loss: 0.0474
Epoch 39/100
362/362 - 1s - loss: 0.0363 - val_loss: 0.0472
Epoch 40/100
362/362 - 1s - loss: 0.0366 - val_loss: 0.0477
Epoch 41/100
362/362 - 1s - loss: 0.0359 - val_loss: 0.0488
Epoch 42/100
362/362 - 1s - loss: 0.0356 - val_loss: 0.0476
Epoch 43/100
362/362 - 1s - loss: 0.0361 - val_loss: 0.0474
Epoch 44/100
362/362 - 1s - loss: 0.0357 - val_loss: 0.0476
Epoch 45/100
362/362 - 1s - loss: 0.0360 - val_loss: 0.0480
Epoch 46/100
362/362 - 1s - loss: 0.0351 - val_loss: 0.0476
Epoch 47/100
362/362 - 1s - loss: 0.0354 - val_loss: 0.0475
Epoch 48/100
362/362 - 1s - loss: 0.0348 - val_loss: 0.0487
Epoch 49/100
362/362 - 1s - loss: 0.0348 - val_loss: 0.0475
Epoch 50/100
362/362 - 1s - loss: 0.0351 - val_loss: 0.0477
Epoch 51/100
362/362 - 1s - loss: 0.0344 - val_loss: 0.0481
Epoch 52/100
362/362 - 1s - loss: 0.0348 - val_loss: 0.0482
Epoch 53/100
362/362 - 1s - loss: 0.0346 - val_loss: 0.0480
Epoch 54/100
362/362 - 1s - loss: 0.0344 - val_loss: 0.0487
Epoch 55/100
362/362 - 1s - loss: 0.0331 - val_loss: 0.0490
Epoch 56/100
362/362 - 1s - loss: 0.0339 - val_loss: 0.0488
Epoch 57/100
362/362 - 1s - loss: 0.0334 - val_loss: 0.0486
Epoch 58/100
362/362 - 1s - loss: 0.0336 - val_loss: 0.0497
Epoch 59/100
362/362 - 1s - loss: 0.0335 - val_loss: 0.0488
Epoch 60/100
362/362 - 1s - loss: 0.0336 - val_loss: 0.0485
Epoch 61/100
362/362 - 1s - loss: 0.0336 - val_loss: 0.0490
Epoch 62/100
362/362 - 1s - loss: 0.0327 - val_loss: 0.0493
Epoch 63/100
362/362 - 1s - loss: 0.0327 - val_loss: 0.0488
Epoch 64/100
362/362 - 1s - loss: 0.0331 - val_loss: 0.0489
Epoch 65/100
362/362 - 1s - loss: 0.0327 - val_loss: 0.0486
Epoch 66/100
362/362 - 1s - loss: 0.0324 - val_loss: 0.0500
Epoch 67/100
362/362 - 1s - loss: 0.0326 - val_loss: 0.0493
Epoch 68/100
362/362 - 1s - loss: 0.0320 - val_loss: 0.0499
Epoch 69/100
362/362 - 1s - loss: 0.0323 - val_loss: 0.0497
Epoch 70/100
362/362 - 1s - loss: 0.0321 - val_loss: 0.0495
Epoch 71/100
362/362 - 1s - loss: 0.0323 - val_loss: 0.0492
Epoch 72/100
362/362 - 1s - loss: 0.0321 - val_loss: 0.0487
Epoch 73/100
362/362 - 1s - loss: 0.0324 - val_loss: 0.0491
Epoch 74/100
362/362 - 1s - loss: 0.0325 - val_loss: 0.0487
Epoch 75/100
362/362 - 1s - loss: 0.0315 - val_loss: 0.0496
Epoch 76/100
362/362 - 1s - loss: 0.0317 - val_loss: 0.0503
Epoch 77/100
362/362 - 1s - loss: 0.0318 - val_loss: 0.0486
Epoch 78/100
362/362 - 1s - loss: 0.0312 - val_loss: 0.0504
Epoch 79/100
362/362 - 1s - loss: 0.0312 - val_loss: 0.0493
Epoch 80/100
362/362 - 1s - loss: 0.0315 - val_loss: 0.0490
Epoch 81/100
362/362 - 1s - loss: 0.0307 - val_loss: 0.0502
Epoch 82/100
362/362 - 1s - loss: 0.0310 - val_loss: 0.0499
Epoch 83/100
362/362 - 1s - loss: 0.0312 - val_loss: 0.0499
Epoch 84/100
362/362 - 1s - loss: 0.0306 - val_loss: 0.0507
Epoch 85/100
362/362 - 1s - loss: 0.0311 - val_loss: 0.0506
Epoch 86/100
362/362 - 1s - loss: 0.0305 - val_loss: 0.0506
Epoch 87/100
362/362 - 1s - loss: 0.0309 - val_loss: 0.0503
Epoch 88/100
362/362 - 1s - loss: 0.0314 - val_loss: 0.0504
Epoch 89/100
362/362 - 1s - loss: 0.0308 - val_loss: 0.0508
Epoch 90/100
362/362 - 1s - loss: 0.0304 - val_loss: 0.0500
Epoch 91/100
362/362 - 1s - loss: 0.0306 - val_loss: 0.0497
Epoch 92/100
362/362 - 1s - loss: 0.0305 - val_loss: 0.0510
Epoch 93/100
362/362 - 1s - loss: 0.0308 - val_loss: 0.0502
Epoch 94/100
362/362 - 1s - loss: 0.0300 - val_loss: 0.0504
Epoch 95/100
362/362 - 1s - loss: 0.0310 - val_loss: 0.0509
Epoch 96/100
362/362 - 1s - loss: 0.0306 - val_loss: 0.0509
Epoch 97/100
362/362 - 1s - loss: 0.0298 - val_loss: 0.0509
Epoch 98/100
362/362 - 1s - loss: 0.0303 - val_loss: 0.0503
Epoch 99/100
362/362 - 1s - loss: 0.0296 - val_loss: 0.0509
Epoch 100/100
362/362 - 1s - loss: 0.0301 - val_loss: 0.0509

The first model performed best, settling around a mean squared logarithmic error of 0.0231 (though it seems even after setting random_state inside train_test_split and seed inside the dropout layers, there's still a bit of entropy left in the training of the model, so if you run this notebook yourself, the course of your training may look a little different). Apparently the additional records in the first dataset did more to aid in training than the additional metrics in the subsequent sets. And the dropout layers didn't stop the third model from overfitting anyway.

sns.lineplot(x=range(1, 101), y=history_1["loss"], label="loss")
sns.lineplot(x=range(1, 101), y=history_1["val_loss"], label="val_loss")
plt.xlabel("epoch")
plt.title("Model 1 loss metrics during training")
plt.show()

Saving the final model

First I need to create the final model, training model_1's architecture on the full dataset. Then I'll save the model to disk with its save function and save the data transformer using joblib so I can use it in the API.

import joblib

_, final_model, final_transformer = run_pipeline(
    loans_1, onehot_cols, ordinal_cols, batch_size=128, validate=False
)

final_model.save("loan_risk_model")
joblib.dump(final_transformer, "data_transformer.joblib")

Epoch 1/100
8674/8674 - 17s - loss: 0.0268
Epoch 2/100
8674/8674 - 17s - loss: 0.0234
Epoch 3/100
8674/8674 - 17s - loss: 0.0233
Epoch 4/100
8674/8674 - 17s - loss: 0.0233
Epoch 5/100
8674/8674 - 17s - loss: 0.0232
Epoch 6/100
8674/8674 - 17s - loss: 0.0232
Epoch 7/100
8674/8674 - 17s - loss: 0.0232
Epoch 8/100
8674/8674 - 17s - loss: 0.0232
Epoch 9/100
8674/8674 - 17s - loss: 0.0232
Epoch 10/100
8674/8674 - 22s - loss: 0.0231
Epoch 11/100
8674/8674 - 17s - loss: 0.0232
Epoch 12/100
8674/8674 - 17s - loss: 0.0231
Epoch 13/100
8674/8674 - 17s - loss: 0.0231
Epoch 14/100
8674/8674 - 17s - loss: 0.0231
Epoch 15/100
8674/8674 - 17s - loss: 0.0231
Epoch 16/100
8674/8674 - 17s - loss: 0.0231
Epoch 17/100
8674/8674 - 17s - loss: 0.0232
Epoch 18/100
8674/8674 - 19s - loss: 0.0231
Epoch 19/100
8674/8674 - 18s - loss: 0.0231
Epoch 20/100
8674/8674 - 17s - loss: 0.0231
Epoch 21/100
8674/8674 - 16s - loss: 0.0231
Epoch 22/100
8674/8674 - 17s - loss: 0.0231
Epoch 23/100
8674/8674 - 17s - loss: 0.0231
Epoch 24/100
8674/8674 - 17s - loss: 0.0231
Epoch 25/100
8674/8674 - 16s - loss: 0.0231
Epoch 26/100
8674/8674 - 17s - loss: 0.0231
Epoch 27/100
8674/8674 - 18s - loss: 0.0231
Epoch 28/100
8674/8674 - 18s - loss: 0.0231
Epoch 29/100
8674/8674 - 18s - loss: 0.0231
Epoch 30/100
8674/8674 - 17s - loss: 0.0231
Epoch 31/100
8674/8674 - 17s - loss: 0.0231
Epoch 32/100
8674/8674 - 17s - loss: 0.0231
Epoch 33/100
8674/8674 - 17s - loss: 0.0231
Epoch 34/100
8674/8674 - 17s - loss: 0.0231
Epoch 35/100
8674/8674 - 17s - loss: 0.0231
Epoch 36/100
8674/8674 - 17s - loss: 0.0231
Epoch 37/100
8674/8674 - 19s - loss: 0.0231
Epoch 38/100
8674/8674 - 17s - loss: 0.0231
Epoch 39/100
8674/8674 - 17s - loss: 0.0231
Epoch 40/100
8674/8674 - 17s - loss: 0.0231
Epoch 41/100
8674/8674 - 17s - loss: 0.0231
Epoch 42/100
8674/8674 - 17s - loss: 0.0231
Epoch 43/100
8674/8674 - 17s - loss: 0.0231
Epoch 44/100
8674/8674 - 18s - loss: 0.0231
Epoch 45/100
8674/8674 - 18s - loss: 0.0231
Epoch 46/100
8674/8674 - 17s - loss: 0.0231
Epoch 47/100
8674/8674 - 18s - loss: 0.0231
Epoch 48/100
8674/8674 - 17s - loss: 0.0231
Epoch 49/100
8674/8674 - 17s - loss: 0.0231
Epoch 50/100
8674/8674 - 17s - loss: 0.0231
Epoch 51/100
8674/8674 - 17s - loss: 0.0231
Epoch 52/100
8674/8674 - 16s - loss: 0.0231
Epoch 53/100
8674/8674 - 17s - loss: 0.0231
Epoch 54/100
8674/8674 - 17s - loss: 0.0231
Epoch 55/100
8674/8674 - 18s - loss: 0.0231
Epoch 56/100
8674/8674 - 16s - loss: 0.0231
Epoch 57/100
8674/8674 - 17s - loss: 0.0231
Epoch 58/100
8674/8674 - 17s - loss: 0.0231
Epoch 59/100
8674/8674 - 16s - loss: 0.0231
Epoch 60/100
8674/8674 - 16s - loss: 0.0230
Epoch 61/100
8674/8674 - 17s - loss: 0.0231
Epoch 62/100
8674/8674 - 17s - loss: 0.0231
Epoch 63/100
8674/8674 - 18s - loss: 0.0231
Epoch 64/100
8674/8674 - 18s - loss: 0.0231
Epoch 65/100
8674/8674 - 17s - loss: 0.0231
Epoch 66/100
8674/8674 - 17s - loss: 0.0231
Epoch 67/100
8674/8674 - 17s - loss: 0.0231
Epoch 68/100
8674/8674 - 17s - loss: 0.0231
Epoch 69/100
8674/8674 - 17s - loss: 0.0231
Epoch 70/100
8674/8674 - 16s - loss: 0.0231
Epoch 71/100
8674/8674 - 17s - loss: 0.0231
Epoch 72/100
8674/8674 - 17s - loss: 0.0231
Epoch 73/100
8674/8674 - 19s - loss: 0.0231
Epoch 74/100
8674/8674 - 19s - loss: 0.0231
Epoch 75/100
8674/8674 - 17s - loss: 0.0231
Epoch 76/100
8674/8674 - 17s - loss: 0.0231
Epoch 77/100
8674/8674 - 17s - loss: 0.0231
Epoch 78/100
8674/8674 - 17s - loss: 0.0231
Epoch 79/100
8674/8674 - 17s - loss: 0.0231
Epoch 80/100
8674/8674 - 17s - loss: 0.0231
Epoch 81/100
8674/8674 - 18s - loss: 0.0230
Epoch 82/100
8674/8674 - 17s - loss: 0.0230
Epoch 83/100
8674/8674 - 17s - loss: 0.0230
Epoch 84/100
8674/8674 - 17s - loss: 0.0231
Epoch 85/100
8674/8674 - 17s - loss: 0.0231
Epoch 86/100
8674/8674 - 17s - loss: 0.0231
Epoch 87/100
8674/8674 - 17s - loss: 0.0230
Epoch 88/100
8674/8674 - 16s - loss: 0.0231
Epoch 89/100
8674/8674 - 17s - loss: 0.0231
Epoch 90/100
8674/8674 - 17s - loss: 0.0230
Epoch 91/100
8674/8674 - 17s - loss: 0.0230
Epoch 92/100
8674/8674 - 19s - loss: 0.0231
Epoch 93/100
8674/8674 - 19s - loss: 0.0231
Epoch 94/100
8674/8674 - 17s - loss: 0.0231
Epoch 95/100
8674/8674 - 17s - loss: 0.0230
Epoch 96/100
8674/8674 - 18s - loss: 0.0231
Epoch 97/100
8674/8674 - 17s - loss: 0.0231
Epoch 98/100
8674/8674 - 17s - loss: 0.0231
Epoch 99/100
8674/8674 - 19s - loss: 0.0231
Epoch 100/100
8674/8674 - 17s - loss: 0.0231

['data_transformer.joblib']

Building the API

I first tried building this API and its demonstrational front end on Glitch, which, officially, only supports Node.js back ends, but unofficially you can get a Python server running there (which I've done before using Flask). When I was almost finished, though, I tried importing TensorFlow to load my model, and it was then that I discovered that unlike Node.js dependencies, Python dependencies get installed to your project's disk space on Glitch, and not even their pro plan provides enough space to contain the entire TensorFlow library. Which totally makes sense—I certainly wasn't using the platform as intended.

Then I discovered PythonAnywhere! They have plenty of common Python libraries already installed out-of-the-box, including TensorFlow, so I got everything working perfectly there.

So head on over if you'd like to check it out; the front end includes a form where you can fill in all the parameters for the API request, and there are a couple of buttons that let you fill the form with typical examples from the dataset (since there are a lot of fields to fill in). Or you can send a GET request to https://tywmick.pythonanywhere.com/api/predict if you really want to include every parameter in your query string. In either case, you're also more than welcome to take a look at its source on GitHub.

One of the best/worst things about machine learning is that your models always have room for improvement. I mentioned a couple ideas along the way above for how I could improve the model in the future, but what's the first thing you would tweak? I'd love to hear in the discussion below.

# Exports for "Can I Grade Loans Better than LendingClub?"
expected.rename("expected_return", inplace=True)
loans_for_eval = loans_1.join([loans_raw[["issue_d", "grade", "sub_grade"]], expected])
joblib.dump(loans_for_eval, "loans_for_eval.joblib")

# Exports for "Improving Loan Risk Prediction With Natural Language Processing"
loans_for_nlp = loans_1.join(loans_raw[["issue_d", "title", "desc"]])
joblib.dump(loans_for_nlp, "loans_for_nlp.joblib")

['loans_for_nlp.joblib']

How to make a browser console log wait until after a page reload

Ty Mick — Mon, 01 Jun 2020 22:45:35 +0000

Ever wanted to create a browser console log that persists after a page reloads? Sure, that isn't a problem if you enable the "preserve log" option in your developer console, but hear me out.

Say you have a JavaScript framework with a development server that usually hot-reloads when you update files while the server is running. In the few cases where hot reloading isn't possible and the page has to fully reload, you want to warn your users and explain why this is happening. It'd be nice to just log a warning to the console with console.warn(), but the moment it shows up in the console, wouldn't you know, the page reloads, and the browser clears it. Even if your user is eagle-eyed enough to notice your warning flash on the screen for a fraction of a second, it certainly isn't there long enough to actually read.

You could say that if any of your users don't know where the "preserve log" button is then that's their problem, but you're kinder than that. You'd rather help them fall into the Pit of Success.

As it happens, that's the very issue I ran into the other day when strolling through my favorite open source project, Next.js.

In my naïveté, I first tried just moving the console.warn() to the line after window.location.reload() and crossing my fingers, but that didn't help. I tried googling things like "console log after page reload", but that only gave me instructions for how to turn on "preserve log", which I already knew how to do.

Here's what worked. Where the console.warn() statement stood before, I instead saved the text content of the warning to the window's sessionStorage in a key called "consoleWarnAfterReload":

sessionStorage.setItem(
  "consoleWarnAfterReload",
  "Dear me, the page had to fully reload!"
);

Then, near the top of a file involved in the loading of a new page, I added a few lines that check that same sessionStorage key. If the key exists, it logs the contents of the warning to the console and then removes the item from sessionStorage:

if (sessionStorage && sessionStorage.getItem("consoleWarnAfterReload")) {
  console.warn(sessionStorage.getItem("consoleWarnAfterReload"));
  sessionStorage.removeItem("consoleWarnAfterReload");
}

That did the trick!

Know another way to make console logs wait until after a page reloads? Let me know in the discussion below—I'd love to hear about it!