Predicting the Financial Health of Small Businesses with Python

Introduction

Small businesses keep many African economies going, but it is often a struggle for their owners when it comes to getting loans, insurance and other financial services. If we could tell from a few survey answers which businesses are doing well and which are struggling, the right people could get help faster.

That was the question behind this project. The data came from the data.org Financial Health Prediction Challenge on Zindi, and the goal was to predict whether a business has low, medium or high financial health.

This article is about the thinking behind the project. I explain what I did at each stage and why. The full notebook, with all the code, is on GitHub.

A note before we start: I am still learning, many of the decisions below are my own judgement calls. If you spot something I got wrong, feel free to point it out in the comments.

The data

The file covered 9,618 small business owners in Eswatini, Lesotho, Malawi and Zimbabwe, with 39 columns each. The questions covered the owner's age and gender, the business's turnover and age, which financial products they used (mobile money, loans, insurance), how they handled money and how they felt about the future.

Every business had been placed into one of three groups using a Financial Health Index, which looks at savings and assets, debt and repayment ability, resilience to shocks, and access to credit.

Exploring the data

Before anything else, it is important to understand the information you are working with. The dataset was accompanied by a variable definitions sheet, which clarified what each column contained. After going through it, I performed basic checks on the data. These checked the size of the data, looked for duplicate rows, counted missing values and looked at how the numbers were spread out:

fhp_df.shape                              # how many rows and columns
fhp_df.duplicated().sum()                 # how many duplicate rows
(fhp_df.isna().mean() * 100).round(1)     # share of missing values in each column

I wrapped these checks, plus a few more, into one function that could be reused on any dataset. You can find it in the notebook.

There were no duplicates, but some columns were missing almost half of their answers, and the money columns contained extreme outliers that sat very far from the rest.

Cleaning the Data

Before cleaning, I split the columns into two groups: categorical (text) columns and numerical columns, because the different data types often have different problems and different solutions. This section is about the categorical columns, numerical columns are tackled later.

cat_columns = fhp_df.select_dtypes(include=['str', 'object']).columns.to_list()

Survey answers are almost never tidy, and in this case some had extra spaces or capital letters, trimming the spaces and lowercasing everything fixed most of that:

for cols in cat_columns:
    fhp_df[cols] = fhp_df[cols].str.strip().str.lower()

The rest I found by printing the unique answers in every categorical column and reading them. That turned up a problem that took me too long to figure out: answers that looked identical but used different apostrophe characters, a straight one (') and a curly one (’). There were also several ways of writing "don't know", a hidden invisible character in one answer, and a stray 0 that I read as "no". I merged each group of look-alikes into one spelling.

Two of my decisions were based on assumptions: that 0 means "no", and that a plain "yes" means "yes, always".

Missing data

I used 35% as a cut-off, with columns missing more than 35% of their answers were handled differently from the rest. Overall, twelve columns were above it, missing between 37% and 47% of their answers:

cat_df = fhp_df.select_dtypes(include=['object', 'str'])
missing_columns = round((cat_df.isna().sum() / len(cat_df)) * 100)
missing_columns[missing_columns > 35]

I checked the gaps country by country, and realized Lesotho had no answers at all for eleven of the categorical columns, such as gender, credit card ownership and insurance. Other countries had a few empty columns too. This was probably because those questions were not asked there:

missing_by_country = cat_df.isna().groupby(fhp_df['country']).mean() * 100
missing_by_country.T[(missing_by_country.T > 35).any(axis=1)].round(1)

This informed my decision to apply the 35% cut-off within each country instead of overall. So a column gets its own category, "unknown", if more than 35% of its answers are missing in at least one country, because not answering can be information in itself. On the training data, that covered 25 of the 31 categorical columns. The remaining six columns had almost no gaps (at most 0.1% missing), so I decided to fill them with the mode.

The 35% cut-off, and applying it country by country, were my own judgement calls and not fixed rules, so a slightly different rule would be defensible.

At this stage I only looked at how much was missing and made these decisions. Filling the missing values was done after splitting the train and test data.

Standardizing Currency

The biggest decision was about money. Income, expenses and turnover were recorded in each country's own currency, so 2,000 in one country was not worth the same as 2,000 in another.
I wrote a small function that looks up each row's country, finds that country's exchange rate and converts the amount to US dollars:

def convert_to_usd(row, column):
    rates = {'eswatini': 0.062, 'lesotho': 0.062, 'zimbabwe': 0.037, 'malawi': 0.00058}
    return row[column] * rates[row['country']]

fhp_df['personal_income_usd'] = fhp_df.apply(convert_to_usd, column='personal_income', axis=1)

These are approximate, fixed rates, one per country which is a simplification, because exchange rates change over time. A more accurate version would use the rates from the survey period.

Outliers

The typical (median) business turnover was about $174, but the average was about $2,321, because a few very large businesses pulled it up. That one fact shaped several later choices. I filled gaps using the median rather than the average, and I picked methods that extreme values do not throw off.

Columns / Entries that were dropped

I dropped the number of months a business had been running, because almost half of it was missing (all of Lesotho and almost all of Malawi had no answers for it) and the number of years already said the same thing. I also dropped business expenses, because I could not tell whether the amounts were monthly or yearly.

Personal income was reported monthly and turnover yearly, so an owner whose monthly income was higher than their business's whole-year turnover I considered to be almost certainly an error. 2,254 rows had the income higher and I removed them.

Correlation analysis and statistical tests

Before building models, I wanted to determine whether the survey answers were linked to financial health at all. The right statistical test depends on the type of data being compared, so I used four:

• Spearman correlation: I chose Spearman over Pearson correlation because the money columns contained extreme outliers. Spearman works with the rank order of the values instead of the values themselves, so a few extreme values cannot dominate the result. Personal income and business turnover were strongly linked (about 0.75), while owner age and business age were more weakly linked (about 0.39).
• Chi-square test of independence: to see whether the mix of answers to categorical variables differed between the low, medium and high groups.
• Kruskal-Wallis test: to see whether a numerical column, such as income, typically differs between the three groups. I chose it over ANOVA because it is non parametric, and the numerical column distributions violated the assumptions of normality.
• Variance Inflation Factor(VIF): to check that no numerical column was repeating the information in the others. A VIF of 1 means a column adds its own information, values above 5 are a warning sign, and values above 10 are serious. Mine ranged from about 1.02 to 1.23, so I kept every numerical column.

For the Chi-square and Kruskal-Wallis tests, a result is called "significant" when the difference is unlikely to be down to chance.

Almost everything came out as significant: 30 of the 31 categorical columns and all four numerical columns. The one exception was covid_essential_service, which was empty for all of Malawi, so the test could only use the other three countries. The same applies to every column that one country did not answer, the tests say nothing about that country.

With thousands of rows, even tiny differences show up as significant, so significance level only tells you that a link exists, not how strong it is. Because almost everything passed, these tests did not help me narrow down my list of columns, so I kept them all.

Separating the target

Before splitting, I separated the columns into the features and the target. The features are the survey answers the model uses to make its prediction, and the target is the financial health group I wanted it to predict. The target had to be removed from the features. If it stayed in, the model would be handed the answer it is supposed to predict, and it would look brilliant without learning anything useful. I also dropped ID, because it is only a label and says nothing about a business, and age_group, because it is just owner_age sorted into bands, so it repeated information that was already there.

X = fhp_df.drop(columns=['Target'])   # the features: what the model uses
y = fhp_df['Target']                  # the target: what the model predicts

Splitting the data

I split the businesses into two sets: a training set and a test set. The training set is what the models learn from. The test set is kept hidden until the end and is used to check how the models do on businesses they have never seen.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

I used an 80/20 split: 5,891 businesses for training and 1,473 for testing. This is a common starting point, not a fixed rule, so it was a judgement call. The best split depends on the size of the data and on the purpose of the model. With millions of rows, a much smaller test set is enough. With very small datasets, people often repeat the split several times instead, which is called cross-validation.

The random_state=42 setting fixes the random shuffle, so anyone who runs the notebook gets exactly the same split.

Filling missing values

Missing values are best filled after splitting the data, whether the fill is the median, the mode or a placeholder such as "unknown".

The reason is data leakage, i.e. when information from outside the training data influences the model, the test scores look better than they really are, and the model does worse on genuinely new data. Filling gaps can cause this, because the median and the mode are worked out from the data. If they come from all the businesses, the test businesses have helped to decide them. So I worked out everything from the training businesses only (including which columns got "unknown") and then used those same values to fill the gaps in both the training and test sets, just as a brand-new business would be handled in real use.

train_medians = X_train[numm_cols].median()                    # worked out from the training data only
X_train[numm_cols] = X_train[numm_cols].fillna(train_medians)
X_test[numm_cols] = X_test[numm_cols].fillna(train_medians)    # the test data uses the training medians

Scaling and encoding

I used label encoding to convert the target column into numerical labels that the models can work with. For the categorical feature columns, I used one-hot encoding to turn each category into its own yes-or-no column, represented by 1s and 0s. I then aligned the test data columns with the training data columns so that both datasets had the same features.

I scaled the numerical columns so that variables measured in thousands would not automatically outweigh those measured in tens. I used RobustScaler because it is less sensitive to extreme values, given the highly skewed money data.

The same rule applies throughout: fit on the training data, then only transform the test data.

scaler = RobustScaler()
X_train[numm_cols] = scaler.fit_transform(X_train[numm_cols])   # learn from the training data
X_test[numm_cols] = scaler.transform(X_test[numm_cols])         # only apply to the test data

The models

I started with simple models and worked up to more complex ones, because a simple baseline tells you whether the complexity is worth it. I tried Logistic Regression, a Decision Tree, K-Nearest Neighbours, Random Forest, Gradient Boosting, XGBoost and an SVM. For each one, I used the same four steps: create the model, train it on the training data, predict on the test data, and print a classification report.

The scores I looked at were accuracy (how many businesses were classified correctly), macro F1 (an average score across the three groups, so the small high group counts as much as the large low group), and recall on the high group (how many truly healthy businesses the model found).

The results

Model	Accuracy	Macro F1	Recall on high
Logistic Regression	0.86	0.75	0.41
Decision Tree	0.87	0.77	0.53
K-Nearest Neighbours	0.77 - 0.79	0.62	0.24 - 0.28
Random Forest	0.81	0.59	0.13
Gradient Boosting	0.87	0.79	0.54
XGBoost	0.87	0.78	0.51
SVM	0.83	0.66	0.24

The test set had 1,473 businesses, and only 79 of them were in the high group. For K-Nearest Neighbours I tried three ways of measuring distance (Euclidean, Manhattan and Minkowski). The Euclidean and Minkowski results were identical, because Minkowski with its default setting is the same as Euclidean, so the table shows the range.

Gradient Boosting did best overall, with a macro F1 of 0.79, followed closely by XGBoost (0.78) and the Decision Tree (0.77). Gradient Boosting also found the most high businesses (recall 0.54), but that still means it missed almost half of them. The Random Forest shows the trade-off clearly: every time it said a business was high it was right (precision 1.00), but it said so so rarely that it found only 13% of them. The simpler models did well too. Logistic Regression and the Decision Tree came close to the boosting models, and they are much easier to explain.

The gaps between the top models are small and come from a single split, so I would not read too much into who came first.

One lesson stands out. Accuracy can flatter a model. About 65% of the businesses are in the low group, so a model that mostly guesses "low" already looks good. The high group is only just over 5% of the data, so it is the real test. Four of the models scored between 0.86 and 0.87 on accuracy, yet their recall on the high group ranged from 0.41 to 0.54. That is why I looked at the macro F1 score and at how well each model found the high group, not at accuracy alone. The competition itself is scored with F1, which is a useful hint that accuracy alone is not the right yardstick.

These scores come from my own train/test split, so they cannot be compared with the Zindi leaderboard.

Limitations

• Questions that were not asked in every country. For the columns that a country did not answer, the model cannot learn what those answers mean for that country. The "unknown" category also lines up with the country, so the model sees the country twice.
• The country itself. The share of high businesses differs a lot between countries (over 12% in Eswatini, under 1% in Lesotho), so the model may rely heavily on where a business is, rather than on what it does.
• The split and the settings. I used one random split and fixed settings for each model. A stratified split, cross-validation and tuning the settings would give more reliable results.

Wrapping up

The biggest lesson was that the hard part was not choosing a clever algorithm. It was understanding the data: noticing the apostrophes, converting the currencies, questioning the odd numbers and judging the models on the hardest group instead of the easy one.

You can find the full notebook, the README and the data here: Small Businesses Financial Health Prediction.

If you try your own version, or find a better way of doing something, I would love to hear about it in the comments.