Skrub: A Python Library for Machine Learning with Messy Data

When working with machine learning, we often spend more time preparing data than training models. This becomes particularly frustrating when a dataset contains high-cardinality categorical variables, inconsistent strings, or information spread across multiple tables.

Skrub is a relatively lesser-known Python library that tackles exactly this part of the machine-learning workflow.

Install it with:

pip install skrub

Skrub works alongside familiar libraries such as pandas and scikit-learn. Its focus is not on providing another collection of machine-learning algorithms, but on making tabular data easier to represent and use in ML pipelines.

Handling difficult categorical variables

Consider a dataset containing:

company
-------------------------
Microsoft
Microsoft Corp
Microsoft Corporation
MICROSOFT

A conventional categorical encoder can treat these as four unrelated categories. However, the strings contain useful similarities.

Skrub provides "MinHashEncoder", which can transform strings into a fixed-size numerical representation based on their character-level similarities:

from skrub import MinHashEncoder

encoder = MinHashEncoder(n_components=10)

X_encoded = encoder.fit_transform(
    df[["company"]]
)

This can be useful when categorical variables contain many unique values or variations caused by spelling, formatting, abbreviations, or inconsistent data entry.

What about thousands of categories?

Suppose a dataset contains 100,000 product names. One-hot encoding could potentially create an extremely large feature matrix.

Skrub provides alternatives such as "MinHashEncoder" and "GapEncoder". "GapEncoder" is particularly interesting because it can discover latent patterns in categorical strings instead of simply assigning a separate representation to every category.

For example, occupations such as "Data Scientist", "Data Analyst", "Software Engineer", and "Financial Analyst" contain textual patterns that ordinary one-hot encoding does not exploit.

from skrub import GapEncoder

encoder = GapEncoder(n_components=10)

X_encoded = encoder.fit_transform(
    df[["occupation"]]
)

Working with multiple tables

Real-world data is also rarely contained in one clean dataframe. A project may have separate customer, transaction, product, and payment tables.

Skrub includes tools such as "Joiner" and "AggJoiner" for constructing useful features from related tables. This makes it interesting for database-heavy machine-learning projects where feature engineering involves aggregating information across relationships rather than simply transforming individual columns.

Why not just use scikit-learn?

You can—and you probably should.

Skrub is designed to complement the scikit-learn ecosystem rather than replace it. You can use Skrub to prepare or represent difficult tabular data and then pass the resulting features into familiar scikit-learn estimators.

That makes the workflow quite natural:

from sklearn.ensemble import RandomForestRegressor
from skrub import MinHashEncoder

encoder = MinHashEncoder(n_components=20)

X_train_encoded = encoder.fit_transform(
    X_train[["company"]]
)

model = RandomForestRegressor(
    random_state=42
)

model.fit(X_train_encoded, y_train)

The important idea is that feature representation is part of machine learning. A sophisticated model cannot compensate for every weakness in the way raw information is represented.

When is Skrub worth trying?

Skrub is particularly interesting when your data contains:

• high-cardinality categorical variables;
• messy or inconsistent strings;
• company, product, customer, or occupation names;
• several related tables;
• mixed-type tabular data; or
• preprocessing that is becoming difficult to manage with standard encoders.

It is not a replacement for pandas, NumPy, or scikit-learn. Instead, it fills a more specialized gap: turning difficult real-world tables into representations that machine-learning models can work with.

That makes Skrub a useful library to know—even if it isn't yet one of the names that immediately comes to mind when someone says "Python machine learning."

Have you used Skrub in a machine-learning project?