I Kept Doing the Same Sanity Check on Every Small Dataset, So I Turned It Into a Package

After writing up two small projects - one predicting data science salaries, one checking whether Punxsutawney Phil's shadow call means anything - I noticed I'd done the same manual step both times: train a couple of models, check the test-set R², then immediately re-run everything with cross-validation before trusting that number. On a dataset with a few hundred rows, a single train/test split can make a weaker model look like the winner purely by chance, and I didn't want to keep catching that by hand.

So I turned the check into a small package: (https://github.com/DostonUr/modelcheck).

What it does

You give it your data and a dictionary of models, and it does the comparison I was doing by hand:

from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from modelcheck import compare_regressors

result = compare_regressors(
    X, y,
    models={
        "Linear Regression": LinearRegression(),
        "Random Forest": RandomForestRegressor(n_estimators=300, random_state=42),
    },
    cv=5,
)
print(result)

It reports the held-out test score for each model, the 5-fold cross-validation spread (mean, std, min, max) for each, and — the part I actually built this for - it automatically flags two specific situations:

1. A model's cross-validation score bounces around a lot fold to fold, meaning the single test-set number shouldn't be trusted as precise.
2. Two models' CV score ranges overlap heavily, meaning whichever one scored higher on the held-out split isn't necessarily the real winner.

Proving it actually works, not just on a toy example

I re-ran both of my earlier projects through it instead of hand-writing the checks again.

On the salary data, it reproduced the exact numbers from the original article - Linear Regression R² 0.273 vs. Random Forest R² 0.24 on the test split - and it correctly generated the overlap warning without me writing that logic manually this time:

"Linear Regression" and "Random Forest" have heavily overlapping CV score ranges. The higher mean R2 alone is not strong evidence that "Linear Regression" is genuinely better on this data.

Then I tried something I hadn't done in the original groundhog article: instead of a hand-computed t-test, I set up a regression comparing a model that uses Phil's shadow call against a DummyRegressor that just always predicts the historical average, and ran both through modelcheck. If the shadow call carried real information, the model using it should have clearly beaten the dummy. Instead, the dummy baseline scored better (R² of ‑0.046 vs. ‑0.061), both were negative, and the tool flagged the same overlap warning again. That's the same "no real signal" conclusion from the original article, reached through a completely different comparison method — which is a more convincing confirmation than either check on its own.

Why I'm putting this out instead of keeping it as a personal script

A one-off script only proves I can write code. A package other people can install and actually use is a different kind of evidence of the work — it has to have real tests, has to handle inputs I didn't originally write it for, and has to be documented well enough that a stranger can pick it up. It's a small tool, but it's built to be used, not just read.

Try it

pip install git+https://github.com/DostonUr/modelcheck.git

Code, tests, and both worked examples: github.com/DostonUr/modelcheck

If you've hit the same problem - a small dataset making one model look better than it really is - I'd like to hear how you've handled it, or what this package is missing. Reply here or find me on
(https://www.linkedin.com/in/doston-urinov/).