DEV Community

Cover image for scm2cpp-learn: 44 scikit-learn estimators, same results, up to ~1000x faster
Hirotaka Niitsuma
Hirotaka Niitsuma

Posted on Edited on

scm2cpp-learn: 44 scikit-learn estimators, same results, up to ~1000x faster

#python #opensource #performance #machinelearning

I'd like to share an open source (MIT) project I've been working on: scm2cpp-learn, drop-in replacements for scikit-learn estimators.

What it is

  • 44 scikit-learn estimators (Lasso, LogisticRegression, RFE, PCA, SVM, ...) implementing the same algorithms with the same results — coefficients agree to rounding error
  • 1.5x to ~1000x faster (e.g. RFE 1,047x; MultiTaskElasticNetCV ~100x on a single core)
  • scikit-learn API compatible: works in Pipeline and GridSearchCV, passes check_estimator
  • Prebuilt wheels for Linux, macOS and Windows; optional GPU (CUDA) 
pip install scm2cpp-learn
Enter fullscreen mode Exit fullscreen mode

Usage

Swap the import and keep the rest of your code:

from scm2cpp_learn import LBFGSLogisticRegression as LogisticRegression

clf = LogisticRegression(C=1.0).fit(X, y)   # same iterations and coefficients as scikit-learn
Enter fullscreen mode Exit fullscreen mode

Benchmarks

Bar chart of speed-ups over scikit-learn, from 1.8x (NMF) to 1,047x (RFE)

Speed-up over scikit-learn 1.9 on one core (i9-10900X); GPU rows on an RTX 4090. Full conditions for everymachinelearning row are in the README.

Why is it faster?

The algorithms are unchanged — the computation is reorganized:

  • Lasso cross-validation: the Gram matrix (X'X) is computed once, and each fold's matrix is obtained by subtraction, so the original data is never scanned again.
  • RFE: instead of refitting the model every round, a Cholesky factor is downdated as each feature is removed.
  • GPU: many penalties are solved in parallel, one CUDA block each.

The twist

The algorithms are written in Scheme and translated into readable C++ by my compiler, scm2cpp, which derives these optimizations automatically from the naive version.

Limitations

Not everything is faster: PCA with many features, for example, can be slower, and the kernels are single-threaded, so Intel's sklearnex with many cores wins on some estimators. All of this is documented in the README, including a comparison with sklearnex.

Links

GitHub logo niitsuma / scm2cpp

Scheme to readable C++. scikit-learn's algorithms written plainly in Scheme come out fast: the translator derives Gram and loop-invariant products, hands them to BLAS/cuBLAS and turns loops into CUDA kernels. pip install scm2cpp-learn (wheels for Linux, macOS, Windows).

Scm2Cpp

tests Open In Colab

Japanese (README.ja.md) · 中文 (README.zh.md)

Scm2Cpp translates a subset of Scheme into C++ that a person can read and edit, and scikit-learn's machine-learning algorithms, written in it plainly, come out fast. The translator derives the fast form of an algorithm from the naive one (the Gram matrix hoisted out of a coordinate descent, a loop-invariant product formed once), hands the products to BLAS or cuBLAS, and turns parallel loops into CUDA kernels The estimators ship on PyPI as scm2cpp-learn (scm2cpp-lasso up to 0.17.0; that name still installs it), with compiled wheels for Linux macOS and Windows (no compiler, no Racket).

Machine-learning algorithms it speeds up

Each is scikit-learn's own algorithm written in the subset, checked against scikit-learn's iterations and results. Times against scikit-learn on one core (i9-10900X, an RTX 4090 for the device rows, scikit-learn 1.9); the conditions are in the section named on the right.

algorithm, n
…
View on GitHub

Feedback is very welcome!

Top comments (0)