DEV Community: Murat Genç

You've Seen the Pipeline. Now Meet the Matrix: The One `Vec` Behind the 400 Shrink

Murat Genç — Sun, 02 Aug 2026 06:18:39 +0000

How a single contiguous allocation — and a type system that won't let you feed strings to a scaler — is the real reason datarust fits in 2.3 megabytes.

In the last post I showed you the whole datarust workflow: impute, scale, one-hot, train a logistic regression, evaluate, and save it as JSON — all without a Python runtime in sight. The Docker image shrank from ~900 MB to ~8 MB, and the binary was 2.3 MB.

But I skimmed over something important. I kept saying "the flat memory layout" as if it were a detail. It isn't. It's the whole bet.

Every scaler, every encoder, every model, every metric in datarust runs on top of one data structure. If you understand that structure — why it looks the way it does and what it refuses to let you do — the rest of the library stops being magic.

So let's zoom in. Meet Matrix.

Two containers, on purpose

Real data is mixed. Numbers in one column, strings in the next. In Python, everything flows through one giant numpy.ndarray or a pandas.DataFrame, and the type system just... shrugs. A string column next to a float column gets coerced into object dtype. You'll find out at training time, in the form of an error message three frames deep.

datarust does the opposite. It splits your data into two types at the source:

use datarust::Matrix;
use datarust::matrix::StrMatrix;

let numeric = Matrix::new(vec![
    vec![3.0,  85.0, 24.0],
    vec![12.0, 70.0, 31.0],
    vec![f64::NAN, 95.0, 45.0],
])?;

let categorical = StrMatrix::from_strings(vec![
    vec!["MonthToMonth"],
    vec!["OneYear"],
    vec!["MonthToMonth"],
])?;

Matrix is f64 only. StrMatrix is strings only. They are different types, and the compiler will refuse to compile a program that hands a string column to a scaler. Not at runtime — at compile time. In the last post I called this "putting on glasses for the first time." Let me show you what it actually buys you.

The ColumnTransformer API is built on that split:

ct.add_numeric("scaled", vec![0, 1],
    TransformerKind::StandardScaler(StandardScaler::new()));          // OK
ct.add_categorical("encoded", vec![0],
    CategoricalTransformerKind::OneHotEncoder(OneHotEncoder::new())); // OK
// ct.add_numeric("nope", vec![0],
//     CategoricalTransformerKind::OneHotEncoder(...));               // does not compile

A OneHotEncoder expects strings. The type system makes that a compile error on a numeric column. sklearn can't do this — every column name is just a string, and OneHotEncoder will happily run on floats if you don't read the docs carefully.

Construction is a lie detector

Here's the thing that surprised me most when I actually read the source: Matrix::new doesn't just store your data. It validates it.

let bad = Matrix::new(vec![
    vec![1.0, 2.0, 3.0],
    vec![4.0, 5.0],
]);

This returns an error:

ShapeMismatch { expected: "3 columns", actual: "2 columns at row 1" }

Not a panic. Not NaN silently appearing in column 3. Not a jagged array that flows downstream and poisons your model. A precise, recoverable error, at the moment of construction.

In pandas, a ragged column silently becomes object dtype. In numpy, you get a cryptic error deep in some array-conversion path. In datarust, the constructor is the bouncer, and it's the only place the check needs to happen. From then on, every function that takes a &Matrix can trust the shape without re-checking.

There's a matching from_flat for when you already have a contiguous buffer, and it validates the element count matches the declared shape:

let m = Matrix::from_flat(2, 3, vec![1.0, 2.0, 3.0, 4.0, 5.0, 6.0])?;
assert_eq!(m.nrows(), 2);
assert_eq!(m.ncols(), 3);
assert_eq!(m.get(1, 2), 6.0);

Even the allocation is checked: rows.checked_mul(cols) — so you can't ask for a matrix so large that rows * cols overflows usize and wraps around into a buffer too small. I know that sounds paranoid. It's also the difference between a library that panics in production and one that returns Err.

The one allocation

Now for the part I waved my hands at last time. Here's the internal layout:

pub struct Matrix {
    data: Vec<f64>,   // one contiguous buffer, row-major
    rows: usize,
    cols: usize,
}

That's it. Not Vec<Vec<f64>> — no allocation per row, no pointers-to-pointers. Element (i, j) lives at data[i * cols + j]. Your 50,000×200 matrix is one contiguous Vec<f64> of ten million floats, in one heap allocation.

Why does that matter? Three reasons:

Cache locality. When a scaler walks a row, every element is adjacent in memory. With Vec<Vec<f64>>, each row is a separate allocation, and walking rows means jumping between unrelated memory pages.
Auto-vectorization. as_slice() hands the compiler a plain &[f64]. Modern CPUs get to use SIMD, and LLVM gets to prove the loop has no aliasing.
The Python boundary doesn't exist. This is the quiet killer. In a numpy pipeline, every scaler.transform() crosses from Python into C and back, materializing Python float objects for the results. In datarust, transform is a Rust function operating on a Rust buffer. The whole pipeline never leaves native code.

This wasn't always the case. Pre-0.3, Matrix was Vec<Vec<f64>>. When we switched to the flat layout, the same workloads got dramatically faster with no algorithm changes:

Workload (50,000 × 200)	`Vec<Vec<f64>>` (v0.2)	flat (v0.3, default)
`StandardScaler`	115 ms	8.4 ms
`MinMaxScaler`	81 ms	12.2 ms
`RobustScaler`	459 ms	137 ms
Pipeline (3 scalers)	662 ms	152 ms
PCA	1056 ms	1008 ms
`OneHotEncoder`	88 ms	98 ms

RobustScaler got 3.4× faster, Pipeline 4.4×. Same math, same data — only the memory layout changed. Flat beats fancy.

The hot loops are designed around that layout. You almost never call get(i, j) in an inner loop; you grab a row slice and iterate:

for row in m.iter_rows() {      // yields &[f64], zero-copy
    for &v in row {
        // ...
    }
}

row(i) returns a &[f64] slice, and as_slice() exposes the whole buffer for the tightest loops. get exists for the times you need it — and it's bounds-checked in release mode, with a precise panic message. If you want to be defensive, checked_get returns Option<f64> instead.

NaN is a type problem, so we treat it like one

Python has a lovely tradition of letting NaN flow through a pipeline until it silently infects your coefficients. You train, your model returns nan, and you spend an afternoon bisecting which column did it.

datarust's answer is three validators, each with a distinct contract:

m.validate_no_nan()?;       // rejects NaN, allows ±inf — for data that must be complete
m.validate_no_infinite()?;  // rejects ±inf, allows NaN — for imputers
m.validate_finite()?;       // rejects both — for models and metrics

The subtle one is the middle: validate_no_infinite. Imputers like SimpleImputer and KnnImputer are allowed to see NaN — that's the missing-value marker they exist to fix. But an infinity is not a missing value, it's a broken number. So imputers call validate_no_infinite, not validate_no_nan. The distinction is baked into the API, and it means the error message tells you which kind of dirty data you're dealing with.

When the validation fails, the error tells you exactly where:

InvalidInput("NaN value at position (2, 0)")

Row 2, column 0. Not "somewhere." You go straight to the bad cell.

And this is exactly why the preprocessing order matters. The Matrix in the previous post had a missing tenure value in row 2 — a NaN that a scaler would refuse to touch. You can't skip the imputer and hope the scaler tolerates it:

use datarust::imputer::{ImputeStrategy, SimpleImputer};

let imputed = SimpleImputer::new(ImputeStrategy::Mean).fit_transform(&numeric)?;

let table = Table::new(imputed, categorical)?;
let mut ct = ColumnTransformer::new()
    .remainder(Remainder::Drop)
    .add_numeric("scaled", vec![0, 1],
        TransformerKind::StandardScaler(StandardScaler::new()))
    .add_categorical("encoded", vec![0],
        CategoricalTransformerKind::OneHotEncoder(
            OneHotEncoder::new().handle_unknown(HandleUnknown::Ignore),
        ));

let x = ct.fit_transform(&table)?;  // imputed → scaled → encoded, one step

Without the imputer, that last line fails loudly — invalid input: non-finite value NaN at position (2, 0) — instead of silently training a model on broken numbers. The validation isn't there to annoy you. It's there to force the pipeline to be honest about what it handles.

The sparse sibling

One-hot encoding a high-cardinality column produces a lot of zeros. Storing all of them as f64 is waste. So there's a third container: SparseMatrix, a Compressed Sparse Row matrix mirroring scipy.sparse.csr_matrix.

use datarust::matrix::SparseMatrix;

let sp = SparseMatrix::from_triplets(
    3, 4,
    &[(0, 0, 1.0), (1, 2, 3.0), (2, 3, 5.0), (0, 0, 2.0)],
)?;

Notice something: I passed (0, 0) twice. datarust sums duplicate coordinates (1.0 + 2.0 = 3.0), drops entries that sum to zero, and sorts each row by column index — so the invariants a reader expects are always true. And if you hand it a malformed CSR array, it rejects it with a specific message instead of indexing out of bounds later. The constructor validates indptr, column ranges, and per-row ordering. Bad data is rejected here, not at the model.

What it doesn't do (yet)

Honesty section, because the tone of these posts is "here's what I learned," not "buy my perfect library."

StrMatrix is not flat. It's still Vec<Vec<String>>, one heap allocation per string. It's fine for categorical columns (which are wide and short), but it's a deliberate asymmetry, and it shows: string-heavy workloads don't get the same cache wins as numeric ones.
No views. select_rows and select_columns copy. In numpy you'd get a view with fancy indexing; here, correctness and simplicity win over aliasing.
No f32. It's f64 everywhere. An f32 mode would halve memory for big datasets, but it's a sweeping refactor across every estimator. It's on the roadmap, not in the crate.
No NumPy interop. The serde feature preserves a nested-JSON wire format for Matrix, so fitted pipelines round-trip through save_json/load_json — but there's no .npy reader yet. That's explicitly listed as under consideration.

Why this is the core feature

Every number in every example in the first article passed through Matrix. The 179–620× speedups against sklearn's ColumnTransformer? Partly the flat buffer, partly never crossing the Python/C boundary. The type-safety that caught a OneHotEncoder on a numeric column before it could run? That's Matrix vs StrMatrix being different types. The JSON model that loads in a WASM module or an ARM embedded target? That's a buffer you can serialize and a shape you can trust.

In other words: the 400× Docker shrink isn't the story. The one Vec<f64> that makes it possible — and the constructor that refuses to let you put garbage in it — is the story.

If you want to see the data structure that everything else is built on, the full source is in crates/datarust/src/matrix.rs — 1,400 lines including tests, and the tests are where the invariants are documented. Or just run the pipeline from the last post and know that every number in it started as a flat buffer that checked its own shape.

datarust is MIT-licensed and available on crates.io, with docs at datarust.dev. The roadmap — including the f32 refactor and the .npy reader — lives at github.com/genc-murat/datarust. If you want to help make Matrix generic over float types, now's a good time.

I Replaced My sklearn Pipeline With Pure Rust. The Docker Image Shrank 400x

Murat Genç — Sat, 01 Aug 2026 14:37:07 +0000

How datarust gives you StandardScaler, OneHotEncoder, LogisticRegression, and JSON model persistence — without BLAS, Python, or the guilt of a 900MB Docker image.

There's a moment in every ML project where the notebook stops being fun and the deployment starts being painful.

You've got your Pipeline. You've got your metrics. You hand it to the platform team, and they ask: "Why does this need 900 megabytes?"

Because scikit-learn pulls in NumPy, SciPy, joblib, threadpoolctl, and a Fortran BLAS library. That's why.

I've been in that meeting. I've been the person who built the beautiful sklearn pipeline and then watched it drown in Docker layers. So I started wondering: what if the entire preprocessing + classical ML workflow existed as a pure Rust library — zero BLAS, zero Python runtime — that compiled to a single static binary?

That library is datarust. It just hit v0.6, and I want to show you the same workflow you'd write at work — mixed data types, preprocessing, training, evaluation, serialization — but without the Python tax.

The setup: two commands, zero configuration

cargo new churn_classifier && cd churn_classifier
cargo add datarust --features datasets,serde

That's it. No pip install. No virtual environment. No fighting with BLAS backends. The default build has zero external dependencies.

The data: mixed types, real problems

Real data is messy. You've got numbers (tenure, monthly_charge), categories (contract_type), missing values, and different scales. In datarust, data lives in two containers: Matrix for numbers and StrMatrix for strings.

They're deliberately separate types — the compiler won't let you accidentally feed a string column to a scaler.

use datarust::Matrix;
use datarust::matrix::StrMatrix;

// Numeric: tenure (months), monthly_charge ($), age
let numeric = Matrix::new(vec![
    vec![3.0,  85.0, 24.0],
    vec![12.0, 70.0, 31.0],
    vec![f64::NAN, 95.0, 45.0],   // missing tenure
    vec![48.0, 55.0, 52.0],
])?;

// Categorical: contract_type
let categorical = StrMatrix::from_strings(vec![
    vec!["MonthToMonth"],
    vec!["OneYear"],
    vec!["MonthToMonth"],
    vec!["TwoYear"],
])?;

// Target: 1 = churned, 0 = stayed
let y = vec![1.0, 0.0, 1.0, 0.0];

Already, something nice has happened: Matrix::new validates that all rows have the same length. No silent broadcasting bugs. If your data is ragged, you find out at construction, not three functions deep.

Preprocessing: the part that usually hurts

This is where most Rust ML libraries tap out. They give you a model — maybe a linear regression — but ask you to handle scaling, encoding, and imputation yourself. datarust takes the sklearn approach: these are first-class citizens.

I want to do four things:

Impute the missing tenure value (replace with the mean)
Scale the numeric columns to zero mean / unit variance
One-hot encode the contract type
Do all of the above in a single, composable step

Here's the ColumnTransformer:

use datarust::compose::{ColumnTransformer, Remainder, Table};
use datarust::encoder::{HandleUnknown, OneHotEncoder};
use datarust::imputer::{ImputeStrategy, SimpleImputer};
use datarust::scaler::StandardScaler;
use datarust::transformer_kind::TransformerKind;
use datarust::CategoricalTransformerKind;

let table = Table::new(numeric, categorical)?;

let mut ct = ColumnTransformer::new()
    .remainder(Remainder::Drop)
    .add_numeric(
        "tenure_imputed",
        vec![0],           // column 0 of numeric matrix
        TransformerKind::SimpleImputer(SimpleImputer::new(ImputeStrategy::Mean)),
    )
    .add_numeric(
        "features_scaled",
        vec![1, 2],        // columns 1, 2 of numeric matrix
        TransformerKind::StandardScaler(StandardScaler::new()),
    )
    .add_categorical(
        "contract_encoded",
        vec![0],           // column 0 of categorical matrix
        CategoricalTransformerKind::OneHotEncoder(
            OneHotEncoder::new().handle_unknown(HandleUnknown::Ignore),
        ),
    );

let x = ct.fit_transform(&table)?;

That's it. The output x is a single numeric Matrix — imputed, scaled, encoded — ready to feed into any model.

There's one design choice here that I genuinely love: the type system enforces correctness. add_numeric takes a TransformerKind, add_categorical takes a CategoricalTransformerKind. You can't accidentally put a OneHotEncoder (which expects strings) on a numeric column. The compiler catches it.

After years of sklearn's everything-is-a-string-column-name dynamism, this feels like putting on glasses for the first time.

Training the model

Now for the classifier. datarust v0.6 ships LogisticRegression with both binary and multiclass support:

use datarust::linear_model::{LogisticRegression, LogisticSolver};
use datarust::traits::Predictor;

let mut model = LogisticRegression::new()
    .with_solver(LogisticSolver::Svd)
    .with_max_iter(100);

model.fit(&x, &y)?;
let predictions = model.predict(&x)?;
let probabilities = model.predict_proba(&x)?;  // (n, 2) matrix

The model uses Newton-Raphson (IRLS for binary, full multinomial for multiclass) under the hood — the same algorithm sklearn uses, implemented in pure Rust with a Cholesky or SVD linear solver. No BLAS. No LAPACK.

And here's something I didn't expect to care about: it's fast. On benchmarks comparing against scikit-learn 1.6 on an M-series Mac, datarust's ColumnTransformer was 179–620× faster than sklearn's on the same data. Part of that is the flat memory layout (a single contiguous Vec<f64> instead of Python objects), and part is simply not crossing the Python/C boundary on every operation.

I'm not claiming datarust beats sklearn everywhere — sklearn's PCA (which calls LAPACK) is dramatically faster on large matrices. But for preprocessing and linear models, pure Rust holds its own.

Evaluation: don't stop at accuracy

A common mistake is to stop at accuracy. For churn — where maybe 20% of customers leave — accuracy is misleading. A model that predicts "nobody churns" is 80% accurate and completely useless.

use datarust::metrics::classification::*;

let acc = accuracy_score(&y, &predictions)?;
let prec = precision_score(&y, &predictions)?;
let rec = recall_score(&y, &predictions)?;
let f1 = f1_score(&y, &predictions)?;
let auc = roc_auc_score(&y, &probabilities.column(1))?;
let ll = log_loss(&y, &probabilities.column(1), 1e-15)?;

For this churn problem, I care most about recall — how many of the customers who will churn did I catch? A false positive (offering a discount to someone who wasn't leaving) costs a little; a false negative (losing a customer I could have saved) costs a lot.

And if this were a multiclass problem, the same metrics auto-detect the label count and switch to macro-averaging. No average='macro' parameter needed.

Cross-validate properly

Evaluating on training data is the oldest sin in machine learning. Let's do it properly:

use datarust::model_selection::StratifiedKFold;

let skf = StratifiedKFold::new()
    .with_n_splits(5)
    .with_shuffle(true)
    .with_random_state(42);

let mut fold_accuracies = Vec::new();
for (train_idx, test_idx) in skf.split(&y)? {
    let x_train = x.select_rows(&train_idx)?;
    let x_test = x.select_rows(&test_idx)?;
    let y_train: Vec<f64> = train_idx.iter().map(|&i| y[i]).collect();
    let y_test: Vec<f64> = test_idx.iter().map(|&i| y[i]).collect();

    let mut fold_model = LogisticRegression::new().with_solver(LogisticSolver::Svd);
    fold_model.fit(&x_train, &y_train)?;
    let fold_pred = fold_model.predict(&x_test)?;
    fold_accuracies.push(accuracy_score(&y_test, &fold_pred)?);
}

Stratified means each fold preserves the churn/stay ratio — critical for imbalanced data.

Save the model: JSON, not pickle

You trained the model. Now you need to deploy it. In Python, you'd joblib.dump a pickle file — a binary blob that only Python can read, tied to the exact library versions that created it.

datarust uses JSON:

use datarust::pipeline::Pipeline;

let mut pipe = Pipeline::new()
    .push("scaler", TransformerKind::StandardScaler(StandardScaler::new()))
    .with_estimator(LogisticRegression::new());

pipe.fit(&x, &y)?;

// Save to disk
datarust::serialize::save_json(&pipe, "churn_model.json")?;

// ... later, in the serving binary ...
let model: SupervisedPipeline<LogisticRegression> =
    datarust::serialize::load_json("churn_model.json")?;

// Already fitted — predict immediately
let predictions = model.predict(&new_data)?;

The saved model is human-readable JSON. You can cat it. You can diff it. You can load it in a Rust service, a CLI tool, or a WASM module — anything that can parse JSON and link the datarust crate.

Try that with a pickle file.

The deployment payoff

The entire workflow above — preprocessing, training, evaluation, cross-validation, serialization — runs in a single Rust binary with zero external dependencies by default.

$ cargo build --release
$ ls -lh target/release/churn_service
-rwxr-xr-x  2.3MB  churn_service

That 2.3-megabyte binary contains the entire ML pipeline.

You can:

Cross-compile it to any target Rust supports (Linux, macOS, Windows, WASM, ARM)
Run it in the browser via WASM — load the JSON model, call predict, return the result
Embed it in a microservice written in Go, Node, or Python via FFI
Ship it as a CLI tool that data scientists can run locally without installing Python

Compare that to a Python sklearn Docker image:

	datarust	Python + sklearn
Binary size	2.3 MB	~30 MB (venv)
Docker image	~8 MB	~900 MB
WASM support	native	pyodide (~30 MB)
External deps	0	NumPy, SciPy, joblib

What's not there yet

I'm not going to pretend this library does everything sklearn does. Here's what's missing:

No trees or ensembles. No RandomForest, no GradientBoosting. This is the biggest gap, planned for v0.7.
No SVM. No SVC with RBF kernels.
No deep learning. That's deliberate — candle and burn own that space in Rust.
PCA is slower than sklearn's on large matrices. sklearn calls LAPACK; datarust uses a pure-Rust Jacobi eigensolver.
No CSV reader built-in. You construct the Matrix yourself. (A csv feature is planned.)

If your workflow absolutely needs RandomForest or a transformer, datarust isn't there yet. But if you're doing logistic regression, linear models, clustering, or — especially — preprocessing pipelines that need to run outside Python, it's ready today.

The full feature set

Here's what v0.6 gives you, all with zero dependencies:

Preprocessing:
StandardScaler, MinMaxScaler, RobustScaler, MaxAbsScaler, Normalizer, Binarizer, KBinsDiscretizer, QuantileTransformer, PowerTransformer

Encoding:
OneHotEncoder, OrdinalEncoder, LabelEncoder, TargetEncoder, FrequencyEncoder

Imputation:
SimpleImputer (mean/median/most_frequent/constant), KnnImputer

Feature engineering:
PolynomialFeatures, VarianceThreshold, SelectKBest, PCA, TruncatedSVD

Models:
LinearRegression, Ridge, Lasso, LogisticRegression, KMeans

Metrics:
MSE, MAE, RMSE, R², accuracy, precision, recall, F1, ROC-AUC, PR-AUC, log loss, confusion matrix, Cohen's kappa, Matthews correlation, silhouette score

Utilities:
Pipeline, ColumnTransformer, KFold, StratifiedKFold, cross_val_score, train_test_split, JSON serialization

Get started

cargo add datarust --features datasets,serde

If you've ever wanted to run an sklearn-style pipeline without the Python tax, give it a try.

datarust is MIT-licensed and available on crates.io, with full documentation at genc-murat.github.io/datarust. The roadmap lives at github.com/genc-murat/datarust — if you want to contribute a DecisionTree or a CountVectorizer, now's a good time.