DEV Community

kirandeepjassal-crypto
kirandeepjassal-crypto

Posted on Originally published at prepstack.co.in

Core ML Concepts for Developers: Datasets, Cleaning, Leak-Free Splits, sklearn Pipelines & Evaluation (LightGBM)

Most "ML basics" articles teach you to call model.fit(X, y) and pretend that's the job. The actual job — where 80% of your time goes and 100% of the mistakes happen — is everything around fit: figuring out what your dataset really is, fixing what's broken, splitting it without leaking, picking the right metric, and noticing when your "great" model is just memorising garbage. This is the missing manual. (Full version on the original — link at the end.)

TL;DR

  • Data > model. A mediocre model on clean, leak-free, well-split data beats a SOTA model on bad data every time.
  • The pipeline: raw data → clean → split → preprocess (fit on train, transform everywhere) → train → evaluate → iterate. Every step has a "wrong way" that silently inflates your score.
  • Pick the right metric. Accuracy is wrong for imbalanced classification. RMSE is wrong for skewed regression. ROC-AUC is wrong for ranking.
  • Use sklearn.pipeline.Pipeline + ColumnTransformer. They prevent leakage by design. Manual preprocessing is how production ML breaks.
  • Validate the way you'll deploy. Time-series prod = time-series CV. Tenant-scoped prod = group-aware CV. Random CV on time-series gives a fantasy score.

1. What ML actually is

ML is a function-approximation game: you have inputs X, outputs y, and a dataset of (X, y) pairs you trust to be representative. You search for f such that f(X) ≈ y and generalises to new data you haven't seen. That "generalises" is the whole reason ML is hard — memorising the training set is easy; predicting next month's data is not. Three things must be true: the signal exists (y depends on X in a learnable way), the data is representative (train looks like prod), and you can measure success honestly.

2. Recognise your dataset type

If your data is... Treat as Reach for
CSV / SQL table Tabular sklearn, LightGBM/XGBoost/CatBoost (trees beat DL here)
Natural-language strings Text transformers, sentence-transformers, TF-IDF + linear
JPEGs / PNGs Image PyTorch + torchvision, pretrained ResNet/ViT (transfer learning)
WAV / MP3 Audio librosa, Whisper, Wav2Vec2
Indexed by timestamp, order matters Time series Prophet, statsmodels, XGBoost on lag features
Relationships matter as much as nodes Graph PyTorch Geometric, DGL

Most real projects are mixed — tabular features + a text column + a timestamp. Identify the dominant structure and build around it.

3. The pipeline (five steps, in order)

Raw data -> Clean -> Split -> Preprocess -> Train -> Evaluate
                                 ^                       |
                                 +--- fit on train ------+
                                     transform val/test
Enter fullscreen mode Exit fullscreen mode

The most important arrow is the feedback loop: preprocessing is fit on training data and applied to validation/test, never the other way round. Get that wrong and your score is a fantasy.

4. Cleaning — the unglamorous 60%

Handle missing values honestly (drop rows, drop columns, or impute using training-set stats only — and add a was_missing indicator, because missingness is often informative). Drop duplicates (df.sort_values('updated_at').drop_duplicates(subset=['id'], keep='last')). Treat outliers carefully — they can be the actual signal (fraud, rare events), so don't blindly cap. Add cheap sanity asserts (assert (df['age'] >= 0).all()). And check class balance now — if positives are <10%, accuracy and naive resampling will mislead you.

5. Splitting — where leakage starts

You need three sets: train (fit params), validation (tune / pick the model), test (touch once, at the very end). Pick the split that matches your data:

# Random (IID) — stratify to keep class ratios
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

# Time-aware — a random split here lets the model peek at the future
cutoff = df['date'].quantile(0.80)
train, test = df[df['date'] < cutoff], df[df['date'] >= cutoff]

# Group-aware — every user ends up entirely in train OR test
from sklearn.model_selection import GroupShuffleSplit
train_idx, test_idx = next(GroupShuffleSplit(test_size=0.2, random_state=42).split(X, y, groups=df['user_id']))
Enter fullscreen mode Exit fullscreen mode

The four leakage failure modes, each of which inflates your score and tanks production: (1) info from val/test leaking into preprocessing (scaling on the full dataset before splitting); (2) time leakage (training on data from after the prediction time); (3) group leakage (same user in both splits); (4) target leakage (a feature only available after the target is known).

6. Preprocessing — fit on train, transform everywhere

Scale numerics (StandardScaler / RobustScaler when outliers are present), encode categoricals (OneHotEncoder for low cardinality, TargetEncoder for high), vectorise text (TF-IDF or sentence embeddings), and engineer features (date parts, ratios like debt_to_income, aggregations, lag features — domain knowledge wins here). The key: never scaler.fit(X_full) before splitting. Wrap it all in a Pipeline + ColumnTransformer:

preprocessor = ColumnTransformer([
    ('num', Pipeline([('imputer', SimpleImputer(strategy='median')),
                      ('scaler',  StandardScaler())]), num_cols),
    ('cat', Pipeline([('imputer', SimpleImputer(strategy='constant', fill_value='UNKNOWN')),
                      ('ohe',     OneHotEncoder(handle_unknown='ignore', sparse_output=False))]), cat_cols),
])
model = Pipeline([('preprocess', preprocessor),
                  ('clf', LogisticRegression(max_iter=1000, class_weight='balanced'))])
model.fit(X_train, y_train)
Enter fullscreen mode Exit fullscreen mode

The pipeline object is the model — joblib.dump it, reload it for serving, and you cannot accidentally apply different preprocessing in production.

7. Choosing a model — start simple, always

Task Start with If you need more
Tabular classification/regression Logistic/Linear baseline LightGBM / XGBoost / CatBoost
Image classification Pretrained ResNet-50 (transfer) Vision Transformer
Text classification TF-IDF + Logistic Pretrained BERT/DistilBERT
Time-series forecasting Naive baseline, ARIMA LightGBM w/ lag features, Prophet

Always ship the baseline first. A 5-line logistic regression at 0.78 AUC tells you the problem is learnable. A 200-line transformer at 0.79 tells you you wasted a week. And watch bias vs variance: big train/val gap = variance (regularise, more data, simpler model); bad on both = bias (bigger model, better features, less regularisation).

8. Evaluation — the part that decides everything

Pick the metric the task requires, not the one your library prints by default.

The accuracy trap: if 97% of your data is "not default" and the model predicts "not default" for everyone, accuracy is 97% — useless model, great-looking number. For imbalanced classification, lead with PR-AUC and recall. Use precision when false positives are costly, recall when false negatives are, ROC-AUC when you'll pick the threshold later, log loss / calibration when probabilities matter. Always look at the confusion matrix.

y_proba = model.predict_proba(X_val)[:, 1]
print('ROC AUC :', roc_auc_score(y_val, y_proba))
print('PR  AUC :', average_precision_score(y_val, y_proba))   # lead with this when positives are rare
print(classification_report(y_val, model.predict(X_val)))
Enter fullscreen mode Exit fullscreen mode

For regression: MAE (robust), RMSE (penalises big errors), R² — and np.log1p(y) before training for skewed targets. Cross-validate the way you'll deploy: StratifiedKFold (classification), GroupKFold (grouped), TimeSeriesSplit (train always before validate). Tune with grid (small), random (medium), or Optuna (large/expensive).

9. The full end-to-end example

Loan-default prediction — tabular, imbalanced, mixed numeric + categorical, time-aware split, LightGBM in a leak-proof Pipeline:

# Time-aware split by application_date quantiles
train = df[df['application_date'] <  df['application_date'].quantile(0.70)]
val   = df[(df['application_date'] >= q70) & (df['application_date'] < q85)]
test  = df[df['application_date'] >= q85]

clf = lgb.LGBMClassifier(n_estimators=500, learning_rate=0.05, num_leaves=31,
                         is_unbalance=True, random_state=42)
# ... preprocessor.fit(X_train); transform val/test; early stopping on val AUC ...
# Evaluate with ROC-AUC + PR-AUC + classification_report + confusion_matrix.
# Touch the TEST set exactly once, at the very end.
Enter fullscreen mode Exit fullscreen mode

Every concept is in there: time-aware split, imputation inside the Pipeline (no leakage), class-weighted training (imbalance), early stopping (free regularisation), PR-AUC alongside ROC-AUC, test set touched once.

10. From notebook to production

The model artifact IS the pipeline (joblib.dump(model, 'model.pkl')) — preprocessing travels with it, so no train/prod skew. Monitor feature drift (inputs shift) and concept drift (the X→y relationship changes). Retrain on a cadence that matches the data (quarterly for stable sensors, daily/weekly for consumer behaviour, continuous for fraud). And A/B test new models — route 5% of traffic to v2 and measure business metrics, because a val score is necessary but not sufficient.

The mental model

Machine learning is engineering with statistics inside it. The model is the last 10% of the work; everything before it — understanding your data, splitting it honestly, engineering features, picking the metric that matches the business — is what separates a model that ships from a notebook that doesn't. Three habits: ship a baseline first; suspect leakage by default (every great-looking score has leakage until proven otherwise); build the pipeline once, retrain forever.

Go build the pipeline.


This is a condensed version. The full guide — with every section in depth, the dataset-type code, all the cleaning/splitting/preprocessing snippets, the complete metrics treatment, calibration, and the full runnable LightGBM loan-default pipeline — is on PrepStack:

→ Core AI/ML Concepts — Datasets, Preprocessing, and the Full Pipeline from Data to Evaluation

Top comments (0)