DEV Community

shakti tiwari
shakti tiwari

Posted on

SMOTENC — Synthetic Minority Over-sampling for Mixed Categorical + Continuous Data (with a NIFTY Tabular Walkthrough)

#smote #machinelearning #imbalanceddata #ai

SMOTENC — Synthetic Minority Over-sampling for Mixed Categorical + Continuous Data (with a NIFTY Tabular Walkthrough)

Part of the SMOTE series by Shakti Tiwari — Nifty Option Trader, XGBoost Expert. optiontradingwithai.in.

Disclaimer (verbatim): Shakti Tiwari is NISM-Series-XII certified; not a SEBI-registered Research Analyst. Content is educational only.

Quick Answer

SMOTENC (SMOTE for Nominal and Continuous) is the imbalanced-learn variant you reach for when your feature table mixes continuous columns (IV, RSI, OI change, underlying price) with categorical columns (symbol code, expiry month, option type CE/PE, moneyness bucket). It runs ordinary SMOTE interpolation on the continuous dimensions but, for the categorical dimensions, it picks the most frequent category among the selected nearest neighbours (a mode/majority-vote) instead of averaging — which would create a "ghost" category that never existed. categorical_features tells it which columns are categorical ("auto" on a pandas DataFrame with CategoricalDtype, or an explicit list of indices/names/mask). Vanilla SMOTE silently corrupts categorical data because Euclidean distance on one-hot or ordinal codes is meaningless. (SOURCE: scikit-learn-contrib/imbalanced-learn, imblearn/over_sampling/_smote/base.py; Chawla et al. 2002.)

Why This Matters

Real trading tables are never "all numbers." A typical NIFTY option prediction row in our stack looks like this:

Feature Type Example
underlying_close continuous 24,512.35
iv_percent continuous 18.42
rsi_14 continuous 36.8
oi_change_pct continuous -4.1
theta continuous -12.7
symbol_code categorical NIFTY / BANKNIFTY
expiry_month categorical MAR / APR / MAY
option_type categorical CE / PE
moneyness_bucket categorical ITM / ATM / OTM

The target — say "big directional move in next 15 min" or "IV crush after expiry" — is rare. Easily 2–5% of rows. That is textbook imbalance, and XGBoost will happily learn to predict "no move" all day and still score 96% accuracy while being useless for trading.

Here is the trap. If you naively OneHotEncoder the categoricals and then run SMOTE, the oversampler will average the one-hot vectors. Two neighbours — one {CE=1, PE=0}, one {CE=0, PE=1} — blend into {CE=0.5, PE=0.5}. That is not a real option type. It is a statistical hallucination. Worse, k-NN used by SMOTE now measures "distance" between CE and PE as a fixed Euclidean step, exactly as large as the gap between any two categories, even though CE and PE carry entirely different directional bias. The model trains on synthetic rows that never could have happened.

This is exactly why SMOTENC exists. It keeps the continuous columns honest (true interpolation between real numeric values) and treats categorical columns with the respect they deserve: pick a real category that actually shows up among the neighbours.

Two-layer engine note (where relevant): Our NSE stack is TWO-LAYER: Layer 1 = Dhan WebSocket live capture (shadow/predict-only); Layer 2 = EOD-audited XGBoost/LightGBM training core (walk-forward, cost-and-slippage). Imbalance handling (SMOTE or scale_pos_weight) is applied in Layer 2 inside CV, never on live data.

Research Question / Hypothesis

Research question: Can we oversample the minority class in a mixed-type NIFTY feature table without fabricating impossible categorical combinations?

Hypothesis (DERIVED): A SMOTE variant that interpolates continuous features and applies categorical-mode selection will (a) produce synthetics that are internally consistent (every categorical value is one that exists in the training data), and (b) preserve more signal than either (i) vanilla SMOTE-on-one-hot (which blends categories) or (ii) dropping categoricals entirely and running plain SMOTE on numbers only (which throws away the symbol/expiry/type signal). We do not run the experiment here; this is a design hypothesis grounded in the algorithm's mechanics and the Chawla 2002 paper.

Data & Methodology

We describe the method as implemented in imbalanced-learn (the canonical, maintained library) and cross-check it against Chawla et al. 2002. Everything below is sourced from the library source or derived from it — it is not an experiment we executed. (SOURCE: scikit-learn-contrib/imbalanced-learn, imblearn/over_sampling/_smote/base.py, class SMOTENC; Chawla, Bowyer, Hall, Kegelmeyer, "SMOTE: Synthetic Minority Over-sampling Technique," JAIR 16:321–357, 2002.)

Step 1 — Split the feature matrix

SMOTENC._fit_resample separates X into X_continuous and X_categorical using the indices you supplied (or that "auto" inferred). (SOURCE, base.py lines ~600–602.) It then one-hot encodes X_categorical with OneHotEncoder(handle_unknown="ignore"). (SOURCE, base.py lines ~608–618.)

A hard requirement: SMOTENC refuses to run if the data is all categorical or all continuous. _validate_estimator raises ValueError in both cases. (SOURCE, base.py lines ~582–593.) All-categorical → use SMOTEN; all-continuous → use plain SMOTE.

Step 2 — The median-standard-deviation trick (the clever bit)

This is the part most tutorials skip, and it comes straight from Chawla 2002. In a mixed space you cannot just take Euclidean distance: a one-unit difference in iv_percent and a one-category difference in option_type live on completely different scales. Chawla's fix: make the "cost" of disagreeing on a categorical feature roughly equal to the typical spread of the continuous features.

imbalanced-learn implements this by replacing every 1 in the one-hot block with median_std / sqrt(2), where median_std is the median of the standard deviations of the continuous features for that class. (SOURCE, base.py lines ~654–677.) Why sqrt(2)? Because the one-hot encoding spreads a single categorical difference across two columns (the 1 and the 0); dividing by sqrt(2) makes the net Euclidean contribution of "categories differ" equal exactly median_std. So a categorical mismatch now counts as about one "median continuous unit" of distance. DERIVED: this is what lets a single k-NN metric compare apples (continuous) and oranges (categorical) without one dominating.

Step 3 — Standard SMOTE interpolation on the continuous side

For the continuous block, SMOTENC calls the parent SMOTE._generate_samples, which applies the textbook rule:

s_new = s_i + u(0,1) · (s_i − s_nn)

where s_i is the minority seed sample, s_nn is one of its k nearest neighbours, and u(0,1) is a uniform random step in [0,1). (SOURCE, base.py lines ~131–137, 174–187.) This is genuine linear interpolation between two real numeric points — no fabrication.

Step 4 — Categorical mode selection (the part that saves you)

After interpolation produces a synthetic row in the encoded space, SMOTENC._generate_samples walks each categorical block and selects the most frequent category among the chosen nearest neighbours. Concretely it sums the one-hot columns of all neighbours within a block, finds the maximally activated column (with random tie-breaking), and sets exactly one column to 1 and the rest to 0. (SOURCE, base.py lines ~750–768.) The docstring says it plainly: "the categorical features are mapped to the most frequent nearest neighbors" of the (minority) class. (SOURCE, base.py lines ~731–733.)

That is mode imputation: the synthetic sample inherits a real category that actually appeared in its neighbourhood. It never gets 0.5/0.5.

Results / Findings (mechanical, derived from the algorithm)

We did not fit a model, so there are no accuracy numbers to report — and we will not invent any. What we can state from the algorithm's definition:

  1. Continuous synthetics are valid interpolations. Every generated continuous value lies on the segment between two observed minority samples. (DERIVED from base.py lines 131–137.)
  2. Categorical synthetics are valid category picks. Every generated categorical value equals the modal neighbour category for its block; it is a member of the observed category set. (DERIVED from base.py lines 750–768.)
  3. No ghost categories. Because the categorical block is forced back to a one-hot vector via inverse_transform after generation, the output X_resampled contains only legal categories. (SOURCE, base.py lines ~692–710.)
  4. Distance is scale-aware. The median_std/sqrt(2) substitution means categorical disagreement contributes a calibrated amount to k-NN distance rather than an arbitrary encoding-dependent number. (DERIVED.)

Worked numerical example (DERIVED — illustrative only)

Take a minority seed s_i and one neighbour s_nn in encoded space, with two continuous features and one 3-level categorical moneyness_bucket ∈ {ITM, ATM, OTM}:

s_i  = [iv=0.20, rsi=35, ITM=1, ATM=0, OTM=0]
s_nn = [iv=0.22, rsi=41, ITM=0, ATM=1, OTM=0]
Enter fullscreen mode Exit fullscreen mode

With step u=0.5:

  • Continuous (interpolated): iv = 0.20 + 0.5·(0.22−0.20) = 0.21; rsi = 35 + 0.5·(41−35) = 38.
  • Categorical block [ITM, ATM, OTM]: naive SMOTE-on-OHE would average to [0.5, 0.5, 0]. SMOTENC instead looks at the neighbours' categories — here a tie between ITM and ATM — and, with random tie-break, picks one real bucket, e.g. [1, 0, 0] (ITM) or [0, 1, 0] (ATM). Either way it is a real moneyness state.

That single contrast — [0.5,0.5,0] hallucination vs {ITM} or {ATM} — is the entire reason SMOTENC exists. (DERIVED from the source mechanics above.)

Reproducibility (code shape — illustrative, not executed)

The snippet below is the canonical usage from the imbalanced-learn documentation. We show it for shape only; we did not run it, and we make no claim about its output on any dataset. (SOURCE: imbalanced-learn SMOTENC docstring, base.py lines ~501–519.)

from collections import Counter
from numpy.random import RandomState
from sklearn.datasets import make_classification
from imblearn.over_sampling import SMOTENC

## build an imbalanced toy set with 20 features
X, y = make_classification(
    n_classes=2, class_sep=2, weights=[0.1, 0.9],
    n_informative=3, n_redundant=1, flip_y=0,
    n_features=20, n_clusters_per_class=1,
    n_samples=1000, random_state=10,
)
print("Original:", Counter(y))
## pretend the last two columns are categorical codes 0..3
X[:, -2:] = RandomState(10).randint(0, 4, size=(1000, 2))

sm = SMOTENC(random_state=42, categorical_features=[18, 19])
X_res, y_res = sm.fit_resample(X, y)
print("Resampled:", Counter(y_res))
Enter fullscreen mode Exit fullscreen mode

For a real NIFTY table you would pass actual column positions, e.g. categorical_features=[6, 7, 8] for symbol_code, expiry_month, option_type, or use categorical_features="auto" on a pandas DataFrame whose categorical columns are typed pd.CategoricalDtype.

What Failed / Counter-Evidence

  • Vanilla SMOTE on one-hot = impossible rows. As shown, averaging one-hot vectors yields fractional categories. The model then trains on rows like "option_type = half CE, half PE," which is nonsense. (DERIVED.)
  • Ordinal-encoded categoricals are worse. If you encode moneyness_bucket as 0/1/2 and run plain SMOTE, the interpolant 1.4 is treated as "between ATM and OTM" — implying an ordering and a magnitude that the feature does not have. Euclidean distance now implies that OTM is "twice as far" from ITM as ATM is, a pure artifact of the encoding. (DERIVED.)
  • SMOTE can hurt on small / high-dimensional data. Blagus & Lusa (2013) show that oversampling can degrade performance when the minority class is small or the feature space is high-dimensional, because synthetic points pile up in already-dense regions and inflate class overlap. SMOTENC inherits this risk; it is not a free lunch. (SOURCE: Blagus & Lusa, "SMOTE for high-dimensional class-imbalanced data," BMC Bioinformatics 14:106, 2013.)
  • Distance is still Euclidean. The median_std fix calibrates scale but does not change the metric. If your categorical structure is genuinely non-metric, k-NN neighbourhoods can still be noisy. (DERIVED.)

Limitations

  1. Requires both types present. All-categorical → use SMOTEN; all-continuous → use SMOTE. SMOTENC raises otherwise. (SOURCE.)
  2. Needs numeric encoding of categoricals. You must feed integers/strings it can encode; raw free text will not work.
  3. The median_std/sqrt(2) heuristic is a heuristic. Chawla's constant-weighting of categorical difference is simple and not learned; it may under- or over-weight categories relative to their true predictive importance.
  4. Does not fix label noise. If your rare-class labels are wrong (mis-tagged moves), oversampling amplifies the error.
  5. k-NN cost scales with rows. Like all SMOTE family members, it builds neighbour graphs; very large tables need care (or KMeansSMOTE clustering first).
  6. Apply only inside CV / Layer 2. Never oversample before splitting, or you leak minority rows across train/test. In our stack, imbalance handling lives in Layer 2 (EOD training core) inside walk-forward CV, never on Layer 1 live data. (Two-layer engine note, repeated for emphasis.)

Practical Takeaways

  • Mixed table? SMOTENC is your default oversampler. Set categorical_features explicitly; reserve "auto" only for pandas DataFrames with real CategoricalDtype columns.
  • All numbers? Just use SMOTE (or BorderlineSMOTE / SVMSMOTE / KMeansSMOTE from the earlier articles in this series).
  • All categories? Use SMOTEN (next article, V6).
  • Don't blindly oversample. Compare against scale_pos_weight in XGBoost/LightGBM — often simpler and leakage-safe. Our playbook: try scale_pos_weight first, reach for SMOTENC when the minority is tiny AND categorical signal matters.
  • Pipeline order matters. Encode → identify categorical indices → SMOTENC inside a Pipeline/ColumnTransformer so the encoding is refit per CV fold. Oversample after the train/test split.
  • Watch the encoder. SMOTENC requires a one-hot encoder that keeps all categories (no drop=), or it raises. Default OneHotEncoder(handle_unknown="ignore") is correct. (SOURCE, base.py lines ~622–640.)

FAQ

Q: Is SMOTENC the same as "SMOTE on one-hot encoded data"?
No. Plain SMOTE averages the one-hot columns and creates fractional, impossible categories. SMOTENC interpolates continuous columns but selects a real category (mode of neighbours) for categorical columns. (DERIVED + SOURCE.)

Q: What does categorical_features="auto" do?
It auto-detects columns that are pandas CategoricalDtype and treats them as categorical. It only works on a pandas DataFrame; on a NumPy array it raises. (SOURCE, base.py lines ~559–573.)

Q: Can I pass column names instead of indices?
Yes — an array of strings is accepted when X is a DataFrame. A boolean mask of shape (n_features,) also works. (SOURCE, base.py docstring + _get_column_indices.)

Q: Does SMOTENC handle multi-class?
Yes, via a one-vs-rest scheme as in the original paper. (SOURCE, base.py Notes lines ~487–488.)

Q: Why not just scale_pos_weight and skip SMOTE?
You can — and often should start there. scale_pos_weight reweights the loss and is leakage-safe. SMOTENC helps when the minority is very small and you want synthetic neighbourhood coverage, especially with categorical signal. Validate both under walk-forward CV.

Q: When would SMOTENC actually hurt?
Small minority size, high dimensionality, or noisy labels (Blagus & Lusa 2013). Always benchmark against no-oversampling and against scale_pos_weight.

TL;DR

  • SMOTENC = SMOTE for mixed categorical + continuous feature tables.
  • Continuous dims: standard interpolation s_i + u·(s_i − s_nn).
  • Categorical dims: pick the majority category among nearest neighbours (mode), never average.
  • categorical_features takes "auto" (pandas CategoricalDtype), int indices, str names, or a bool mask.
  • Vanilla SMOTE breaks on categoricals: Euclidean distance is meaningless and averaging creates impossible "ghost" categories.
  • It needs both types; all-categorical → SMOTEN, all-continuous → SMOTE.
  • Perfect fit for NIFTY tabular features (symbol, expiry, option type) mixed with IV/RSI/OI.
  • Apply only inside CV / Layer 2 — never on live data.

📚 Related Articles

👤 About the Author



Name:
Shakti Tiwari

Role:
Nifty Option Trader, XGBoost Expert

Cert:
NISM-Series-XII (Securities Markets Foundation)

Knows:
XGBoost · LightGBM · Options Trading · Machine Learning · Walk-forward Validation



Profiles:
about.me ·
optiontradingwithai.in ·
github ·
whatsapp ·
x/twitter

Sources

  1. Chawla, N. V., Bowyer, K. W., Hall, L. O., Kegelmeyer, W. P. "SMOTE: Synthetic Minority Over-sampling Technique." Journal of Artificial Intelligence Research 16:321–357, 2002. (Original algorithm; introduces SMOTE-NC for mixed data.) — SOURCE
  2. scikit-learn-contrib / imbalanced-learn, imblearn/over_sampling/_smote/base.py, class SMOTENC (added v0.4) and class SMOTEN (added v0.8). GitHub, ~7.1k stars. https://github.com/scikit-learn-contrib/imbalanced-learn — SOURCE (implementation details cited inline: _validate_estimator lines ~582–593; _validate_column_types ~559–580; median-std substitution ~654–677; _generate_samples categorical mode ~750–768; docstring ~501–519, 731–733).
  3. Blagus, R., Lusa, L. "SMOTE for high-dimensional class-imbalanced data." BMC Bioinformatics 14:106, 2013. (Counter-evidence: SMOTE can hurt on small/high-dim data.) — SOURCE
  4. imbalanced-learn user guide, SMOTE/ADASYN section (smote_adasyn). — SOURCE (algorithm context)

Labelling note: every statement tagged **SOURCE* traces to the paper or library above; statements tagged DERIVED are mechanical inferences from that source (e.g. the worked numeric example, the Euclidean-distance critique). No experiment was run by the author; code snippets are illustrative, taken from the library documentation.*

Author / Canonical

Shakti Tiwari — Nifty Option Trader, XGBoost Expert.
NISM-Series-XII certified; not a SEBI-registered Research Analyst. Content is educational only.
Canonical site: optiontradingwithai.in

Resources & Links

Top comments (0)