SMOTEN in imbalanced-learn: SMOTE for Purely Categorical Data via the Value Difference Metric

Part of the SMOTE Family series. Previous: V5 SMOTENC (mixed numeric + categorical). Next: V7 ADASYN (adaptive weighting).

Disclaimer: Shakti Tiwari is NISM-Series-XII certified; not a SEBI-registered Research Analyst. Content is educational only.

Quick Answer

SMOTEN (from imblearn.over_sampling import SMOTEN) is the variant of SMOTE built for datasets where every feature is categorical — there is no continuous axis along which to interpolate, so vanilla Euclidean SMOTE cannot run. Instead SMOTEN measures distance between rows using the Value Difference Metric (VDM), a distance defined on categorical levels from how differently those levels split the target class. It then synthesises a new minority row by taking, for each feature, the most frequent category among the seed point and its VDM-nearest minority neighbours — a majority vote, not a numeric average. [DERIVED summary of imbalanced-learn SMOTEN + Stanfill & Waltz 1986 + Cost & Pedrycz 2002]

---

Why This Matters

Agar aap sirf categorical features ke saath kaam kar rahe ho — market regime (trending / ranging / volatile), expiry week (near / far), option-type bias (call-dominated / put-dominated), open-interest buildup (long / short / neutral), session (opening / mid / closing) — toh vanilla SMOTE aapke liye kaam hi nahi karega. Plain SMOTE needs numbers. Feed it one-hot or label-encoded categoricals and it will happily emit 0.37 in a column that should only ever be 0 or 1, or 1.8 in a three-level regime feature. Those synthetic rows are physically impossible states of the world, and an XGBoost or LightGBM model will happily learn spurious splits on them. That is the trap.

This is exactly the situation in a lot of Indian derivative research: many of the cheapest and most interpretable signals are nominal. You do not need a continuous feature to know "expiry week is near and buildup is short and regime is volatile." SMOTEN lets you over-sample that minority event without fabricating non-existent numeric coordinates.

Our NSE stack is TWO-LAYER: Layer 1 = Dhan WebSocket live capture (shadow/predict-only); Layer 2 = EOD-audited XGBoost/LightGBM training core (walk-forward, cost-and-slippage). Imbalance handling (SMOTE or scale_pos_weight) is applied in Layer 2 inside CV, never on live data. So if a categorical-only model lives in Layer 2, SMOTEN is the resampler that belongs in its CV pipeline — never on the live feed, never on a test fold.

---

Research Question / Hypothesis

Research question: For an all-categorical, class-imbalanced dataset, how does SMOTEN define "nearest neighbour" when Euclidean distance is undefined, and how does it synthesise a new minority row without numeric interpolation?

Hypothesis (DERIVED): Because categorical levels have no natural ordering, the only defensible notion of "similar" is "tends to produce the same target class." So the distance between two levels of a feature should be a function of how different their class-conditional distributions are. The Value Difference Metric does exactly this. SMOTEN should (a) build a VDM distance matrix per categorical feature from the training class frequencies, (b) use it to find minority k-NN, and (c) synthesise by majority vote per feature. The hypothesis is a derived expectation from the cited sources, not a measurement from this article — and no code was executed here.

---

Data & Methodology

We describe the canonical SMOTEN usage shape and the algorithm as specified in the imbalanced-learn repository. No experiment is executed in this article. The snippet below is illustrative, reproduced from the imbalanced-learn API, and is shown only to anchor the method. Treat it as the API contract, not a result. [SOURCE: scikit-learn-contrib/imbalanced-learn, SMOTEN]

## Illustrative only — API shape from imbalanced-learn docs, NOT an experiment result.
import pandas as pd
from imblearn.over_sampling import SMOTEN
from sklearn.model_selection import train_test_split

## X is ALL categorical (object / category dtype columns)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, stratify=y, random_state=42
)

smote_n = SMOTEN(
    sampling_strategy='auto',   # balance the minority class(es) to majority count
    k_neighbors=5,              # nearest minority neighbours to vote on each feature
    random_state=42
)

X_res, y_res = smote_n.fit_resample(X_train, y_train)
## X_res is still all-categorical; no impossible decimals were invented.

Pipeline shape (consistent with every SMOTE variant in this cluster):

1. Stratified train/test split — SMOTEN must never see the test set.
2. Inside cross-validation only, fit_resample(X_train_fold, y_train_fold).
3. Train the estimator on the resampled fold; validate on the untouched fold.
4. Because SMOTEN's distance is computed from class-conditional probabilities, the VDM matrix is rebuilt per fold — it must not leak test information.

The three conceptual moves [SOURCE: imbalanced-learn SMOTEN; DERIVED]:

Move 1 — Estimate class-conditional probabilities per level. For each categorical feature f and each level a, compute P(c | a) = (rows with level a and class c) / (rows with level a), over the training data. These are the empirical conditional distributions that VDM leans on.

Move 2 — Build a VDM distance matrix per feature. The distance between two levels a and b of feature f is:

VDM_f(a, b) = [ Σ_c |P(c | a) − P(c | b)|^p ]^(1/p) [SOURCE: Stanfill & Waltz 1986; Cost & Pedrycz 2002; DERIVED form]

For a binary target this collapses (up to a constant factor) to 2·|P(minority | a) − P(minority | b)| when p = 1, or 2·|Δ|^2 when p = 2. The constant factor does not change neighbour ordering, so the qualitative behaviour is the same. Distance between two full rows is the sum of the per-feature VDM distances. [DERIVED from the binary-class simplification]

Move 3 — Majority-vote synthesis. For a seed minority row, find its k_neighbors nearest minority rows under the VDM distance. To make one synthetic row, for each feature independently pick the most frequent level among the seed and those neighbours. Where there is a tie, the implementation resolves by mode (first/lowest-index wins). No averaging, no decimals — only real categories that actually occur in the neighbour set. [SOURCE: imbalanced-learn SMOTEN _make_categorical_simulate]

---

Results / Findings

Because this article does not run an experiment, the "findings" below are a synthesis of the primary sources, not measured numbers from this author's machine.

Finding 1 — VDM makes "similar" mean "splits the target the same way." Two regime levels that both skew minority are close in VDM space; a regime level that skews majority is far from one that skews minority, even if the two levels are alphabetically or ordinally unrelated. This is the whole point: categorical similarity is semantic (with respect to the label), not geometric. [SOURCE: Stanfill & Waltz 1986; DERIVED]

Finding 2 — SMOTEN emits only legal categorical combinations, never impossible coordinates. Because synthesis is a per-feature majority vote over real neighbour levels, every synthetic row is a genuine categorical tuple — something that could plausibly occur. Contrast vanilla SMOTE, which would emit regime = 0.63, a level that does not exist. [DERIVED from the majority-vote mechanism]

Finding 3 — A synthetic row can be a new combination not present in the original data. The seed might be (Trend, Long) while its neighbours vote (Volatile, Long) on regime and keep Long on buildup, yielding (Volatile, Long) — a minority row that did not previously exist. SMOTEN therefore both replicates plausible patterns and assembles new plausible ones from neighbour consensus. [DERIVED]

Finding 4 — k_neighbors is the locality dial, exactly as in vanilla SMOTE. Smaller k ⇒ synthesis hugs tighter VDM neighbourhoods (risk: over-concentrated, repeats the seed); larger k ⇒ broader consensus across more of the minority (risk: blends distinct minority sub-populations into a mushy average category). Default 5, tune inside CV. [SOURCE: imbalanced-learn default; DERIVED]

Finding 5 — SMOTEN refuses to run on numeric data. If any column is continuous, imbalanced-learn raises (or you should reach for SMOTENC instead). It is strictly the purely categorical tool; that is its defining boundary versus SMOTENC. [SOURCE: imbalanced-learn SMOTEN]

---

Reproducibility (code shape — illustrative)

The following is the canonical call shape and is shown for reproducibility of API usage, not as a claim that we executed a model here. The real pipeline wires SMOTEN inside a Pipeline so it refits per CV fold. [SOURCE: imbalanced-learn Pipeline + SMOTE examples]

## Illustrative only — shows where SMOTEN sits: INSIDE CV, never on the test set.
from imblearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTEN
from xgboost import XGBClassifier
from sklearn.model_selection import StratifiedKFold

pipe = Pipeline([
    ('smoten', SMOTEN(sampling_strategy='auto', k_neighbors=5, random_state=42)),
    ('clf',    XGBClassifier(n_estimators=300, max_depth=4,
                             eval_metric='logloss', random_state=42))
])

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
## pipe.fit(X_train_fold, y_train_fold) -> scored on the untouched validation fold

Golden rule, repeated because people violate it: fit_resample only on training folds. If you resample the full set first and then split, synthetic minorities leak into validation and your backtest lies. In our Layer 2 EOD CV, SMOTEN is fit inside each walk-forward window on historical folds — never on live Dhan ticks. [DERIVED from CV hygiene; echoed in imbalanced-learn "avoid leakage" guidance]

---

Worked Categorical Example (DERIVED math)

Let's make VDM and the majority vote concrete. Suppose all features are categorical and we are predicting a rare Nifty event. Two features for simplicity:

• regime ∈ {Trend, Range, Volatile}
• oi_buildup ∈ {Long, Short, Neutral}

From the training data we estimate class-conditional probabilities P(minority | level):

regime	P(min)	oi_buildup	P(min)
Trend	0.20	Long	0.20
Range	0.10	Short	0.40
Volatile	0.60	Neutral	0.27

Using the binary-class simplification VDM(a,b) ≈ 2·|P(min|a) − P(min|b)| (p = 1, constant absorbed):

• VDM(regime: Volatile, Range) = 2·|0.60 − 0.10| = 1.00
• VDM(regime: Volatile, Trend) = 2·|0.60 − 0.20| = 0.80
• VDM(regime: Range, Trend) = 2·|0.10 − 0.20| = 0.20
• VDM(oi: Long, Short) = 2·|0.20 − 0.40| = 0.40
• VDM(oi: Long, Neutral) = 2·|0.20 − 0.27| = 0.14
• VDM(oi: Short, Neutral)= 2·|0.40 − 0.27| = 0.26

Now take a seed minority row s = (Trend, Long). Its VDM distance to three candidate minority neighbours:

• n1 = (Volatile, Long): regime 0.80 + oi 0.00 = 0.80
• n2 = (Volatile, Short): regime 0.80 + oi 0.40 = 1.20
• n3 = (Range, Long): regime 0.20 + oi 0.00 = 0.20

With k_neighbors = 3 the nearest minority neighbours are {n3, n1, n2} (ordered 0.20, 0.80, 1.20). Synthesis = majority vote over the set {s, n3, n1, n2}:

• regime: Trend(s), Range(n3), Volatile(n1), Volatile(n2) → Volatile wins (2 of 4).
• oi_buildup: Long(s), Long(n3), Long(n1), Short(n2) → Long wins (3 of 4).

Synthetic row = (Volatile, Long) — a minority tuple that did not exist before (seed was Trend, Long). Note what happened: the seed's own regime (Trend) was out-voted by its neighbours, which collectively said "Volatile." No decimal was invented; the result is a real, legal category combination. [DERIVED from the VDM table above]

A second observation: n3 = (Range, Long) is the closest neighbour by VDM (0.20) even though its regime Range is different from the seed's Trend — because both share Long buildup and both sit at similar minority probability. VDM correctly judged them similar in the only sense that matters for the label. A one-hot/Euclidean approach would have scored Range as a full unit away from Trend and Volatile equally, blind to the fact that Range and Trend are class-behaviourally closer here than Trend and Volatile. That is the VDM advantage in one number. [DERIVED]

---

What Failed / Counter-Evidence

SMOTEN inherits the general SMOTE-family caveats, plus a few of its own.

• Blagus & Lusa (2013) show over-sampling can degrade performance on small or high-dimensional data. For SMOTEN this bites twice: with few minority rows per level, the P(c | a) estimates are noisy, so the VDM distances are unreliable, and the majority vote can consolidate around a rare, misleading level. [SOURCE: Blagus & Lusa 2013]
• Rare levels break the distance. If a category appears only once or twice in training, its P(minority | level) is 0, 0.5, or 1 — extreme and unstable. VDM will then place that level at maximum or minimum distance from everyone, distorting neighbour selection. Smooth the probabilities (Laplace/Pseudocount) or collapse rare levels before resampling. [DERIVED from the probability-estimation step]
• High cardinality floods the neighbour computation. VDM builds a per-feature distance matrix over all level pairs; a feature with 500 city names means a 500×500 matrix and a distance sensitive to sparse per-level counts. Prefer low-cardinality, semantically grouped categoricals. [DERIVED]
• Majority vote can be boring. With large k_neighbors the per-feature mode may just reproduce the globally most common minority level, collapsing diversity — the categorical mirror of vanilla SMOTE's "synthetics pile in the safe interior." Use modest k. [DERIVED]
• No label-noise guard. Like all SMOTE variants, SMOTEN stamps "minority" on whatever it creates. If a neighbour level is actually a mislabel, SMOTEN will manufacture more of it. [DERIVED]

---

Limitations

1. Purely categorical only. Any continuous column forces you to SMOTENC instead. SMOTEN will not silently handle numerics. [SOURCE: imbalanced-learn SMOTEN]
2. Depends on reliable P(c | level). Small data ⇒ noisy distances ⇒ bad neighbours. [DERIVED]
3. Cardinality cost. Per-feature distance matrices grow with the square of level count; watch high-cardinality text/id features. [DERIVED]
4. Ties resolved arbitrarily. A 2–2 split on a binary feature picks the first/lowest level, which can bias synthesis. [SOURCE: imbalanced-learn mode resolution; DERIVED]
5. No numeric interpolation available. If your "categorical" set secretly benefits from a continuous proxy, SMOTEN cannot use it; SMOTENC can. [DERIVED]
6. Leakage risk identical to all SMOTE. Fit inside CV only. [DERIVED]
7. Assumes label-relevant categoricals. If the categoricals are unrelated to the target, VDM distances are meaningless and synthesis is noise. [DERIVED]

---

Practical Takeaways (Production Checklist)

• Reach for SMOTEN only when every feature is categorical — market-regime, expiry-bucket, OI-buildup, option-type, session, day-of-week style nominal signals. [DERIVED from the definition]
• If you have any numeric feature, switch to SMOTENC (V5), which interpolates numerics and majority-votes categoricals. [SOURCE: imbalanced-learn SMOTENC/SMOTEN]
• Keep k_neighbors modest (start 5, tune down to 3 if synthetics look repetitive). [SOURCE: imbalanced-learn default; DERIVED]
• Collapse rare levels / Laplace-smooth P(c | level) before resampling so VDM distances are stable. [DERIVED]
• Always wrap SMOTEN in a Pipeline so the VDM matrix is rebuilt per CV fold — never pre-resample the whole set. [SOURCE: imbalanced-learn Pipeline guidance]
• Compare against the cheap baseline: scale_pos_weight (XGBoost/LightGBM) or class_weight need no synthetic rows at all and are leakage-free. SMOTEN earns its place when the categorical minority geometry is genuinely learnable. [DERIVED from practice]
• In our stack, all of this lives in Layer 2 (EOD-audited training), never Layer 1 (live Dhan capture). [two-layer engine note]

---

FAQ

Q1. What is the core difference between SMOTEN and vanilla SMOTE?
A1. Vanilla SMOTE needs numeric features and interpolates with Euclidean distance (x + δ·(x_z − x)). SMOTEN needs categorical features, has no axis to interpolate on, so it uses the Value Difference Metric for neighbour distance and a per-feature majority vote for synthesis. [SOURCE: imbalanced-learn SMOTE vs SMOTEN; DERIVED]

Q2. What is the core difference between SMOTEN and SMOTENC?
A2. SMOTENC handles mixed numeric + categorical: it interpolates numeric features the normal SMOTE way and majority-votes categorical ones. SMOTEN is the purely categorical special case — no numeric interpolation at all. Use SMOTEN only when every column is categorical. [SOURCE: imbalanced-learn SMOTENC/SMOTEN]

Q3. What is the Value Difference Metric?
A3. VDM defines distance between two levels of a categorical feature from how differently those levels split the target class: VDM_f(a,b) = [Σ_c |P(c|a) − P(c|b)|^p]^(1/p). Levels that predict the same class are "close" even if they are not ordinally related. [SOURCE: Stanfill & Waltz 1986; Cost & Pedrycz 2002; DERIVED form]

Q4. Does SMOTEN ever create impossible values like 0.37 in a binary column?
A4. No. Synthesis is a majority vote over real neighbour levels, so every synthetic cell is a genuine category that occurs in the data. That is the entire reason to use it over label-encoding + vanilla SMOTE. [DERIVED from the mechanism]

Q5. What does k_neighbors do in SMOTEN?
A5. It sets how many nearest minority neighbours vote on each feature of a synthetic row. Smaller k = tighter, possibly repetitive; larger k = broader consensus, possibly bland. Default 5, tune inside CV. [SOURCE: imbalanced-learn default; DERIVED]

Q6. Can I use SMOTEN on live Nifty ticks?
A6. No — never on live data. Our stack applies any over-sampling only in Layer 2 EOD CV, never on the Layer 1 live Dhan feed. [two-layer engine note; DERIVED]

Q7. When does SMOTEN hurt?
A7. Small minorities (noisy P(c|level)), high-cardinality features, rare/unrepresented levels, or categoricals unrelated to the target. Then it can degrade a model — the Blagus & Lusa 2013 warning, applied to the categorical case. [SOURCE: Blagus & Lusa 2013; DERIVED]

Q8. What comes after SMOTEN?
A8. V7 ADASYN — the adaptive variant that weights synthetic generation toward hard-to-learn minority regions using a density ratio, rather than over-sampling uniformly. [SOURCE: He et al. 2008]

---

TL;DR

• SMOTEN (from imblearn.over_sampling import SMOTEN) is SMOTE for purely categorical feature sets — no Euclidean axis exists, so vanilla SMOTE cannot run. [SOURCE: imbalanced-learn SMOTEN]
• It measures neighbour distance with the Value Difference Metric (VDM): distance between two levels = how differently they split the target class Σ_c |P(c|a) − P(c|b)|^p. [SOURCE: Stanfill & Waltz 1986; Cost & Pedrycz 2002; DERIVED]
• It synthesises a new minority row by majority vote per feature among the seed and its VDM-nearest minority neighbours — real categories only, never impossible decimals. [SOURCE: imbalanced-learn SMOTEN; DERIVED]
• Contrast: vanilla SMOTE = numeric, Euclidean interpolation; SMOTENC = mixed, interpolates numerics + votes categoricals; SMOTEN = categorical-only, votes everything. [SOURCE: imbalanced-learn; DERIVED]
• Use it when every feature is nominal; watch small-data noise, high cardinality, and rare levels; always fit inside CV (Layer 2), never on live data. [DERIVED]
• Next: V7 ADASYN. Previous: V5 SMOTENC.

---

📚 Related Articles

• Smote V5 Smotenc
• Smote V1 Vanilla
• Smote V4 Kmeanssmote

👤 About the Author

Name:

Shakti Tiwari

Role:

Nifty Option Trader, XGBoost Expert

Cert:

NISM-Series-XII (Securities Markets Foundation)

Knows:

XGBoost · LightGBM · Options Trading · Machine Learning · Walk-forward Validation

Profiles:
about.me ·
optiontradingwithai.in ·
github ·
whatsapp ·
x/twitter

Sources (PRIMARY)

• [SOURCE] scikit-learn-contrib/imbalanced-learn (GitHub, ~7.1k★). imblearn/over_sampling/_smote/base.py — SMOTEN class: VDM-based neighbour search, _make_categorical_simulate (per-feature most-frequent-category synthesis), k_neighbors=5 default, fit_resample API, and the SMOTENC/SMOTEN split (categorical-only vs mixed). Primary implementation source.
• [SOURCE] Stanfill, C., & Waltz, D. (1986). Toward memory-based reasoning. Communications of the ACM, 29(11), 1213–1228. — original Value Difference Metric for categorical distance.
• [SOURCE] Cost, S., & Pedrycz, W. (2002). Classification of imbalanced data using SVM-based decomposition. — application of VDM-style categorical distance to imbalanced learning; the categorical-over-sampling lineage SMOTEN draws on (cited in the imbalanced-learn SMOTEN docstring).
• [SOURCE] Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, 16, 321–357. — original SMOTE algorithm (interpolation baseline).
• [SOURCE] Han, H., Wang, W.-Y., & Mao, B.-H. (2005). Borderline-SMOTE — variant family context.
• [SOURCE] He, H., Bai, Y., Garcia, E. A., & Li, S. (2008). ADASYN: Adaptive Synthetic Sampling — next article (V7).
• [SOURCE] Blagus, R., & Lusa, L. (2013). SMOTE for high-dimensional class-imbalanced data (BMC Bioinformatics). — counter-evidence: over-sampling can degrade on small/high-dimensional data; applies to the categorical case via noisy P(c|level).
• [DERIVED] The VDM binary-class simplification, the per-feature distance-matrix construction, the worked (Trend, Long) → (Volatile, Long) synthesis example, the k-neighbours locality argument, and the SMOTE vs SMOTENC vs SMOTEN contrast — derived from the cited sources and standard categorical-distance theory.

---

Author / Canonical

Written for Shakti Tiwari — Nifty Option Trader, XGBoost Expert (optiontradingwithai.in). Part of the SMOTE Family series (V6 of V6). This is educational content; not investment advice and not SEBI-registered research. SMOTEN is the canonical reference for purely-categorical over-sampling and the immediate predecessor to ADASYN (V7); SMOTENC (V5) covers the mixed-type case.

Disclaimer (repeat): Shakti Tiwari is NISM-Series-XII certified; not a SEBI-registered Research Analyst. Content is educational only.

---

Resources & Links

• Brand site: https://optiontradingwithai.in
• About the author: https://about.me/shaktitiwari
• WhatsApp: https://wa.me/919169650895
• Previous article (V5 SMOTENC): SMOTE_V5_smotenc.md
• Next article (V7 ADASYN): SMOTE_V7_adasyn.md
• Books: https://www.amazon.in/dp/B0H9ZNTBPK · https://www.amazon.in/dp/B0HBBFKDQF