A categorical column has to become numbers before almost any model can use it, and the four common ways of doing that produce matrices of radically different width with radically different leakage risk. Here they all are, on the same column.
One column, four encodings
Take a marketing table with a column channel holding five values — organic, paid_search, email, affiliate, referral — and a binary target converted. Suppose 10,000 rows, with these per-category conversion rates and counts:
channel n converted rate
organic 5,200 416 0.080
paid_search 2,900 319 0.110
email 1,400 210 0.150
affiliate 440 22 0.050
referral 60 15 0.250
----
global 10,000 982 0.0982
Every encoding below is a different answer to one question: how much of that table do you hand to the model, and how much of it did the model already have access to through the label?
One-hot: width you pay for
One-hot encoding replaces the column with one binary indicator per category. Five categories become five columns, or four if you drop one to avoid perfect collinearity in a linear model. A row with channel=email becomes [0, 0, 1, 0, 0].
The property that matters is that it is lossless and order-free. It asserts nothing about the relationship between categories, which is exactly right for a nominal variable where affiliate is not “between” email and referral in any sense. The cost is width and sparsity. Five is nothing; a postcode column with 8,000 distinct values becomes 8,000 columns, most of which are zero in every row, and a tree-based model has to spend one split per indicator to isolate a single category.
Ordinal encoding — mapping the five values to 0 through 4 — is the cheap alternative and is usually wrong for a nominal column, because it tells a linear or distance-based model that referral is four units away from organic and one from affiliate. Tree ensembles are less damaged by this than people expect, since a tree can carve an arbitrary integer range into intervals given enough splits, but it still spends depth doing so. The distinction between a genuinely ordered category and an arbitrarily numbered one is worth getting explicit, and is covered in the difference between ordinal and nominal encoding.
Target encoding and the leak it hides
Target encoding, introduced by Daniele Micci-Barreca in ACM SIGKDD Explorations 3(1) in 2001, replaces each category with a statistic of the target computed within that category — usually the mean. The column stays one column wide no matter how many categories there are, which is the entire appeal.
The naive version replaces email with 0.150 and referral with 0.250 and is a disaster, for a reason thereferral row makes obvious. Sixty rows, fifteen conversions: 0.250 is an estimate with an enormous standard error, and if any of those sixty rows is in the training set then its own label has contributed to the number the model sees on that row. That is target leakage, and it produces a feature that looks brilliant in training and collapses in production.
Two mechanisms fix it, and a usable implementation applies both. Smoothing shrinks each category mean toward the global mean in proportion to how little data supports it. With a smoothing weight m, the encoded value is:
encoded(c) = (n_c * mean_c + m * global_mean) / (n_c + m)
referral, m = 20: (60 * 0.250 + 20 * 0.0982) / 80 = 0.2130
email, m = 20: (1400 * 0.150 + 20 * 0.0982) / 1420 = 0.1493
The 60-row category moves a long way toward the global rate; the 1,400-row category barely moves. That is the correct behaviour and it is a plain Bayesian shrinkage.
The second mechanism is cross-fitting: the encoding applied to a training row must be computed from folds that exclude that row. scikit-learn’s TargetEncoder, added in version 1.3, does this by default with cv=5, and its documentation is explicit that fit(X, y).transform(X) does not equal fit_transform(X, y) because only the latter cross-fits, and that calling fit then transform on training data is discouraged for exactly that reason.
The smooth default in that implementation is "auto", an empirical Bayes estimate rather than a fixed number, and library defaults move between versions. Check the version you have installed rather than trusting a value quoted anywhere, including here.
Learned embeddings
An embedding replaces each category with a learned vector of fixed length d, trained jointly with the model. The five channels become a 5×d lookup table; with d=3 that is fifteen learned parameters and each row contributes three columns to the input.
The dimensionality comparison is the point. For five categories: one-hot gives five columns, target encoding gives one, an embedding with d=3 gives three. For 8,000 postcodes: one-hot gives 8,000, target encoding still gives one, and an embedding with d=16 gives sixteen input columns backed by 128,000 learned parameters. Embeddings are the only one of the three whose output width is decoupled from cardinality without collapsing the category to a single scalar summary of the target.
What you buy for that is the ability to represent similarity between categories that no rule told the model about — two postcodes with similar behaviour end up near each other in the space. What you pay is that the vectors only mean anything after training, so they cannot be computed once and reused across models, and they need enough rows per category to be estimated at all. A category seen twice gets a near-random vector.
What changes at high cardinality
- Unseen categories at inference. One-hot silently produces an all-zero row unless you configured a handler; target encoding needs an explicit fallback to the global mean; an embedding table needs a reserved index. All three are decisions, and the default is often to raise an error in production at 3am.
- Rare categories are noise, not signal. Grouping everything below a count threshold into a single
otherbucket before encoding is usually a larger win than the choice of encoder. The threshold is a hyperparameter; treat it as one. - Native handling may beat all of it. XGBoost can split on a categorical feature directly, and its
max_cat_to_onehotparameter decides per feature whether to use a one-hot-style split or to partition the categories between the child nodes. Above the threshold it partitions, which is what makes high-cardinality columns tractable without you encoding them at all. The docs note theexacttree method does not support categorical features.
Choosing by model, not by taste
The encoder interacts with the model class, and picking one in the abstract is how people end up with a pipeline that is worse than doing nothing. Linear and distance-based models need one-hot or embeddings, because they read magnitude directly and an ordinal code is a false claim about distance. Gradient-boosted trees do well with native categorical splits where available and with target encoding otherwise, and are the least sensitive of the three families to a mediocre choice. Neural networks are where embeddings earn their keep, since the table is trained end to end anyway.
Whatever you choose, fit the encoder inside the cross-validation loop and never before it. An encoder fitted on the full dataset and then split has already carried information across the split, and the symptom is a validation score you cannot reproduce on new data — the pattern described in detecting target leakage. Numeric columns have their own version of this trap, covered in numeric feature scaling.
Top comments (0)