Encoding Categorical Features

Q: When should I use one-hot encoding versus label encoding?

Use one-hot encoding for nominal categories that have no natural order, so the model does not infer a false ranking. Use label encoding for ordinal features that do have an order, such as low, medium, high, or when the downstream model is tree-based and only uses thresholds. For linear and distance-based models on unordered categories, prefer one-hot.

Q: How do I handle categorical features with very many unique values?

One-hot encoding becomes impractical when a column has thousands of distinct values because it creates too many sparse columns. Consider target or frequency encoding, feature hashing, grouping rare categories into an other bucket, or learned embeddings if you are training a neural network.

Q: What is target encoding and why is it risky?

Target encoding replaces each category with a statistic of the target, such as the mean label for that category. It is compact and powerful but leaks the target into the feature if computed on the same rows used for training. To use it safely, compute the encoding within cross-validation folds or with smoothing and out-of-fold estimates.

Q: What happens when a category appears in the test set but not in training?

An encoder fit only on training data will not have a mapping for the unseen category and may error or produce a missing value. Configure encoders to handle unknown categories gracefully, for example by mapping them to a zero vector or a reserved other category, so inference does not fail.

Q: Does one-hot encoding cause any problems for linear models?

One-hot columns are perfectly collinear because they sum to one, which can make a plain linear regression's coefficients unstable. Dropping one reference category removes the redundancy. With regularization the issue is usually minor, but dropping a level keeps coefficients interpretable.

Data Prep & Model EvaluationBeginner~5 min

Encoding Categorical Features — Turn categories into numbers models can use.

Models eat numbers, not words. Encoding turns categories like “red, green, blue” into numeric form — but the wrong encoding can invent a fake ordering that misleads the model.

Design matrix — 8 × 1

Apple
Apple
Banana
Banana
Cherry
Cherry
Date
Date

code

Model error by encoding (click one to use it)

Predicted vs actual shelf life (days)

Encoding controls

Data

Column

No order exists. Apple is not "less than" Cherry, and any single number column has to invent one.

Cardinality (distinct categories)4

Rows per category2

Target noise0.6

Model

Encoding

Design matrix8 × 1

Cells that are 025%

Playback

Step 0 / 2

Speed

Categories
Encoded columns
Model fit
↻

Step 0 of 2 — 8 rows of the raw fruit column, 4 distinct values — nothing a model can multiply yet

Click any bar in the error chart to switch to that encoding, and hover a matrix row to find it in the predicted-vs-actual plot.

Break it

The idea in plain words

Models eat numbers, not words. Encoding turns categories like “Red, Green, Blue” into numeric form — but the wrong choice invents structure that isn’t there. Label encoding assigns 0, 1, 2, which tells the model the categories are ordered and evenly spaced.

One-hot encoding instead gives each category its own binary column, so every pair is equally distant. For unordered categories that’s the honest representation.

Now, the math

Under label encoding the model reads a false distance and ordering:

d(\text{Red},\text{Blue}) = 2 \neq 1 = d(\text{Red},\text{Green})

One-hot makes every distinct pair equidistant:

d(i, j) = \sqrt{2}\quad \forall\, i \neq j

▸ Show the derivation

A linear model multiplies the encoded value by a weight, so label codes force the effect of “Blue” to be exactly twice that of “Green.” One-hot lets each category get its own independent weight, removing the artificial order — at the cost of one extra column per category.

Trace it by hand

Four data rows with one categorical feature, city, taking three values: Delhi, Paris, Tokyo. Every number below is exact — no rounding involved.

Step 1 — the raw column

row	city
1	Delhi
2	Paris
3	Tokyo
4	Delhi

A model can’t multiply “Tokyo” by a weight, so the strings must become numbers.

Step 2 — label encoding invents an order

Assign Delhi = 0, Paris = 1, Tokyo = 2. The codes immediately claim Tokyo is “twice” Paris and that Tokyo is farther from Delhi than Paris is:

d(\text{Delhi},\text{Tokyo}) = |0 - 2| = 2 \neq 1 = d(\text{Delhi},\text{Paris})

A linear model multiplies the code by one weight w, so Tokyo’s effect is forced to be exactly 2w — double Paris’s w — purely because of an arbitrary alphabetical numbering. city = 2 > city = 1 is a fake ordering.

Step 3 — one-hot encoding: one column per category

row	city_Delhi	city_Paris	city_Tokyo
1	1	0	0
2	0	1	0
3	0	0	1
4	1	0	0

k = 3 \ \text{categories} \;\Rightarrow\; 3 \ \text{columns} \quad (\text{or } k-1 = 2 \ \text{with drop-first})

Now every pair of distinct cities sits at the same distance √2 and gets its own independent weight. Drop-first removes one redundant column: if city_Paris and city_Tokyo are both 0, the row must be Delhi.

What just happened: the same four rows produced two different geometries. Label encoding smuggled in the claim Tokyo > Paris > Delhi with Tokyo twice as far from Delhi; one-hot spent k = 3 columns to make every city equidistant and independently weighted — the honest choice for unordered categories.

Now Break It

Try this: Label encoding unordered categories invents a fake numeric order the model treats as meaningful.

Control: Encoding selector (set to label encoding)

What happens: Fake ordering! Label encoding tells the model red < green < blue — an order that doesn’t exist.

Where encoding categorical features is used

Encoding categorical features turns labels like country, product category, or browser type into numbers a model can process. One-hot encoding is the safe default for nominal categories with no inherent order: a color feature becomes separate red, green, and blue indicator columns, so the model never assumes green is somehow between red and blue. For high-cardinality columns such as zip code or user id, one-hot encoding explodes the feature count, so practitioners turn to alternatives like target encoding, hashing, or learned embeddings. Recommendation systems and large-scale ad-click models routinely embed millions of categorical ids into dense vectors, which is far more compact and lets the model learn similarity between categories.

The classic mistake is applying integer label encoding to unordered categories and feeding them to a linear model or distance-based method. Mapping cat to 1, dog to 2, and bird to 3 falsely tells the model that bird is three times cat and that dog sits between them, injecting a false ordering. Label encoding is appropriate for genuinely ordinal features like small, medium, large, and it is also fine for tree models that only compare thresholds. A second pitfall is target encoding computed on the full dataset, which leaks the label into the feature; it must be fit inside cross-validation folds. Finally, always plan for categories in the test set that never appeared in training so the pipeline does not crash on unseen values.

Frequently asked questions

When should I use one-hot encoding versus label encoding?

Use one-hot encoding for nominal categories that have no natural order, so the model does not infer a false ranking. Use label encoding for ordinal features that do have an order, such as low, medium, high, or when the downstream model is tree-based and only uses thresholds. For linear and distance-based models on unordered categories, prefer one-hot.

How do I handle categorical features with very many unique values?

One-hot encoding becomes impractical when a column has thousands of distinct values because it creates too many sparse columns. Consider target or frequency encoding, feature hashing, grouping rare categories into an other bucket, or learned embeddings if you are training a neural network.

What is target encoding and why is it risky?

Target encoding replaces each category with a statistic of the target, such as the mean label for that category. It is compact and powerful but leaks the target into the feature if computed on the same rows used for training. To use it safely, compute the encoding within cross-validation folds or with smoothing and out-of-fold estimates.

What happens when a category appears in the test set but not in training?

An encoder fit only on training data will not have a mapping for the unseen category and may error or produce a missing value. Configure encoders to handle unknown categories gracefully, for example by mapping them to a zero vector or a reserved other category, so inference does not fail.

Does one-hot encoding cause any problems for linear models?

One-hot columns are perfectly collinear because they sum to one, which can make a plain linear regression's coefficients unstable. Dropping one reference category removes the redundancy. With regularization the issue is usually minor, but dropping a level keeps coefficients interpretable.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.