Feature Scaling

Data Prep & Model EvaluationBeginner~5 min

Feature Scaling — Put features on the same scale so no one dominates.

If one feature ranges 0–1 and another 0–10,000, distance- and gradient-based models get dominated by the big one. Scaling puts every feature on equal footing.

Raw — 98 steps

Standardize — 17 steps

Descent path
Minimum
Loss (darker = higher)

Before → after, standardize scaling
Row	Bedrooms	Lot size	Bedrooms′	Lot size′
#1	2.00	10.80	-1.57	-1.06
#2	3.00	16.80	-0.52	-0.56
#3	4.00	30.00	0.52	0.56
#4	3.00	13.20	-0.52	-0.86
#5	5.00	45.60	1.57	1.87
#6	4.00	24.00	0.52	0.05
Typical spread (IQR)	1.00	14.40	1.04	1.21

Scaling controls

Data

Table

Two honest columns. Only the recording unit differs, so scaling should fix it outright.

Unit blow-up on Lot size12×

Recording lot size in a finer unit multiplies every number in that column. The data is unchanged; only its scale is.

Model

Scaler

Learning rate0.28

Step budget300

Raw spread ratio14.4×

After scaling1.2×

Playback

Step 0 / 98

Speed

Overshoot the steep axis
Crawl down the valley
Settled
↻

Step 0 of 98 — raw bowl at (1.20, 8.60) — loss 43.83; standardize bowl at (1.20, 8.60) — loss 13.84

Drag the highlighted start dot on either bowl to relaunch descent from anywhere; hover any step to ring the same step on the other bowl.

Break it

The idea in plain words

If one feature ranges 0–1 and another 0–10,000, the big one dominates any distance- or gradient-based model. Scaling puts every feature on equal footing. On the loss surface, unscaled features make skewed, stretched contours; scaled features make near-circular ones.

The identical gradient descent zig-zags hopelessly across the skewed valley but walks straight down the circular one — same math, opposite outcome. It’s why scaling matters for kNN and PCA.

Now, the math

Standardization rescales each feature to zero mean and unit variance:

z = \frac{x - \mu}{\sigma}

$\mu$: the feature’s mean.
$\sigma$: its standard deviation.

▸ Show the derivation

The convergence speed of gradient descent depends on the condition number of the loss (the ratio of largest to smallest curvature). Unequal feature scales inflate that ratio, forcing tiny steps along the steep axis; standardizing equalizes the curvatures, so a single learning rate works in every direction.

Trace it by hand

One feature with three values: 10, 20, 60. We standardize it to zero mean and unit variance, then min-max scale the same three points. Standard deviation is the population version (divide by n = 3); results rounded to 2 decimals.

Compute the mean
$\mu = \frac{10 + 20 + 60}{3} = \frac{90}{3} = 30$
Compute the standard deviation
$\sigma = \sqrt{\frac{(-20)^2 + (-10)^2 + 30^2}{3}} = \sqrt{\frac{1400}{3}} \approx 21.6$
Deviations from the mean are minus 20, minus 10 and plus 30; their squares are 400, 100 and 900. The lone large value 60 dominates the spread.
Standardize each value
$z = \frac{x - \mu}{\sigma}: \quad \frac{10 - 30}{21.6} \approx -0.93, \quad \frac{20 - 30}{21.6} \approx -0.46, \quad \frac{60 - 30}{21.6} \approx 1.39$
Check: the three z-scores now have mean 0 and standard deviation 1, whatever units x started in.
Min-max scale the same points
$x' = \frac{x - 10}{60 - 10}: \quad 0, \quad 0.2, \quad 1$
Min-max pins the endpoints to exactly 0 and 1; standardization instead centers on the mean. Both put this feature on the same footing as any other.

What just happened: Raw values 10, 20, 60 became z-scores of minus 0.93, minus 0.46, plus 1.39 or min-max values 0, 0.2, 1 — same relative positions, but now on a scale where no single feature can dominate a distance or a gradient step.

Now Break It

Try this: Unscaled features make distance-based methods obsess over the large-magnitude feature.

Control: Scaling toggle (turn off)

What happens: Scale domination! Without scaling, the large-magnitude feature drowns out all the others.

Where feature scaling is used

Feature scaling matters most for algorithms that measure distance or rely on gradient steps. In a k-nearest-neighbors credit model, income measured in tens of thousands would completely swamp age measured in years, so the nearest neighbor is chosen almost entirely by income unless both features are scaled. Support vector machines, k-means clustering, principal component analysis, and neural networks trained with gradient descent all converge faster and behave more sensibly when inputs share a comparable range. In production, a common example is combining a website's session duration in seconds with number of pages viewed; without scaling, the raw seconds dominate the distance metric and the page-count signal is effectively ignored by the model.

A frequent and damaging pitfall is fitting the scaler on the entire dataset before splitting, which leaks statistics from the test set into training. The correct order is to split first, fit the scaler on the training data only, then apply that same fitted transform to the validation and test sets. A second misconception is that every model needs scaling. Tree-based methods like decision trees, random forests, and gradient boosting split on one feature at a time using thresholds, so they are invariant to monotonic rescaling and gain nothing from it. Scaling also does not fix skew or outliers by itself; standardization still leaves a heavy tail heavy, which is why log transforms or robust scalers are sometimes better choices.

Frequently asked questions

What is the difference between standardization and normalization?

Standardization subtracts the mean and divides by the standard deviation, giving each feature roughly zero mean and unit variance, with no fixed bounds. Normalization, usually min-max scaling, rescales values into a fixed range such as 0 to 1. Standardization is more robust to outliers and is the common default; min-max is useful when you need bounded inputs, for example for certain neural network layers.

Do I need to scale features for decision trees or random forests?

No. Tree-based models split on thresholds of individual features and are unaffected by monotonic rescaling, so scaling neither helps nor hurts them. You mainly need scaling for distance-based and gradient-based models such as KNN, SVMs, k-means, PCA, and neural networks.

Should I scale the target variable too?

For most classifiers the target is a label and is never scaled. In regression you can optionally scale the target to help optimization, but you must remember to invert the transform on predictions before interpreting them. It is usually optional rather than required.

Why is it wrong to scale before splitting the data?

Fitting a scaler on all the data lets the mean and variance of the test set influence the training transform, a form of data leakage. This produces optimistic evaluation scores that do not hold up in deployment. Always split first, fit on the training set, then apply the same transform to validation and test data.

How should I handle outliers when scaling?

Standard and min-max scaling are both sensitive to extreme values, which can compress the rest of the data into a narrow band. A robust scaler that uses the median and interquartile range is more resistant, and applying a log transform first can tame heavy-tailed features before scaling.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.