Ridge Regression (L2)

RegressionIntermediate~7 min

Ridge Regression (L2) — Shrink coefficients toward zero to reduce variance.

Ridge regression adds a penalty for large coefficients. It gently shrinks every weight toward zero, trading a little bias for a big drop in variance — taming wild overfit models.

Coefficient path — drag to set λ

Coefficients at λ = 0.00100 (click one to trace it)

Ridge controls

Data

Design

A comfortable design: more rows than columns, so plain least squares is stable.

Model

L2 constraint (circle)

Regularization λ0.00100

Largest |w|3.782

Non-zero6 / 6

Playback

Step 0 / 59

Speed

Weak penalty
Balanced
Strong penalty
↻

Step 0 of 59 — λ = 0.00100 — largest |w| 3.78, 6 of 6 coefficients still meaningfully non-zero, error 1.06

Break it

The idea in plain words

Ridge regression adds a price for large coefficients. Instead of only minimizing error, it minimizes error plus the summed squares of the weights, so the fit trades a little bias for a big drop in variance — taming the wild swings a high-degree fit is prone to.

Turn the penalty λ up and every coefficient shrinks smoothly toward zero, but none ever reaches it exactly. Geometrically, the round L2 constraint has no corners for a coefficient to snap to — the crucial difference from lasso.

Now, the math

Ridge minimizes squared error plus an L2 penalty on the weights:

J = \tfrac{1}{n}\sum_i (y_i - \hat{y}_i)^2 + \lambda \sum_j w_j^2

This still has a closed form — just a nudged normal equation:

\mathbf{w} = (X^\top X + \lambda I)^{-1} X^\top \mathbf{y}

$\lambda$: the regularization strength — how hard large weights are penalized.
$\sum_j w_j^2$: the L2 penalty — the squared length of the weight vector.
$\lambda I$: the ridge added to the diagonal, which also fixes ill-conditioning.

▸ Show the derivation

Adding λI to X^⊤X shifts every eigenvalue up by λ, so directions of low data variance (which cause instability) are damped most. As λ → ∞ the solution collapses toward the all-zero vector and the model predicts the mean of y — the failure you can drive with the slider.

Trace it by hand

A tiny two-feature problem you can shrink by hand: standardized columns x1 equal to (-1, -1, 1, 1) and x2 equal to (-1, 1, -1, 1), with targets y equal to 2 x1 plus 0.5 x2 and zero noise — so any change in the weights is caused purely by the penalty. Solved with the repo's ridge closed form, which scales the normal equations by 1 over n.

Step 1 — the unpenalized solution
$\tfrac{1}{n}X^\top X = I, \qquad \mathbf{z} = \tfrac{1}{n}X^\top \mathbf{y} = (2,\ 0.5), \qquad \lambda = 0: \ \mathbf{w} = (2,\ 0.5)$
Because the two columns are orthogonal with variance 1, X transpose X over n is the identity — every matrix formula collapses to per-weight arithmetic.
Step 2 — the closed form becomes a divide
$\mathbf{w} = \left(\tfrac{1}{n}X^\top X + \lambda I\right)^{-1} \tfrac{1}{n}X^\top \mathbf{y} = \frac{\mathbf{z}}{1+\lambda}$
Adding lambda down the diagonal is the nudged normal equation from the equations section — here the nudge is the whole story.
Step 3 — turn lambda up
$\lambda = 1: \ \mathbf{w} = \left(\tfrac{2}{2},\ \tfrac{0.5}{2}\right) = (1,\ 0.25), \qquad \lambda = 4: \ \mathbf{w} = \left(\tfrac{2}{5},\ \tfrac{0.5}{5}\right) = (0.4,\ 0.1)$
Both weights shrink by the same factor 1 over 1 plus lambda: a 50 percent cut at lambda 1, an 80 percent cut at lambda 4.
Step 4 — the cost of shrinkage
$\text{MSE}(\lambda{=}0) = 0 \;\to\; \text{MSE}(\lambda{=}1) = 1.0625 \;\to\; \text{MSE}(\lambda{=}4) = 2.72$
On this noiseless data shrinkage only hurts the fit; on noisy data the same bias is what buys a bigger drop in variance.
Step 5 — zero is never reached
$w_2 = \frac{0.5}{1+\lambda} > 0 \quad \text{for every finite } \lambda$
Division shrinks smoothly but never lands on zero exactly — compare the lasso walk-through, which subtracts instead and does hit zero.

What just happened: Ridge multiplied every weight by 1 over 1 plus lambda: (2, 0.5) became (1, 0.25) and then (0.4, 0.1). The small weight got proportionally smaller but stayed alive — L2 shrinks, it never selects.

Now Break It

Try this: Enormous λ crushes every coefficient to near zero — the model becomes a flat line ignoring the data.

Control: Lambda slider (set to maximum)

What happens: Over-regularized! λ is so large every coefficient is crushed to zero — the model underfits badly.

Where ridge regression (l2) is used

Ridge regression shines whenever features are numerous or correlated, situations where ordinary least squares becomes unstable. Genomics is a classic example: predicting a trait from thousands of correlated gene expression levels with only hundreds of samples would give wildly unstable ordinary coefficients, but the L2 penalty in ridge regression tames them into a stable, usable model. Finance teams use it to build factor models where many economic indicators move together, and marketing analysts apply it when advertising channels are strongly correlated. It is also common in chemometrics, where spectra have many overlapping wavelengths. Because ridge keeps every feature but shrinks their weights smoothly, it is favored when you believe most inputs contribute a little and you want reliable, low-variance predictions rather than an explicit subset of features.

A common misconception is that ridge regression performs feature selection; it does not. The L2 penalty shrinks coefficients toward zero but almost never sets any exactly to zero, so all features stay in the model, which is why ridge differs fundamentally from lasso. Another pitfall is forgetting to standardize features before fitting, since the penalty acts on coefficient size and unscaled features are penalized unfairly, distorting the result. People also misread the penalty strength: too large and the model underfits by shrinking everything toward the mean, too small and it barely regularizes, so the strength must be chosen by cross-validation rather than guessed. Ridge also leaves the intercept unpenalized by convention.

Frequently asked questions

What is ridge regression?

Ridge regression is linear regression with an added L2 penalty that discourages large coefficients by adding the sum of squared coefficients to the loss. This shrinks the coefficients toward zero, trading a little bias for a large reduction in variance. The result is a more stable model that generalizes better when features are many or correlated.

Why does ridge regression not set coefficients to exactly zero?

The L2 penalty is a smooth, rounded constraint, so its gradient shrinks coefficients proportionally and gently, never forcing them all the way to zero. Geometrically the constraint region is a circle or sphere with no corners on the axes, so the solution rarely lands exactly on zero. That is why ridge shrinks but keeps every feature instead of selecting a subset.

How is ridge regression different from lasso?

Ridge uses an L2 penalty and shrinks all coefficients smoothly, keeping every feature, while lasso uses an L1 penalty that can drive coefficients exactly to zero for feature selection. Ridge usually handles correlated features more gracefully by sharing weight among them, whereas lasso tends to pick one and drop the rest. Choose ridge when you expect many small contributions and lasso when you expect a sparse set of important features.

How do I choose the ridge penalty strength?

Tune the penalty, often called lambda or alpha, with cross-validation, trying a range of values on a logarithmic scale and picking the one with the lowest validation error. A larger value regularizes more strongly and increases bias, while a smaller value behaves closer to ordinary least squares. Always standardize features first so the penalty applies fairly across them.

When should I use ridge regression?

Reach for ridge when you have many correlated predictors, more features than is comfortable relative to your sample size, or an ordinary least squares fit with unstable, exploding coefficients. It is ideal when you believe most features carry some signal and want to keep them all while controlling variance. If you instead need a small, interpretable subset of features, lasso is a better fit.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.