Lasso Regression (L1)

RegressionIntermediate~7 min

Lasso Regression (L1) — Drive some coefficients exactly to zero for feature selection.

Lasso uses a different penalty than ridge — one that pushes weak coefficients all the way to exactly zero. That means it doesn’t just shrink features, it deletes them, doing automatic feature selection.

Coefficient path — drag to set λ

Coefficients at λ = 0.00100 (click one to trace it)

Lasso controls

Data

Design

Four features carry real signal, two are pure noise — the two lasso should delete.

Model

L1 constraint (diamond)

Penalty

Regularization λ0.00100

Largest |w|3.816

Non-zero6 / 6

Diamond corners lie on the axes — the loss contour meets a corner and a coefficient snaps to exactly zero.

Playback

What to step through

Step 0 / 59

Speed

Weak penalty
Balanced
Strong penalty
↻

Step 0 of 59 — λ = 0.00100 — largest |w| 3.82, 6 of 6 coefficients still non-zero, error 1.06

Break it

The idea in plain words

Lasso penalizes the absolute value of the weights instead of their squares. That one change lets it drive weak coefficients all the way to exactly zero — so it doesn’t just shrink features like ridge, it deletes them, performing automatic feature selection.

The reason is geometric: the L1 constraint region is a diamond with sharp corners on the axes. The elliptical loss contours almost always first touch that region at a corner, where one coordinate is zero. Toggle between ridge and lasso to see the circle-vs-diamond difference directly.

Now, the math

Lasso swaps the squared penalty for an absolute-value (L1) penalty:

J = \tfrac{1}{n}\sum_i (y_i - \hat{y}_i)^2 + \lambda \sum_j |w_j|

There’s no closed form; coordinate descent applies a soft-threshold to each weight:

w_j = \operatorname{sign}(z_j)\,\max(|z_j| - \lambda,\ 0)

$\sum_j |w_j|$: the L1 penalty — the diamond-shaped constraint that yields sparsity.
$z_j$: the least-squares update for weight j before penalizing.
$\lambda$: the threshold — any weight with |z| below it is set to exactly zero.

▸ Show the derivation

Soft-thresholding is the exact solution of the one-coordinate lasso problem: whenever a weight’s correlation with the residual falls below λ, the subgradient of |w| pins it to zero. Cycling this over all coordinates converges to the global optimum. With correlated features, though, lasso keeps one and zeroes the others somewhat arbitrarily — which motivates elastic net.

Trace it by hand

The exact same tiny problem as the ridge walk-through: columns x1 equal to (-1, -1, 1, 1) and x2 equal to (-1, 1, -1, 1), targets y equal to 2 x1 plus 0.5 x2, so the per-weight least-squares values are z equal to (2, 0.5). Now the penalty is L1 and the solver is the repo's coordinate descent with soft-thresholding.

Step 1 — start from the unpenalized weights
$\lambda = 0: \quad \mathbf{w} = (z_1,\ z_2) = (2,\ 0.5)$
With orthonormal columns, each coordinate update sees exactly zj — the least-squares value for that weight from the page equation.
Step 2 — soft-threshold with a small lambda
$\lambda = 0.25: \quad w_1 = \operatorname{sign}(2)\max(|2| - 0.25,\ 0) = 1.75, \qquad w_2 = \max(0.5 - 0.25,\ 0) = 0.25$
Lasso subtracts lambda from every weight's magnitude — a flat tax, unlike ridge's percentage cut.
Step 3 — raise lambda past the small weight
$\lambda = 1: \quad w_1 = \max(2 - 1,\ 0) = 1, \qquad w_2 = \operatorname{sign}(0.5)\max(|0.5| - 1,\ 0) = 0$
0.5 minus 1 is negative, so the max clamps w two to exactly 0 — not merely small. Feature x2 has been deleted from the model.
Step 4 — ridge on the same problem, same lambda
$\text{ridge},\ \lambda = 1: \quad \mathbf{w} = (1,\ 0.25) \qquad \text{vs.} \qquad \text{lasso},\ \lambda = 1: \quad \mathbf{w} = (1,\ 0)$
Identical data, identical lambda: ridge divides 0.5 down to 0.25, lasso subtracts past zero and clamps. That is the circle-versus-diamond corner, in numbers.

What just happened: Soft-thresholding turned (2, 0.5) into (1, 0): any weight whose least-squares value is smaller in magnitude than lambda gets set to exactly zero. Lasso performed feature selection with nothing but subtract-and-clamp arithmetic.

Now Break It

Try this: With correlated features, lasso arbitrarily picks one and zeroes the rest, which can be unstable.

Control: Lambda slider with correlated features enabled

What happens: Instability! With correlated features, lasso arbitrarily keeps one and drops the others.

Where lasso regression (l1) is used

Lasso regression is prized when you want a model that is not just accurate but also sparse and interpretable, because its L1 penalty drives many coefficients exactly to zero and effectively selects features. In genomics and biomarker discovery, researchers use it to pick a handful of relevant genes from thousands of candidates. Text classification and sentiment analysis rely on it to keep only the most predictive words out of huge vocabularies. Signal processing uses the same sparsity idea in compressed sensing to reconstruct signals from few measurements. Economists and social scientists use lasso to prune large sets of candidate variables down to a defensible short list. Whenever storage, cost, or explanation matters and you suspect only a few inputs truly drive the outcome, lasso regression delivers a compact model.

A key misconception is that the features lasso keeps are the definitively correct ones. When predictors are correlated, lasso tends to arbitrarily pick one and zero out the others, so the selected set can change with small perturbations in the data and should not be read as a stable causal story. Another pitfall is that lasso can be unstable or underperform ridge when features are highly collinear, which is precisely why elastic net was invented. People also forget that lasso, like ridge, requires standardized features so the L1 penalty treats them fairly, and that the penalty strength must be cross-validated. Finally, a zeroed coefficient means unselected under this penalty, not proven irrelevant.

Frequently asked questions

What is lasso regression?

Lasso regression is linear regression with an L1 penalty that adds the sum of the absolute values of the coefficients to the loss. This penalty shrinks coefficients and pushes many of them to exactly zero, so the model both regularizes and performs automatic feature selection. The name stands for least absolute shrinkage and selection operator.

Why does lasso set some coefficients exactly to zero?

The L1 penalty has a diamond-shaped constraint region with sharp corners lying on the axes, and the least squares solution often meets it at one of those corners, where some coefficients are exactly zero. Unlike the smooth L2 penalty, the absolute-value penalty keeps a constant push toward zero regardless of how small a coefficient is. This corner geometry is what produces sparse solutions.

When should I use lasso instead of ridge?

Use lasso when you believe only a small subset of features truly matters and you want the model to identify and keep them while discarding the rest. It produces sparse, interpretable models and is helpful when you have far more features than samples. Prefer ridge when you expect many features to each contribute a little, especially if they are strongly correlated.

What are the limitations of lasso with correlated features?

When several features are highly correlated, lasso tends to select just one of them somewhat arbitrarily and drive the others to zero, which can make the chosen set unstable across samples. It can also select at most as many features as you have samples, a limit in very wide datasets. Elastic net addresses both issues by combining L1 with an L2 term.

Does lasso require feature scaling?

Yes. Because the L1 penalty is applied to coefficient magnitudes, features on larger scales would be penalized less and features on smaller scales more, biasing which ones get zeroed. Standardizing each feature to comparable scale before fitting ensures the penalty treats them fairly. The intercept is typically left unpenalized.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.