Linear Regression

RegressionBeginner~7 min

Linear Regression — Linear regression is a supervised learning algorithm that fits the best straight line through data to predict a continuous value, choosing the slope and intercept that minimize the squared error (ordinary least squares).

Linear regression finds the single straight line that gets as close as possible to all your data points at once. It’s the simplest model — and often the first one you should try.

Data points
Fitted line
Residuals
Outlier / current r²

Residual per point — click a bar to step there

Regression controls

Data

Shape of the data

A straight signal with even noise — exactly the shape a line is built for.

Sample size n24

Noise level1.00×

Model

Slope0.50

Intercept3.00

Mean squared error0.665

Least-squares slope0.700

Playback

Step 0 / 24

Speed

Place the line
Measure one residual
Total the squares
↻

Step 0 of 24 — line placed at slope 0.50, intercept 3.00 — nothing measured yet, so the error total is 0

Break it

Fit without it: slope 0.700. Fit with everything: slope 0.700.

The idea in plain words

Linear regression draws the one straight line that sits closest to all the points at once. “Closest” has a precise meaning: measure each point’s vertical distance to the line (its residual), square those distances so big misses count more, and pick the line that makes the total as small as possible.

Because every point pulls on that total, a single far-out leverage point can drag the whole line toward it — the same outlier sensitivity you can feel in loss functions. When there’s no closed form, the line is found with gradient descent.

Now, the math

The model is a straight line:

\hat{y} = \beta_1 x + \beta_0

It minimizes the mean squared error, which has a closed-form slope:

\beta_1 = \frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2}

$\hat{y}$: the predicted value (a point on the line).
$\beta_1$: the slope — how much ŷ changes per unit of x.
$\beta_0$: the intercept — the prediction when x = 0.
$\bar{x},\ \bar{y}$: the mean of the inputs and outputs.

▸ Show the derivation

Setting the derivative of MSE with respect to each β to zero yields the “normal equations,” whose solution is the closed form above. Because it depends on sums over every point, one extreme leverage point can shift the slope substantially — the fragility you can trigger by adding an outlier.

Trace it by hand

Four points: (1, 2), (2, 3), (3, 5), (4, 6). We fit the least-squares line with nothing but sums — every number below is exact, no rounding needed.

Step 1 — find the means
$\bar{x} = \frac{1+2+3+4}{4} = 2.5, \qquad \bar{y} = \frac{2+3+5+6}{4} = 4$
Step 2 — accumulate the slope's numerator and denominator
$\sum_i (x_i-\bar{x})(y_i-\bar{y}) = 3 + 0.5 + 0.5 + 3 = 7, \qquad \sum_i (x_i-\bar{x})^2 = 2.25 + 0.25 + 0.25 + 2.25 = 5$
Each point contributes one product to each sum: the first point gives (1 minus 2.5) times (2 minus 4), which is 3, and so on.
Step 3 — slope and intercept
$\beta_1 = \frac{7}{5} = 1.4, \qquad \beta_0 = \bar{y} - \beta_1\bar{x} = 4 - 1.4 \times 2.5 = 0.5$
Solving the full normal equations with the repo's Cholesky solver returns exactly the same pair: intercept 0.5, slope 1.4.
Step 4 — check the residuals
$\hat{y} = 1.4x + 0.5: \quad (1.9,\ 3.3,\ 4.7,\ 6.1), \qquad \text{MSE} = \tfrac{1}{4}\left(0.1^2 + 0.3^2 + 0.3^2 + 0.1^2\right) = 0.05$
The residuals are 0.1, minus 0.3, 0.3, minus 0.1 — they sum to zero, which is always true once an intercept is fit.
Step 5 — use the line
$\hat{y}(5) = 1.4 \times 5 + 0.5 = 7.5$

What just happened: Two sums fully determined the line: slope 7 over 5 equals 1.4 and intercept 0.5. No other line can beat its MSE of 0.05 on these four points — that is exactly what least squares means.

Now Break It

Try this: Add one extreme leverage outlier — it drags the entire regression line off course.

Control: Click to add a point far from the cluster

What happens: Leverage point detected! One outlier is pulling the entire line toward it, ruining the fit for everyone else.

Where linear regression is used

Linear regression is the workhorse behind an enormous range of practical predictions because it is fast, interpretable, and easy to audit. Epidemiologists use it to relate exposures like air pollution or smoking to health outcomes while adjusting for confounders such as age and income. Economists estimate how consumer spending responds to changes in interest rates, and real-estate platforms price homes from square footage, location, and bedroom count. In manufacturing, engineers fit sensor readings to product quality to catch drift on a production line. Marketing teams use it to attribute sales lift to advertising spend across channels. Because each coefficient has a clear per-unit meaning, linear regression is often preferred in regulated fields like finance and medicine where decisions must be explained.

Two misconceptions trip up beginners. First, people assume a high correlation or a good fit proves causation; in reality linear regression only quantifies association, and a strong coefficient can vanish once you add a confounding variable, so causal claims require experimental design or careful causal methods. Second, many think linear regression can only model straight lines. The model is linear in its parameters, not necessarily in the raw inputs, so you can fit curves by adding transformed features like logarithms or interaction terms. A common pitfall is ignoring the assumptions: influential outliers, correlated errors, and non-constant variance can all distort the fit, so residual plots should always be inspected before trusting the coefficients.

Frequently asked questions

What is linear regression in simple terms?

Linear regression finds the straight line, or flat plane in higher dimensions, that best predicts a numeric target from one or more input features. It does this by choosing coefficients that minimize the sum of squared differences between predicted and actual values. Each coefficient tells you how much the prediction changes when its feature increases by one unit, holding the others fixed.

What does ordinary least squares actually minimize?

Ordinary least squares minimizes the sum of the squared vertical distances between each data point and the fitted line, called the residuals. Squaring penalizes large errors more heavily and produces a single closed-form solution. Because errors are squared, least squares is sensitive to outliers, which can pull the line noticeably.

What assumptions does linear regression make?

The classic assumptions are a linear relationship between features and the target, independent errors, constant error variance (homoscedasticity), and approximately normal residuals for valid confidence intervals. It also assumes features are not perfectly collinear. Violations do not always ruin predictions, but they can make coefficient estimates and their uncertainty misleading.

How do I interpret a regression coefficient?

A coefficient is the expected change in the target for a one-unit increase in that feature, while all other features are held constant. The intercept is the predicted value when every feature equals zero, which may or may not be meaningful. Comparing coefficient magnitudes only makes sense if features are on comparable scales, so standardizing inputs first helps.

When should I not use linear regression?

Avoid plain linear regression when the true relationship is strongly nonlinear and cannot be fixed with feature transformations, or when the target is a category or count rather than a continuous number. Heavy outliers, many correlated predictors, or non-constant variance also call for robust or regularized alternatives. In those cases models like logistic regression, tree ensembles, or regularized regression usually work better.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.