Polynomial Regression

RegressionBeginner~6 min

Polynomial Regression — Fit curves by adding polynomial features.

What if your data curves? Polynomial regression bends the line into a curve by feeding the model powers of the input. But bend it too far and it starts chasing noise.

Training data
Polynomial fit
True function
Extrapolation zone

Error vs degree (log scale — click to set the degree)

Polynomial controls

Data

Shape of the data

A genuine bend in the signal — degree 1 cannot reach it, and degree 3 or 4 nails it.

Training points n14

Noise level1.00×

Held-out test points

Model

Polynomial degree1

Train error0.716

Test error1.869

Drag any training point on the plot — every degree refits, and the U below moves with it.

Playback

Step 0 / 14

Speed

Too stiff
About right
Chasing noise
↻

Step 0 of 14 — degree 1 on 14 points — train error 0.716, test error 1.869; the test-error minimum on this sample sits at degree 5

Break it

The idea in plain words

A straight line can only go straight. Polynomial regression bends it into a curve by feeding the model extra features — the input squared, cubed, and so on — then fitting a linear combination of them. The degree controls how many wiggles the curve is allowed.

Raise the degree and the curve threads the training points ever more tightly, but between and especially beyond them it can rocket to absurd values. That gap between fitting the data and fitting the world is overfitting, and it motivates the bias–variance tradeoff.

Now, the math

The model is linear in the coefficients but polynomial in the input:

\hat{y} = w_0 + w_1 x + w_2 x^2 + \cdots + w_d x^d

Stacking the powers of each x into a design matrix, the least-squares fit is:

\mathbf{w} = (X^\top X)^{-1} X^\top \mathbf{y}

$d$: the degree — the highest power of x, set by the slider.
$w_j$: the coefficient on the j-th power of x.
$X$: the Vandermonde design matrix of powers of each input.

▸ Show the derivation

The columns of powers of x become nearly parallel at high degree, so X^⊤X is badly conditioned — the fit is computed on a normalized input to stay numerically stable. Even so, a degree-15 polynomial through 10 points interpolates every one exactly yet oscillates wildly in between, the Runge phenomenon you can trigger with the degree slider.

Trace it by hand

Four points that lie exactly on a parabola: (0, 1), (1, 3), (2, 7), (3, 13), generated by y equals 1 plus x plus x squared. We fit degree 1 and degree 2 with the same least-squares machinery — the repo's normal-equations solver — and let the numbers expose the difference.

Step 1 — the best straight line falls short
$d=1: \quad \hat{y} = w_0 + w_1 x = 0 + 4x, \qquad \text{MSE} = \tfrac{1}{4}\left(1^2 + (-1)^2 + (-1)^2 + 1^2\right) = 1$
The line's residuals are plus 1, minus 1, minus 1, plus 1 — an up-down-down-up pattern no straight line can remove.
Step 2 — add an x-squared column to the design matrix
$X = \begin{bmatrix} 1 & 0 & 0 \\ 1 & 1 & 1 \\ 1 & 2 & 4 \\ 1 & 3 & 9 \end{bmatrix}, \qquad X^\top X = \begin{bmatrix} 4 & 6 & 14 \\ 6 & 14 & 36 \\ 14 & 36 & 98 \end{bmatrix}, \qquad X^\top \mathbf{y} = \begin{bmatrix} 24 \\ 56 \\ 148 \end{bmatrix}$
Each row of X holds one point's powers: 1, x, x squared. The model stays linear in the coefficients — only the features changed.
Step 3 — solve the normal equations
$\mathbf{w} = (X^\top X)^{-1} X^\top \mathbf{y} = (1,\ 1,\ 1) \;\Rightarrow\; \hat{y} = 1 + x + x^2$
The solver returns 1.000, 1.000, 1.000 to three decimals — it recovered the exact parabola the data came from.
Step 4 — compare the errors
$\text{MSE}_{d=1} = 1, \qquad \text{MSE}_{d=2} \approx 0$
The degree-2 MSE prints as 3.5 times 10 to the minus 30 — zero up to floating-point dust. One extra feature removed all the error because the truth really was quadratic.

What just happened: The same normal equations fit both models; the extra x-squared column dropped the MSE from 1 to 0. The catch: with only 4 points a degree-3 fit would also reach zero training error, so a perfect fit alone never proves you found the true curve.

Now Break It

Try this: Degree 15 wiggles through every point perfectly on train data but oscillates wildly between them.

Control: Degree slider (set to maximum)

What happens: Overfitting! The high-degree curve threads every training point but wiggles wildly — it memorized noise.

Where polynomial regression is used

Polynomial regression extends the linear model by adding powers of the inputs, letting a single feature bend to capture curved trends while still using ordinary least squares under the hood. Physicists and engineers use it to fit calibration curves for sensors whose response is not perfectly linear, and chemists model reaction yield as a function of temperature where the relationship peaks and then falls. In economics, quadratic terms capture diminishing returns, such as how output rises with labor but eventually plateaus. Growth curves in biology, dose-response modeling in pharmacology, and trajectory smoothing in sports analytics all lean on polynomial terms. Because it reuses the linear regression machinery, polynomial regression is a quick, transparent way to add flexibility before reaching for more complex nonlinear models.

The biggest misconception is that a higher-degree polynomial is always better because it fits the training data more closely. In truth, high-degree polynomial regression overfits: it chases noise, wiggles wildly between points, and behaves erratically at the edges of the data, a problem known as Runge's phenomenon. The correction is to keep the degree low, validate on held-out data, or use splines that fit local pieces instead. A second pitfall is numerical instability, because raw powers of large inputs produce highly correlated, huge-magnitude columns; centering and scaling the features, or using orthogonal polynomials, keeps the fit well conditioned and the coefficients meaningful.

Frequently asked questions

What is polynomial regression?

Polynomial regression models the target as a weighted sum of a feature and its powers, such as the value, its square, and its cube. This lets the fitted curve bend to follow nonlinear patterns while still being solved as a linear regression in the expanded features. The model is linear in its coefficients even though it produces a curved prediction.

Is polynomial regression still a linear model?

Yes, it is linear in the parameters, which is what matters mathematically. You create new columns like x squared and x cubed, then fit them exactly as you would any linear regression. The curve you see is nonlinear in the original input, but the estimation problem remains a standard least squares fit.

How do I choose the polynomial degree?

Pick the degree using cross-validation rather than by looking at training error, which always drops as the degree rises. Plot validation error against degree and choose the point where it stops improving or begins to climb. In practice degrees of two or three suffice for most smooth relationships, and very high degrees usually signal overfitting.

Why does a high-degree polynomial overfit?

A high-degree polynomial has enough flexibility to pass near every training point, so it starts modeling random noise instead of the underlying trend. This makes it swing sharply between points and especially at the ends of the data range, hurting predictions on new data. Regularization, a lower degree, or splines reduce this instability.

When should I use splines instead of polynomial regression?

Use splines when a single global polynomial cannot capture a relationship that changes shape across the input range without a very high degree. Splines fit smooth low-degree pieces joined at knots, giving local flexibility while avoiding the wild edge behavior of high-degree polynomials. They are a better default for flexible, stable nonlinear curve fitting.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.