Overfitting & Underfitting

FoundationsBeginner~6 min

Overfitting & Underfitting — Underfitting is when a model is too simple to capture the pattern (high error everywhere); overfitting is when it is so flexible it memorizes noise (low train error, high test error). The sweet spot minimizes test error.

Slide a model from a flat line to a deranged wiggle and watch two curves: training error keeps falling, but test error dips then rises — the single most important picture in machine learning, drawn by your own hand.

Train points
Test points
Model
Truth

Train vs test error — the U (click a degree to jump there)

Complexity controls

Data

Shape of the signal

A smooth bend. Low degrees genuinely cannot reach it, so the U has a real left arm.

Training points10

Noise1.0×

Model

Model complexity — Balanceddegree 1

Train error0.577

Test error5.619

Playback

Step 0 / 11

Speed

Underfit
Balanced
Overfit
↻

Step 0 of 11 — degree 1 on 10 training points — train error 0.577, test error 5.619, gap 5.042 (test error bottoms out at degree 1)

Break it

The idea in plain words

Slide the model from a flat line to a wild wiggle and watch two numbers. Training error only ever falls — a flexible model can always hug the points it has seen. Test error, measured on points it hasn’t, dips and then rises. Too simple underfits; too complex overfits.

The bottom of that test-error U is the sweet spot. Overshoot it and the model is memorizing noise, not the pattern — the same trap you can trigger with polynomial degree, and the reason for the bias–variance tradeoff.

Now, the math

Generalization is measured by the gap between two errors:

\text{gap} = E_{\text{test}} - E_{\text{train}}

$E_{\text{train}}$: error on the data the model was fit to — falls with complexity.
$E_{\text{test}}$: error on held-out data — dips then rises, forming the U.

▸ Show the derivation

A model with enough parameters can drive training error to zero by interpolating every point, but those extra degrees of freedom fit the random noise in the sample. On fresh data that noise is different, so the wiggles that helped on the training set now hurt — test error climbs even as training error keeps falling.

Trace it by hand

Using the repo's own polynomial demo data: 10 noisy training points sampled from the curve 2 sin(0.6 x) + 0.3 x, plus 40 independently-noised test points. We fit polynomials of degree 1, 4, and 9 by least squares and report each MSE, rounded to 2 decimals.

Degree 1: too simple
$d = 1: \quad E_{\text{train}} = 0.82, \qquad E_{\text{test}} = 0.66$
A straight line cannot follow the sine wave, so it misses train and test data alike. Test even lands slightly below train here, pure luck of the noise.
Degree 4: the sweet spot
$d = 4: \quad E_{\text{train}} = 0.08, \qquad E_{\text{test}} = 0.48, \qquad \text{gap} = 0.40$
Enough bend to capture the true curve; test error is at its lowest of the three fits.
Degree 9: memorize the sample
$d = 9: \quad E_{\text{train}} \approx 0, \qquad E_{\text{test}} = 9.90, \qquad \text{gap} = 9.90$
Ten coefficients through ten points interpolates them exactly — the computed training MSE is about 5e-21, pure floating-point dust.
The two curves diverge
$E_{\text{train}}: 0.82 \to 0.08 \to 0 \qquad E_{\text{test}}: 0.66 \to 0.48 \to 9.90$
Training error is monotone in complexity; test error traces the U whose bottom is the model you actually want.

What just happened: The degree-9 fit scores a perfect 0 on the points it saw and 9.90 on points it did not — 20 times worse than degree 4's 0.48. Training error alone rewarded the memorizer; only the held-out gap exposed it.

Now Break It

Try this: Max out complexity — train error hits zero while test error explodes; resample and the overfit curve flails.

Control: Complexity slider (set to maximum), then resample

What happens: Overfitting! Train error is near zero but test error has exploded — the model memorized the noise.

Where overfitting & underfitting is used

Overfitting and underfitting describe the two ways a model can miss the mark, and both appear constantly in practice. An underfit model is too simple to capture the pattern: a straight line trying to fit clearly curved sales data, or a tiny network that cannot learn to read handwritten digits, shows high error on both training and test data. An overfit model instead memorizes noise: a deep decision tree that perfectly classifies every training email but flags legitimate messages as spam, or a stock-prediction model that fits historical wiggles that never repeat. Teams fighting overfitting reach for regularization, dropout, early stopping, cross-validation, and simply gathering more data. The classic U-shaped curve, where test error falls then rises as complexity grows, guides the search for the sweet spot in between.

A common misconception is that low training error means a good model. Training error near zero often signals overfitting, not success, because the model has learned quirks of the training set that will not generalize; the honest measure is error on held-out data. A second pitfall is treating more complexity or more training as always better. Past a point, extra capacity or extra epochs push a model from the healthy region into overfitting, which is exactly why early stopping and validation curves matter. It also helps to remember that overfitting and underfitting sit on the bias-variance tradeoff: underfitting is high bias, overfitting is high variance, and good generalization balances the two rather than eliminating either alone.

Frequently asked questions

What is the difference between overfitting and underfitting?

Underfitting happens when a model is too simple to capture the underlying pattern, so it performs poorly on both training and test data. Overfitting happens when a model is too complex and learns noise specific to the training set, so it does well on training data but poorly on new data. The goal is the middle ground that generalizes well.

How do I know if my model is overfitting?

Compare performance on training data with performance on a held-out validation or test set. If training error is low but validation error is much higher, the model is overfitting. Watching the two curves during training and stopping when validation error starts rising is a reliable way to catch it.

How can I prevent overfitting?

Common remedies include gathering more or more varied training data, simplifying the model, and adding regularization such as L2 penalties or dropout. Early stopping halts training before the model starts memorizing noise, and cross-validation gives a more honest estimate of generalization. Data augmentation is especially effective for images and audio.

How do I fix underfitting?

Underfitting means the model lacks the capacity or training to capture the pattern. Fixes include using a more expressive model, adding useful features, reducing excessive regularization, and training longer. If both training and test error remain high, more capacity or better features is usually the answer rather than more regularization.

What is the bias-variance tradeoff?

Bias is error from overly simple assumptions that cause underfitting, while variance is error from excessive sensitivity to the training data that causes overfitting. Reducing one often increases the other, so the tradeoff is about balancing them. The best generalization comes from the sweet spot where total error, the sum of bias and variance effects, is lowest.

Does low training error mean my model is good?

Not by itself. Very low training error can be a warning sign of overfitting, because the model may have memorized noise rather than learned a general rule. The meaningful measure of quality is performance on data the model has not seen. Always evaluate on a held-out test set before trusting the results.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.