Bias–Variance Tradeoff

RegressionIntermediate~8 min

Bias–Variance Tradeoff — Balance underfitting against overfitting.

Every model error splits into two parts: bias (too simple, misses the pattern) and variance (too complex, chases noise). You can’t eliminate both — you can only find the sweet spot between them.

True function
Fitted models
Average model
Your sample & fit

Bias² vs variance across complexity (log scale — click to set the degree)

Bias²
Variance
Total error

Bias–variance controls

Data

Sampling regime

σ = 0.6 labels. Enough scatter that flexible models start chasing it, but the signal is still plainly there.

Label noise σ0.60

Number of resamples25

Model

Model complexitydegree 1

Error decomposition

Bias² 0.49

Variance 0.04

Noise 0.36

Playback

Step 0 / 11

Speed

Bias dominates
Balanced
Variance dominates
↻

Step 0 of 11 — degree 1 over 25 resamples at σ = 0.60 — bias² 0.49, variance 0.04, irreducible noise 0.36, total 0.90; the total bottoms out at degree 3

Break it

The idea in plain words

Refit the same model on many resampled datasets and overlay every fitted curve. A simple model gives a tight bundle that misses the true shape — that consistent offset is bias. A complex model gives curves that scatter wildly around the truth — that spread is variance.

You can’t drive both to zero at once: lowering one usually raises the other. Total error is their sum plus irreducible noise, and the best model sits where the two balance — the quantitative version of overfitting vs underfitting.

Now, the math

Expected prediction error decomposes exactly into three parts:

\mathbb{E}[(y - \hat{f})^2] = \underbrace{(\bar{f} - f)^2}_{\text{Bias}^2} + \underbrace{\mathbb{E}[(\hat{f} - \bar{f})^2]}_{\text{Variance}} + \underbrace{\sigma^2}_{\text{Noise}}

$\bar{f}$: the average prediction across all resampled fits.
$f$: the true underlying function.
$\sigma^2$: irreducible noise — the floor no model can beat.

▸ Show the derivation

Bias² measures how far the average model is from the truth; variance measures how much individual fits jump around that average when the training data changes. Increasing complexity lets the average track the truth (less bias) but makes each fit chase its own sample’s noise (more variance). The slider lets you watch the spaghetti tighten then explode.

Trace it by hand

The repo's decomposition experiment traced with real numbers: resample 50 training sets of 12 points from the true curve f of x equals 2 sin(0.6 x) plus 0.3 x with noise sigma equal to 0.6, fit a degree-d polynomial to each, and average the decomposition over the evaluation grid. Values rounded to 4 decimals.

Step 1 — the floor no model can beat
$\sigma^2 = 0.6^2 = 0.36$
Every total below contains this 0.36 — it is randomness in the data itself, not a modeling error.
Step 2 — degree 1: a tight bundle in the wrong place
$d=1: \quad \underbrace{0.5111}_{\text{Bias}^2} + \underbrace{0.0444}_{\text{Variance}} + \underbrace{0.36}_{\sigma^2} = 0.9156$
All 50 straight lines agree closely with each other — variance is tiny — but they all miss the sine bend by the same amount, so bias dominates.
Step 3 — degree 9: scattered around the truth
$d=9: \quad \underbrace{0.0026}_{\text{Bias}^2} + \underbrace{0.2860}_{\text{Variance}} + \underbrace{0.36}_{\sigma^2} = 0.6487$
The average of the 50 wiggly fits tracks the truth almost perfectly, but each individual fit chases its own sample's noise: bias fell nearly 200-fold while variance grew more than 6-fold.
Step 4 — degree 3 balances the two
$d=3: \quad \underbrace{0.0074}_{\text{Bias}^2} + \underbrace{0.1008}_{\text{Variance}} + \underbrace{0.36}_{\sigma^2} = 0.4682$
Neither term is at its own minimum, but their sum is: 0.4682 beats 0.9156 for degree 1 and 0.6487 for degree 9.
Step 5 — read the tradeoff
$\text{Bias}^2: 0.5111 \to 0.0074 \to 0.0026, \qquad \text{Variance}: 0.0444 \to 0.1008 \to 0.2860$
One complexity knob, two opposite responses — that is why total error is U-shaped and the best model sits in the middle.

What just happened: Complexity moved error between the two buckets: degree 1 pays 0.51 in bias squared, degree 9 pays 0.29 in variance, and degree 3 pays the least combined at 0.4682. The 0.36 noise floor stayed untouched throughout — no model choice can spend it away.

Now Break It

Try this: At maximum complexity, train error hits zero but test error explodes — the classic overfitting gap.

Control: Complexity slider (push to either extreme)

What happens: The generalization gap! Train error is near zero but test error is huge — high variance overfitting.

Where bias–variance tradeoff is used

The bias-variance tradeoff is the conceptual backbone of every model-selection decision a practitioner makes, so it shows up everywhere rather than in one application. When a data science team tunes the depth of a decision tree, the number of neighbors in k-nearest neighbors, or the strength of a regularization penalty, they are navigating this tradeoff explicitly. In forecasting demand for retail inventory, an overly simple model misses seasonal swings (high bias), while an overly flexible one memorizes past noise and fails next quarter (high variance). Understanding it guides how much data to collect, whether to add features, and when to use ensembles. Techniques like bagging reduce variance and boosting reduces bias, and knowing which lever to pull is what the bias-variance tradeoff teaches.

A frequent misconception is that bias and variance are properties you can read off a single trained model; in fact they describe how predictions behave on average across many datasets drawn from the same source, which is why they are estimated with resampling like cross-validation. Another pitfall is believing you must always accept a strict tradeoff where reducing one raises the other. Adding more training data, better features, or ensemble methods can lower variance without increasing bias, shifting the whole error curve down. Finally, people conflate underfitting and overfitting with training accuracy alone; the reliable signal is the gap between training and validation error, not either number in isolation.

Frequently asked questions

What is the bias-variance tradeoff?

It describes how a model's expected error splits into bias, the error from wrong simplifying assumptions, and variance, the error from sensitivity to the particular training sample. Simple models tend to have high bias and low variance, while flexible models have low bias and high variance. The goal is to balance the two so that total error on new data is smallest.

What is the difference between bias and variance?

Bias is systematic error: the model consistently misses the true pattern because it is too simple, causing underfitting. Variance is instability: the model changes a lot when trained on different samples because it is too flexible, causing overfitting. Bias shows up as poor performance everywhere, while variance shows up as a large gap between training and test performance.

How do I tell if my model has high bias or high variance?

Compare training and validation error. High training error that stays close to validation error signals high bias and underfitting. Low training error with much higher validation error signals high variance and overfitting. Learning curves that plot both errors against training set size make this diagnosis clear.

How can I reduce variance without increasing bias?

Collect more training data, since a flexible model stabilizes as it sees more examples. Ensemble methods like bagging and random forests average many models to cut variance while keeping bias roughly the same. Better features and mild regularization can also help lower one component without badly inflating the other.

Does irreducible error fit into the tradeoff?

Yes. Total expected error is bias squared plus variance plus irreducible error, the noise inherent in the data that no model can remove. You can trade bias against variance, but the irreducible part sets a floor on how accurate any model can be. Recognizing it prevents chasing impossible accuracy targets.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.