Bias–Variance Tradeoff
Bias–Variance Tradeoff — Balance underfitting against overfitting.
Every model error splits into two parts: bias (too simple, misses the pattern) and variance (too complex, chases noise). You can’t eliminate both — you can only find the sweet spot between them.
- True function
- Fitted models
- Average model
- Your sample & fit
The idea in plain words
Refit the same model on many resampled datasets and overlay every fitted curve. A simple model gives a tight bundle that misses the true shape — that consistent offset is bias. A complex model gives curves that scatter wildly around the truth — that spread is variance.
You can’t drive both to zero at once: lowering one usually raises the other. Total error is their sum plus irreducible noise, and the best model sits where the two balance — the quantitative version of overfitting vs underfitting.
Now, the math
Expected prediction error decomposes exactly into three parts:
- the average prediction across all resampled fits.
- the true underlying function.
- irreducible noise — the floor no model can beat.
▸ Show the derivation
Bias² measures how far the average model is from the truth; variance measures how much individual fits jump around that average when the training data changes. Increasing complexity lets the average track the truth (less bias) but makes each fit chase its own sample’s noise (more variance). The slider lets you watch the spaghetti tighten then explode.
Now Break It
Try this: At maximum complexity, train error hits zero but test error explodes — the classic overfitting gap.
Control: Complexity slider (push to either extreme)
Last updated .