Skip to content
ML Visualization

Cross-Validation

Data Prep & Model EvaluationIntermediate~7 min

Cross-Validation — Rotate the holdout set across k folds for a stable estimate.

A single train/test split can be lucky or unlucky. k-fold cross-validation rotates the holdout across k slices of the data and averages, giving a far more stable estimate of performance.

Fold 1 of 5 — 5 held out, 19 trained on
  • Trained on
  • Held out this fold
  • Fold model

Data split into 5 folds — drag across the strip to add or remove folds

val
2
3
4
5

Validation error per fold (hover a bar to find its points)

Cross-validation controls

Data
Shape of the data

A wave a polynomial can genuinely chase — degree and fold count both bite here.

24
1.0×
Model
Strategy
5
4
CV score0.00
Spread (±1 sd)0.00
Playback
Step 0 / 14
Speed
  1. Hold out a fold
  2. Fit on the rest
  3. Score the holdout

Step 0 of 14 — Fold 1 of 5: set aside 5 points

Hover any bar in the fold chart to light up exactly which points that fold held out.

Break it

The idea in plain words

A single train/test split can be lucky or unlucky. k-fold cross-validation rotates the validation set across k slices of the data, scores each, and averages — a far more stable estimate of performance.

You watch the validation fold sweep and the per-fold scores accumulate into a mean ± spread. Too few folds (k = 2) gives a noisy estimate that swings run to run; more folds tighten it, at more compute.

Now, the math

The cross-validation score is the mean of the per-fold errors:

CV=1k∑f=1kEf  ±  std(Ef)\text{CV} = \frac{1}{k}\sum_{f=1}^{k} E_f \;\pm\; \text{std}(E_f)
kk
the number of folds the data is split into.
EfE_f
the error when fold f is the validation set.
▸ Show the derivation

Every point is used for validation exactly once and for training k−1 times, so the estimate uses all the data while never testing on training points. The spread across folds is itself informative — a large std warns that the score is sensitive to which data you held out.

Trace it by hand

The same ten targets y = 2, 4, 4, 6, 6, 8, 8, 10, 10, 12 with k = 5 folds: fold 1 holds out points 1-2, fold 2 points 3-4, and so on to fold 5 with points 9-10. The model again predicts the training mean, and E is the mean squared error on the held-out fold. Std rounded to 1 decimal.

  1. Slice the data into 5 folds

    F1={2,4},F2={4,6},F3={6,8},F4={8,10},F5={10,12}F_1 = \{2, 4\}, \quad F_2 = \{4, 6\}, \quad F_3 = \{6, 8\}, \quad F_4 = \{8, 10\}, \quad F_5 = \{10, 12\}

    Each fold takes a turn as the validation set while the other 8 points train the model.

  2. Score fold 1

    y^=648=8,E1=(2−8)2+(4−8)22=36+162=26\hat{y} = \frac{64}{8} = 8, \qquad E_1 = \frac{(2 - 8)^2 + (4 - 8)^2}{2} = \frac{36 + 16}{2} = 26

    Folds 2 to 5 repeat the same recipe with their own training means of 7.5, 7, 6.5 and 6.

  3. Collect all five fold errors

    E1=26,E2=7.25,E3=1,E4=7.25,E5=26E_1 = 26, \quad E_2 = 7.25, \quad E_3 = 1, \quad E_4 = 7.25, \quad E_5 = 26

    A single split could have reported anything from 1 to 26 for this same model and data.

  4. Average the folds

    CV=15∑f=15Ef=26+7.25+1+7.25+265=13.5\text{CV} = \frac{1}{5} \sum_{f=1}^{5} E_f = \frac{26 + 7.25 + 1 + 7.25 + 26}{5} = 13.5

  5. Measure the spread

    std(Ef)=156.25+39.06+156.25+39.06+156.255≈10.5\text{std}(E_f) = \sqrt{\frac{156.25 + 39.06 + 156.25 + 39.06 + 156.25}{5}} \approx 10.5

    Squared deviations of each fold error from 13.5. The final report is 13.5 plus or minus 10.5.

What just happened: Depending on which two points were held out, a single split scored anywhere from 1 to 26. Five-fold CV distills that into 13.5 plus or minus 10.5 — a fairer average, with a spread that honestly warns how unstable the estimate is on only ten points.

Now Break It

Try this: Too few folds gives a noisy estimate; too many is slow and high-variance per fold.

Control: Number of folds slider

What happens: Noisy estimate! With too few folds the cross-validation score swings wildly run to run.

Where cross-validation is used

Cross-validation gives a more stable estimate of model performance than a single train/test split by rotating the held-out portion across the data. In k-fold cross-validation you divide the data into k equal parts, train on k minus one of them, and test on the remaining fold, repeating until every fold has served as the test set once. Averaging the k scores smooths out the luck of any one split, which is especially valuable on small or medium datasets where a single split can swing widely. It is the standard tool for hyperparameter tuning: grid and random search evaluate each candidate configuration with cross-validation so the chosen settings are not tuned to one arbitrary partition.

The most damaging cross-validation mistake is leakage from preprocessing done outside the loop. If you scale, impute, or select features on the full dataset and then cross-validate, each fold's test data has already influenced the transform, inflating scores. Wrap the entire preprocessing sequence in a pipeline so it is refit inside every fold. Data structure also dictates the scheme: for imbalanced targets use stratified k-fold, for grouped data such as repeated measurements per patient use grouped folds so no individual spans train and test, and for time series use forward-chaining splits rather than shuffled folds. Finally, more folds reduce bias but cost more compute; leave-one-out is thorough but can be noisy and expensive.

Frequently asked questions

How many folds should I use?
Five or ten folds are the common defaults and usually offer a good balance between reliable estimates and reasonable compute. More folds mean each model trains on more data, reducing bias, but they take longer and the test folds get smaller. Choose based on dataset size and how expensive training is.
What is the difference between cross-validation and a train/test split?
A single train/test split evaluates the model once on one held-out portion, which can be noisy. Cross-validation rotates the held-out fold across the whole dataset and averages the results, giving a more stable estimate and using all the data for both training and evaluation across the folds.
What is leave-one-out cross-validation?
Leave-one-out is k-fold taken to the extreme, where k equals the number of samples, so each model is tested on a single example. It uses nearly all the data for training and has low bias, but it is computationally expensive and its estimates can have high variance. It is mainly used for very small datasets.
Do I still need a separate test set if I use cross-validation?
If you use cross-validation to select hyperparameters, its score is optimistic for the chosen model because you picked the best configuration on it. Keeping a final untouched test set, or using nested cross-validation, gives an unbiased estimate of the tuned model's performance.
How do I cross-validate time series data?
Standard shuffled k-fold is wrong for time series because it trains on future data. Use forward-chaining, where each fold trains on all data up to a point and tests on the next block, so the model is always evaluated on the future relative to its training window.
← Back to all visualizationsContinue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated .