Skip to content
ML Visualization

Train/Test Split

Data Prep & Model EvaluationBeginner~5 min

Train/Test Split — Hold out data to measure real generalization.

If you grade a model on the same data it studied, of course it aces the test. A train/test split holds out unseen data so you measure how well the model actually generalizes.

Drag the divider to set the train / test split (17 train · 11 test)

  • Train
  • Test
  • Model
  • Not dealt yet
Train and test error as points are dealt
  • Train error
  • Test error

Split controls

Data
Shape of the data

A wave the polynomial can genuinely chase, so degree and split interact.

28
1.0×
Model
60% / 40%
Stratify on the target
5
Train error0.00
Test error0.00
Playback
Step 0 / 28
Speed
  1. Deal a point
  2. Fit on train only
  3. Score the holdout

Step 0 of 28 — 28 points on the table, none dealt yet — target split 60% train

Break it

The model fits on train only; test points reveal the honest error. Turn on leakage — include the test points in training — and the test score becomes a lie.

The idea in plain words

If you grade a model on the same data it studied, of course it aces the test. A train/test split holds out unseen data so you can measure how well the model actually generalizes — the honest number that overfitting would otherwise hide.

The catch is leakage: accidentally letting test data into training makes the test score a lie. And if the training set is tiny, the fit becomes unstable and changes every reshuffle.

Now, the math

Generalization is judged on the held-out test set, not the training set:

Etest=1∣T∣∑i∈T(yi−y^i)2E_{\text{test}} = \tfrac{1}{|T|}\sum_{i \in T} (y_i - \hat{y}_i)^2
TT
the test set — never touched during fitting.
y^i\hat{y}_i
the model’s prediction, trained only on the train split.
▸ Show the derivation

A fair test error estimates performance on future data only if the test set is truly independent. Leakage (fitting on test points, or letting information cross over) collapses that gap and yields an over-optimistic score that won’t hold up in production.

Trace it by hand

Ten targets y = 2, 4, 4, 6, 6, 8, 8, 10, 10, 12 and the simplest model there is: predict the mean of the training set. We hold out 2 of the 10 points (an 80/20 split) and compare an ordered split against a shuffled one. All arithmetic is exact.

  1. Fit on the training split

    y^=2+4+4+6+6+8+8+108=488=6\hat{y} = \frac{2 + 4 + 4 + 6 + 6 + 8 + 8 + 10}{8} = \frac{48}{8} = 6

    Ordered split: the test set T is the last two points, y equals 10 and 12. The model never sees them.

  2. Score on the training set

    Etrain=16+4+4+0+0+4+4+168=6E_{\text{train}} = \frac{16 + 4 + 4 + 0 + 0 + 4 + 4 + 16}{8} = 6

  3. Score on the held-out test set

    Etest=(10−6)2+(12−6)22=16+362=26E_{\text{test}} = \frac{(10 - 6)^2 + (12 - 6)^2}{2} = \frac{16 + 36}{2} = 26

    More than four times the training error — holding out the two largest values made the split unrepresentative.

  4. Shuffle before splitting

    y^=568=7,Etest=(4−7)2+(10−7)22=9+92=9\hat{y} = \frac{56}{8} = 7, \qquad E_{\text{test}} = \frac{(4 - 7)^2 + (10 - 7)^2}{2} = \frac{9 + 9}{2} = 9

    A shuffled test set, y equals 4 and 10, spans the range instead of sitting at one end.

What just happened: Same ten points, same model — but the ordered split reported a test error of 26 while a shuffled split reported 9 (against a training error of 6). Which points land in the test set changes the verdict, which is why splits are shuffled and why cross-validation averages several of them.

Now Break It

Try this: Leaking test data into training (or too small a test set) makes the score meaningless.

Control: Test-set size slider (set to near zero)

What happens: Unreliable estimate! A tiny test set gives a noisy score you can’t trust.

Where train/test split is used

A train/test split measures how well a model generalizes to data it has never seen. You train on one portion and evaluate on a held-out portion that the model never touched during fitting, giving an honest estimate of real-world performance. This directly exposes overfitting: a model that memorizes noise scores near-perfectly on training data but poorly on the test set. For time series such as stock prices or demand forecasting, a random split is wrong because it lets the model peek at the future to predict the past; instead you split chronologically, training on earlier data and testing on later data. For imbalanced problems like fraud detection, stratified splitting preserves the rare-class proportion in both halves.

A subtle but serious pitfall is tuning hyperparameters against the test set. Every time you look at the test score and adjust the model, information leaks and the estimate becomes optimistic; the test set should be touched once, at the very end. The fix is a three-way split into train, validation, and test, or cross-validation on the training portion. Another common error is fitting any preprocessing, such as scalers, imputers, or feature selectors, on the combined data before splitting, which contaminates the test set. Grouped data also needs care: if the same patient or user appears in both splits, the model can cheat by recognizing the individual rather than learning the underlying pattern.

Frequently asked questions

What is a good train/test split ratio?
Common choices are 80/20 or 70/30, but there is no universal rule. Larger datasets can afford a smaller test fraction because even a small percentage is many examples, while very small datasets often benefit from cross-validation instead of a single split to get a more stable estimate.
Why do I need a validation set if I already have a test set?
The validation set is used to tune hyperparameters and compare models, while the test set is reserved for a single final evaluation. If you tune against the test set, its score becomes optimistic because information about it has leaked into your choices. Keeping them separate preserves an honest final estimate.
How should I split time series data?
Never split time series randomly, because that lets the model train on future observations to predict the past. Split chronologically so training data comes before the test period, mirroring how the model will be used in production. Rolling or expanding window schemes extend this idea for cross-validation.
What is stratified splitting and when do I need it?
Stratified splitting preserves the class proportions of the whole dataset in both the training and test sets. It is important for imbalanced classification, where a random split might leave very few or zero rare-class examples in one part, making evaluation unreliable.
Should I set a random seed when splitting?
Yes, fixing the random seed makes the split reproducible so you and others get the same partition every run. That said, results from a single seed can vary, so for small datasets it is wise to also check stability across multiple seeds or use cross-validation.
← Back to all visualizationsContinue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated .