Random Forest

EnsemblesIntermediate~7 min

Random Forest — Bag decision trees with random feature subsets.

A random forest takes bagging one step further: each tree not only sees a different sample of the data but also a random subset of features at each split. This decorrelates the trees and makes the forest even more robust.

Tree 1 on its own — sample only

Forest vote — 0 trees

Class 0
Class 1
Out-of-bag (hollow)

Out-of-bag error % vs iteration

Tree disagreement % vs iteration

One point per tree added. Disagreement is the average vote split across the plane — 0% means every tree already says the same thing everywhere.

Forest controls

Data

Dataset

Points per class20

Spread1.0×

Add points as

Model

Features per split

Number of trees18

Max depth per tree8

Bootstrap seed11

Playback

Step 0 / 35

Speed

Resample
Grow + vote
↻

Step 0 of 35 — tree 1 — resampled 40 points with replacement, 14 left out-of-bag; every split may look at 1 of the 2 features

Hover a point on the left plot for its draw count and the forest’s current vote on it. Drag a point on the right plot, or click empty space to drop a new one, and the forest retrains.

Break it

The idea in plain words

A random forest takes bagging one step further: each tree not only sees a different resample of the data but also considers only a random subset of features at each split. This decorrelates the trees, so averaging them helps far more.

Out-of-bag error — scoring each point using only the trees that didn’t train on it — gives a free validation estimate. If you let every split see all features, the trees become near-identical and the ensemble stops improving.

Now, the math

Averaging correlated trees only reduces variance so far:

\text{Var} = \rho\,\sigma^2 + \frac{1-\rho}{B}\sigma^2

$\rho$: the correlation between trees — feature subsampling lowers it.
$\sigma^2$: the variance of a single tree.

▸ Show the derivation

As B → ∞ the second term vanishes but the first, ρσ², remains — so the only way to keep reducing variance is to lower ρ. Restricting each split to a random feature subset does exactly that, at the cost of a little more bias per tree. Using all features sends ρ → 1 and erases the benefit.

Trace it by hand

Let us put real numbers into the variance formula. Dataset: 16 points in two overlapping blobs (class 0 around (3.5, 3.5), class 1 around (6.5, 6.5), spread 1.6), depth-3 trees, and the test point (5, 5) halfway between the blobs. Over 2000 seeded replications (mulberry32) we record each tree's 0-or-1 vote at (5, 5) and measure sigma squared and rho; estimates are rounded to 2 decimals before plugging in.

Step 1 — measure one tree, then a bagged pair
$\sigma^2 = 0.25, \qquad \rho_{\text{bag}} = 0.31$
A lone tree's vote at (5, 5) flips between 0 and 1 from sample to sample, so its variance sits at the maximum 0.25. Bagged trees may use both features at every split, and their votes correlate at 0.31.
Step 2 — variance of a 3-tree bagged average
$\text{Var} = \rho\,\sigma^2 + \frac{1-\rho}{B}\sigma^2 = 0.31 \times 0.25 + \frac{0.69}{3} \times 0.25 = 0.0775 + 0.0575 = 0.135$
Averaging already halves the single-tree variance of 0.25 — but notice the 0.0775 term does not shrink with B.
Step 3 — feature subsampling decorrelates the trees
$\rho_{\text{RF}} = 0.22 \;\Rightarrow\; \text{Var} = 0.22 \times 0.25 + \frac{0.78}{3} \times 0.25 = 0.055 + 0.065 = 0.12$
Forest trees see only 1 of the 2 features at each split, so trees grown on the same data disagree more: sigma squared stayed at 0.25 while rho fell from 0.31 to 0.22.
Step 4 — the floor that more trees cannot break
$B \to \infty: \quad \text{Var} \to \rho\,\sigma^2 = \begin{cases} 0.31 \times 0.25 = 0.0775 & \text{bagging} \\ 0.22 \times 0.25 = 0.055 & \text{forest} \end{cases}$
Adding trees kills the second term only. The correlated part survives forever, so the only way to lower the floor is to lower rho — exactly what feature subsampling does.
Step 5 — the same story in three actual votes
$\text{bagged votes} = (1, 1, 1), \qquad \text{forest votes} = (1, 1, 0) \;\Rightarrow\; \bar{T} = \tfrac{2}{3}$
One seeded run (dataset seed 10007, ensemble seed 42): the bagged trio moves in lockstep, so averaging clones buys little; the forest trio disagrees at (5, 5) — that visible diversity is what the smaller rho was measuring.

What just happened: With sigma squared pinned at 0.25, cutting the tree correlation from 0.31 to 0.22 lowered the 3-tree variance from 0.135 to 0.12 and the infinite-tree floor from 0.0775 to 0.055 — feature subsampling, not more trees, is what moved the floor.

Now Break It

Try this: Using all features per split makes every tree nearly identical, defeating the ensemble.

Control: Features-per-split slider (set to all)

What happens: Trees too correlated! Using every feature per split makes all trees alike — no diversity gain.

Where random forest is used

Random forests are a workhorse for tabular data across industry. Banks use them for default and churn prediction, insurers for claim triage, and biologists for gene-expression classification, partly because they handle mixed numeric and categorical features with minimal preprocessing and no scaling. Their built-in feature-importance scores make them a popular first pass for understanding which variables matter before a more tuned model is built. In fraud and recommendation pipelines they serve as strong baselines and as feature generators. Because trees train independently, a random forest scales across cores easily, and the out-of-bag error gives a quick internal estimate of accuracy. They are often the default answer when someone wants a good tabular model with little tuning effort.

A common misconception is that random forests cannot overfit. They can: with very deep trees on small or noisy datasets they will memorize, and adding more trees does not fix that, it only stabilizes the averaged prediction. More trees never hurt accuracy but do cost time and memory, so the number of trees is a compute knob, not a regularizer. The real regularization comes from tree depth, minimum leaf size, and the number of features sampled per split. Another pitfall is trusting the default impurity-based feature importances, which are biased toward high-cardinality and continuous features; permutation importance on held-out data is more reliable. Finally, forests extrapolate poorly, predicting flat values outside the range of the training targets.

Frequently asked questions

What is a random forest?

A random forest is a bagging ensemble of decision trees with an extra twist: at each split, the tree considers only a random subset of the features rather than all of them. Each tree trains on a bootstrap sample of the rows, and predictions are combined by voting for classification or averaging for regression. The feature sampling decorrelates the trees, which is what makes the ensemble strong.

How is a random forest different from plain bagging of trees?

Plain bagging lets every tree consider all features at each split, so a few dominant features get chosen repeatedly and the trees end up correlated. A random forest restricts each split to a random feature subset, forcing trees to explore different structure and become more independent. That extra decorrelation reduces the ensemble variance more than bagging alone.

Do more trees cause a random forest to overfit?

No. Adding trees only refines the averaged prediction and makes it more stable; the error curve flattens rather than getting worse. Overfitting in a random forest comes from individual trees being too deep or leaves too small on noisy data, not from the count of trees. Treat the number of trees as a speed-versus-stability trade-off.

How does a random forest measure feature importance?

The default method sums how much each feature reduces impurity across all splits that use it, but this is biased toward continuous and high-cardinality features. A more trustworthy alternative is permutation importance, which shuffles one feature at a time and measures how much accuracy drops on held-out data. Prefer permutation importance when the ranking matters for decisions.

When should I choose a random forest?

Random forests are a strong default for medium-sized tabular problems where you want good accuracy with little tuning and some interpretability. They handle mixed feature types and missing-ish data gracefully and resist overfitting better than a single tree. If you need the last few points of accuracy or are ranking, gradient boosting usually edges them out.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.