Skip to content
ML Visualization

Gradient Boosting

EnsemblesAdvanced~8 min

Gradient Boosting — Fit each new tree to the residual errors of the last.

Gradient boosting builds an ensemble by having each new tree predict the residual errors left over by the current ensemble. Add them up with a small learning rate and the errors shrink round after round.

Residuals round 1 is about to fit — RMS 1.22

  • Data
  • Ensemble fit
  • Stump before shrinkage
  • Residuals
Squared error vs boosting round
tr 1.49te 1.98

Boosting controls

Data
Dataset

22 evenly spaced points, light noise — nearly every residual is real signal.

22
0.7
Model
24
0.30

Every tree here is a single split. Small η means each one barely moves the fit, so it takes many rounds — and the extra rounds are what buy the smoothness.

Playback
Step 0 / 47
Speed
  1. Fit the residuals
  2. Add η·h

Step 0 of 47 — round 1 — the ensemble leaves residuals of RMS 1.22; the next stump cuts at x = 5.44 and predicts 0.85 left, -1.02 right

Hover a point to read its prediction and residual. Drag any point, or click empty space to add one, and the model re-fits from round 1.

Break it

The idea in plain words

Gradient boosting builds its ensemble by having each new tree predict the residual errors left over by the current model. Add that tree with a small learning rate, and the leftover error shrinks. Repeat, and the fit tightens round after round.

The red arrows are the residuals each new stump chases. Unlike AdaBoost’s reweighting, this is literally gradient descent in function space. Too high a learning rate with too many rounds overfits — test error starts to climb.

Now, the math

Each stage adds a shrunken tree fit to the current residuals:

Fm(x)=Fm−1(x)+η hm(x)F_m(x) = F_{m-1}(x) + \eta\, h_m(x)
hmh_m
the m-th tree, fit to the residuals of the current ensemble.
η\eta
the learning rate (shrinkage) — small steps generalize better.
▸ Show the derivation

For squared-error loss the negative gradient at each point is exactly the residual y − F(x), so fitting a tree to the residuals is a gradient-descent step in function space. Shrinkage (small η) trades more rounds for better generalization; large η with many rounds memorizes the training noise.

Trace it by hand

Four points — (1, 1), (2, 2), (3, 5), (4, 6) — fit with depth-1 regression trees (stumps) and learning rate η = 0.5, exactly the algorithm the interactive runs. Every number below is exact.

Step 1 — start from the mean

F0(x)=yˉ=1+2+5+64=3.5F_0(x) = \bar{y} = \frac{1 + 2 + 5 + 6}{4} = 3.5

The residuals y − F₀ are (−2.5, −1.5, 1.5, 2.5) — for squared error, exactly the negative gradient at each point.

Step 2 — fit stump h₁ to the residuals, then take a shrunken step

h1(x)={−2x≤2.5+2x>2.5F1(x)=F0(x)+0.5 h1(x)h_1(x) = \begin{cases} -2 & x \le 2.5 \\ +2 & x > 2.5 \end{cases} \qquad F_1(x) = F_0(x) + 0.5\,h_1(x)
xyF₀y − F₀h₁F₁y − F₁
113.5-2.5-22.5-1.5
223.5-1.5-22.5-0.5
353.51.524.50.5
463.52.524.51.5

The best split is x ≤ 2.5; the leaf values −2 and +2 are the residual means on each side. After the η = 0.5 step, every residual shrank by 1.

Step 3 — round 2 chases the new residuals

h2(x)={−1x≤2.5+1x>2.5F2(x)=F1(x)+0.5 h2(x)h_2(x) = \begin{cases} -1 & x \le 2.5 \\ +1 & x > 2.5 \end{cases} \qquad F_2(x) = F_1(x) + 0.5\,h_2(x)

F₂ predicts (2, 2, 5, 5), leaving residuals (−1, 0, 0, 1). Two points are now fit exactly.

Step 4 — watch the error shrink

MSE:  4.25  →  h1    1.25  →  h2    0.50\text{MSE}: \; 4.25 \;\xrightarrow{\;h_1\;}\; 1.25 \;\xrightarrow{\;h_2\;}\; 0.50

What just happened: no tree ever saw y directly — each one was fit purely to the previous ensemble’s leftover errors, and the η = 0.5 shrinkage meant each round only closed half the gap. The largest residual fell 2.5 → 1.5 → 1.0, which is gradient descent in function space, one small step per round.

Now Break It

Try this: A high learning rate with many rounds overshoots and overfits the training residuals.

Control: Learning rate slider (set high)

What happens: Overfitting the residuals! A high learning rate with many rounds memorizes noise.

Where gradient boosting is used

Gradient boosting is the reigning champion of tabular machine learning, and its optimized implementations, XGBoost, LightGBM, and CatBoost, dominate structured-data Kaggle competitions and power real production systems. It sits behind many search and ad ranking models via learning-to-rank objectives, credit-risk and fraud scoring where calibrated probabilities matter, demand forecasting, and click-through prediction. Its appeal is accuracy on heterogeneous features with support for custom loss functions, monotonic constraints, and native handling of missing values in the modern libraries. Because it fits each new tree to the errors the current ensemble still makes, gradient boosting keeps chipping away at bias, which is why a well-tuned booster usually edges out a random forest on the same tabular problem, at the cost of more careful tuning.

The main misconception is that gradient boosting fits each tree to the raw residuals. For squared-error loss that happens to be true, but in general each tree fits the negative gradient of the chosen loss function evaluated at the current predictions, which is what lets it target log-loss, quantile, or ranking objectives. A second pitfall is confusing it with bagging-based random forests: here trees are grown sequentially and are dependent, so they cannot be trained in parallel across the ensemble, and deeper individual trees make overfitting easier rather than safer. Control that with a small learning rate, shallow trees, subsampling of rows and columns, and early stopping on a validation set. Without regularization and enough rounds at a low learning rate, gradient boosting will happily memorize noise.

Frequently asked questions

What is gradient boosting?
Gradient boosting builds an ensemble of decision trees one at a time, where each new tree is trained to correct the errors of the trees built so far. Formally, each tree fits the negative gradient of a chosen loss function with respect to the current predictions, and its output is added to the ensemble scaled by a learning rate. Repeating this gradually reduces the model's bias.
Does gradient boosting really fit the residuals?
Only for squared-error regression, where the negative gradient equals the residual. In general each tree fits the negative gradient of whatever loss you chose, such as log-loss for classification or a quantile loss for prediction intervals. Thinking in gradients is what lets the same algorithm handle many objectives.
What is the difference between gradient boosting and random forests?
Random forests use bagging: independent trees trained in parallel and averaged to reduce variance. Gradient boosting trains dependent trees sequentially, each fixing the previous ensemble's errors to reduce bias. Boosting usually reaches higher accuracy but needs more careful tuning and is easier to overfit.
What does the learning rate do in gradient boosting?
The learning rate, or shrinkage, scales down each tree's contribution before it is added to the ensemble. A smaller learning rate makes the model learn more slowly and generalize better, but it requires more trees to reach the same fit. Tuning learning rate together with the number of trees and using early stopping is the standard recipe.
How do I stop gradient boosting from overfitting?
Use a small learning rate with enough trees, keep individual trees shallow, and add row and column subsampling so each tree sees only part of the data. Early stopping on a validation set halts training once the validation score stops improving. Regularization terms on leaf weights, available in libraries like XGBoost, help further.
Why is gradient boosting so popular on tabular data?
It delivers state-of-the-art accuracy on heterogeneous numeric and categorical features with relatively little preprocessing, and modern libraries handle missing values, custom losses, and monotonic constraints. Because it reduces bias directly, a tuned booster usually beats other tabular models. That combination is why XGBoost, LightGBM, and CatBoost dominate structured-data competitions and many production pipelines.
← Back to all visualizationsContinue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated .