AdaBoost

EnsemblesAdvanced~8 min

AdaBoost — Chain weak learners, each fixing the last one’s mistakes.

Boosting builds an ensemble sequentially. Each new weak learner focuses on the examples the previous ones got wrong, re-weighting the hard cases until the combined model is strong.

Class 0
Class 1
This round’s stump
Point weight

Ensemble train error % vs iteration

AdaBoost controls

Data

Dataset

Points per class20

Spread1.0×

Label noise0%

Add points as

Model

Boosting rounds16

Weighted error ε0.050

Stump weight α1.47

A stump that is barely better than a coin flip (ε near 0.5) earns α near 0 — the ensemble almost ignores it.

Playback

Step 0 / 31

Speed

Fit a stump
Reweight
↻

Step 0 of 31 — round 1 — the weighted search picked “class 1 when x > 4.81”, weighted error ε = 0.050, so α = ½ln((1−ε)/ε) = 1.47

Point size is its sample weight. Hover one to read the weight itself. Drag any point, or click empty space to drop a new one, and boosting re-runs from round 1.

Break it

The idea in plain words

Boosting builds an ensemble sequentially rather than in parallel. AdaBoost starts with equal weights on every point, fits a weak learner (a one-split stump), then increases the weight of the points it got wrong so the next stump focuses on them.

Watch the hard points swell round by round and the combined boundary bend to catch them. But with label noise, AdaBoost obsesses over impossible points — it can never classify them — and overfits.

Now, the math

Each stump’s vote weight α depends on its weighted error ε:

\alpha_t = \tfrac{1}{2}\ln\frac{1 - \varepsilon_t}{\varepsilon_t}

$\varepsilon_t$: the stump’s weighted error rate this round.
$\alpha_t$: its vote weight — larger when the stump is more accurate.

▸ Show the derivation

After each round, misclassified points have their weights scaled up by a factor of e^α and correct ones scaled down, then renormalized — so the next stump is trained on a distribution that emphasizes the current mistakes. The final classifier is the α-weighted vote of all stumps.

Trace it by hand

Eight points with labels +1 or -1: four negatives P1 (1,5), P2 (2,2), P3 (3,7), P4 (4,4) and four positives P5 (2,6), P6 (7,3), P7 (8,8), P8 (9,5). P5 sits deep inside the negative cluster — the hard point. Every weight starts at 1/8 = 0.125. The run is fully deterministic; values are shown to 3-4 decimals.

Step 1 — best first stump and its weighted error
$h_1(x) = \operatorname{sign}(x_1 - 5.5), \qquad \varepsilon_1 = \sum_{\text{missed}} w_i = 0.125$
The stump predicting +1 when x1 is greater than 5.5 gets 7 of 8 points right; only the hard point P5 (at x1 = 2) is missed, and it carries weight 1/8.
Step 2 — its vote weight from the alpha formula
$\alpha_1 = \tfrac{1}{2}\ln\frac{1 - \varepsilon_1}{\varepsilon_1} = \tfrac{1}{2}\ln\frac{0.875}{0.125} = \tfrac{1}{2}\ln 7 = 0.973$
A small error earns a big say: this stump will cast a vote of weight 0.973 in the final ensemble.
Step 3 — reweight: the mistake swells, correct points shrink
$w_{P_5} = \frac{0.125 \, e^{0.973}}{Z} = \frac{0.3307}{0.6614} = 0.5, \qquad w_{\text{correct}} = \frac{0.125 \, e^{-0.973}}{0.6614} = 0.0714$
e to the alpha is 2.646 and e to the minus alpha is 0.378; after dividing by the normalizer Z = 0.6614, the one missed point holds exactly half of all the weight. AdaBoost always rebalances so the mistakes carry one half.
Step 4 — round 2 chases the hard point
$h_2(x) = \operatorname{sign}(x_1 - 1.5), \qquad \varepsilon_2 = 3 \times 0.0714 = 0.214, \qquad \alpha_2 = \tfrac{1}{2}\ln\frac{0.786}{0.214} = 0.650$
With P5 worth 0.5, the cheapest stump is one that classifies it correctly, even at the cost of missing three light points worth 0.0714 each. Its higher error buys it a smaller alpha.
Step 5 — weights after round 2 and the ensemble so far
$w = (0.045,\; 0.167,\; 0.167,\; 0.167,\; 0.318,\; 0.045,\; 0.045,\; 0.045), \qquad \text{train error} = \tfrac{1}{8}$
For P5 the alpha-weighted vote is 0.650 for and 0.973 against, so it is still outvoted and its weight keeps climbing. With noisy labels this same single-minded chase is exactly how AdaBoost overfits.

What just happened: Two passes of the alpha formula turned the real errors 0.125 and 0.214 into vote weights 0.973 and 0.650, and one exponential reweighting pushed the missed point from 12.5 percent to 50 percent of the total weight — the mechanism that makes every new stump chase the last one's mistakes.

Now Break It

Try this: Too many rounds on noisy data starts fitting the noise — boosting can overfit outliers.

Control: Number of rounds slider (set high on noisy data)

What happens: Boosting the noise! Too many rounds and AdaBoost obsesses over noisy outliers, overfitting.

Where adaboost is used

AdaBoost, short for adaptive boosting, was the algorithm that made boosting practical, and its most famous deployment is the Viola-Jones face detector that put real-time face detection in early digital cameras. There it chained thousands of tiny threshold classifiers into a fast cascade. Beyond vision, AdaBoost with shallow decision stumps remains a compact, interpretable choice for binary classification tasks like spam filtering and churn flags, and it often appears as a teaching bridge toward gradient boosting. Because each round reweights the training examples toward the ones still being misclassified, AdaBoost concentrates effort where the current ensemble is weak, building a strong committee from learners that are individually barely better than chance.

The classic misconception is that AdaBoost never overfits because early experiments showed test error still dropping after training error hit zero. In practice it does overfit, especially on noisy data or with mislabeled points, because its exponential loss piles ever more weight on hard or corrupted examples until they dominate. That sensitivity to label noise and outliers is AdaBoost's main weakness, and it is why gradient boosting with a robust loss is often preferred today. Another confusion is treating AdaBoost and gradient boosting as unrelated: AdaBoost is essentially a special case of gradient boosting using exponential loss, where example reweighting is the practical stand-in for fitting the loss gradient. Keep base learners weak, since boosting strong learners tends to overfit fast.

Frequently asked questions

What is AdaBoost?

AdaBoost is a boosting algorithm that combines many weak classifiers, often one-split decision stumps, into a single strong classifier. It trains them one at a time, and after each round it increases the weight of the examples that were misclassified so the next learner focuses on them. The final prediction is a weighted vote, where more accurate learners get a larger say.

How does AdaBoost decide the weight of each weak learner?

After training a weak learner, AdaBoost measures its weighted error rate on the current example weights. Learners with lower error receive a larger coefficient in the final vote, while a learner no better than chance gets near zero weight. This way accurate learners dominate the committee and weak ones contribute little.

Does AdaBoost overfit?

Yes, it can, despite early results suggesting it was resistant. AdaBoost uses an exponential loss that keeps up-weighting the hardest examples, so on noisy data or with mislabeled points it eventually chases those points and generalizes worse. Limiting the number of rounds or the base-learner complexity helps control this.

Why is AdaBoost sensitive to noise and outliers?

Every time an example is misclassified its weight grows, and the exponential loss makes that growth aggressive. Genuinely mislabeled or outlier points get misclassified round after round, so their weights balloon and later learners bend the model toward fitting them. This is the main reason gradient boosting with a robust loss is often preferred on messy data.

How is AdaBoost related to gradient boosting?

AdaBoost can be viewed as a special case of gradient boosting that uses the exponential loss function. In that framing, reweighting the misclassified examples is the practical equivalent of fitting each new learner to the gradient of the loss. Gradient boosting generalizes the idea to any differentiable loss, which is why it is more flexible.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.