Bagging vs Boosting
Bagging trains many models independently on random resamples of the data and averages their predictions; boosting trains models one after another, each focusing on the examples the previous ones got wrong. Bagging is a variance-reduction strategy: it takes unstable, overfit-prone learners like deep decision trees and cancels their noise by averaging. Boosting is a bias-reduction strategy: it takes weak, underfit learners like decision stumps and combines them into a strong model by making every new learner correct the ensemble's remaining mistakes.
AdaBoost is the classic boosting algorithm and a useful concrete anchor. It maintains a weight on every training example, raises the weights of misclassified points after each round, and trains the next weak learner on the reweighted data, so hard cases get progressively more attention. Each learner also receives a vote proportional to its accuracy. Compare that to bagging, where every model sees an equally random bootstrap sample, trains in complete ignorance of its peers, and gets an equal vote. One paradigm is a committee of independents; the other is a relay team.
The practical consequences follow directly. Bagging is robust, parallel, and hard to overfit, but it can only polish what its base learner already captures. Boosting can turn barely-better-than-chance learners into highly accurate models, but its obsession with hard examples makes it sensitive to label noise and outliers, and it needs a stopping rule. Modern gradient boosting generalizes AdaBoost's reweighting idea into fitting residual errors under an arbitrary loss, and dominates tabular leaderboards today.
Side by side
Core idea
What it reduces
Base learners
How models interact
Parallelism
Overfitting behavior
Sensitivity to noisy labels
Key hyperparameters
When it fails
| Dimension | Bagging | AdaBoost |
|---|---|---|
| Core idea | Train many models independently on bootstrap resamples and aggregate by averaging or majority vote. | Train weak learners sequentially, reweighting the data so each new learner concentrates on previously misclassified examples. |
| What it reduces | Variance; averaging cancels the noise of unstable learners, while their shared bias remains untouched. | Bias; each round removes systematic error, though pushing too far eventually raises variance. |
| Base learners | Strong, low-bias, high-variance models, typically deep decision trees grown near purity. | Weak, high-bias models, classically decision stumps or very shallow trees that barely beat chance. |
| How models interact | They do not; each model trains blind to the others, and all get an equal vote at prediction time. | Tightly coupled; each learner's training data weighting depends on every previous learner's mistakes, and votes are weighted by accuracy. |
| Parallelism | Fully parallel; models can train simultaneously on separate cores or machines. | Inherently sequential; round t cannot start until round t-1 has finished and the weights are updated. |
| Overfitting behavior | Very resistant; adding more models cannot overfit further, it only stabilizes the average. | Can overfit with enough rounds, especially on noisy data; the number of rounds must be validated or early-stopped. |
| Sensitivity to noisy labels | Robust; a mislabeled point appears in some resamples and not others, and averaging dilutes its pull. | Sensitive; a mislabeled point is permanently hard, so its weight grows round after round and the ensemble contorts to fit it. |
| Key hyperparameters | Number of models and base-learner depth; both are forgiving, and more models is almost always safe. | Number of rounds, learner depth, and in modern variants a learning rate; these interact and need validation. |
| When it fails | When the base learner is biased: averaging a hundred underfit models yields the same underfit answer. | When labels are noisy or the data is small: the hard-example focus amplifies noise into confident mistakes. |
| Flagship algorithms | Random forest, which adds random feature subsets per split to further decorrelate the bagged trees. | AdaBoost historically, and gradient boosting machines such as XGBoost and LightGBM as the modern standard. |
When to use Bagging
- Your base model is unstable and overfits, such as a deep decision tree, and you want to stabilize it without changing its bias.
- Labels are noisy or outliers are common, and you need an ensemble that dilutes bad points instead of fixating on them.
- You want parallel training across cores with no sequential bottleneck and no stopping rule to babysit.
- You want a robust result with minimal tuning; number of models is the only knob that really matters and more is safe.
- You also want free out-of-bag error estimates, since each model's held-out bootstrap points give validation without a split.
When to use AdaBoost
- Your base learner underfits, and you need to build accuracy up from weak models rather than average strong ones.
- The data is reasonably clean, so concentrating on hard examples sharpens the boundary instead of chasing label noise.
- You want a simple, nearly hyperparameter-free introduction to boosting; classic AdaBoost needs little beyond the round count.
- You want to see boosting's mechanics explicitly, since the example weights make each round's focus visible and teachable.
- You are stepping toward gradient boosting, which generalizes the same sequential idea to arbitrary losses and is the production standard.
The bottom line
Diagnose your base learner first: if it overfits, bag it; if it underfits, boost it. In practice that means random forests, bagging's flagship, when you want robustness, noisy-label tolerance, parallel training, and near-zero tuning, and boosting when you want maximum accuracy on reasonably clean tabular data and can validate the number of rounds. For serious production work, prefer gradient boosting over classic AdaBoost, since it generalizes the same sequential error-correction idea to arbitrary loss functions with better regularization. And remember the paradigms are not enemies: a random forest baseline followed by a tuned gradient boosting model is the standard one-two punch for tabular machine learning.