Ensembles & Boosting
A single decision tree overfits; a shallow one underfits. Ensembles escape the dilemma by combining many imperfect trees into one vote, and they dominate tabular data to this day. Two recipes: random forests train hundreds of trees in parallel on random slices of the data and average them, so their individual mistakes cancel. Boosting trains trees in sequence — each new tree is fitted mostly to the examples the ensemble so far gets wrong, so every addition attacks the remaining errors. Gradient-boosted trees (XGBoost, LightGBM) are boosting's modern form and still win most tabular competitions.
Add weak stumps one at a time. Red-ringed points are the current mistakes — watch each new cut land near them, and accuracy climb from one crude guess to a sharp combined boundary.
One stump alone: a crude cut that gets the easy points and butchers the messy middle.
Variants & real-world flavors
Random Forest trains hundreds of deep trees in parallel, each on a bootstrapped sample of the data with a random subset of features per split, then averages — robust, hard to overfit, nearly tuning-free. Extra Trees adds more randomness to the splits themselves, trading a little bias for lower variance. On the boosting side, AdaBoost (1995) reweights misclassified samples each round; gradient boosting generalises it to fitting each new tree to the current errors' gradient. Its modern engines — XGBoost, LightGBM, CatBoost — add regularisation, histogram tricks, and categorical handling, and remain the default winners on tabular data, routinely beating neural networks there.
Check yourself
Every stump in the ensemble is a terrible classifier on its own. Why does their weighted vote keep improving as you add more — and what must be true about their mistakes for that to work?
Go deeper (free): StatQuest — Gradient Boost explained (video) ↗