Ensembling in machine learning, explained in 5 lines

Module 01 · Machine Learning · 55 concepts

In depth / 15 lines

Ensembling, in more detail

Ensembling combines several models rather than betting everything on one.

It works because different models make different mistakes.

Average their predictions and the shared signal survives while the errors partly cancel.

The gain is largest when the models are genuinely different from each other.

Averaging ten near-identical models buys almost nothing.

Bagging trains each model on a different random sample of the data.

The models are independent and can be trained in parallel.

Random forest is bagging applied to decision trees, with random feature choices added.

It mainly reduces variance, which is why it tames trees that would otherwise overfit.

Boosting trains models one after another, each focused on what the last got wrong.

It mainly reduces bias, and it can overfit if you let it run too long.

Gradient boosting and XGBoost are the widely used implementations.

Stacking trains a small model to decide how much to trust each of the others.

Ensembles win most competitions and cost more to run and to explain.

Even averaging one model trained with several random seeds usually helps a little.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Ensembling

Why averaging models works at all

\mathbb{E}\big[(\bar{f} - y)^2\big] = \frac{1}{B}\overline{\mathrm{err}} + \left(1 - \frac{1}{B}\right)\overline{\mathrm{cov}}

$B$ : how many models are averaged.
$\overline{\mathrm{err}}$ : the average error of a single model.
$\overline{\mathrm{cov}}$ : the average covariance between the models' errors.

The ensemble beats its members only to the extent that their mistakes are uncorrelated.

Average ten copies of the same model and you get the same model back. Average ten models that fail on different examples and the failures cancel.

Every ensembling method is therefore a method for manufacturing disagreement.

The vote made exact

P(\text{ensemble wrong}) = \sum_{k > B/2}^{B} \binom{B}{k} p^k (1-p)^{B-k}

$p$ : the error rate of one independent model.

With 11 independent models each wrong 30 percent of the time, the majority vote is wrong about 8 percent of the time.

That calculation assumes independence, which real models never have. It is the ceiling, not the result.

The three ways to build one

Bagging trains the same algorithm on different bootstrap samples and averages them, which attacks variance and leaves bias alone.

Boosting trains models in sequence, each one fitted to what the last got wrong, which attacks bias and can raise variance.

Stacking trains different algorithms on the same data and then trains a small model to combine their outputs.

\hat{y} = g\big(f_1(\mathbf{x}), f_2(\mathbf{x}), \dots, f_M(\mathbf{x})\big)

The combiner $g$ must be fitted on out-of-fold predictions. Fitted on the same rows the base models saw, it learns to trust whichever one memorised hardest.

One stacked ensemble, in order

Split the training data into $k$ folds.
Train each base model on $k-1$ folds and predict the held-out fold.
Collect those out-of-fold predictions into a new feature matrix.
Train the combiner on that matrix against the true labels.
Retrain each base model on all the data for use at prediction time.

What it costs

$B$ models cost $B$ times the training, $B$ times the memory and $B$ times the latency, for a gain that is usually a point or two.

An ensemble also cannot be explained the way its members could, and averaging well calibrated models produces something less confident than any of them.

That trade is worth it in a competition and often not worth it in a service answering requests.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Why does averaging several models usually beat the best single one?

Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →