TracksStudyAbout
Sign in
Start studying
Module 1 · Machine Learning Module contents
Module 1  ·  Card 40 / 55intermediate

Gradient Boosting

Gradient boosting builds trees one after another, not all at once.

Each new tree is trained to correct the errors the previous ones made.

The predictions of all the trees are added together.

It usually beats random forest in accuracy on tabular data.

It is slower to train and more sensitive to hyperparameters.

Back Next
Learn in depth ↓
Module 01 · Machine Learning · 55 concepts
In depth  /  15 lines

Gradient Boosting, in more detail

Boosting builds trees in a queue, each one fixing the last one's mistakes.

The first tree makes a rough attempt at the answer.

You then measure what it got wrong, row by row.

The second tree is trained to predict those errors, not the original target.

Adding its output to the first tree's output improves the total.

Repeat that a few hundred times and the errors keep shrinking.

Each tree is deliberately small and weak, often only a few levels deep.

A learning rate shrinks every tree's contribution so no single one takes over.

A smaller learning rate needs more trees but usually ends up better.

Because it keeps chasing errors, it will happily overfit if you let it.

Early stopping on a validation set is the standard defence.

It is very sensitive to its settings, unlike random forest.

Trees are built one after another, so training is harder to spread across cores.

On tabular data it is usually the most accurate thing available.

It is why so many competition winners never touch a neural network.

Want the full technical article? Wikipedia ↗
Deep dive

What is underneath Gradient Boosting

Fitting the next model to what is left over

Boosting builds an additive model: the prediction is a running sum, and each new tree is trained on the error the sum has so far.

Fm(x)=Fm−1(x)+η hm(x)F_m(\mathbf{x}) = F_{m-1}(\mathbf{x}) + \eta \, h_m(\mathbf{x})
  • FmF_m: the ensemble after mm rounds.
  • hmh_m: the tree added in round mm.
  • η\eta: the learning rate, which shrinks every tree's contribution.

The first model is usually just a constant, the mean of the target, and every tree after it is a correction.

The residual is a gradient

ri=−[∂L(yi,F(xi))∂F(xi)]F=Fm−1r_i = -\left[ \frac{\partial \mathcal{L}(y_i, F(\mathbf{x}_i))}{\partial F(\mathbf{x}_i)} \right]_{F = F_{m-1}}

Each tree is fitted to the negative gradient of the loss with respect to the current prediction, evaluated at every training point.

L=12(y−F)2  ⟹  ri=yi−Fm−1(xi)\mathcal{L} = \frac{1}{2}(y - F)^2 \implies r_i = y_i - F_{m-1}(\mathbf{x}_i)

For squared error that gradient is exactly the residual, which is where the plain-English description comes from.

For any other loss it is not a residual at all, and that generality is the whole point of the word gradient in the name.

Boosting is gradient descent, with each step taken in the space of functions rather than the space of parameters.

One round, in order

  1. Compute the negative gradient of the loss at every training point.
  2. Fit a shallow regression tree to those values.
  3. Choose the value each leaf should output, by line search on the loss.
  4. Add that tree to the ensemble, scaled by the learning rate.
  5. Repeat until the validation score stops improving.

Why the trees are kept small

depth∈[3,8]\text{depth} \in [3, 8]

Each tree only has to capture one correction, so a stump or a small tree is enough and a deep one fits noise that the next round then has to undo.

η≈0.01 to 0.1\eta \approx 0.01 \text{ to } 0.1

A small learning rate makes each correction timid, which needs more rounds and reliably generalises better.

Learning rate and number of trees trade directly against each other. Halve η\eta and you need roughly twice as many rounds to reach the same fit.

Why it overfits where a forest does not

A forest averages independent trees, so adding trees can only help and BB is not a tuning parameter.

Boosting is sequential, so adding rounds keeps reducing training loss past the point where validation loss turns.

The number of rounds is therefore the main regularisation dial, and it is set by early stopping on a validation fold rather than chosen in advance.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
Which algorithm builds trees sequentially, each correcting the previous trees' errors?
Concepts these five lines lean on
HyperparametersEarly StoppingTraining / Validation / Test SetsRandom ForestRows and Columns · Data ScienceAccuracyOverfitting
Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close