Bias and Variance in machine learning

Module 01 · Machine Learning · 55 concepts

In depth / 15 lines

Bias and Variance, in more detail

Model error comes from three places: bias, variance, and noise.

Noise is randomness in the data, and no model can remove it.

Bias means the model is wrong in the same way every time.

Variance means the model is wrong in a different way every time.

Picture drawing a straight line through data that curves.

It misses in the same places no matter which rows you use.

That is high bias, and it means the model is too simple.

Now picture a very deep decision tree.

Change a few rows and it gives you a completely different tree.

That is high variance, and it means the model is too flexible.

Making a model more powerful lowers bias and raises variance.

Making it simpler does the reverse.

To tell which one you have, compare training error with validation error.

Both bad means bias, so try a stronger model or better features.

A wide gap between them means variance, so get more data or simplify.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Bias and Variance

It is a theorem, not a metaphor

For squared error, the expected error on an unseen point decomposes exactly into three pieces, and the decomposition is an identity rather than an intuition.

\mathbb{E}\left[ (y - \hat{f}(x))^2 \right] = \underbrace{\left( \mathbb{E}[\hat{f}(x)] - f(x) \right)^2}_{\text{bias}^2} + \underbrace{\mathbb{E}\left[ \left( \hat{f}(x) - \mathbb{E}[\hat{f}(x)] \right)^2 \right]}_{\text{variance}} + \underbrace{\sigma^2}_{\text{noise}}

$f(x)$ : the true function.
$\hat{f}(x)$ : the function this particular training set produced.
The expectation is over training sets, not over examples.

That last point is what makes the decomposition hard to see: it asks what would happen if you retrained on many different samples of the same size.

Reading the three terms

Bias is how far the average model is from the truth. A straight line fitted to a curve has high bias no matter how much data you give it.

Variance is how much the model moves when the training set changes. A deep tree grown to purity has high variance: resample the data and you get a different tree.

Noise is the irreducible term. It is the part of $y$ that $x$ simply does not determine, and no model can remove it.

\text{total error} \geq \sigma^2

That floor is why chasing zero error is chasing a label error rate, and why a model that reaches it has memorised the noise.

The trade

Making a model more flexible lowers bias and raises variance. Making it simpler does the reverse.

Underfitting is high bias: training error and test error are both high and close together.
Overfitting is high variance: training error is low, test error is much higher.
The gap between the two curves is variance.
The height of the training curve is bias plus noise.
Where the sum is smallest is the model you want, and it is rarely at either extreme.

What each fix actually moves

Regularisation, fewer features and a shallower model all reduce variance at the cost of bias.

More training data reduces variance without touching bias, which is why it is the fix that costs nothing in accuracy.

Bagging averages many high-variance models, which cuts variance while leaving bias roughly where it was. That is the whole idea behind random forests.

Boosting does the opposite: it stacks many high-bias models to drive bias down, and has to be regularised to keep the variance it accumulates in check.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Retraining on slightly different rows gives you a very different model. Which is high?

Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →