Regularization in machine learning

Module 01 · Machine Learning · 55 concepts

In depth / 15 lines

Regularization, in more detail

Regularization is a way of telling a model to stay humble.

It adds a penalty for complexity to the thing being minimised.

Now the model must trade accuracy on training data against staying simple.

L2 shrinks every weight toward zero without ever quite arriving.

L1 pushes weak weights all the way to zero and drops those features entirely.

So L1 gives you feature selection for free, which helps when you have many columns.

The strength of the penalty is a setting you choose and tune.

Too little and it does nothing, too much and the model underfits.

It matters most when you have more parameters than the data can support.

With enormous amounts of data it matters much less.

Dropout is regularization for neural networks under another name.

Early stopping is regularization too, since it limits how far the fit can go.

Even adding noise or extra augmented data works the same way.

Scale your features first, or these penalties punish some columns unfairly.

The point is never a lower training score, it is a better score on new data.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Regularization

Add a price for being large

Regularisation adds a term to the loss that grows with the size of the weights, so the fit has to justify every unit of complexity it buys.

\mathcal{L}_{\text{reg}} = \mathcal{L}_{\text{data}} + \lambda \, \Omega(\mathbf{w})

$\mathcal{L}_{\text{data}}$ : the loss you actually care about.
$\Omega$ : the penalty on the weights.
$\lambda$ : how much you care about the penalty relative to the fit.

At $\lambda = 0$ the penalty vanishes and you are back to the unregularised model. As $\lambda$ grows the weights are pulled towards zero, and in the limit they all are zero.

The two penalties

\Omega_{L2}(\mathbf{w}) = \sum_{j=1}^{d} w_j^2 \qquad \Omega_{L1}(\mathbf{w}) = \sum_{j=1}^{d} \left| w_j \right|

L2 is called ridge, weight decay, or Tikhonov regularisation depending on who is speaking. L1 is called lasso.

They sound like minor variants of each other and they behave completely differently.

Why L1 produces zeros and L2 does not

Look at what each penalty contributes to the gradient.

\frac{\partial}{\partial w_j} \lambda w_j^2 = 2\lambda w_j \qquad \frac{\partial}{\partial w_j} \lambda \left| w_j \right| = \lambda \, \mathrm{sign}(w_j)

L2's pull is proportional to the weight, so it shrinks fast when the weight is large and slows to nothing as the weight approaches zero. It never quite arrives.

L1's pull is a constant $\lambda$ regardless of how small the weight already is. A weight whose data gradient is smaller than $\lambda$ gets pushed to exactly zero and stays there.

That is why L1 does feature selection and L2 does not: L1 removes features, L2 merely quietens them.

What L2 does to the closed form

\hat{\mathbf{w}} = \left( X^{\top}X + \lambda I \right)^{-1} X^{\top}\mathbf{y}

Adding $\lambda$ down the diagonal makes the matrix invertible even when features are perfectly correlated, which is where the name ridge comes from.

One fit, in order

Scale the features first. A penalty on raw weights is meaningless if one feature is in metres and another in millimetres.
Add the penalty term to the loss.
Choose $\lambda$ by cross-validation, not by intuition.
Never penalise the intercept: shrinking it towards zero asserts that the target's mean is zero.
Check what survived. With L1, the zeros are a result in themselves.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What does an L1 penalty do that L2 does not?

Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →