R squared in machine learning, explained in 5 lines

Module 01 · Machine Learning · 55 concepts

In depth / 15 lines

R squared, in more detail

R squared compares your model against a very lazy baseline.

That baseline is simply predicting the average value every single time.

A score of zero means your model matched that and did no better.

A score of one means it predicted everything perfectly.

A negative score means it did worse than the average, which happens more than people expect.

Because it is a ratio it has no units, so it travels between problems.

That is its main advantage over RMSE and MAE.

It is often described as the share of variation the model explains.

That phrasing is fine, but it says nothing at all about cause.

A high R squared does not mean your features cause the outcome.

It always rises when you add a feature, even a completely random one.

Adjusted R squared exists to penalise that, and it is the fairer number.

A high score on training data and a low one on test data is the usual overfitting signal.

It is also sensitive to how much the target varies in your sample.

The same model scores well on a varied sample and badly on a narrow one.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath R squared

The formula

R^2 = 1 - \frac{SS_{\text{res}}}{SS_{\text{tot}}} = 1 - \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i (y_i - \bar{y})^2}

$SS_{\text{res}}$ : the squared error your model makes.
$SS_{\text{tot}}$ : the squared error you would make by always predicting the mean.
$\bar{y}$ : the mean of the true values.

So the ratio is "how much error is left, as a fraction of the error you started with", and $R^2$ is one minus that.

What the numbers mean

$R^2 = 1$ means every residual is zero: a perfect fit.

$R^2 = 0$ means your model is exactly as good as predicting the average every time, having learned nothing.

R^2 < 0 \iff SS_{\text{res}} > SS_{\text{tot}}

It can be negative, and this surprises people who remember it as "the square of something".

A negative value means the model is worse than the mean, which happens routinely on a test set when a model has overfitted.

Why it is unitless, and what that buys

Both sums are in the target's units squared, so they cancel.

That makes $R^2$ comparable across problems in a way RMSE never is: 0.85 means the same thing on house prices and on rainfall.

It also makes it useless for telling you whether the model is good enough to use. $R^2 = 0.9$ on a target with huge variance can still be an unusably large absolute error.

Quote it alongside RMSE, never instead of it.

The trap: it never goes down

Adding any feature, even a column of random noise, cannot increase $SS_{\text{res}}$ on the training set.

So $R^2$ on training data rises monotonically with model complexity and is worthless as a model-selection criterion.

R^2_{\text{adj}} = 1 - \left( 1 - R^2 \right) \frac{n - 1}{n - p - 1}

$n$ : number of examples.
$p$ : number of features.

Adjusted $R^2$ penalises each added feature, so it falls when a new one does not pay for itself.

Fit the model.
Compute $R^2$ on held-out data, not on the training set.
Check whether it is negative, which is a real result and not a bug.
Report RMSE beside it so the absolute scale is visible.
Use adjusted $R^2$ only when comparing models with different numbers of features on the same data.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What does a negative R squared mean?

Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →