Class Imbalance in machine learning

Module 01 · Machine Learning · 55 concepts

In depth / 15 lines

Class Imbalance, in more detail

Class imbalance is when one outcome is far more common than another.

Fraud, disease, defects and churn are all naturally rare.

One case in a thousand is ordinary in fraud detection.

A model that always predicts not fraud is then 99.9 per cent accurate and completely useless.

So the first fix is not the model but the metric.

Accuracy hides the failure, and precision, recall and PR-AUC expose it.

The confusion matrix shows immediately that the rare class is never predicted.

Class weighting tells the model that a mistake on the rare class costs more.

That is usually the first thing to try, since it changes nothing about the data.

Oversampling repeats rare rows, and risks the model memorising those exact rows.

SMOTE instead invents new rare rows between existing ones, which generalises better.

Undersampling throws away common rows, which is wasteful but sometimes practical.

Any resampling must happen after the train and test split, and only on the training part.

Resample the test set and your score measures a world that does not exist.

Adjusting the decision threshold is often the simplest lever of all.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Class Imbalance

Why accuracy stops being a measurement

\text{accuracy} = 1 - \pi_{+}

$\pi_{+}$ : the share of the data that is positive.

On a dataset that is 1 percent positive, a model that always answers negative scores 99 percent, and it has learned nothing.

The loss has the same problem. Summed over the data, the 99 percent of easy negatives contribute almost all the gradient and the rare class barely moves the weights.

So imbalance is not a data problem that happens to hurt the metric. It is a weighting problem in the objective, and the repairs all change a weight somewhere.

Reweighting the loss

\mathcal{L} = - \frac{1}{n} \sum_{i=1}^{n} w_{y_i} \log \hat{p}_{i, y_i}

w_c = \frac{n}{C \cdot n_c}

$n_c$ : how many samples carry class $c$ .
$C$ : the number of classes.

Weighting each class inversely to its frequency makes the rare class contribute as much total gradient as the common one, which is one line of code in every framework.

\mathcal{L}_{\mathrm{focal}} = -(1 - \hat{p}_t)^{\gamma} \log \hat{p}_t

Focal loss reweights by difficulty instead of by class: an example the model already gets right has $\hat{p}_t$ near 1, so its contribution is scaled to almost nothing.

With $\gamma = 2$ , an example predicted at 0.9 contributes a hundredth of what it would have. The gradient goes to what is still wrong.

Resampling, and what it costs

Random undersampling throws away majority rows. It is fast, it balances exactly, and it discards real data the model could have learned from.

Random oversampling duplicates minority rows. It keeps everything and gives the model the same few examples repeatedly, which is a direct invitation to memorise them.

\mathbf{x}_{\mathrm{new}} = \mathbf{x}_i + \delta (\mathbf{x}_j - \mathbf{x}_i), \quad \delta \sim U(0, 1)

SMOTE interpolates between a minority point and one of its minority neighbours, so the new points are new rather than copies.

It assumes the line between two minority points is also minority, which fails where the classes interleave, and it must be applied inside the cross-validation loop or it leaks.

Moving the threshold instead

\hat{y} = \mathbf{1}\big[\hat{p} > \tau\big]

A probability model does not have to be retrained at all. The default $\tau = 0.5$ is a convention, not a result, and it is the wrong one under imbalance.

\tau^{*} = \frac{c_{\mathrm{FP}}}{c_{\mathrm{FP}} + c_{\mathrm{FN}}}

If a missed positive costs ten times a false alarm, the optimal threshold is about 0.09, and choosing it takes no training at all.

This is the first thing to try, because it is free, reversible, and it leaves the model calibrated.

One imbalanced problem, in order

Check the base rate and pick a metric that does not reward the majority, such as PR-AUC or recall at a fixed precision.
Train once with no correction at all, as the baseline everything else must beat.
Add class weights, which cost nothing and usually capture most of the gain.
Tune the decision threshold on a validation set against the real costs.
Only then reach for resampling, and keep it strictly inside the cross-validation folds.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Your data is 99 percent negative and the model predicts negative every time. What happens?

Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →