Data Augmentation in computer vision

Module 03 · Computer Vision · 22 concepts

In depth / 15 lines

Data Augmentation, in more detail

Data augmentation manufactures new training examples by altering the ones you have.

Flip an image horizontally and it is still a cat, but the pixels are all different.

The label is unchanged while the input is new, which is exactly what a model needs.

Rotation, cropping, scaling, brightness and colour shifts are the usual operations.

Each teaches the model that identity survives that particular change.

Without it, a model trained on centred well-lit photographs fails on anything else.

It is among the cheapest and most effective defences against overfitting in vision.

Transformations are applied on the fly, so the model rarely sees the exact same image twice.

The choice must respect the task, which is where people go wrong.

Flipping horizontally is fine for animals and wrong for text or for road signs.

Flipping vertically is fine for satellite images and wrong for photographs of people.

Colour shifts are wrong if colour is the thing being classified, as in skin lesions.

Mixup and CutMix blend two images and their labels, which sounds odd and works well.

Augment the training set only, since a transformed test set measures a different task.

Augmentation is not a substitute for data, since it cannot add information you never had.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Data Augmentation

Enlarging the dataset with transformations that preserve the label

(\mathbf{x}, y) \longrightarrow (T(\mathbf{x}), y)

$T$ : a random transformation applied at load time.

A flipped photograph of a cat is still a cat, so the pair is still valid training data and the model has seen one more example.

The condition is the label. Flipping a photograph horizontally is safe. Flipping a digit is not, because a mirrored 2 is not a 2.

\mathbb{E}_{T}\big[\mathcal{L}(f(T(\mathbf{x})), y)\big]

Training minimises the loss averaged over the transformations, which teaches the model to be invariant to exactly the ones you chose.

That makes augmentation a way of stating what should not matter, and everything it leaves out is something the model may still key on.

The standard set, and what each one buys

Random crop with padding, which teaches tolerance to position and framing.
Horizontal flip, which doubles the data for free on anything without handedness.
Colour jitter on brightness, contrast and saturation, for lighting.
Random rotation of a few degrees, for camera tilt.
Random erasing of a rectangle, which forces the model to use more than one part of the object.

Each is applied with a probability rather than always, so the model still sees clean examples and the distribution is widened rather than shifted.

Mixup, which breaks the rule deliberately

\tilde{\mathbf{x}} = \lambda \mathbf{x}_i + (1 - \lambda) \mathbf{x}_j

\tilde{y} = \lambda y_i + (1 - \lambda) y_j

$\lambda \sim \mathrm{Beta}(\alpha, \alpha)$ , usually with $\alpha$ between 0.2 and 0.4.

Mixup blends two images and blends their labels by the same weight, producing a training example that looks like nothing in the real world.

It works because it forces the model's output to move linearly between classes, which suppresses the overconfident regions a network otherwise invents between its training points.

\tilde{\mathbf{x}} = M \odot \mathbf{x}_i + (1 - M) \odot \mathbf{x}_j, \qquad \lambda = 1 - \frac{|M|}{HW}

CutMix pastes a rectangle of one image into another and weights the labels by the area, which keeps local statistics realistic while still mixing.

Both give a point or two on image classification and both make the model better calibrated, which is the less advertised half of the result.

Where it belongs in the pipeline

Augmentation applies to the training set only. Augmenting the validation set measures the model on a distribution it will never be asked about.

\text{test time augmentation} : \hat{y} = \frac{1}{K}\sum_{k=1}^{K} f\big(T_k(\mathbf{x})\big)

The exception is test time augmentation, where several transformed copies are averaged at inference to buy a small gain for $K$ times the cost.

Transformations are applied on the fly in the data loader rather than saved to disk, so every epoch sees a different sample and the storage cost is zero.

How to tell it has gone too far

\mathcal{L}_{\mathrm{train}} > \mathcal{L}_{\mathrm{val}}

Training loss above validation loss is the signature of augmentation that is too aggressive, since the model is being scored on distorted images and evaluated on clean ones.

A transformation that can destroy the label is worse than no augmentation, because it teaches the model that the wrong answer is sometimes correct.

Strong augmentation also needs longer training. Its whole effect is to make the problem harder, and a schedule tuned without it will stop before the gain arrives.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Why does flipping an image while keeping its label reduce overfitting?

Computer Vision

22 concepts in this module, then a graded test with every answer explained.

Study this module →