Autoencoders in generative AI, explained in 5 lines

Module 12 · Generative AI · 18 concepts

In depth / 15 lines

Autoencoders, in more detail

An autoencoder is trained to output exactly what it was given.

That sounds pointless until you see the constraint in the middle.

The encoder squeezes the input down to a much smaller vector.

The decoder has to rebuild the original from that alone.

The narrow point is the bottleneck, and it is where the learning happens.

To reconstruct well through a small bottleneck, only what matters can be kept.

The model is forced to discover an efficient representation on its own.

Nobody labels anything, since the input is its own target.

The compressed vector is the useful product, not the reconstruction.

It can compress, though dedicated codecs usually beat it.

A denoising autoencoder is given a corrupted input and asked for the clean original.

That forces it to learn structure rather than copy pixels.

Anomaly detection is the neatest application.

Train on normal data only, and anything reconstructed badly is unlike what it saw.

It cannot generate new samples reliably, since its latent space has gaps, which is what VAEs fix.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Autoencoders

Learning to copy, badly on purpose

\hat{\mathbf{x}} = g\big(f(\mathbf{x})\big), \qquad \mathcal{L} = \|\mathbf{x} - \hat{\mathbf{x}}\|^2

$f$ : the encoder, mapping input to a code.
$g$ : the decoder, mapping the code back.

The target is the input itself, so no labels are needed and any pile of data is a training set.

\mathbf{z} = f(\mathbf{x}) \in \mathbb{R}^{k}, \qquad k \ll d

Copying is trivial unless something prevents it. The bottleneck is that constraint: the code is too small to hold the input, so the network must decide what to keep.

What it keeps is the structure that repeats across the dataset, which is exactly what a useful representation is.

The linear case is PCA

f(\mathbf{x}) = W\mathbf{x}, \qquad g(\mathbf{z}) = W^{\top}\mathbf{z}

With linear layers and squared error, the optimal solution spans the same subspace as the top $k$ principal components.

The autoencoder does not recover the components individually, only the subspace, since any rotation inside it gives the same reconstruction.

The whole gain from a deep autoencoder is that the surface it fits can bend, so it captures structure a plane cannot.

The other ways to force a choice

\mathcal{L} = \|\mathbf{x} - g(f(\tilde{\mathbf{x}}))\|^2, \quad \tilde{\mathbf{x}} = \mathbf{x} + \boldsymbol{\epsilon}

A denoising autoencoder corrupts the input and asks for the clean version, which stops it learning the identity even when the code is large.

\mathcal{L} = \|\mathbf{x} - \hat{\mathbf{x}}\|^2 + \lambda \|\mathbf{z}\|_1

A sparse autoencoder penalises active units instead of limiting their number, so the code is wide and only a few entries fire per input.

That variant is what is now used to read features out of language models, where a very wide sparse code separates concepts a dense activation had mixed together.

One training run, in order

Choose the code size, which is the main decision.
Encode a batch, decode it, and compute the reconstruction error.
Backpropagate through both halves at once, since they train jointly.
Watch the reconstruction on held-out data rather than on the training set.
Discard the decoder if the representation is the product.

What it is and is not good for

As a codec it loses to purpose-built compression, because it is learned for one distribution and has no rate control.

\text{anomaly detection} : \|\mathbf{x} - \hat{\mathbf{x}}\| \text{ as the score}

It is genuinely useful here. Trained on normal data only, it reconstructs normal inputs well and unusual ones badly, and that error is an unsupervised anomaly score.

The plain autoencoder cannot generate.

Its code space has no structure, so a randomly chosen $\mathbf{z}$ decodes to nothing meaningful, and the regions between two valid codes are undefined.

Fixing that is exactly what the variational autoencoder does, by making the code a distribution rather than a point.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What forces an autoencoder to learn something useful rather than copying its input?

Generative AI

18 concepts in this module, then a graded test with every answer explained.

Study this module →