GANs in generative AI, explained in 5 lines

Module 12 · Generative AI · 18 concepts

In depth / 15 lines

GANs, in more detail

A GAN trains two networks in competition, which was a genuinely new idea in 2014.

The generator produces fake samples from random noise.

The discriminator is shown real and fake samples and must tell them apart.

The discriminator improves at spotting fakes.

The generator improves at fooling it.

Each one's progress makes the other's job harder, which is the engine.

At equilibrium the fakes are indistinguishable and the discriminator is guessing.

The generator never sees a real image, and it only sees whether it was caught.

That produced remarkably sharp images, far better than the VAEs of the time.

Training is notoriously unstable, since two networks chasing each other rarely settle.

One can overpower the other, and if the discriminator wins outright the generator stops learning.

Mode collapse is the classic failure, where the generator produces only a few outputs.

Many tricks exist to stabilise it, including Wasserstein loss and gradient penalties.

Diffusion models largely replaced GANs, being stabler and more controllable.

GANs remain useful where speed matters, since they generate in one pass rather than fifty.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath GANs

Two networks with opposite objectives

\min_{G} \max_{D} \; \mathbb{E}_{\mathbf{x} \sim p_{\mathrm{data}}}\big[\log D(\mathbf{x})\big] + \mathbb{E}_{\mathbf{z} \sim p_{\mathbf{z}}}\big[\log\big(1 - D(G(\mathbf{z}))\big)\big]

$D$ : the discriminator, which outputs the probability that its input is real.
$G$ : the generator, which maps noise to a sample.

The discriminator maximises that expression by telling real from fake. The generator minimises it by producing samples the discriminator calls real.

There is no reconstruction term and no likelihood. The only signal the generator ever receives is the discriminator's opinion.

The optimum, and what it says

D^{*}(\mathbf{x}) = \frac{p_{\mathrm{data}}(\mathbf{x})}{p_{\mathrm{data}}(\mathbf{x}) + p_{g}(\mathbf{x})}

For a fixed generator, the best discriminator is that ratio, which is 0.5 everywhere when the two distributions match.

\min_{G} \; 2 \, D_{\mathrm{JS}}\big(p_{\mathrm{data}} \Vert p_{g}\big) - \log 4

Substituting it back shows the generator is minimising the Jensen-Shannon divergence, so the objective is a divergence between distributions in disguise.

That is the theoretical result, and it assumes the discriminator is trained to optimality at every step, which it never is.

The gradient problem at the start

D(G(\mathbf{z})) \approx 0 \implies \nabla_{G} \log\big(1 - D(G(\mathbf{z}))\big) \approx 0

Early in training the discriminator wins easily, and the generator's gradient vanishes exactly when it needs the most guidance.

\max_{G} \; \mathbb{E}\big[\log D(G(\mathbf{z}))\big]

The non-saturating loss flips the objective to maximising the log of being called real, which has the same fixed point and a strong gradient when the generator is losing.

Every implementation uses this version. The minimax form in the paper is the analysis, not the code.

One training step, in order

Sample a batch of real data and a batch of noise.
Generate fakes and update the discriminator on both, with real labelled 1 and fake 0.
Sample fresh noise and generate again.
Update the generator through the discriminator, with the fakes labelled 1.
Keep the two updates balanced, since either network winning outright stalls the other.

The discriminator's weights are frozen during step four. The gradient passes through it to reach the generator, and updating it there would be training it to be wrong.

Why it is hard to train

There is no convergence guarantee.

This is a two-player game rather than a minimisation, so the parameters can cycle indefinitely without either player improving.

p_{g} \to \text{a few modes}

Mode collapse is the characteristic failure: the generator finds a handful of outputs that fool the discriminator and stops producing anything else.

The loss values say almost nothing about sample quality, which is why GAN papers report FID and show images rather than plotting a curve.

\mathcal{L} = \mathbb{E}[D(\mathbf{x})] - \mathbb{E}[D(G(\mathbf{z}))]

Wasserstein GAN replaces the divergence with the earth mover's distance and constrains the discriminator to be Lipschitz.

That gives a loss which correlates with sample quality and a gradient that does not vanish.

Diffusion models displaced GANs for image generation largely because they train by a stable regression objective.

They pay for it in sampling, needing many forward passes to produce one image where a GAN needs one.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What makes GAN training unstable?

Generative AI

18 concepts in this module, then a graded test with every answer explained.

Study this module →