A variational autoencoder learns a distribution rather than a single point.
The encoder outputs a mean and a variance for the latent vector.
That makes the latent space smooth and continuous.
You can sample from it to generate genuinely new examples.
Outputs tend to be blurrier than those from GANs or diffusion models.
A variational autoencoder learns a distribution instead of a single point.
The encoder outputs a mean and a spread rather than one vector.
The actual code is then sampled from that small cloud.
That sampling is the whole difference from a plain autoencoder.
Because the input maps to a region, nearby codes must decode to sensible outputs.
The latent space becomes smooth and continuous with no dead gaps.
So you can sample a random point and get a genuinely new example.
You can also walk between two points and watch one thing morph into another.
A plain autoencoder cannot do that, since between its codes lies nonsense.
The loss has two parts pulling against each other.
Reconstruction error wants the output to match the input.
The KL term pulls the learned distributions towards a standard normal, keeping the space tidy.
The balance between them decides whether you get sharp copies or usable structure.
Outputs are famously blurry, because averaging over the sampled region smooths detail away.
GANs and diffusion models produce sharper images, and the VAE survives as the compressor inside latent diffusion.
The encoder outputs a mean and a variance rather than a point, so one input maps to a cloud in the latent space.
Sampling from that cloud during training means nearby codes must decode to plausible outputs, which is what fills in the gaps a plain autoencoder leaves.
A prior is imposed on the whole latent space, so a vector drawn from a standard normal decodes to something valid, and the model can generate.
The likelihood of the data is intractable, so training maximises a lower bound on it, which is what the E, L and B stand for.
The first term is reconstruction: decode the sampled code and score the original against it.
The second is a regulariser pulling each input's cloud towards the prior, and it is the price paid for a latent space that is continuous.
For two Gaussians that divergence has a closed form, which is the practical reason the Gaussian assumption is made at all.
A sampling step has no gradient, so backpropagation cannot pass through it as written.
Moving the randomness into an input the parameters do not depend on makes the sampled code a deterministic function of the encoder's outputs.
The gradient then flows through it normally.
That one substitution is what made variational inference trainable by gradient descent, and it is the contribution of the 2013 paper.
Log variance is predicted rather than variance, so the output can be any real number and the exponential keeps it positive.
If the decoder is powerful enough to do well without the code, the cheapest way to lower the KL term is to set equal to the prior and ignore entirely.
The model then generates fine and learns nothing in its latent space, which is a silent failure since the loss looks healthy.
Annealing from zero upward during training is the standard defence, and it is why almost every implementation has a warmup on the KL term.
The blurriness of VAE samples is the other known cost. Averaging over the posterior under a Gaussian likelihood is exactly an averaging operation, and averaged images are soft.
One question on this card. Every option explains itself once you pick, right or wrong.
18 concepts in this module, then a graded test with every answer explained.