U-Net is a segmentation architecture shaped like the letter U.
The left side downsamples to capture context.
The right side upsamples back to full resolution.
Skip connections carry fine detail across from left to right.
It was designed for medical images and works well with small datasets.
U-Net is a segmentation architecture named after the shape of its diagram.
The left arm downsamples, halving resolution and doubling channels at each step.
That builds up context, so the network understands what it is looking at.
The right arm upsamples back to the original resolution to produce the mask.
The problem is that downsampling has already destroyed the fine detail the mask needs.
Skip connections solve it by carrying feature maps straight across from left to right.
Each upsampling step is handed the matching high-resolution map from the way down.
So the network combines what it is with where exactly it is.
Those horizontal links are what make the output sharp instead of blobby.
It was designed in 2015 for biomedical images, where datasets are tiny.
It works well with a few dozen annotated images plus heavy augmentation.
That efficiency is why it spread far beyond medicine.
It is still a strong baseline for any dense prediction task.
The same shape reappears inside diffusion models, which denoise images step by step.
Ten years on, a U-Net remains a sensible first thing to try on a new segmentation problem.
A classifier collapses an image to one label, so throwing away spatial detail is free. Segmentation has to return a label per pixel, so the detail has to come back.
The output is the same size as the input, with one channel per class rather than three colours.
That forces an architecture that goes down and then comes back up, which is where the U in the name comes from.
The encoder is an ordinary convolutional stack. Each stage convolves twice and pools once, halving the resolution and doubling the channels.
The decoder mirrors it. Each stage upsamples, halves the channels, and convolves twice.
The skip connection is the part that matters: before convolving, the decoder concatenates the encoder's map from the same resolution onto its own.
The encoder's deep layers know what is in the image and have lost where, because five poolings reduced 256 pixels to 8.
Its early layers know exactly where every edge is and almost nothing about what they belong to.
The decoder needs both at once, and concatenation gives it both rather than forcing one path to carry them.
Concatenation rather than addition is deliberate. Adding would force the two to share a representation, while concatenating lets the following convolution learn how to weigh them.
Pixelwise cross-entropy treats every pixel as its own classification problem, which works and is badly imbalanced when the object is small.
Dice loss measures overlap instead, so it is insensitive to how much background there is, and it is the standard choice in medical imaging for that reason.
The two are usually summed, because Dice alone has an unstable gradient when the prediction is almost empty and cross-entropy alone ignores the shape.
U-Net was published in 2015 for cell images with about 30 training examples, and it still wins on small datasets.
The reason is the skip connections and heavy augmentation rather than depth.
The same shape now sits inside every diffusion model, where the noise predictor is a U-Net with attention added.
One question on this card. Every option explains itself once you pick, right or wrong.
22 concepts in this module, then a graded test with every answer explained.