A Vision Transformer splits the image into fixed-size patches.
Each patch is flattened and embedded like a token in a sentence.
Self-attention then lets every patch look at every other patch.
This captures long-range relationships that convolution reaches only slowly.
ViTs need more data than CNNs but scale better once you have it.
A vision transformer applies the language architecture to images with almost no changes.
The image is cut into fixed patches, typically 16 by 16 pixels.
A 224 by 224 image becomes 196 patches.
Each patch is flattened and projected into a vector, exactly like a word embedding.
Position information is added, since attention has no idea about arrangement.
Self-attention then lets every patch attend to every other patch.
A patch in the top corner relates directly to one in the bottom corner, in a single step.
A CNN needs many layers to connect those two places.
The catch is that a transformer assumes far less about images than a CNN does.
It does not know that nearby pixels are related or that patterns repeat across positions.
Everything a CNN builds in must instead be learned from data.
So ViTs underperform CNNs on small datasets and overtake them on very large ones.
Pretraining on hundreds of millions of images is what unlocked them.
Hybrid designs and hierarchical variants such as Swin narrowed the data requirement.
Treating images as sequences also made multimodal models much easier to build.
A transformer takes a sequence of vectors. An image is a grid of pixels, so the only real question is how to turn one into the other.
A 224 by 224 image cut into 16 by 16 patches gives 196 patches, and each patch is flattened into a vector of numbers.
That projection is equivalent to a single convolution with kernel size and stride , which is exactly how it is implemented.
After that line, nothing about the model is specific to images. It is the same encoder used for text.
Self-attention is permutation invariant, so without positions the model sees a bag of patches and cannot tell a face from its parts rearranged.
The embedding is learned rather than derived, and interpolating it is what allows a model trained at 224 pixels to be fine-tuned at 384.
The extra token belongs to no patch. It attends to all of them and its final state is the image representation the classifier reads, which avoids having to pool the patches by hand.
Attention is quadratic in the sequence length, and the sequence length is quadratic in the image side.
Halving the patch size quadruples the tokens and multiplies the attention cost by sixteen.
That is why 16 is the usual compromise, and why high resolution vision transformers use windowed attention instead.
A convolution assumes neighbouring pixels are related and that a pattern means the same thing anywhere. A vision transformer assumes neither and has to learn both.
That is why it loses to a convolutional network on ImageNet alone and wins decisively once pretrained on a hundred million images.
The gain is reach. A convolutional network builds up its receptive field layer by layer, while every patch here can attend to every other patch in the first block.
Hybrids now dominate in practice: a convolutional stem for the cheap local structure, attention above it for the long-range relationships.
One question on this card. Every option explains itself once you pick, right or wrong.
22 concepts in this module, then a graded test with every answer explained.