CLIP trains an image encoder and a text encoder side by side.
It learns to place a picture and its caption close together in one shared space.
Mismatched pairs are pushed apart, using millions of image and caption pairs from the web.
That lets you classify an image against any list of words with no task-specific training.
It is the component that connects text prompts to pictures in image generation models.
CLIP trains an image encoder and a text encoder side by side.
Each turns its input into a vector, and both land in one shared space.
Training uses hundreds of millions of image and caption pairs scraped from the web.
For each batch, matching pairs are pulled together and mismatched ones pushed apart.
That is contrastive learning, and the negatives come free from the rest of the batch.
No labels were needed, since the caption alongside the image is the supervision.
The result is a space where a photograph of a dog sits near the words a photo of a dog.
That enables zero-shot classification against any list of words.
Embed the image, embed each candidate label, and take the closest.
No task-specific training is required, which was the striking result in 2021.
It also enables image search by text and text search by image.
Its most consequential use is inside image generation, connecting prompts to pictures.
The text encoder from CLIP is what conditions the denoising in many diffusion models.
It inherits the biases of web captions, which is well documented.
It is also weak at counting, spatial relations and reading text in images.
Two separate encoders produce vectors in the same space, and the only thing tying them together is the training objective.
A batch of pairs gives an matrix of similarities, and the task is to pick the correct partner in each row and in each column.
It is cross-entropy over the batch, with the diagonal as the labels, applied in both directions.
Every other caption in the batch is a negative, so a batch of 32,768 gives tens of thousands of negatives for free.
CLIP was trained at that scale, and the result degrades sharply at small batch sizes because the task becomes too easy to teach anything fine-grained.
The training set was 400 million image and text pairs.
The data came from the web rather than from labelled datasets, which is the other half of why it works: the supervision is the caption somebody already wrote.
Write one sentence per class, embed them, and classify an image by which sentence it is closest to.
The usual template is "a photo of a {class}".
The prompt template matters by several points, because the text encoder saw captions rather than bare nouns during training.
No training is needed for a new label set, which is the property that made the model interesting: the classifier is defined by words at inference time.
Searching a photo library by description is the direct application, and it needs no model at query time beyond one text embedding.
Text-to-image models use the CLIP text encoder to turn a prompt into the vectors their cross-attention reads, which is why prompt phrasing behaves the way it does in those tools.
It also serves as an automatic metric for how well a generated image matches its prompt, with the known circularity that models conditioned on CLIP score well by construction.
Its weaknesses are specific: counting, spatial relations, and text inside images are all poor, because contrastive captions rarely depend on them.
One question on this card. Every option explains itself once you pick, right or wrong.
18 concepts in this module, then a graded test with every answer explained.