A vision-language model connects a vision encoder to a language model.
Image patches are projected into the language model's token space.
The model then treats them much like text tokens.
It can caption images, answer visual questions and read documents.
CLIP-style contrastive training is what aligns the two spaces.
A vision-language model joins a vision encoder to a language model.
The vision side turns an image into a set of feature vectors.
A projection layer maps those into the language model's token space.
The language model then processes them alongside text tokens.
So an image becomes, in effect, a few hundred unusual words in the prompt.
That is why one model can answer questions about a picture in normal language.
The projection is the piece that has to be trained to align the two sides.
Often the vision encoder and language model are frozen and only the connector is trained.
That is far cheaper than training a multimodal model from scratch.
CLIP-style pretraining is usually what gives the vision encoder text-aligned features.
Captioning, visual question answering and document understanding are the main uses.
Reading a chart or a form and returning structured data is the commercially valuable case.
Resolution is a real constraint, since images are downsampled before encoding.
Small text and fine detail are frequently lost, and the model may confabulate rather than admit it.
Tiling an image into regions is the usual workaround for high-resolution documents.
One question on this card. Every option explains itself once you pick, right or wrong.
18 concepts in this module, then a graded test with every answer explained.