vision languige models

Vision-Language Models: Teaching AI to See and Talk
Show a photo of a crumpled bottle to a normal image classifier and it can only say "plastic" or "glass" from a fixed list. Show the same photo to a vision-language model and you can ask, "What is this made of, and can it be recycled?" It answers in plain sentences. That jump, from picking a label to understanding a picture in words, is what vision-language models (VLMs) are about.
In this post I explain what VLMs are, the two main ways they are built, how they are trained, and where they still fail.
What is a vision-language model?
A VLM is a model that understands images and text together. You give it an image, some text, or both, and it can answer in text or tell you how well an image and a sentence match.
The core problem is that an image is a grid of numbers with no words in it, and a sentence is a sequence of symbols with no pixels in it. A VLM has to connect the two. When it reads the word "bottle", it must link it to bottle-shaped patterns in the pixels. This is called grounding.
Two building blocks you may already know
• The image side: a computer vision model cuts the picture into small squares, called patches, and turns each square into a list of numbers.
• The text side: a language model cuts the sentence into small pieces, called tokens, and turns each piece into a list of numbers.
Now both the picture and the sentence are just lists of numbers. That is why they can be combined: the model only has to learn how the two kinds of lists relate to each other.
Family 1: matching images and text (CLIP)
CLIP, from OpenAI in 2021, has two encoders: an image encoder (a ViT) and a text encoder (a Transformer). Both map their input into the same vector space.
It was trained on about 400 million image and caption pairs from the internet. In each training batch, CLIP:

1. Encodes all the images and all the captions.
2. Compares every image with every caption, giving a table of similarity scores.
3. Learns to give high scores to the correct pairs (the diagonal of the table) and low scores to all the wrong pairs. This is called contrastive learning. Nobody had to label the images with classes, because the captions already describe them. Large batches matter because every other caption in the batch acts as a wrong example for each image. The zero-shot trick The best-known use of CLIP is zero-shot classification, which means classifying images without training on your labels:
4. Write a sentence for each class, such as "a photo of a glass bottle" and "a photo of a cardboard box".
5. Turn each sentence into a vector, and turn the image into a vector.
6. Choose the class whose sentence vector is closest to the image vector. To add a new class, you only write a new sentence. You do not need new labelled images or a new training run. The wording of the sentence still matters, and "a photo of a glass bottle" can work better than the bare word "glass". Family 2: image tokens inside a language model (LLaVA and friends) CLIP can compare, but it cannot write. Most modern assistants that can chat about images work differently, with three parts:
7. A vision encoder, usually a pretrained ViT, that produces patch vectors.
8. A connector (a small projection network) that translates those vectors into the language model's embedding space.
9. A large language model (LLM) that reads the result and writes the answer one token at a time. Here is a simple example. The picture is cut into 576 small squares. A small connector converts each square into the same kind of number list that the language model already understands. These converted squares are called visual tokens. The language model reads them first and your question second: Picture (576 visual tokens) + question ("What material is this?") → the model writes the answer. The language model does not know which pieces came from a picture. During training it learned that the first group of pieces describes a picture, and it uses them to write its answer. There are other designs. BLIP-2 squeezes the picture into just a few tokens, which is cheaper but can lose small details. Flamingo lets the text layers look back at the picture while they write. How these models are trained A typical recipe for a generative VLM has two main stages:
10. Learn to translate: keep the picture model and the language model fixed, and train only the connector on pictures with captions.
11. Learn to follow instructions: train the language model on examples of a picture, a question and a good answer, such as "describe this picture". While training, the model is graded only on the answer it writes. The picture and the question are just context. What VLMs can do • Describe an image in words (captioning). • Answer questions about an image (visual question answering). • Read text in photos, charts and documents. • Classify images from text prompts, with no training on your labels. • Search images using a sentence. Where they still fail VLMs are impressive, but they make mistakes that are worth knowing about: • Making things up (hallucination). The language model has strong guesses about what usually appears in a scene. When the picture is unclear, those guesses can win, and it may describe an object that is not there. • Counting. It is hard to count many small objects exactly. • Position words. Words like "left of" and "behind" are only weakly understood. • Small details. Tiny text and tiny objects can be lost, because the picture is shrunk to a fixed size. Some systems cut large pictures into pieces to help, which costs more computing power. • Word order. CLIP-style models can confuse "a dog chasing a cat" with "a cat chasing a dog". Cost matters too. More visual tokens mean more computing power and slower answers, so designers have to balance detail against cost. Try it yourself You can test CLIP on your own photos without writing any code. Any tool that lets you try CLIP works the same way:
12. Pick a photo, for example a glass bottle.
13. Write three or four short descriptions, such as "a photo of a glass bottle" and "a photo of a cardboard box".
14. Ask CLIP which description fits the photo best. It gives each one a score.
15. Change the wording and see how the scores change. The wording matters, so you can learn a lot by changing the descriptions and watching how the scores move. It is a quick way to see how much the text side of the model shapes its answer. Key takeaways • A VLM connects images and text so that one system can work with both. • CLIP matches images and captions in a shared space, and it can classify from text prompts alone. • LLaVA-style models turn an image into visual tokens and feed them to a language model, which then writes the answer. • Training usually aligns the connector first, then tunes the model on instructions. • VLMs can hallucinate, miscount and miss small details, so their answers should be checked. Vision Transformers gave computers a way to read images as sequences. Vision-language models take the next step and let those sequences meet language, which is why they are such an active area of AI right now.