Suprisingly two sleeping cats taught me Vision Language models

I wanted to understand vision language models (VLMs), the AI that looks at a photo and answers questions about it. So I started with a tiny grid of numbers and ended up quizzing an AI about two sleeping cats.

Everything below runs free in Google Colab.

An image is just numbers

import numpy as np

tiny = np.zeros((8, 8))
tiny[:, 4:] = 9   # left dark, right bright
print(tiny)

Output:

[[0. 0. 0. 0. 9. 9. 9. 9.]
 [0. 0. 0. 0. 9. 9. 9. 9.]
 [0. 0. 0. 0. 9. 9. 9. 9.]
 [0. 0. 0. 0. 9. 9. 9. 9.]
 [0. 0. 0. 0. 9. 9. 9. 9.]
 [0. 0. 0. 0. 9. 9. 9. 9.]
 [0. 0. 0. 0. 9. 9. 9. 9.]
 [0. 0. 0. 0. 9. 9. 9. 9.]]

Every row is 0 0 0 0 9 9 9 9. There's an edge, but the computer needs a way to find it.

A filter finds the edge

kernel = np.array([-1, 0, 1])   # right pixel minus left pixel
out = np.array([[np.sum(tiny[r, c:c+3] * kernel) for c in range(6)] for r in range(8)])
print(out)

Output:

[[0. 0. 9. 9. 0. 0.]
 [0. 0. 9. 9. 0. 0.]
 [0. 0. 9. 9. 0. 0.]
 [0. 0. 9. 9. 0. 0.]
 [0. 0. 9. 9. 0. 0.]
 [0. 0. 9. 9. 0. 0.]
 [0. 0. 9. 9. 0. 0.]
 [0. 0. 9. 9. 0. 0.]]

Each row becomes 0 0 9 9 0 0. Flat areas give 0, and only the edge lights up.

The same filter on a real photo

import requests
from PIL import Image

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
img = Image.open(requests.get(url, stream=True).raw).convert("RGB")

gray = np.array(img.convert("L")).astype(float)
edges = np.abs(gray[:, 2:] - gray[:, :-2])
Image.fromarray(np.clip(edges, 0, 255).astype("uint8"))

Output:

The smooth blanket went black, while the fur, ears, paws and remotes lit up. This is the core idea behind a CNN, except it learns thousands of filters instead of us writing one.

Testing a real vision language model

from transformers import AutoProcessor, AutoModelForImageTextToText

model_id = "HuggingFaceTB/SmolVLM-256M-Instruct"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id)

def ask(q, image):
    msgs = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": q}]}]
    prompt = processor.apply_chat_template(msgs, add_generation_prompt=True)
    inputs = processor(text=prompt, images=[image], return_tensors="pt")
    print("input tokens:", inputs["input_ids"].shape[1])
    out = model.generate(**inputs, max_new_tokens=80)
    return processor.batch_decode(out, skip_special_tokens=True)[0].split("Assistant:")[-1]

Then I asked it four questions about the photo:

for q in ["Describe the image in detail.",
          "How many cats are there?",
          "Is there a dog in this image?",
          "Where are the remote controls?"]:
    print("\nQ:", q)
    print("A:", ask(q, img))

Output:

Q: Describe the image in detail.
input tokens: 876
A: The image depicts two cats lying on a pink, fabric-covered surface.
The cats are positioned side by side, facing opposite directions. The cat
on the left is larger and has a lighter fur pattern, while the cat on the
right is smaller and has a darker fur pattern.

Q: How many cats are there?
input tokens: 876
A: There are two cats in the image.

Q: Is there a dog in this image?
input tokens: 878
A: No, there is no dog present in this image.

Q: Where are the remote controls?
input tokens: 876
A: The remote controls are on a pink color cloth.

My questions were about 6 words, but the model took in 876 tokens. Nearly all of them were the image.

The scorecard

Question	Result
How many cats?	✅ Two
Is there a dog?	✅ No
Describe the image	❌ Swapped which cat was on the left
Where are the remotes?	❌ "On a pink cloth" (they're on the couch)

The big lesson

It sounded just as confident when it was wrong. A fluent answer isn't proof of a correct one, so always check it against the image.

Photo: COCO dataset. Model: SmolVLM by Hugging Face.