After weeks of teaching computers to read in NLP, our course moved on to teaching them to see. Here’s what I learned, and what happened when I tested real models on my own photos from Rwanda.
A photo is just numbers
A computer doesn’t see people or objects, only numbers. My photo of people painting is 547 × 365 pixels, and each pixel has 3 values (red, green, blue) from 0 to 255: almost 600,000 numbers. One pixel, [226, 168, 43], was mostly red and green, which makes yellow: it was on a yellow shirt.
Filters find edges
A filter is a small grid of weights that slides over the image, multiplying and adding pixel values. The Sobel filter subtracts the left side from the right, so flat areas give 0 and edges give big numbers. A CNN uses this same idea, but it learns its own filters from data instead of us designing them.
Testing models on my photos
I used three pretrained YOLO models on a painting photo and a photo of agaseke baskets:
Classification called the painting scene a “marimba” (39.8%), probably because of the wooden easels, and the agaseke a “soap dispenser” (21%).
Detection found 4 people when there were only 3 (one woman got two boxes), and called the baskets “bottles,” since it has no “basket” category.
Segmentation showed objects covering 38% of the image.
Then I made the photo harder. The model still found 3 people when the photo was very dark, saw a “horse” when it was upside down, and found nothing when it was blurry or low resolution.
What I learned
The biggest lesson: a model only knows what it was trained on. These models learned mostly from photos taken outside Africa, so they had never seen an agaseke. The fix is local data, used to fine-tune a pretrained model.
This is already happening across Africa: Nuru spots cassava diseases from leaf photos, Smile ID checks identities from selfies, Open Buildings maps buildings from satellite images, and BLAIVA, built in Kigali, describes the world to people who are blind.
Key takeaways
- An image is a grid of numbers: height × width × 3.
- Filters find patterns; CNNs learn them.
- Classification, detection and segmentation give more and more detail.
- Models fail on things unlike their training data, so local data matters.
Top comments (1)
Great article, Nadia! I really liked how you connected computer vision to our previous NLP lessons and used local examples like agaseke baskets to demonstrate the limitations of pretrained models. The part where the model confused an agaseke with a soap dispenser was particularly interesting!
One suggestion would be to explore why the model made certain incorrect predictions. For example, why did it detect four people instead of three, and could adjusting the confidence threshold or using a different model improve the results? It would also be interesting to see a comparison of how different YOLO models perform on the same images.
Overall, I enjoyed how you explained the concepts through practical experiments and connected them to real-world challenges in Africa. Great work!