DEV Community: abijuru diana

VISION LANGUAGE MODEL

abijuru diana — Sat, 24 Oct 2026 20:28:06 +0000

VLM:Vision Language Transformers, is an artifiacl intelligence that can process visual information, such as images, and language, such as text, and learn how they relate to each other.
A VLM is an AI model designed to connect visual information, such as images, with language, such as captions, questions, and answers. Understanding how these models learn requires exploring their training data, their learning objectives, and the loss functions that guide their improvement.

Vision-Language model can learn relationships between what it sees in an image and what people say about
for example:

Input: an image of dog with a ball
Question: what is happening
Output:a dog running on the grass after the ball.

How does a VLM work?

Vision encoder:Turns the image into numerical features,
Multimodal connection:Helps the model combine visual and language information,
Language model:Can use that information to answer questions or describe the image.

Types of vision language models
Vision language models can be categorised based on how they process and connect visual and textual information.

Vision-to-text Models this models focus on generationg textual descriptions or answering questions based on visual inputs.
Text-to-Vision Models
text to vision models generate or modify visual content based on textual descriptions.
Cross-modal retrieval Models
Cross-modal retrieval models use one type of data, such as text or images to search and retrieve information from another modality.

Image search using text
alows users to retrieve relevant images using textual queries.
Query: mountaiin view
Output:images of mountain
Text search using images
uses an imge as input to retrieve related text, descriptions or documents .
Input:Image of a car
Output: different descriptions about the car

LET'S LEARN COMPUTER VISION

abijuru diana — Tue, 13 Oct 2026 11:32:55 +0000

KEY NOTE:A person can just once see an image or a picture and at a moment be able to describe where, which and what is in that picture.

--->You can probably ask youself how Machines just get to see an image and be able to describe all features about it right away!!

This is what we are going to see: "computer vision:how machines see,process and understand images "

What is Computer Vision?

Computer vision is a field of artificial intelligence (AI) that teaches computers to see, process, and understand visual information from digital images and videos

for example: when you uplaod a car-photo, A computer doesn't automatically know what a car is just because it sees a picture. A machine learning model must learn patterns from training images.

humans just see a picture as it is but machine just see it in numbers

what computer vision do?

Image classifiction

Predicts what an image contains
Object detection

Identifies objects and locates them wih bounding boxes
Image segmentation

Classifies individual pixels into regions or objects.
Video analysis and tracking

Detects objects across video frames and follows their movement

Images as tensors in pytorch

You have already learned about tensors in PyTorch. A tensor is a data structure that can store numbers in one or more dimensions.

Different shapes , according to the image:
1.Grayscale image
--->usually represented by 2D array of pixel values

Shape:(height * width)

2.Color image
---> usually represented by three colors channels(Red,Green,Blue)

Shape:(height*width*3)

3.A batch of color images
---> several images grouped together for model processing

shape:(batch*channels*height*width)

how to load the image in pytorch

We'll use three tools:

Pillow (PIL): opens and reads image files.
PyTorch: stores image data in tensors and prepares it for a model.
Matplotlib: displays the image and helps us visualize it.

1.Prepare the environment
import torch import numpy as np import matplotlib.pyplot as plt from PIL import Image from google.colab import files

Library ** Purpose**
torch Creates and manipulates tensors
numpy Works with numerical arrays
matplotlib Displays images
PIL.Image Opens and reads image files
files Lets you upload a file to Colab

2.upload a photo
uploaded = files.upload()

this will provide you with upload option.

3.Three-channel color image
image = Image.open(filename).convert("RGB")

The convert("RGB") instruction ensures that we work with a three-channel color image, even if the original file was grayscale or used a different color format.

4.Convert the image into numbers
# Convert the image into a NumPy array image_array = np.array(image)

Computer Vision: Filters and Finding Edges

Computer Vision techniques process those numbers to find useful patterns, such as:

Brightness and colors: Which parts are light, dark, or different colors?

Edges: Where does the cat end and the background begin?

Textures and shapes: What patterns form the ears, eyes, and body?

Objects: Does the image contain a cat, a person, a car, or something else?

What is an image filter?
--->A filter is a mathematical operation applied to an image's pixels.

Different kernel values produce different effects: some smooth an image, some detect edges, and others emphasize fine details.

Sobel X and Sobel Y: detecting changes in two directions:

Sobel X and Sobel Y are not two types of objects or two separate kinds of images. They are two filters that measure brightness changes in different directions.

--->The Sobel operator uses two (3 \times 3) kernels.

     Sx=[
         [-1  0  1],
         [-2  0  2],
         [-1  0  1]
       ]
  ----> Sobel X measures how quickly brightness changes as we move from left to right

     Sy=[
         [-1 -2 -1],
         [0   0  0],
         [1  2  1]
        ]
    ---->Sobel Y measures how quickly brightness changes as we move from top to bottom.

How does a Sobel filter actually calculate an edge?
eg:Suppose the left side is dark and the right side is bright:

I=[
[10 10 100],
[10 10 100],
[10 10 100]
]

Grayscale: removing color information

It simplifies the image when color is not the main thing we need. We can focus on brightness differences, shapes, and boundaries.

example:
Pic1: Original color image

Pic2: Grayscale image

A color image commonly has three channels: Red, Green, and Blue (RGB). Each pixel has three values.

A grayscale image represents each pixel with one brightness value rather than three color values.

Blur: reducing noise and tiny details

Blur makes an image smoother by combining information from neighboring pixels.

Why do we use blur before detection?
For example: Imagine we have a photo of leaf. and the leaf has a clear boundary but its surface contains some details and also other things such as small trees and others.

if we detect edges immediately , the computer might detect both leaf boundaary and mant tiny surface details yet we want a leaf.

Soo **Blurring **reduces some of those small variations, allowing the stronger boundaries to stand out.

1.Original photo of a leaf 2.Blurred Image

2.Blurred Image

KEY NOTE:too much blur can remove real details and make edges less precise.

What is an edge?

An edge is a location where image intensity changes significantly.
Consider these grayscale pixel values across a row:
[
[20,\ 20,\ 22,\ 21,\ 220,\ 225,\ 224]
]

The values change only slightly at first, then jump from (21) to (220).
That large change could indicate a boundary between a dark object and a bright background.

Types of edges:

Vertical edges:A boundary that runs vertically, such as the left or right side of a door.
Horizontal edges:A boundary that runs horizontally, such as the top of a table.
Diagonal edges:Boundaries that run diagonally, such as a sloping roof or the side of a tilted object.
Soft or gradual edges:Brightness changes gradually rather than suddenly, for example because of shadows, soft lighting, or a blurry boundary.

Sharpening: making details stand out

Sharpening emphasizes local differences between neighboring pixels. It can make boundaries and fine details appear clearer.

eg1: without shapening:

eg2:with shapening:

What does sharpening contribute?

Makes some boundaries look more defined.
Emphasizes small details and textures.
Can make a slightly soft image appear clearer.

Take this:
The big idea to remember is that filters let us transform an image into a form that emphasizes particular information. Grayscale focuses on brightness, blur smooths variations, Sobel measures directional changes, Canny extracts refined edges, and sharpening emphasizes local detail. These are foundational tools for understanding how more advanced Computer Vision systems process visual information.

Still learning, still exploring, and growing in Computer Vision.......

word embeddings in NLP

abijuru diana — Wed, 30 Sep 2026 14:42:48 +0000

what is word embeddings?

word embedding is techniques of NLP that mapps the text into lists of vectors to represent word's meaning.

how does word embeddings works?

numerical conversion

vector space

semantic clustering

Why does this word embeddings matters?

word embedding matters because they were the step to make machine work with meaning instead of just spelling.

Ex: before word embeddings machine could take like "good" and "great" as unrelated data but after word embeddings model can treat them as related because of the "vector space"

they enable transfer learning.
--> word embeddings can be reuse in many small project,the model starts from the general langauge knowledge instead of starting from scracth. you dont need to relearn what langauge mean everytime you run a new project

they are the sourced applications

Word embedding Methods:

Normally in word embedddings we have the 3-be-called-approaches :

count/frequency-based :This methods describe how often the words are in the sentence.
...This includes:
- BoW (Bag-Of-Word)
- TF-IDF(Term Frequency-Inverse Document Frequency)
- Co-occurrence Matrix
prediction-based :Methods use neural networks to learn word representations based on surrounding context.
...This includes:
- Word2Vec
- GloVe
- FastText
Contextual embeddings: This method generates dynamic vectors that change depending on the surrounding sentence, effectively handling polysemy
...This includes:
-ELMo (Embeddings from Language Models)
-Transformers models(BERT, GPT

FOCUS: Prediction-Based Methods:

Word2Vec Here we train neural network to guess which words appear near each other.Word2Vec looks at the context of words in a large text dataset. Words that appear in similar settings get vector coordinates close to each other in a multi-dimensional space. It uses two main architectures:

• CBOW (Continuous Bag of Words): Predicts a target center word using its surrounding context words. It works faster for frequent words.
• Skip-gram: Predicts the surrounding context words by taking a single target center word as input. It performs better with smaller datasets and rare words.

example of codes:
`from gensim.models import Word2Vec

sentences = [
["Paul", "is", "driving", "the", "car"],
["mary", "is", "driving", "a", "biycle"],
["Cat", "drank", "the", "whole", "milk",]

]
model = Word2Vec(
sentences,
vector_size=100,
window=5,
min_count=1,
sg=1,

negative=5,
epochs=50,
)

print(model.wv["driving"])

print(model.wv.most_similar("driving"))

print(model.wv.similarity("biycle", "car"))
`

from gensim.models import Word2Vec

1.-->Load the Word2Vec tool

sentences = [[...], [...], [...]]

2.-->A list of sentences, each a list of words

Word2Vec(sentences, ...)

3.-->Learn a vector for every word

model.wv[...], .most_similar(), .similarity()

4.-->Look at the results

What i learned:

1.What is embeddings
2.How is word embedding giving high volume to today's tech
3.How is it applied through different ways
4.Word2Vec approach

Sources

https://d2l.ai/index.html : from the book of dive into deep learning

-Chatgpt : some informations were prompted from chatgpt

Google