Embeddings from First Principle : How “Kitten” finds “Cat” (and “Gato”)

Ritwick Biswas — Tue, 13 Oct 2026 05:01:37 +0000

RAG series · Part 1

TL;DR

An embedding is a list of numbers that represents the meaning of a piece of text.
Similar meanings → nearby vectors. Cosine similarity measures how close they are.
This is the "R" (retrieval) in RAG. Pick the wrong embedding model and your RAG fails silently in other languages.

The problem

Computers can't compare meanings. They can only compare numbers.

Keyword search fails on:

Synonyms: "kitten" vs "cat"
Translations: "cat" vs "gato"

So we need a way to turn meaning into numbers.

The idea

An embedding model turns any text into a fixed-size list of numbers (a vector).

Start with colours

Every colour on your screen is just 3 numbers: Red, Green, Blue.

Orange       → [255, 165,   0]
Dark orange  → [255, 140,   0]   ← numbers close to Orange's
Blue         → [  0,   0, 255]   ← numbers far from both

Orange and dark orange look similar, and their numbers are similar.
Blue looks different, and its numbers are far away.
3 numbers can describe 16 million colours. 3 numbers ≠ 3 colours.

Now swap "colour" for "meaning"

An embedding does the same thing, with more numbers. How many depends on the model:

Model                                   Numbers per text
paraphrase-multilingual-MiniLM-L12-v2   384   (used here)
BERT-base models                        768
OpenAI text-embedding-3-small           1536
OpenAI text-embedding-3-large           3072
Google gemini-embedding-001             3072

More numbers can capture finer detail, but they cost more storage and make search slower.

With our model, that's 384 numbers:

"The cat sat on the mat."         → [ 0.43, -0.38, -0.23, ... ]   (384 numbers)
"El gato duerme en la alfombra."  → [  ...  similar numbers ... ]   ← same meaning, close
"Stock markets fell sharply."     → [  ... very different ...   ]   ← different meaning, far

Colours                          Embeddings
3 numbers (R, G, B)              384 / 768 / 1536 ... (depends on model)
similar colours → close numbers  similar meanings → close numbers
orange ≈ dark orange             cat ≈ gato
orange ≠ blue                    cat ≠ stock market

384 is the model's output size (its dimensions). It's fixed by the model's design, not by how many words or characters the text has.
A single number means nothing on its own. Meaning only shows up when you compare one vector with another.

Comparing: cosine similarity

Cosine similarity measures the angle between two vectors:

1 → same direction (same meaning)
0 → unrelated

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

It ignores vector length, so a short question can still match a long paragraph.

Build it (≈20 lines)

from sentence_transformers import SentenceTransformer
import numpy as np

docs = [
    "The cat sat on the mat.",
    "Dogs are loyal pets.",
    "Stock markets fell sharply today.",
    "El gato duerme en la alfombra.",   # Spanish: "The cat sleeps on the rug."
    "The central bank raised interest rates.",
]
query = "Where is the kitten?"

model = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")
doc_vecs = model.encode(docs)
q_vec = model.encode(query)

scores = [cosine(q_vec, d) for d in doc_vecs]
for score, doc in sorted(zip(scores, docs), reverse=True):
    print(f"{score:.3f}  {doc}")

What I saw

0.560  The cat sat on the mat.
0.480  El gato duerme en la alfombra.     ← Spanish, #2!
0.138  Dogs are loyal pets.
0.049  Stock markets fell sharply today.
0.046  The central bank raised interest rates.

"Kitten" found "cat" with zero shared words.
The Spanish sentence came second. The model never compares "gato" with "cat" letter by letter. It was trained on millions of translation pairs, with the goal of putting both sentences of each pair at the same spot in the vector space. So where a sentence lands depends on its meaning, not its language.

Break it: an English-only model

I ran the same test with all-MiniLM-L6-v2, which was trained only on English:

Model          Query                      Cat (EN)   Gato (ES)
Multilingual   "Where is the kitten?"       0.560      0.480
Multilingual   "¿Dónde está el gatito?"     0.569      0.467
English-only   "Where is the kitten?"       0.418      0.096
English-only   "¿Dónde está el gatito?"    -0.069      0.616  ← !?

The last row is a trap. The English-only model scored the Spanish sentence highest, but not because it understood it. Here's what its tokenizer produced:

query: ['¿', 'don', '##de', 'est', '##a', 'el', 'ga', '##ti', '##to', '?']
doc:   ['el', 'ga', '##to', 'due', '##rm', '##e', 'en', 'la', 'alf', '##om', '##bra', '.']

They share the pieces el, ga and ##to, so this is spelling overlap, not meaning. Meanwhile it ranked the English cat sentence dead last.

Lesson: a model can look multilingual when the query and the document share spelling. Only cross-lingual tests (question in language A, answer in language B) prove it really is.

Gotchas

Only compare vectors from the same model. Each model has its own "map" of meaning. Comparing a vector from model A with one from model B is like comparing GPS coordinates with RGB values. If you switch models, you must re-embed every document.
Max length: This multi-lingual model reads atmost 128 tokens. Anything longer is silently cut-off. Thats why RAG Splits documents into chunks. We can check the max sequence length (chunk size) from the config file of the model we downloaded (or the docs of a provider). I have given an example below

bash
# PS: I am using a mac, so copy pasting this might not work.
# Paste the below in any AI Assistant (Claude, Gemini, Codex) - and ask it to adapt for your system.
cd ~/.cache/huggingface/hub/models--sentence-transformers--paraphrase-multilingual-MiniLM-L12-v2/snapshots/*/ 
cat sentence_bert_config.json

Output :
{
  "max_seq_length": 128, // That is how I got the number 128. 
  "do_lower_case": false
}

The tokenizer is the model's eyes. If it chops a language into meaningless pieces, the model can only match spelling.

Next in the series: Inside the model, or how "Hi" becomes 384 numbers.

DEV Community: Ritwick Biswas