RAG series · Part 1
TL;DR
- An embedding is a list of numbers that represents the meaning of a piece of text.
- Similar meanings → nearby vectors. Cosine similarity measures how close they are.
- This is the "R" (retrieval) in RAG. Pick the wrong embedding model and your RAG fails silently in other languages.
The problem
Computers can't compare meanings. They can only compare numbers.
Keyword search fails on:
- Synonyms: "kitten" vs "cat"
- Translations: "cat" vs "gato"
So we need a way to turn meaning into numbers.
The idea
An embedding model turns any text into a fixed-size list of numbers (a vector).
Start with colours
Every colour on your screen is just 3 numbers: Red, Green, Blue.
Orange → [255, 165, 0]
Dark orange → [255, 140, 0] ← numbers close to Orange's
Blue → [ 0, 0, 255] ← numbers far from both
- Orange and dark orange look similar, and their numbers are similar.
- Blue looks different, and its numbers are far away.
- 3 numbers can describe 16 million colours. 3 numbers ≠ 3 colours.
Now swap "colour" for "meaning"
An embedding does the same thing, with more numbers. How many depends on the model:
Model Numbers per text
paraphrase-multilingual-MiniLM-L12-v2 384 (used here)
BERT-base models 768
OpenAI text-embedding-3-small 1536
OpenAI text-embedding-3-large 3072
Google gemini-embedding-001 3072
More numbers can capture finer detail, but they cost more storage and make search slower.
With our model, that's 384 numbers:
"The cat sat on the mat." → [ 0.43, -0.38, -0.23, ... ] (384 numbers)
"El gato duerme en la alfombra." → [ ... similar numbers ... ] ← same meaning, close
"Stock markets fell sharply." → [ ... very different ... ] ← different meaning, far
Colours Embeddings
3 numbers (R, G, B) 384 / 768 / 1536 ... (depends on model)
similar colours → close numbers similar meanings → close numbers
orange ≈ dark orange cat ≈ gato
orange ≠ blue cat ≠ stock market
- 384 is the model's output size (its dimensions). It's fixed by the model's design, not by how many words or characters the text has.
- A single number means nothing on its own. Meaning only shows up when you compare one vector with another.
Comparing: cosine similarity
Cosine similarity measures the angle between two vectors:
- 1 → same direction (same meaning)
- 0 → unrelated
def cosine(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
It ignores vector length, so a short question can still match a long paragraph.
Build it (≈20 lines)
from sentence_transformers import SentenceTransformer
import numpy as np
docs = [
"The cat sat on the mat.",
"Dogs are loyal pets.",
"Stock markets fell sharply today.",
"El gato duerme en la alfombra.", # Spanish: "The cat sleeps on the rug."
"The central bank raised interest rates.",
]
query = "Where is the kitten?"
model = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")
doc_vecs = model.encode(docs)
q_vec = model.encode(query)
scores = [cosine(q_vec, d) for d in doc_vecs]
for score, doc in sorted(zip(scores, docs), reverse=True):
print(f"{score:.3f} {doc}")
What I saw
0.560 The cat sat on the mat.
0.480 El gato duerme en la alfombra. ← Spanish, #2!
0.138 Dogs are loyal pets.
0.049 Stock markets fell sharply today.
0.046 The central bank raised interest rates.
- "Kitten" found "cat" with zero shared words.
- The Spanish sentence came second. The model never compares "gato" with "cat" letter by letter. It was trained on millions of translation pairs, with the goal of putting both sentences of each pair at the same spot in the vector space. So where a sentence lands depends on its meaning, not its language.
Break it: an English-only model
I ran the same test with all-MiniLM-L6-v2, which was trained only on English:
Model Query Cat (EN) Gato (ES)
Multilingual "Where is the kitten?" 0.560 0.480
Multilingual "¿Dónde está el gatito?" 0.569 0.467
English-only "Where is the kitten?" 0.418 0.096
English-only "¿Dónde está el gatito?" -0.069 0.616 ← !?
The last row is a trap. The English-only model scored the Spanish sentence highest, but not because it understood it. Here's what its tokenizer produced:
query: ['¿', 'don', '##de', 'est', '##a', 'el', 'ga', '##ti', '##to', '?']
doc: ['el', 'ga', '##to', 'due', '##rm', '##e', 'en', 'la', 'alf', '##om', '##bra', '.']
They share the pieces el, ga and ##to, so this is spelling overlap, not meaning. Meanwhile it ranked the English cat sentence dead last.
Lesson: a model can look multilingual when the query and the document share spelling. Only cross-lingual tests (question in language A, answer in language B) prove it really is.
Gotchas
- Only compare vectors from the same model. Each model has its own "map" of meaning. Comparing a vector from model A with one from model B is like comparing GPS coordinates with RGB values. If you switch models, you must re-embed every document.
- Max length: This multi-lingual model reads atmost 128 tokens. Anything longer is silently cut-off. Thats why RAG Splits documents into chunks. We can check the max sequence length (chunk size) from the config file of the model we downloaded (or the docs of a provider). I have given an example below
bash
# PS: I am using a mac, so copy pasting this might not work.
# Paste the below in any AI Assistant (Claude, Gemini, Codex) - and ask it to adapt for your system.
cd ~/.cache/huggingface/hub/models--sentence-transformers--paraphrase-multilingual-MiniLM-L12-v2/snapshots/*/
cat sentence_bert_config.json
Output :
{
"max_seq_length": 128, // That is how I got the number 128.
"do_lower_case": false
}
- The tokenizer is the model's eyes. If it chops a language into meaningless pieces, the model can only match spelling.
Next in the series: Inside the model, or how "Hi" becomes 384 numbers.
Top comments (0)