When I first learned about RAG, embeddings were an abstraction I accepted without questioning. "Sentences get converted to vectors," "similar meanings end up close together in vector space" — all true, but none of it clicked until I actually measured the numbers. So I installed sentence-transformers locally, measured cosine similarities, ran a mini retrieval simulation, and checked what happens with Korean queries specifically.
The short answer: building a Korean RAG system with an English-optimized embedding model dropped accuracy by 67% in my tests. Two out of three queries returned the wrong document at rank 1. This post documents that experiment and how switching to a multilingual model fixed it.
What it actually means to run an embedding model locally with pip
I didn't believe it was possible without a cloud API at first. OpenAI and Gemini embeddings require API keys and round-trips to external servers. But sentence-transformers just downloads model weights locally.
pip install sentence-transformers
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
embedding = model.encode("building an AI agent")
print(embedding.shape) # (384,)
all-MiniLM-L6-v2 downloads from Hugging Face Hub on first run. Load time on my machine: 9.52 seconds. After that, it's cached and loads in under 1 second. Model size is around 22MB — genuinely lightweight.
The vector internals are worth inspecting:
Type: numpy.ndarray
Shape: (384,)
dtype: float32
L2 norm: 1.000000
Min: -0.147123
Max: 0.183166
First 5 dims: [-0.0216, 0.0593, -0.0049, -0.0172, 0.0079]
The L2 norm being exactly 1.0 is interesting. sentence-transformers normalizes embeddings by default, which means cosine similarity and dot product return identical results. This is a nice property — you don't need to remember to normalize before computing similarity.
384 dimensions is a deliberate tradeoff. Modern large embedding models use 1536〜3072 dimensions, which is more expressive but costs more to store and search. For most semantic search use cases, 384 is plenty.
Cosine similarity: what the numbers actually look like
from sentence_transformers.util import cos_sim
pairs = [
("AI agent uses tools to complete tasks",
"An autonomous agent invokes functions to achieve goals"),
("AI agent uses tools to complete tasks",
"The cat sat on the warm mat by the window"),
("How do I install Python packages?",
"What is the command to add a Python library?"),
("How do I install Python packages?",
"The stock market closed higher today"),
]
Measured similarity scores:
0.6489 ████████████ AI agent vs autonomous agent (same meaning, different words)
-0.0112 AI agent vs cat on mat (unrelated)
0.6248 ████████████ install packages vs add library (same meaning)
-0.0163 install packages vs stock market (unrelated)
Semantically similar pairs land at 0.62〜0.65. Unrelated pairs are near zero or slightly negative. The negative values surprised me at first — cosine similarity ranges from -1 to 1, so truly unrelated sentences naturally cluster around 0, sometimes dipping slightly negative.
For context on what "high" looks like in practice: the vector DB benchmark post notes that RAG systems typically use 0.3〜0.5 as a retrieval threshold. So 0.65 is a strong semantic match.
Mini RAG simulation — two out of three queries failed
I ran a retrieval simulation with 10 Korean blog post titles as the knowledge base and 3 Korean queries. Model: English-optimized all-MiniLM-L6-v2.
Knowledge base included titles like:
- "Reduce LLM costs 70% with Claude API Prompt Caching"
- "MCP vs A2A vs Open Responses — agent protocol comparison"
- "Use Node.js built-in SQLite without external packages"
Queries:
- "How do I reduce LLM API costs?"
- "Compare agent-to-agent communication protocols"
- "Lightweight database without Python"
Results:
Query: "How do I reduce LLM API costs?"
#1 [0.5649] Anthropic Message Batches API for bulk processing ← expected: Prompt Caching
#2 [0.4534] Claude API Prompt Caching — 70% cost reduction
Query: "Compare agent-to-agent communication protocols"
#1 [0.5605] AI Agent Observability Guide ← completely wrong topic
(MCP vs A2A vs Open Responses ranked outside top 3)
Query: "Lightweight database without Python"
#1 [0.5172] LangGraph multi-agent state management ← irrelevant
(Node.js SQLite not in top 3)
One query found the right answer at rank 2. The other two were completely off. The "agent protocol comparison" failure is particularly bad — there's a document with "protocol comparison" literally in its title, and the model ranked an observability guide above it. This is what an English model mishandling Korean semantics looks like.
Why English models fail on Korean and how multilingual models fix it
all-MiniLM-L6-v2 comes from the SBERT paper and was trained primarily on English sentence pairs. It can process Korean text but doesn't represent Korean semantics with the same fidelity as English.
paraphrase-multilingual-MiniLM-L12-v2 was trained on parallel corpora across 50+ languages. It's explicitly designed to place the same meaning in different languages close together in vector space.
Same 3 queries, both models:
| Query | English model #1 (correct?) | Multilingual #1 (correct?) |
|---|---|---|
| Reduce LLM API costs | Message Batches [0.453] ✗ | Prompt Caching [0.720] ✓ |
| Agent protocol comparison | Observability [0.561] ✗ | MCP vs A2A [0.647] ✓ |
| Lightweight DB without Python | LangGraph [0.461] ✗ | Node.js SQLite [0.439] ✓ |
English model: 1/3 (33%). Multilingual: 3/3 (100%).
The similarity scores are also telling. The multilingual model assigned 0.720 to the correct cost-reduction document; the English model only managed 0.453 for the same pair. Stronger semantic connection, better ranked.
The RAG architecture post argues that retrieval quality determines generation quality. This experiment shows that principle applies at the embedding model selection step — before you've written a single line of retrieval code.
My conclusion: for Korean or multilingual RAG pipelines, start with a multilingual model. Switching later means re-embedding your entire document collection, which is a meaningful operational cost for large corpora.
Batch encoding: the 2.4x throughput gap
Sequential encoding is the obvious first implementation. Batch encoding is faster for reasons that have to do with CPU/GPU parallelism and memory bandwidth — but how much faster exactly?
sentences = [f"This is test sentence number {i}" for i in range(100)]
# Sequential: 1.075 seconds
for s in sentences:
model.encode(s)
# Batch: 0.455 seconds
model.encode(sentences, batch_size=32, show_progress_bar=False)
Sequential (100 sentences): 1.075s
Batch (100 sentences): 0.455s → 2.4x faster
Throughput (batch): 220 sentences/sec
220 sentences/second on CPU. For 10,000 documents at 5 sentences each, that's 50,000 sentences — about 227 seconds, under 4 minutes. With GPU (CUDA or Apple Silicon MPS), throughput reportedly scales 10〜50x higher, though I didn't test that directly.
The practical implication: use model.encode(batch, batch_size=N) instead of calling encode in a loop. This is especially true for initial indexing jobs.
Putting this into an actual RAG pipeline
Today's experiment covered the R in RAG. A complete pipeline looks like:
# Step 1: Index documents (offline, run once)
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")
documents = load_your_documents()
doc_embeddings = model.encode(documents, batch_size=32)
# Store in a vector DB (Chroma, Qdrant, pgvector)
# Step 2: Retrieve at query time (online)
query_embedding = model.encode(user_query)
# Fetch top-k by cosine similarity from vector DB
# Step 3: Generate with context (online)
context = "\n".join(retrieved_docs)
# Pass context to Claude or another LLM
One practical detail: store the model name as metadata alongside your embeddings. If you ever change models, you need to re-embed everything. Without metadata tracking, you risk mixing vectors from different embedding spaces in the same database — a subtle bug that produces silently degraded retrieval.
What I still don't know and what's next
Limitation 1: My knowledge base had only 10 documents. Real retrieval performance over thousands of documents could look quite different, and I don't have that data yet.
Limitation 2: 3/3 accuracy for the multilingual model is based on 3 test cases. Not statistically meaningful. I'd need hundreds of test pairs to make a confident claim.
Limitation 3: Embeddings have a known weakness: they struggle when specific keywords matter more than semantic similarity. "Without Python" as a constraint is the kind of thing BM25 handles better than dense retrieval. Hybrid search (BM25 + embeddings via reciprocal rank fusion) is the standard solution here, but I haven't implemented it yet.
What's next: testing multilingual-e5-large on the same benchmark, wiring up a persistent ChromaDB store, and measuring hybrid search accuracy against pure vector retrieval.
I went into this experiment thinking embedding models were interchangeable plug-ins. The 67% accuracy gap changed that view. For non-English RAG, model selection is a first-order decision that affects the entire pipeline downstream.
Top comments (0)