Not All Multilingual Embedding Models Are Equal — Testing 5 of Them

Part of the "From Hallucination to Precision" series.

If your RAG documents are in English and your users search in German, French, or Spanish, you already know you need a "multilingual" embedding model. But which one? "Multilingual" isn't one thing — it's a label that covers models of very different sizes, ages, and training approaches, and they don't all perform the same.

I tested five multilingual embedding models against the same task: a small English knowledge base built entirely around one subject — a cat — with several near-duplicate sentences on purpose, keyword queries in three other languages, and one question — did each model retrieve the exact right sentence? All local, via sentence-transformers, no API keys.

This post shows the exact code and the exact numbers it produced.

The five models

Model	Dim	What it is
paraphrase-multilingual-MiniLM-L12-v2	384	Smaller, older (2021), covers 50+ languages
paraphrase-multilingual-mpnet-base-v2	768	Larger version of the same family
intfloat/multilingual-e5-small	384	Newer, competitive multilingual retrieval model
intfloat/multilingual-e5-base	768	Same E5 family, bigger
distiluse-base-multilingual-cased-v2	512	Compact, fast, 50+ languages

All five are multilingual. None are English-only. The question isn't "does multilingual beat English-only" — that's obvious. The question is: which multilingual model actually works, and how much do they differ from each other?

The setup

The knowledge base: 10 English sentences, all about a cat, on purpose.

• 6 near-duplicates, including two intentionally close pairs: "sat on the mat" vs. "sat on the windowsill," and "knocked over a vase" vs. "knocked over a lamp." These sit close together in meaning — a query has to land on the exact right sentence, not just "something about a cat sitting somewhere" or "something the cat knocked over."
• 4 distinct-context sentences: won an award at a pet show, needs a vet checkup, was adopted from a shelter, had an empty food bowl. Still about the same cat, but different enough in topic that they're an easier case.

One subject, varying difficulty — closer to how a real knowledge base often looks: many entries about the same product or feature, worded slightly differently.

Instead of full sentences, queries are short keywords — because that's how people actually search. Four keyword queries per language: German, French, Spanish, plus English as a same-language control. Several deliberately target one half of a close pair (e.g. "knocked over a lamp," not "a vase") to see whether the model finds the exact match or grabs its near-twin.

For every query: embed it, compare it against all 10 corpus sentences with cosine similarity, and see which one wins. Whichever sentence scores highest is what a real RAG pipeline would hand to the LLM as context. Wrong match, wrong context, wrong answer — silently, no crash, no error.

The code

"""
Embedding Model Comparison — 5 multilingual models, cross-lingual retrieval test.
No API keys. Runs fully local. Requires: pip install sentence-transformers
"""

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

MODEL_NAMES = [
    "paraphrase-multilingual-MiniLM-L12-v2",
    "paraphrase-multilingual-mpnet-base-v2",
    "intfloat/multilingual-e5-small",
    "intfloat/multilingual-e5-base",
    "distiluse-base-multilingual-cased-v2",
]

LANGUAGE_ORDER = ["German", "French", "Spanish", "English"]

# Knowledge base: 10 sentences, all about a cat. 6 are near-duplicates,
# including two intentionally close pairs (sat-on-X, knocked-over-X).
# 4 are still cat-related but describe clearly different situations.
CORPUS = [
    "The cat sat on the mat.",                                # 0 - tight cluster
    "The cat sat on the windowsill.",                         # 1 - tight cluster (close to 0)
    "The cat knocked over a vase.",                           # 2 - tight cluster
    "The cat knocked over a lamp.",                           # 3 - tight cluster (close to 2)
    "The cat is sleeping in the sun.",                        # 4 - tight cluster
    "The cat scratched the sofa.",                            # 5 - tight cluster
    "The cat won an award at the pet show.",                  # 6 - distinct context
    "The cat needs to see the vet for a checkup.",            # 7 - distinct context
    "The cat was adopted from a shelter last year.",          # 8 - distinct context
    "The cat's food bowl was empty.",                         # 9 - distinct context
]

# Keyword queries: 4 per language (German, French, Spanish, English control).
# Several deliberately target one half of a close pair (e.g. "knocked over a
# lamp") to see if the model finds the exact match or grabs its near-twin.
QUERIES = [
    {"answer_index": 1, "language": "German", "text": "Katze Fensterbank gesessen"},
    {"answer_index": 3, "language": "German", "text": "Katze Lampe umgeworfen"},
    {"answer_index": 7, "language": "German", "text": "Katze Tierarzt Untersuchung"},
    {"answer_index": 9, "language": "German", "text": "Katze Napf leer"},

    {"answer_index": 0, "language": "French", "text": "chat assis tapis"},
    {"answer_index": 2, "language": "French", "text": "chat renversé vase"},
    {"answer_index": 6, "language": "French", "text": "chat prix concours"},
    {"answer_index": 8, "language": "French", "text": "chat adopté refuge"},

    {"answer_index": 1, "language": "Spanish", "text": "gato sentado ventana"},
    {"answer_index": 5, "language": "Spanish", "text": "gato arañó sofá"},
    {"answer_index": 7, "language": "Spanish", "text": "gato veterinario revisión"},
    {"answer_index": 9, "language": "Spanish", "text": "plato comida vacío"},

    {"answer_index": 3, "language": "English", "text": "cat knocked lamp"},
    {"answer_index": 4, "language": "English", "text": "cat sleeping sun"},
    {"answer_index": 6, "language": "English", "text": "cat award pet show"},
    {"answer_index": 8, "language": "English", "text": "cat adopted shelter"},
]


def get_embedding_dim(model):
    # sentence-transformers renamed this method; support both.
    if hasattr(model, "get_embedding_dimension"):
        return model.get_embedding_dimension()
    return model.get_sentence_embedding_dimension()


def test_one_model(model_name):
    print("=" * 78)
    print(f"MODEL: {model_name}")
    print("=" * 78)

    model = SentenceTransformer(model_name)
    dim = get_embedding_dim(model)
    print(f"Embedding dimension: {dim}\n")

    corpus_embeddings = model.encode(CORPUS, convert_to_tensor=True)
    results = {lang: {"hits": 0, "total": 0} for lang in LANGUAGE_ORDER}

    for query in QUERIES:
        query_embedding = model.encode(query["text"], convert_to_tensor=True)
        scores = cos_sim(query_embedding, corpus_embeddings)[0]
        top_match_index = int(scores.argmax())
        top_match_score = float(scores[top_match_index])

        is_correct = (top_match_index == query["answer_index"])
        language = query["language"]
        results[language]["total"] += 1
        if is_correct:
            results[language]["hits"] += 1

        status = "HIT " if is_correct else "MISS"
        match_percent = (top_match_score + 1) / 2 * 100
        print(f"  [{status}] ({language}) \"{query['text']}\"  ->  \"{CORPUS[top_match_index]}\"  ({match_percent:.0f}% match)")

    print()
    accuracy_by_language = {}
    for language in LANGUAGE_ORDER:
        hits = results[language]["hits"]
        total = results[language]["total"]
        accuracy = hits / total if total else 0.0
        accuracy_by_language[language] = accuracy
        print(f"  {language} accuracy: {hits}/{total} = {accuracy:.0%}")
    print()

    return dim, accuracy_by_language


def print_summary_table(all_results):
    print("=" * 78)
    print("SUMMARY — Retrieval Accuracy by Model and Query Language")
    print("=" * 78)

    header = f"{'Model':32}{'Dim':>6}"
    for language in LANGUAGE_ORDER:
        header += f"{language:>12}"
    print(header)
    print("-" * len(header))

    for model_name, (dim, accuracy_by_language) in all_results.items():
        row = f"{model_name:32}{dim:>6}"
        for language in LANGUAGE_ORDER:
            row += f"{accuracy_by_language[language]:>12.0%}"
        print(row)


def main():
    all_results = {}
    for model_name in MODEL_NAMES:
        dim, accuracy_by_language = test_one_model(model_name)
        all_results[model_name] = (dim, accuracy_by_language)

    print_summary_table(all_results)


if __name__ == "__main__":
    main()

Run it yourself:

pip install sentence-transformers
python3 embedding_comparison.py

The results

Here's the real output, one model at a time.

paraphrase-multilingual-MiniLM-L12-v2

Embedding dimension: 384

  [HIT ] (German) "Katze Fensterbank gesessen"  ->  "The cat sat on the windowsill."  (89% match)
  [HIT ] (German) "Katze Lampe umgeworfen"  ->  "The cat knocked over a lamp."  (93% match)
  [HIT ] (German) "Katze Tierarzt Untersuchung"  ->  "The cat needs to see the vet for a checkup."  (90% match)
  [MISS] (German) "Katze Napf leer"  ->  "The cat was adopted from a shelter last year."  (83% match)
  [HIT ] (French) "chat assis tapis"  ->  "The cat sat on the mat."  (62% match)
  [HIT ] (French) "chat renversé vase"  ->  "The cat knocked over a vase."  (93% match)
  [HIT ] (French) "chat prix concours"  ->  "The cat won an award at the pet show."  (87% match)
  [HIT ] (French) "chat adopté refuge"  ->  "The cat was adopted from a shelter last year."  (86% match)
  [HIT ] (Spanish) "gato sentado ventana"  ->  "The cat sat on the windowsill."  (94% match)
  [HIT ] (Spanish) "gato arañó sofá"  ->  "The cat scratched the sofa."  (95% match)
  [HIT ] (Spanish) "gato veterinario revisión"  ->  "The cat needs to see the vet for a checkup."  (88% match)
  [HIT ] (Spanish) "plato comida vacío"  ->  "The cat's food bowl was empty."  (78% match)
  [HIT ] (English) "cat knocked lamp"  ->  "The cat knocked over a lamp."  (97% match)
  [HIT ] (English) "cat sleeping sun"  ->  "The cat is sleeping in the sun."  (96% match)
  [HIT ] (English) "cat award pet show"  ->  "The cat won an award at the pet show."  (96% match)
  [HIT ] (English) "cat adopted shelter"  ->  "The cat was adopted from a shelter last year."  (93% match)

  German accuracy: 3/4 = 75%
  French accuracy: 4/4 = 100%
  Spanish accuracy: 4/4 = 100%
  English accuracy: 4/4 = 100%

One miss: "Katze Napf leer" (cat's bowl empty) matched to "adopted from a shelter" instead of "food bowl was empty."

paraphrase-multilingual-mpnet-base-v2

Embedding dimension: 768

  [HIT ] (German) "Katze Fensterbank gesessen"  ->  "The cat sat on the windowsill."  (88% match)
  [HIT ] (German) "Katze Lampe umgeworfen"  ->  "The cat knocked over a lamp."  (94% match)
  [HIT ] (German) "Katze Tierarzt Untersuchung"  ->  "The cat needs to see the vet for a checkup."  (94% match)
  [MISS] (German) "Katze Napf leer"  ->  "The cat sat on the mat."  (84% match)
  [MISS] (French) "chat assis tapis"  ->  "The cat sat on the windowsill."  (61% match)
  [HIT ] (French) "chat renversé vase"  ->  "The cat knocked over a vase."  (94% match)
  [HIT ] (French) "chat prix concours"  ->  "The cat won an award at the pet show."  (80% match)
  [HIT ] (French) "chat adopté refuge"  ->  "The cat was adopted from a shelter last year."  (89% match)
  [HIT ] (Spanish) "gato sentado ventana"  ->  "The cat sat on the windowsill."  (95% match)
  [HIT ] (Spanish) "gato arañó sofá"  ->  "The cat scratched the sofa."  (89% match)
  [HIT ] (Spanish) "gato veterinario revisión"  ->  "The cat needs to see the vet for a checkup."  (95% match)
  [HIT ] (Spanish) "plato comida vacío"  ->  "The cat's food bowl was empty."  (80% match)
  [HIT ] (English) "cat knocked lamp"  ->  "The cat knocked over a lamp."  (98% match)
  [HIT ] (English) "cat sleeping sun"  ->  "The cat is sleeping in the sun."  (98% match)
  [HIT ] (English) "cat award pet show"  ->  "The cat won an award at the pet show."  (97% match)
  [HIT ] (English) "cat adopted shelter"  ->  "The cat was adopted from a shelter last year."  (97% match)

  German accuracy: 3/4 = 75%
  French accuracy: 3/4 = 75%
  Spanish accuracy: 4/4 = 100%
  English accuracy: 4/4 = 100%

Two misses: the same "bowl empty" query missed again, plus a French miss where "chat assis tapis" (cat sat mat) — which should have gone to the exact "sat on the mat" sentence — went to "sat on the windowsill" instead. Notably, that's the near-duplicate pair the corpus was designed to test, and the largest model in the lineup got it wrong.

intfloat/multilingual-e5-small

Embedding dimension: 384

  [MISS] (German) "Katze Fensterbank gesessen"  ->  "The cat knocked over a vase."  (92% match)
  [HIT ] (German) "Katze Lampe umgeworfen"  ->  "The cat knocked over a lamp."  (94% match)
  [HIT ] (German) "Katze Tierarzt Untersuchung"  ->  "The cat needs to see the vet for a checkup."  (94% match)
  [MISS] (German) "Katze Napf leer"  ->  "The cat needs to see the vet for a checkup."  (90% match)
  [HIT ] (French) "chat assis tapis"  ->  "The cat sat on the mat."  (89% match)
  [HIT ] (French) "chat renversé vase"  ->  "The cat knocked over a vase."  (94% match)
  [HIT ] (French) "chat prix concours"  ->  "The cat won an award at the pet show."  (92% match)
  [HIT ] (French) "chat adopté refuge"  ->  "The cat was adopted from a shelter last year."  (94% match)
  [HIT ] (Spanish) "gato sentado ventana"  ->  "The cat sat on the windowsill."  (92% match)
  [HIT ] (Spanish) "gato arañó sofá"  ->  "The cat scratched the sofa."  (94% match)
  [HIT ] (Spanish) "gato veterinario revisión"  ->  "The cat needs to see the vet for a checkup."  (93% match)
  [HIT ] (Spanish) "plato comida vacío"  ->  "The cat's food bowl was empty."  (90% match)
  [HIT ] (English) "cat knocked lamp"  ->  "The cat knocked over a lamp."  (98% match)
  [HIT ] (English) "cat sleeping sun"  ->  "The cat is sleeping in the sun."  (97% match)
  [HIT ] (English) "cat award pet show"  ->  "The cat won an award at the pet show."  (97% match)
  [HIT ] (English) "cat adopted shelter"  ->  "The cat was adopted from a shelter last year."  (97% match)

  German accuracy: 2/4 = 50%
  French accuracy: 4/4 = 100%
  Spanish accuracy: 4/4 = 100%
  English accuracy: 4/4 = 100%

The worst German showing of the five: 2 misses out of 4, including "Katze Fensterbank gesessen" (cat sat on windowsill) landing on a completely unrelated action — "knocked over a vase" — at a confident 92%.

intfloat/multilingual-e5-base

Embedding dimension: 768

  [HIT ] (German) "Katze Fensterbank gesessen"  ->  "The cat sat on the windowsill."  (92% match)
  [HIT ] (German) "Katze Lampe umgeworfen"  ->  "The cat knocked over a lamp."  (93% match)
  [HIT ] (German) "Katze Tierarzt Untersuchung"  ->  "The cat needs to see the vet for a checkup."  (93% match)
  [MISS] (German) "Katze Napf leer"  ->  "The cat needs to see the vet for a checkup."  (90% match)
  [HIT ] (French) "chat assis tapis"  ->  "The cat sat on the mat."  (89% match)
  [HIT ] (French) "chat renversé vase"  ->  "The cat knocked over a vase."  (93% match)
  [HIT ] (French) "chat prix concours"  ->  "The cat won an award at the pet show."  (91% match)
  [HIT ] (French) "chat adopté refuge"  ->  "The cat was adopted from a shelter last year."  (92% match)
  [HIT ] (Spanish) "gato sentado ventana"  ->  "The cat sat on the windowsill."  (93% match)
  [HIT ] (Spanish) "gato arañó sofá"  ->  "The cat scratched the sofa."  (92% match)
  [HIT ] (Spanish) "gato veterinario revisión"  ->  "The cat needs to see the vet for a checkup."  (92% match)
  [HIT ] (Spanish) "plato comida vacío"  ->  "The cat's food bowl was empty."  (91% match)
  [HIT ] (English) "cat knocked lamp"  ->  "The cat knocked over a lamp."  (97% match)
  [HIT ] (English) "cat sleeping sun"  ->  "The cat is sleeping in the sun."  (96% match)
  [HIT ] (English) "cat award pet show"  ->  "The cat won an award at the pet show."  (96% match)
  [HIT ] (English) "cat adopted shelter"  ->  "The cat was adopted from a shelter last year."  (95% match)

  German accuracy: 3/4 = 75%
  French accuracy: 4/4 = 100%
  Spanish accuracy: 4/4 = 100%
  English accuracy: 4/4 = 100%

One miss, the same "bowl empty" query that tripped up three of the five models — this time landing on "needs to see the vet" instead.

distiluse-base-multilingual-cased-v2

Embedding dimension: 512

  [HIT ] (German) "Katze Fensterbank gesessen"  ->  "The cat sat on the windowsill."  (88% match)
  [HIT ] (German) "Katze Lampe umgeworfen"  ->  "The cat knocked over a lamp."  (93% match)
  [HIT ] (German) "Katze Tierarzt Untersuchung"  ->  "The cat needs to see the vet for a checkup."  (86% match)
  [HIT ] (German) "Katze Napf leer"  ->  "The cat's food bowl was empty."  (86% match)
  [HIT ] (French) "chat assis tapis"  ->  "The cat sat on the mat."  (75% match)
  [HIT ] (French) "chat renversé vase"  ->  "The cat knocked over a vase."  (86% match)
  [HIT ] (French) "chat prix concours"  ->  "The cat won an award at the pet show."  (74% match)
  [HIT ] (French) "chat adopté refuge"  ->  "The cat was adopted from a shelter last year."  (71% match)
  [HIT ] (Spanish) "gato sentado ventana"  ->  "The cat sat on the windowsill."  (96% match)
  [HIT ] (Spanish) "gato arañó sofá"  ->  "The cat scratched the sofa."  (92% match)
  [HIT ] (Spanish) "gato veterinario revisión"  ->  "The cat needs to see the vet for a checkup."  (84% match)
  [HIT ] (Spanish) "plato comida vacío"  ->  "The cat's food bowl was empty."  (79% match)
  [HIT ] (English) "cat knocked lamp"  ->  "The cat knocked over a lamp."  (96% match)
  [HIT ] (English) "cat sleeping sun"  ->  "The cat is sleeping in the sun."  (96% match)
  [HIT ] (English) "cat award pet show"  ->  "The cat won an award at the pet show."  (92% match)
  [HIT ] (English) "cat adopted shelter"  ->  "The cat was adopted from a shelter last year."  (87% match)

  German accuracy: 4/4 = 100%
  French accuracy: 4/4 = 100%
  Spanish accuracy: 4/4 = 100%
  English accuracy: 4/4 = 100%

Zero misses — the only model that got every single query right, including the "bowl empty" query that tripped up three of the other four models. Its match scores run noticeably lower than the others (71–96% vs. the high-80s/90s elsewhere), but the ranking was always correct, which is what actually matters for retrieval.

Summary table

Model                                   Dim      German      French     Spanish     English
--------------------------------------------------------------------------------------
paraphrase-multilingual-MiniLM-L12-v2   384         75%        100%        100%        100%
paraphrase-multilingual-mpnet-base-v2   768         75%         75%        100%        100%
intfloat/multilingual-e5-small          384         50%        100%        100%        100%
intfloat/multilingual-e5-base           768         75%        100%        100%        100%
distiluse-base-multilingual-cased-v2    512        100%        100%        100%        100%

Note: this is a small demonstration — 16 queries across 5 models — meant to show the direction of the differences, not a production benchmark. With only 4 queries per language, each hit/miss swings a language's accuracy by 25 percentage points. If you're choosing an embedding model for a real multilingual RAG system, test it against your own corpus and real user queries first.

Code walkthrough

The full script above is runnable as-is. Here's what the key steps actually do:

Build the corpus and embed it once

corpus_embeddings = model.encode(CORPUS, convert_to_tensor=True)

This runs once per model. In a real system, this is the step that happens when you ingest documents into your vector store.

Embed each keyword query and compare it against every corpus sentence

query_embedding = model.encode(query["text"], convert_to_tensor=True)
scores = cos_sim(query_embedding, corpus_embeddings)[0]
top_match_index = int(scores.argmax())

cos_sim returns one similarity score per corpus sentence. argmax() picks the index of the highest one — the sentence the model thinks is the closest match.

Check whether the top match was actually correct

is_correct = (top_match_index == query["answer_index"])

Every query already knows which corpus sentence it should match, since the queries were written by hand. This is a simple hit/miss check, rolled up per language into an accuracy percentage.

Rescale the raw score into a readable "match %"

match_percent = (top_match_score + 1) / 2 * 100

Cosine similarity runs from -1 to 1; this rescales it to 0–100% for readability. It's a relative strength score, not a probability — worth keeping in mind given what follows.

Why the same query gets different scores — and different answers

Look at the query that tripped up the most models: "Katze Napf leer" (German, "cat bowl empty"), which should match "The cat's food bowl was empty."

Model	Top match	Score
paraphrase-multilingual-MiniLM-L12-v2	"...adopted from a shelter" (wrong)	83%
paraphrase-multilingual-mpnet-base-v2	"...sat on the mat" (wrong)	84%
intfloat/multilingual-e5-small	"...needs to see the vet" (wrong)	90%
intfloat/multilingual-e5-base	"...needs to see the vet" (wrong)	90%
distiluse-base-multilingual-cased-v2	"...food bowl was empty" (correct)	86%

Unlike a case where every model converges on the same wrong answer, here four different models missed in three different ways — no single "close neighbor" was consistently confused with the target. That's a different, and arguably more concerning, failure mode: it's not one predictable blind spot, it's inconsistent behavior on the exact same short query across models. The only model that got it right, distiluse-base-multilingual-cased-v2, also had the lowest raw scores across the board (dropping as low as 71% on a correct match) — proof that a lower confidence number doesn't mean a worse model. Score magnitude and correctness are simply not the same thing.

This is the practical trap: cosine similarity scores are not on a universal scale, even across models that all carry the "multilingual" label. A similarity threshold you tune for one model is not portable to another — you have to calibrate on your own data, model by model.

The core finding: which multilingual model actually held up

Ranked by overall accuracy across German, French, and Spanish:

1. distiluse-base-multilingual-cased-v2 — 100% across every language. The only model with zero misses anywhere, despite being the "compact, fast" option in the lineup and consistently producing the lowest raw similarity scores.
2. paraphrase-multilingual-MiniLM-L12-v2 and intfloat/multilingual-e5-base — 75% German, 100% everywhere else. Both missed once, on different queries.
3. paraphrase-multilingual-mpnet-base-v2 — 75% German, 75% French, 100% Spanish/English. The largest model tested, and the only one to miss the "sat on the mat vs. sat on the windowsill" distinction — exactly the kind of fine-grained pair this test was built to catch.
4. intfloat/multilingual-e5-small — 50% German, the worst score of the five, missing two of four German queries.

The standout lesson here is that the compact, "budget" model won, and the largest model in the set (mpnet-base-v2, 768 dimensions) came in last. Dimension count and model size told you almost nothing about which model would actually retrieve correctly. This directly contradicts an earlier, smaller-scale run of this same idea, where mpnet-base-v2 was the clean winner — a useful reminder on its own: with only 4 queries per language, a single hit or miss can flip which model looks best, and that instability is itself information, not noise. It means none of these models is decisively, reliably better at this specific hard case — the winner changes depending on exactly which near-duplicate pairs you happen to test.

The practical takeaway

"Multilingual" on a model card is a label, not a guarantee, and neither is "bigger" or "newer." In this run, the smallest, lowest-confidence model won outright, and the largest model made the most mistakes. That's not a universal claim that small models are better — it's a demonstration that you cannot predict ranking from parameter count or release date, and you have to actually test.

Two things follow from that:

1. Benchmark multilingual models against each other on your own languages and your own corpus, with enough queries that one lucky or unlucky match doesn't flip the result. A model that's "multilingual" on paper — even a newer or larger one — can still underperform a smaller model on your specific vocabulary.
2. Thresholds and top-k tuning are model-specific. A 71% match from one model can be correct, while a 92% match from another can be wrong. Don't set a single similarity cutoff and assume it transfers across models.

Try it yourself

The full script is in this post — swap in your own corpus, your own languages, your own models, and see what actually happens before you pick a multilingual embedding model on faith.

---

This post is part of the "From Hallucination to Precision" series on RAG and embeddings. Full code, no API keys, fully reproducible locally.