DEV Community: Ritesh Totlani

I Tested 4 RAG Chunking Strategies Everyone Recommends. 2 Were Quietly Broken.

Ritesh Totlani — Thu, 27 Aug 2026 07:06:49 +0000

If you've researched how to build a RAG system, you've seen the advice: use recursive splitting, or better yet semantic chunking, or better yet hierarchical chunking for the tricky cases. Every article explains why each one should work. Almost none of them show what happens when you actually point these techniques at real documents and check the output.

I did that. Took 140 real pages from Microsoft's Azure Service Bus documentation, ran them through four chunking strategies — recursive splitting, semantic chunking, hierarchical (parent-child) chunking, and Contextual Retrieval — and looked, chunk by chunk, at what came out the other side.

Setup: fully local stack — LM Studio for inference, nomic-embed-text for embeddings, LangChain + ChromaDB for orchestration.

Two of the four had genuine bugs. One had a subtle flaw that persisted even after I tried to prompt my way out of it. Here's what that taught me, with the numbers, the fix, and the code.

All four strategies follow the same basic pattern — load the docs, hand them to a splitter, get chunks back:

from langchain_text_splitters import RecursiveCharacterTextSplitter
# swap this import for SemanticChunker, ParentDocumentRetriever's splitters,
# or the Contextual Retrieval prompt step below — everything else stays the same

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=450,
    chunk_overlap=0,
)
chunks = splitter.split_documents(documents)

What differs between the four is what came out the other side. That's what this post is actually about.

The chunk that wasn't a chunk

The numbers: 1,330 chunks from 140 documents, average 266.2 tokens, 69% landed in the healthy 200-399 token range, max never exceeded the 450-token budget. Solid on paper.

Then I pulled a sample chunk to eyeball the output, expecting a paragraph of Azure documentation. Instead I got this:

title: Advanced Features in Azure Service Bus Messaging
description: This article provides a high-level overview...
ms.topic: concept-article

Metadata. Every single document in the corpus starts with a YAML header like this, and the very first "chunk" the system produced was just that header — no actual content, nothing a user's question could ever meaningfully match against.

No chunking algorithm on earth fixes this. It's not a splitting problem, it's a data problem — and it's exactly the kind of thing that never shows up in a benchmark paper, because benchmark papers use pre-cleaned datasets. Real corpora aren't pre-cleaned.

FRONTMATTER_PATTERN = re.compile(r"^---\n.*?\n---\n", re.DOTALL)
doc.page_content = FRONTMATTER_PATTERN.sub("", doc.page_content, count=1)

Chunk count before the fix: 1,366. After: 1,330. This is the first thing I'd check on any real client corpus — production RAG failures trace back to ungoverned source data more often than to chunking algorithm choice.

The 5,000-token "chunk"

The numbers, same corpus, semantic chunking instead of recursive:

	Recursive	Semantic
Chunks	1,330	692
Average	266.2 tokens	527.8 tokens
Min / Max	3 / 443	0 / 4,972
% in 200-399 range	69.0%	21.2%

Semantic chunking has a good pitch: instead of cutting text at arbitrary character counts, cut it wherever the meaning shifts. Sounds strictly better than mechanical splitting. Those min/max numbers say otherwise.

One document — a technical reference page listing various messaging protocol operations — came out as a single chunk. Not a paragraph. Not a page. The entire document, nearly 5,000 tokens, because the algorithm never found a strong enough topic shift to justify a cut. Reference documentation reads as "the same topic" to a similarity algorithm even when a human would clearly want it broken into digestible pieces.

Alongside it: 4 chunks containing literally nothing — empty strings, silently sitting in what would have become the search index. And the healthy-range rate tells the rest of the story: only 21.2% of semantic chunks landed in the 200-399 token range, against 69.0% for plain recursive splitting on the same corpus.

Neither failure is visible if you only check "did chunking run successfully" and "how many chunks did I get." Both are only visible if you check the extremes, not just the middle of the distribution.

chunks = [c for c in chunks if c.page_content.strip()]

for chunk in chunks:
    if len(encoding.encode(chunk.page_content)) > MAX_CHUNK_SIZE:
        final_chunks.extend(fallback_splitter.split_documents([chunk]))
    else:
        final_chunks.append(chunk)

After the fix: max dropped from 4,972 to 999 tokens, and the healthy-range rate more than doubled to 46.2%. The underlying lesson holds regardless: semantic chunking can fail badly on structurally repetitive content, and reference-style documentation is full of it.

The right answer, ranked dead last

The numbers: query "what are message sessions used for?" against a hierarchical (parent-child) index returned 4 results, each 2,300–4,400 characters — complete, well-formed sections, nothing came back as an orphaned fragment. Full-context return worked exactly as designed.

Here's what actually came back, in order:

Message sequencing and timestamps — a related but different feature. Similar-sounding, wrong concept.
An SDK/API reference table — mentions session-related method names, but it's a code reference, not an explanation.
Request-queue routing with header parameters — not about sessions at all.
"Enable FIFO with Sessions" — opens with "Azure Service Bus sessions enable joint and ordered processing of unbounded sequences of related messages. Use sessions in first in, first out (FIFO) and request-response patterns." That's a direct, textbook answer to the query. It came back last.

The best-matching document in the entire corpus for this question wasn't just outranked — it was buried behind three progressively less relevant results. Dense embedding similarity latched onto surface-level word overlap ("sessions," "sequencing," "FIFO") instead of which document actually answered the question.

This one stung a little, because hierarchical chunking is supposed to solve exactly this class of problem: search on small, precise pieces of text, but hand back the full surrounding section so nothing gets returned context-free. It did that — every result was complete, nothing was an orphaned fragment. It just didn't fix ranking.

Worth noting: the frontmatter bug from earlier shows up here too — the raw preview of ranks 1 and 4 both start with the YAML title/description block, the exact same data-hygiene issue that broke the very first recursive chunk.

retriever = ParentDocumentRetriever(
    vectorstore=vector_store,
    docstore=docstore,
    parent_splitter=parent_splitter,   # 1000 tokens — what the LLM sees
    child_splitter=child_splitter,     # 200 tokens — what gets searched
)
retriever.add_documents(documents)
results = retriever.invoke("what are message sessions used for?")

The lesson isn't "hierarchical chunking doesn't work." It's that completeness and correctness of ranking are two different problems, solved by two different techniques, and fixing one tells you nothing about whether you've fixed the other. Completeness needed parent-child chunking. Ranking needs hybrid search (BM25 + dense), reranking, or metadata-aware filtering — a different fix entirely. A lot of RAG advice conflates the two.

The context that leaked

The numbers: across 3 test documents, the LLM generated 13 topic statements describing chunk 1 of each. Checked against the actual source text: 12 correct, 1 leaked — a point about avoiding stored credentials via OAuth 2.0 that actually belonged to the next chunk over, not the one it was describing.

The technique — Contextual Retrieval — asks an LLM to write a short blurb situating each chunk within its source document, so an isolated piece of text doesn't lose its meaning once separated from its neighbors:

prompt = CONTEXT_PROMPT.invoke({"document": full_document, "chunk": chunk.page_content})
generated_context = chat_model.invoke(prompt).content
chunk.page_content = generated_context + "\n\n" + chunk.page_content

I tightened the prompt, told the model explicitly not to describe content from elsewhere in the document. The leak got smaller — 1 out of 13, not 1 out of 5 — but it didn't disappear.

That's not a bug I could code my way out of — it's structural. The technique works by showing the model the whole document for context. That same visibility is what let it blur a boundary. You can't fully remove the risk without removing the thing that makes the technique useful in the first place. Knowing that trade-off exists — and being honest about it instead of claiming a technique is flawless — is worth more than pretending it isn't there.

Why this is the actual point

None of these are exotic failures. They're the kind of thing that ships quietly in a lot of RAG systems, because "it ran without an error" and "it looks fine in a quick demo" get mistaken for "it works." A system that returns a metadata header instead of an answer, or ranks the wrong document first, doesn't crash — it just quietly gives worse answers, and nobody notices until a user complains.

The fix isn't a smarter algorithm. It's a habit: don't trust a technique's reputation, verify it against your own data, and specifically go looking at the edge cases — the shortest result, the longest result, the ones that don't fit the average. That's what caught every failure in this post. Not one of them showed up in the headline numbers.

If there's a single thing worth taking from this: the strategy that wins on a benchmark isn't the strategy that's right for your documents. The only way to know which one is right for yours is to actually test it — and actually look.

What's the weirdest thing you've found when you actually inspected your own RAG chunks? Drop it below.

Not All Multilingual Embedding Models Are Equal — Testing 5 of Them

Ritesh Totlani — Thu, 20 Aug 2026 07:51:45 +0000

Part of the "From Hallucination to Precision" series.

If your RAG documents are in English and your users search in German, French, or Spanish, you already know you need a "multilingual" embedding model. But which one? "Multilingual" isn't one thing — it's a label that covers models of very different sizes, ages, and training approaches, and they don't all perform the same.

I tested five multilingual embedding models against the same task: a small English knowledge base built entirely around one subject — a cat — with several near-duplicate sentences on purpose, keyword queries in three other languages, and one question — did each model retrieve the exact right sentence? All local, via sentence-transformers, no API keys.

This post shows the exact code and the exact numbers it produced.

The five models

Model	Dim	What it is
`paraphrase-multilingual-MiniLM-L12-v2`	384	Smaller, older (2021), covers 50+ languages
`paraphrase-multilingual-mpnet-base-v2`	768	Larger version of the same family
`intfloat/multilingual-e5-small`	384	Newer, competitive multilingual retrieval model
`intfloat/multilingual-e5-base`	768	Same E5 family, bigger
`distiluse-base-multilingual-cased-v2`	512	Compact, fast, 50+ languages

All five are multilingual. None are English-only. The question isn't "does multilingual beat English-only" — that's obvious. The question is: which multilingual model actually works, and how much do they differ from each other?

The setup

The knowledge base: 10 English sentences, all about a cat, on purpose.

6 near-duplicates, including two intentionally close pairs: "sat on the mat" vs. "sat on the windowsill," and "knocked over a vase" vs. "knocked over a lamp." These sit close together in meaning — a query has to land on the exact right sentence, not just "something about a cat sitting somewhere" or "something the cat knocked over."
4 distinct-context sentences: won an award at a pet show, needs a vet checkup, was adopted from a shelter, had an empty food bowl. Still about the same cat, but different enough in topic that they're an easier case.

One subject, varying difficulty — closer to how a real knowledge base often looks: many entries about the same product or feature, worded slightly differently.

Instead of full sentences, queries are short keywords — because that's how people actually search. Four keyword queries per language: German, French, Spanish, plus English as a same-language control. Several deliberately target one half of a close pair (e.g. "knocked over a lamp," not "a vase") to see whether the model finds the exact match or grabs its near-twin.

For every query: embed it, compare it against all 10 corpus sentences with cosine similarity, and see which one wins. Whichever sentence scores highest is what a real RAG pipeline would hand to the LLM as context. Wrong match, wrong context, wrong answer — silently, no crash, no error.

The code

"""
Embedding Model Comparison — 5 multilingual models, cross-lingual retrieval test.
No API keys. Runs fully local. Requires: pip install sentence-transformers
"""

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

MODEL_NAMES = [
    "paraphrase-multilingual-MiniLM-L12-v2",
    "paraphrase-multilingual-mpnet-base-v2",
    "intfloat/multilingual-e5-small",
    "intfloat/multilingual-e5-base",
    "distiluse-base-multilingual-cased-v2",
]

LANGUAGE_ORDER = ["German", "French", "Spanish", "English"]

# Knowledge base: 10 sentences, all about a cat. 6 are near-duplicates,
# including two intentionally close pairs (sat-on-X, knocked-over-X).
# 4 are still cat-related but describe clearly different situations.
CORPUS = [
    "The cat sat on the mat.",                                # 0 - tight cluster
    "The cat sat on the windowsill.",                         # 1 - tight cluster (close to 0)
    "The cat knocked over a vase.",                           # 2 - tight cluster
    "The cat knocked over a lamp.",                           # 3 - tight cluster (close to 2)
    "The cat is sleeping in the sun.",                        # 4 - tight cluster
    "The cat scratched the sofa.",                            # 5 - tight cluster
    "The cat won an award at the pet show.",                  # 6 - distinct context
    "The cat needs to see the vet for a checkup.",            # 7 - distinct context
    "The cat was adopted from a shelter last year.",          # 8 - distinct context
    "The cat's food bowl was empty.",                         # 9 - distinct context
]

# Keyword queries: 4 per language (German, French, Spanish, English control).
# Several deliberately target one half of a close pair (e.g. "knocked over a
# lamp") to see if the model finds the exact match or grabs its near-twin.
QUERIES = [
    {"answer_index": 1, "language": "German", "text": "Katze Fensterbank gesessen"},
    {"answer_index": 3, "language": "German", "text": "Katze Lampe umgeworfen"},
    {"answer_index": 7, "language": "German", "text": "Katze Tierarzt Untersuchung"},
    {"answer_index": 9, "language": "German", "text": "Katze Napf leer"},

    {"answer_index": 0, "language": "French", "text": "chat assis tapis"},
    {"answer_index": 2, "language": "French", "text": "chat renversé vase"},
    {"answer_index": 6, "language": "French", "text": "chat prix concours"},
    {"answer_index": 8, "language": "French", "text": "chat adopté refuge"},

    {"answer_index": 1, "language": "Spanish", "text": "gato sentado ventana"},
    {"answer_index": 5, "language": "Spanish", "text": "gato arañó sofá"},
    {"answer_index": 7, "language": "Spanish", "text": "gato veterinario revisión"},
    {"answer_index": 9, "language": "Spanish", "text": "plato comida vacío"},

    {"answer_index": 3, "language": "English", "text": "cat knocked lamp"},
    {"answer_index": 4, "language": "English", "text": "cat sleeping sun"},
    {"answer_index": 6, "language": "English", "text": "cat award pet show"},
    {"answer_index": 8, "language": "English", "text": "cat adopted shelter"},
]


def get_embedding_dim(model):
    # sentence-transformers renamed this method; support both.
    if hasattr(model, "get_embedding_dimension"):
        return model.get_embedding_dimension()
    return model.get_sentence_embedding_dimension()


def test_one_model(model_name):
    print("=" * 78)
    print(f"MODEL: {model_name}")
    print("=" * 78)

    model = SentenceTransformer(model_name)
    dim = get_embedding_dim(model)
    print(f"Embedding dimension: {dim}\n")

    corpus_embeddings = model.encode(CORPUS, convert_to_tensor=True)
    results = {lang: {"hits": 0, "total": 0} for lang in LANGUAGE_ORDER}

    for query in QUERIES:
        query_embedding = model.encode(query["text"], convert_to_tensor=True)
        scores = cos_sim(query_embedding, corpus_embeddings)[0]
        top_match_index = int(scores.argmax())
        top_match_score = float(scores[top_match_index])

        is_correct = (top_match_index == query["answer_index"])
        language = query["language"]
        results[language]["total"] += 1
        if is_correct:
            results[language]["hits"] += 1

        status = "HIT " if is_correct else "MISS"
        match_percent = (top_match_score + 1) / 2 * 100
        print(f"  [{status}] ({language}) \"{query['text']}\"  ->  \"{CORPUS[top_match_index]}\"  ({match_percent:.0f}% match)")

    print()
    accuracy_by_language = {}
    for language in LANGUAGE_ORDER:
        hits = results[language]["hits"]
        total = results[language]["total"]
        accuracy = hits / total if total else 0.0
        accuracy_by_language[language] = accuracy
        print(f"  {language} accuracy: {hits}/{total} = {accuracy:.0%}")
    print()

    return dim, accuracy_by_language


def print_summary_table(all_results):
    print("=" * 78)
    print("SUMMARY — Retrieval Accuracy by Model and Query Language")
    print("=" * 78)

    header = f"{'Model':32}{'Dim':>6}"
    for language in LANGUAGE_ORDER:
        header += f"{language:>12}"
    print(header)
    print("-" * len(header))

    for model_name, (dim, accuracy_by_language) in all_results.items():
        row = f"{model_name:32}{dim:>6}"
        for language in LANGUAGE_ORDER:
            row += f"{accuracy_by_language[language]:>12.0%}"
        print(row)


def main():
    all_results = {}
    for model_name in MODEL_NAMES:
        dim, accuracy_by_language = test_one_model(model_name)
        all_results[model_name] = (dim, accuracy_by_language)

    print_summary_table(all_results)


if __name__ == "__main__":
    main()

Run it yourself:

pip install sentence-transformers
python3 embedding_comparison.py

The results

Here's the real output, one model at a time.

`paraphrase-multilingual-MiniLM-L12-v2`

Embedding dimension: 384

  [HIT ] (German) "Katze Fensterbank gesessen"  ->  "The cat sat on the windowsill."  (89% match)
  [HIT ] (German) "Katze Lampe umgeworfen"  ->  "The cat knocked over a lamp."  (93% match)
  [HIT ] (German) "Katze Tierarzt Untersuchung"  ->  "The cat needs to see the vet for a checkup."  (90% match)
  [MISS] (German) "Katze Napf leer"  ->  "The cat was adopted from a shelter last year."  (83% match)
  [HIT ] (French) "chat assis tapis"  ->  "The cat sat on the mat."  (62% match)
  [HIT ] (French) "chat renversé vase"  ->  "The cat knocked over a vase."  (93% match)
  [HIT ] (French) "chat prix concours"  ->  "The cat won an award at the pet show."  (87% match)
  [HIT ] (French) "chat adopté refuge"  ->  "The cat was adopted from a shelter last year."  (86% match)
  [HIT ] (Spanish) "gato sentado ventana"  ->  "The cat sat on the windowsill."  (94% match)
  [HIT ] (Spanish) "gato arañó sofá"  ->  "The cat scratched the sofa."  (95% match)
  [HIT ] (Spanish) "gato veterinario revisión"  ->  "The cat needs to see the vet for a checkup."  (88% match)
  [HIT ] (Spanish) "plato comida vacío"  ->  "The cat's food bowl was empty."  (78% match)
  [HIT ] (English) "cat knocked lamp"  ->  "The cat knocked over a lamp."  (97% match)
  [HIT ] (English) "cat sleeping sun"  ->  "The cat is sleeping in the sun."  (96% match)
  [HIT ] (English) "cat award pet show"  ->  "The cat won an award at the pet show."  (96% match)
  [HIT ] (English) "cat adopted shelter"  ->  "The cat was adopted from a shelter last year."  (93% match)

  German accuracy: 3/4 = 75%
  French accuracy: 4/4 = 100%
  Spanish accuracy: 4/4 = 100%
  English accuracy: 4/4 = 100%

One miss: "Katze Napf leer" (cat's bowl empty) matched to "adopted from a shelter" instead of "food bowl was empty."

`paraphrase-multilingual-mpnet-base-v2`

Embedding dimension: 768

  [HIT ] (German) "Katze Fensterbank gesessen"  ->  "The cat sat on the windowsill."  (88% match)
  [HIT ] (German) "Katze Lampe umgeworfen"  ->  "The cat knocked over a lamp."  (94% match)
  [HIT ] (German) "Katze Tierarzt Untersuchung"  ->  "The cat needs to see the vet for a checkup."  (94% match)
  [MISS] (German) "Katze Napf leer"  ->  "The cat sat on the mat."  (84% match)
  [MISS] (French) "chat assis tapis"  ->  "The cat sat on the windowsill."  (61% match)
  [HIT ] (French) "chat renversé vase"  ->  "The cat knocked over a vase."  (94% match)
  [HIT ] (French) "chat prix concours"  ->  "The cat won an award at the pet show."  (80% match)
  [HIT ] (French) "chat adopté refuge"  ->  "The cat was adopted from a shelter last year."  (89% match)
  [HIT ] (Spanish) "gato sentado ventana"  ->  "The cat sat on the windowsill."  (95% match)
  [HIT ] (Spanish) "gato arañó sofá"  ->  "The cat scratched the sofa."  (89% match)
  [HIT ] (Spanish) "gato veterinario revisión"  ->  "The cat needs to see the vet for a checkup."  (95% match)
  [HIT ] (Spanish) "plato comida vacío"  ->  "The cat's food bowl was empty."  (80% match)
  [HIT ] (English) "cat knocked lamp"  ->  "The cat knocked over a lamp."  (98% match)
  [HIT ] (English) "cat sleeping sun"  ->  "The cat is sleeping in the sun."  (98% match)
  [HIT ] (English) "cat award pet show"  ->  "The cat won an award at the pet show."  (97% match)
  [HIT ] (English) "cat adopted shelter"  ->  "The cat was adopted from a shelter last year."  (97% match)

  German accuracy: 3/4 = 75%
  French accuracy: 3/4 = 75%
  Spanish accuracy: 4/4 = 100%
  English accuracy: 4/4 = 100%

Two misses: the same "bowl empty" query missed again, plus a French miss where "chat assis tapis" (cat sat mat) — which should have gone to the exact "sat on the mat" sentence — went to "sat on the windowsill" instead. Notably, that's the near-duplicate pair the corpus was designed to test, and the largest model in the lineup got it wrong.

`intfloat/multilingual-e5-small`

Embedding dimension: 384

  [MISS] (German) "Katze Fensterbank gesessen"  ->  "The cat knocked over a vase."  (92% match)
  [HIT ] (German) "Katze Lampe umgeworfen"  ->  "The cat knocked over a lamp."  (94% match)
  [HIT ] (German) "Katze Tierarzt Untersuchung"  ->  "The cat needs to see the vet for a checkup."  (94% match)
  [MISS] (German) "Katze Napf leer"  ->  "The cat needs to see the vet for a checkup."  (90% match)
  [HIT ] (French) "chat assis tapis"  ->  "The cat sat on the mat."  (89% match)
  [HIT ] (French) "chat renversé vase"  ->  "The cat knocked over a vase."  (94% match)
  [HIT ] (French) "chat prix concours"  ->  "The cat won an award at the pet show."  (92% match)
  [HIT ] (French) "chat adopté refuge"  ->  "The cat was adopted from a shelter last year."  (94% match)
  [HIT ] (Spanish) "gato sentado ventana"  ->  "The cat sat on the windowsill."  (92% match)
  [HIT ] (Spanish) "gato arañó sofá"  ->  "The cat scratched the sofa."  (94% match)
  [HIT ] (Spanish) "gato veterinario revisión"  ->  "The cat needs to see the vet for a checkup."  (93% match)
  [HIT ] (Spanish) "plato comida vacío"  ->  "The cat's food bowl was empty."  (90% match)
  [HIT ] (English) "cat knocked lamp"  ->  "The cat knocked over a lamp."  (98% match)
  [HIT ] (English) "cat sleeping sun"  ->  "The cat is sleeping in the sun."  (97% match)
  [HIT ] (English) "cat award pet show"  ->  "The cat won an award at the pet show."  (97% match)
  [HIT ] (English) "cat adopted shelter"  ->  "The cat was adopted from a shelter last year."  (97% match)

  German accuracy: 2/4 = 50%
  French accuracy: 4/4 = 100%
  Spanish accuracy: 4/4 = 100%
  English accuracy: 4/4 = 100%

The worst German showing of the five: 2 misses out of 4, including "Katze Fensterbank gesessen" (cat sat on windowsill) landing on a completely unrelated action — "knocked over a vase" — at a confident 92%.

`intfloat/multilingual-e5-base`

Embedding dimension: 768

  [HIT ] (German) "Katze Fensterbank gesessen"  ->  "The cat sat on the windowsill."  (92% match)
  [HIT ] (German) "Katze Lampe umgeworfen"  ->  "The cat knocked over a lamp."  (93% match)
  [HIT ] (German) "Katze Tierarzt Untersuchung"  ->  "The cat needs to see the vet for a checkup."  (93% match)
  [MISS] (German) "Katze Napf leer"  ->  "The cat needs to see the vet for a checkup."  (90% match)
  [HIT ] (French) "chat assis tapis"  ->  "The cat sat on the mat."  (89% match)
  [HIT ] (French) "chat renversé vase"  ->  "The cat knocked over a vase."  (93% match)
  [HIT ] (French) "chat prix concours"  ->  "The cat won an award at the pet show."  (91% match)
  [HIT ] (French) "chat adopté refuge"  ->  "The cat was adopted from a shelter last year."  (92% match)
  [HIT ] (Spanish) "gato sentado ventana"  ->  "The cat sat on the windowsill."  (93% match)
  [HIT ] (Spanish) "gato arañó sofá"  ->  "The cat scratched the sofa."  (92% match)
  [HIT ] (Spanish) "gato veterinario revisión"  ->  "The cat needs to see the vet for a checkup."  (92% match)
  [HIT ] (Spanish) "plato comida vacío"  ->  "The cat's food bowl was empty."  (91% match)
  [HIT ] (English) "cat knocked lamp"  ->  "The cat knocked over a lamp."  (97% match)
  [HIT ] (English) "cat sleeping sun"  ->  "The cat is sleeping in the sun."  (96% match)
  [HIT ] (English) "cat award pet show"  ->  "The cat won an award at the pet show."  (96% match)
  [HIT ] (English) "cat adopted shelter"  ->  "The cat was adopted from a shelter last year."  (95% match)

  German accuracy: 3/4 = 75%
  French accuracy: 4/4 = 100%
  Spanish accuracy: 4/4 = 100%
  English accuracy: 4/4 = 100%

One miss, the same "bowl empty" query that tripped up three of the five models — this time landing on "needs to see the vet" instead.

`distiluse-base-multilingual-cased-v2`

Embedding dimension: 512

  [HIT ] (German) "Katze Fensterbank gesessen"  ->  "The cat sat on the windowsill."  (88% match)
  [HIT ] (German) "Katze Lampe umgeworfen"  ->  "The cat knocked over a lamp."  (93% match)
  [HIT ] (German) "Katze Tierarzt Untersuchung"  ->  "The cat needs to see the vet for a checkup."  (86% match)
  [HIT ] (German) "Katze Napf leer"  ->  "The cat's food bowl was empty."  (86% match)
  [HIT ] (French) "chat assis tapis"  ->  "The cat sat on the mat."  (75% match)
  [HIT ] (French) "chat renversé vase"  ->  "The cat knocked over a vase."  (86% match)
  [HIT ] (French) "chat prix concours"  ->  "The cat won an award at the pet show."  (74% match)
  [HIT ] (French) "chat adopté refuge"  ->  "The cat was adopted from a shelter last year."  (71% match)
  [HIT ] (Spanish) "gato sentado ventana"  ->  "The cat sat on the windowsill."  (96% match)
  [HIT ] (Spanish) "gato arañó sofá"  ->  "The cat scratched the sofa."  (92% match)
  [HIT ] (Spanish) "gato veterinario revisión"  ->  "The cat needs to see the vet for a checkup."  (84% match)
  [HIT ] (Spanish) "plato comida vacío"  ->  "The cat's food bowl was empty."  (79% match)
  [HIT ] (English) "cat knocked lamp"  ->  "The cat knocked over a lamp."  (96% match)
  [HIT ] (English) "cat sleeping sun"  ->  "The cat is sleeping in the sun."  (96% match)
  [HIT ] (English) "cat award pet show"  ->  "The cat won an award at the pet show."  (92% match)
  [HIT ] (English) "cat adopted shelter"  ->  "The cat was adopted from a shelter last year."  (87% match)

  German accuracy: 4/4 = 100%
  French accuracy: 4/4 = 100%
  Spanish accuracy: 4/4 = 100%
  English accuracy: 4/4 = 100%

Zero misses — the only model that got every single query right, including the "bowl empty" query that tripped up three of the other four models. Its match scores run noticeably lower than the others (71–96% vs. the high-80s/90s elsewhere), but the ranking was always correct, which is what actually matters for retrieval.

Summary table

Model                                   Dim      German      French     Spanish     English
--------------------------------------------------------------------------------------
paraphrase-multilingual-MiniLM-L12-v2   384         75%        100%        100%        100%
paraphrase-multilingual-mpnet-base-v2   768         75%         75%        100%        100%
intfloat/multilingual-e5-small          384         50%        100%        100%        100%
intfloat/multilingual-e5-base           768         75%        100%        100%        100%
distiluse-base-multilingual-cased-v2    512        100%        100%        100%        100%

Note: this is a small demonstration — 16 queries across 5 models — meant to show the direction of the differences, not a production benchmark. With only 4 queries per language, each hit/miss swings a language's accuracy by 25 percentage points. If you're choosing an embedding model for a real multilingual RAG system, test it against your own corpus and real user queries first.

Code walkthrough

The full script above is runnable as-is. Here's what the key steps actually do:

Build the corpus and embed it once

corpus_embeddings = model.encode(CORPUS, convert_to_tensor=True)

This runs once per model. In a real system, this is the step that happens when you ingest documents into your vector store.

Embed each keyword query and compare it against every corpus sentence

query_embedding = model.encode(query["text"], convert_to_tensor=True)
scores = cos_sim(query_embedding, corpus_embeddings)[0]
top_match_index = int(scores.argmax())

cos_sim returns one similarity score per corpus sentence. argmax() picks the index of the highest one — the sentence the model thinks is the closest match.

Check whether the top match was actually correct

is_correct = (top_match_index == query["answer_index"])

Every query already knows which corpus sentence it should match, since the queries were written by hand. This is a simple hit/miss check, rolled up per language into an accuracy percentage.

Rescale the raw score into a readable "match %"

match_percent = (top_match_score + 1) / 2 * 100

Cosine similarity runs from -1 to 1; this rescales it to 0–100% for readability. It's a relative strength score, not a probability — worth keeping in mind given what follows.

Why the same query gets different scores — and different answers

Look at the query that tripped up the most models: "Katze Napf leer" (German, "cat bowl empty"), which should match "The cat's food bowl was empty."

Model	Top match	Score
`paraphrase-multilingual-MiniLM-L12-v2`	"...adopted from a shelter" (wrong)	83%
`paraphrase-multilingual-mpnet-base-v2`	"...sat on the mat" (wrong)	84%
`intfloat/multilingual-e5-small`	"...needs to see the vet" (wrong)	90%
`intfloat/multilingual-e5-base`	"...needs to see the vet" (wrong)	90%
`distiluse-base-multilingual-cased-v2`	"...food bowl was empty" (correct)	86%

Unlike a case where every model converges on the same wrong answer, here four different models missed in three different ways — no single "close neighbor" was consistently confused with the target. That's a different, and arguably more concerning, failure mode: it's not one predictable blind spot, it's inconsistent behavior on the exact same short query across models. The only model that got it right, distiluse-base-multilingual-cased-v2, also had the lowest raw scores across the board (dropping as low as 71% on a correct match) — proof that a lower confidence number doesn't mean a worse model. Score magnitude and correctness are simply not the same thing.

This is the practical trap: cosine similarity scores are not on a universal scale, even across models that all carry the "multilingual" label. A similarity threshold you tune for one model is not portable to another — you have to calibrate on your own data, model by model.

The core finding: which multilingual model actually held up

Ranked by overall accuracy across German, French, and Spanish:

distiluse-base-multilingual-cased-v2 — 100% across every language. The only model with zero misses anywhere, despite being the "compact, fast" option in the lineup and consistently producing the lowest raw similarity scores.
paraphrase-multilingual-MiniLM-L12-v2 and intfloat/multilingual-e5-base — 75% German, 100% everywhere else. Both missed once, on different queries.
paraphrase-multilingual-mpnet-base-v2 — 75% German, 75% French, 100% Spanish/English. The largest model tested, and the only one to miss the "sat on the mat vs. sat on the windowsill" distinction — exactly the kind of fine-grained pair this test was built to catch.
intfloat/multilingual-e5-small — 50% German, the worst score of the five, missing two of four German queries.

The standout lesson here is that the compact, "budget" model won, and the largest model in the set (mpnet-base-v2, 768 dimensions) came in last. Dimension count and model size told you almost nothing about which model would actually retrieve correctly. This directly contradicts an earlier, smaller-scale run of this same idea, where mpnet-base-v2 was the clean winner — a useful reminder on its own: with only 4 queries per language, a single hit or miss can flip which model looks best, and that instability is itself information, not noise. It means none of these models is decisively, reliably better at this specific hard case — the winner changes depending on exactly which near-duplicate pairs you happen to test.

The practical takeaway

"Multilingual" on a model card is a label, not a guarantee, and neither is "bigger" or "newer." In this run, the smallest, lowest-confidence model won outright, and the largest model made the most mistakes. That's not a universal claim that small models are better — it's a demonstration that you cannot predict ranking from parameter count or release date, and you have to actually test.

Two things follow from that:

Benchmark multilingual models against each other on your own languages and your own corpus, with enough queries that one lucky or unlucky match doesn't flip the result. A model that's "multilingual" on paper — even a newer or larger one — can still underperform a smaller model on your specific vocabulary.
Thresholds and top-k tuning are model-specific. A 71% match from one model can be correct, while a 92% match from another can be wrong. Don't set a single similarity cutoff and assume it transfers across models.

Try it yourself

The full script is in this post — swap in your own corpus, your own languages, your own models, and see what actually happens before you pick a multilingual embedding model on faith.

This post is part of the "From Hallucination to Precision" series on RAG and embeddings. Full code, no API keys, fully reproducible locally.

Building a RAG System From Scratch — Four Components, One Working Pipeline

Ritesh Totlani — Wed, 12 Aug 2026 11:40:08 +0000

Most RAG tutorials explain the concept. This one shows the code — a complete working pipeline using LangChain, ChromaDB, and a local LLM via LM Studio. No OpenAI API key. No cloud costs.

The business problem: A company has hundreds of pages of HR documentation. Employees ask questions. An AI answers accurately from the actual documents in seconds.

Stack: Python, LangChain, ChromaDB, nomic-embed-text, Qwen 9B via LM Studio

pip install langchain langchain-community langchain-chroma langchain-openai langchain-text-splitters langchain-core openai requests

Make sure LM Studio is running with both models loaded before running the code.

Component 1 — Document Processor

Splits raw text into chunks. Chunk size and overlap determine everything downstream — cut a paragraph wrong and retrieval suffers.

from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.documents import Document
from typing import List

class DocumentProcessor:
    def __init__(self, chunk_size: int = 500, chunk_overlap: int = 50):
        self.splitter = RecursiveCharacterTextSplitter(
            chunk_size=chunk_size,
            chunk_overlap=chunk_overlap,
            separators=["\n\n", "\n", ". ", " ", ""]
        )

    def process(self, text: str, source: str = "") -> List[Document]:
        chunks = self.splitter.split_text(text)
        return [
            Document(
                page_content=chunk,
                metadata={"source": source, "chunk_index": i}
            )
            for i, chunk in enumerate(chunks)
        ]

⚠️ Changing chunk size later means reembedding everything from scratch. Your chunks and embeddings are tightly coupled. Choose carefully the first time.

Component 2 — Embedding Service

Converts text into numerical vectors. LM Studio needs a direct requests call — the standard LangChain OpenAI wrapper sends the wrong input format and throws a 400 error.

from langchain_core.embeddings import Embeddings
import requests

LM_STUDIO_URL = "http://localhost:1234/v1"
EMBED_MODEL   = "nomic-embed-text"

class EmbeddingService(Embeddings):
    def __init__(self):
        self.url   = f"{LM_STUDIO_URL}/embeddings"
        self.model = EMBED_MODEL

    def _embed(self, text: str) -> List[float]:
        response = requests.post(
            self.url,
            json={"model": self.model, "input": text},
            headers={"Authorization": "Bearer lm-studio"}
        )
        response.raise_for_status()
        return response.json()["data"][0]["embedding"]

    def embed_documents(self, texts: List[str]) -> List[List[float]]:
        return [self._embed(text) for text in texts]

    def embed_query(self, text: str) -> List[float]:
        return self._embed(text)

⚠️ Same model must be used for both indexing and querying. They must share the same vector space. Switching models means rebuilding the entire index.

Component 3 — Vector Store

Stores embeddings and retrieves closest matches by meaning — not by keyword. ChromaDB persists to disk so your index survives restarts without reprocessing documents.

from langchain_chroma import Chroma
from typing import Tuple

CHROMA_DIR      = "./chroma_db"
TOP_K           = 4
SCORE_THRESHOLD = 0.3

class VectorStore:
    def __init__(self, embedding_service: EmbeddingService):
        self.store = Chroma(
            collection_name="rag_collection",
            embedding_function=embedding_service,
            persist_directory=CHROMA_DIR
        )

    def add_documents(self, documents: List[Document]) -> None:
        self.store.add_documents(documents)

    def similarity_search(self, query: str) -> List[Tuple[Document, float]]:
        results = self.store.similarity_search_with_relevance_scores(
            query=query,
            k=TOP_K,
        )
        return [(doc, score) for doc, score in results if score >= SCORE_THRESHOLD]

💡 Score threshold of 0.3 works well for small document sets. In production with larger knowledge bases tune this to 0.6–0.75 to filter out loosely relevant chunks.

Component 4 — RAG Pipeline

Orchestrates the full flow. Query in, embed it, retrieve closest chunks, build context, generate grounded answer, return sources.

The most important line in the entire pipeline is the system prompt. Without the strict instruction to answer only from context, the LLM falls back on its general training knowledge — defeating the entire purpose of RAG.

from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage
from typing import Dict

CHAT_MODEL = "qwen/qwen3.5-9b"

class RAGPipeline:
    def __init__(self, vector_store: VectorStore):
        self.vector_store = vector_store
        self.llm = ChatOpenAI(
            model=CHAT_MODEL,
            openai_api_base=LM_STUDIO_URL,
            openai_api_key="lm-studio",
            max_tokens=1000,
            temperature=0.1
        )

    def _build_context(self, documents: List[Tuple[Document, float]]) -> str:
        parts = []
        for doc, score in documents:
            parts.append(
                f"Source: {doc.metadata.get('source', 'unknown')}\n"
                f"Relevance: {score:.2f}\n"
                f"Content: {doc.page_content}"
            )
        return "\n\n---\n\n".join(parts)

    def query(self, question: str) -> Dict:
        results = self.vector_store.similarity_search(question)

        if not results:
            return {
                "answer": "I could not find relevant information to answer this question.",
                "sources": [],
                "chunks_used": 0
            }

        context = self._build_context(results)

        messages = [
            SystemMessage(content=(
                "You are a precise assistant that answers questions "
                "based solely on the provided context. "
                "If the answer is not in the context, say so clearly. "
                "Do not use your general knowledge to supplement the context."
            )),
            HumanMessage(content=(
                f"Context:\n{context}\n\n"
                f"Question: {question}\n\n"
                f"Answer based only on the context above:"
            ))
        ]

        response = self.llm(messages)

        return {
            "answer": response.content,
            "sources": list(set(doc.metadata.get("source") for doc, _ in results)),
            "chunks_used": len(results)
        }

💡 temperature=0.1 keeps answers factual and consistent. Low temperature means the model stays close to what the context says rather than being creative with it.

Run It

if __name__ == "__main__":

    SAMPLE_DOCUMENT = """
    Employee Leave Policy

    Sick Leave:
    Employees are entitled to 10 days of paid sick leave per calendar year.
    Sick leave resets on January 1st each year.
    Unused sick leave cannot be carried over to the next year.
    To apply for sick leave, submit a request through the HR portal.
    A medical certificate is required for sick leave exceeding 3 consecutive days.

    Annual Leave:
    Employees receive 25 days of annual leave per year.
    Annual leave must be approved by the line manager at least 2 weeks in advance.
    Up to 5 unused annual leave days can be carried over to the following year.

    Remote Work Policy:
    Employees may work remotely up to 3 days per week.
    Remote work requires a stable internet connection and a dedicated workspace.
    Core hours of 10:00 to 16:00 must be maintained regardless of location.

    Expense Policy:
    Business travel expenses must be approved before travel.
    Receipts are required for all expenses above 25 euros.
    Expense reports must be submitted within 30 days of the expense.
    Maximum meal allowance is 50 euros per day during business travel.
    """

    # Initialise all four components
    processor  = DocumentProcessor()
    embeddings = EmbeddingService()
    store      = VectorStore(embedding_service=embeddings)
    pipeline   = RAGPipeline(vector_store=store)

    # Index documents — happens once
    documents = processor.process(SAMPLE_DOCUMENT, source="company_policy")
    store.add_documents(documents)

    # Query — happens live for every user question
    questions = [
        "How many sick days am I entitled to per year?",
        "Can I carry over unused annual leave?",
        "How many days can I work remotely?",
        "What is the maximum meal allowance during business travel?",
        "Do I need a medical certificate for sick leave?",
    ]

    for q in questions:
        result = pipeline.query(q)
        print(f"Q: {q}")
        print(f"A: {result['answer']}")
        print(f"   Sources: {result['sources']}")
        print(f"   Chunks used: {result['chunks_used']}")
        print("-" * 60)

Real Output

This is the actual output from running this pipeline locally:

Q: How many sick days am I entitled to per year?
A: Based on the provided context, employees are entitled to 10 days
   of paid sick leave per calendar year.
   Sources: ['company_policy'] · Chunks used: 4

Q: Can I carry over unused annual leave?
A: Yes, up to 5 unused annual leave days can be carried over to
   the following year.
   Sources: ['company_policy'] · Chunks used: 4

Q: How many days can I work remotely?
A: Based on the provided context, employees may work remotely up
   to 3 days per week.
   Sources: ['company_policy'] · Chunks used: 4

Q: What is the maximum meal allowance during business travel?
A: The maximum meal allowance during business travel is 50 euros
   per day.
   Sources: ['company_policy'] · Chunks used: 4

Q: Do I need a medical certificate for sick leave?
A: A medical certificate is required for sick leave exceeding
   3 consecutive days.
   Sources: ['company_policy'] · Chunks used: 4

Every answer accurate. Every answer sourced. No hallucination. No API key. No cloud cost.

Replace SAMPLE_DOCUMENT with your own content and you have a working RAG system on your documents in minutes.

Moving to Azure

Every component maps directly to an Azure service. The pipeline logic stays identical — only the initialisation changes.

Local	Azure Equivalent
nomic-embed-text via LM Studio	AzureOpenAIEmbeddings
Qwen 9B via LM Studio	AzureChatOpenAI
ChromaDB	Azure AI Search

What to Tune Next

This implementation is intentionally minimal. In production add:

Higher score threshold — 0.6 to 0.75 for larger document sets
Reranking — a second model pass to reorder chunks by true relevance
Query expansion — generate multiple question phrasings before searching
Conversation history — maintain context across multiple turns
Observability — log every retrieval and generation for debugging

These are the patterns covered in the next post — where RAG goes from working to production-ready.