Why RAG breaks on Persian text, and what I built to fix it: normalization, half-spaces, spelling variants, and a regression bench

Most RAG tutorials assume the hard parts are the embedding model and the vector database. For Persian, a lot of the damage happens before either of them sees a byte. Two strings that look identical on screen are different code points. A word can be written three ways depending on how the author typed a half-space. A PDF can give you every word backwards. None of these raise an error. Retrieval just quietly gets worse.

I built persian-rag as a Persian-first RAG toolkit for Python. The core install is pure Python with no dependencies, and models, vector stores, parsers, and the HTTP server are optional extras. This post walks through the Persian-specific problems in pipeline order, the design decision behind each fix, and a small benchmark that shows where the fixes help and where they do not.

Problem 1: the same letter, different code points

Persian is written in an extended Arabic script, and several letters have an Arabic twin with a different code point that renders almost the same:

• Arabic Yeh ي (U+064A) and Alef Maksura ى versus Persian Yeh ی (U+06CC)
• Arabic Kaf ك (U+0643) versus Persian Keheh ک (U+06A9)

Text copied from Arabic-locale keyboards, older websites, or some CMS exports uses the Arabic forms. A user typing on a Persian keyboard uses the Persian ones. To BM25, and to many tokenizers, they are different words.

On top of that come three digit systems (ASCII 0-9, Persian ۰-۹, Arabic-Indic ٠-٩), the tatweel (kashida) used for stretching words, optional diacritics, and Arabic presentation forms from PDFs.

normalize handles all of it, driven by one config object:

from persian_rag import NormalizeConfig, normalize

normalize("يك كتاب با اعداد ٤٥٦")
# 'یک کتاب با اعداد ۴۵۶'

normalize("کِتاب", NormalizeConfig(remove_diacritics=True, digits="ascii"))
# 'کتاب'

The order of operations inside normalize is deliberate:

if options.compatibility_forms:
    text = unicodedata.normalize("NFKC", text)

text = text.translate(_INVISIBLE)

if options.unify_arabic_chars:
    text = text.translate(_ARABIC_TO_PERSIAN)

if options.remove_tatweel:
    text = text.replace(_TATWEEL, "")

if options.remove_diacritics:
    text = _DIACRITICS_RE.sub("", text)

text = _map_digits(text, options.digits)

if options.clean_punctuation:
    text = _clean_punctuation(text, options.punctuation)

text = _apply_zwnj_policy(text, options)
text = _normalize_whitespace(text)

NFKC runs first because it folds presentation forms (contextual glyphs such as U+FEE3) into base letters, and every later step expects base letters. Invisible characters such as bidirectional marks are stripped next so they cannot hide inside words. Half-space handling runs after punctuation cleanup, and whitespace is collapsed last, so neither step can undo the other.

Two design rules hold the whole thing together:

1. normalize is idempotent for a fixed config. normalize(normalize(t)) == normalize(t). That makes it safe to call at every layer without worrying about double application.
2. Documents and queries must use the same config. build_retriever and HybridRetriever apply it to both automatically, and a saved index stores the config and restores it on load. A mismatch here is the single easiest way to lose recall without noticing.

What it deliberately does not do: آ and ئ stay as they are. Folding them would merge words that are genuinely different, and that is a spelling question, not a code point question. It is handled later, at query time.

Problem 2: the half-space

The zero-width non-joiner (ZWNJ, U+200C), which Persian speakers call the half-space, is the most Persian-specific problem in the pipeline. It separates parts of a word visually without breaking it into two words. The verb prefix می, the plural suffix ها, and the comparative suffixes تر and ترین are joined to their word with it.

In practice people type it three ways: with a half-space, with a normal space, or attached with nothing in between. A search for the plural of "book" can miss a document because the author used a space where the query used a half-space.

persian-rag does two things. First, it corrects the common cases toward the half-space form, with regular expressions anchored on known prefixes and suffixes:

_PREFIX_SPACE_RE = re.compile(r"(?<![\w\u200c])(نمی|می|بی)[ \t]+(?=[\u0600-\u06FF])")
_SUFFIX_SPACE_RE = re.compile(r"(?<=[\u0600-\u06FF])[ \t]+(هایی|های|ها|ترین|تر)(?![\w\u200c])")

Then it cleans up the half-spaces themselves: repeated ones collapse into one, spaces around them are removed, and half-spaces next to punctuation or at the edges of a string are dropped.

normalize("كتاب هاي مي روم ٤٥٦")
# 'کتاب\u200cهای می\u200cروم ۴۵۶'

Arabic letters, spaced suffix, spaced prefix, and Arabic-Indic digits all land on one canonical form.

The policy is configurable with zwnj="normalize" (the default), "remove", or "space", because some downstream systems cannot handle the character at all. That choice has consequences in the tokenizer, which is the next problem.

Problem 3: tokenization

A tokenizer that splits on whitespace and punctuation, or one that treats U+200C as a separator, turns a single Persian word into two tokens. For BM25 that is a real loss: the suffix ها (plural) is extremely common, so it carries almost no IDF weight, and the stem token is shared with the singular form.

The tokenizer in persian-rag keeps the half-space inside the token:

# Letters only. Digits, diacritics, and Arabic punctuation stay out of words
# so "سلام،" does not become a single token.
_TOKEN_RE = re.compile(
    r"[0-9۰-۹٠-٩]+(?:[.\u066b][0-9۰-۹٠-٩]+)?|"
    r"[A-Za-z]+(?:'[A-Za-z]+)?|"
    r"[\u0621-\u063A\u0641-\u064A\u066E\u0671-\u06D3\u06FA-\u06FF\u200c]+"
)

Because normalization runs first, the spaced and half-spaced forms converge:

tokenize("می\u200cروم")   # ['می\u200cروم']
tokenize("می روم")        # ['می\u200cروم']
tokenize("میروم")         # ['میروم']

The first two become the same token. The third, written fully attached, does not. I chose not to split or join attached forms in the tokenizer, because doing that reliably needs a morphological analyzer, and a wrong split is worse than a missed match. Attached forms are handled by query expansion instead.

If you choose zwnj="space", you get what most generic tokenizers give you:

tokenize("کتاب\u200cها", config=NormalizeConfig(zwnj="space"))
# ['کتاب', 'ها']

Numbers are separate tokens, including decimals with the Arabic decimal separator (U+066B), and Latin tokens are lowercased so mixed Persian and English product names still match.

Problem 4: PDFs that store Persian backwards

Many Persian PDFs, especially older ones, store lines in visual order: the glyphs are in the order they appear on screen from left to right, so every Persian word comes out of the extractor reversed, while numbers and Latin words inside the line read normally. Indexing that text produces an index of reversed words that no query will ever match.

The loader detects this per page with a score built from two signals:

def visual_order_score(text: str) -> int:
    score = 0
    for word in _WORD_RE.findall(text):
        if len(word) > 1:
            first, last = _form(word[0]), _form(word[-1])
            if first == "FINAL" or last == "INITIAL":
                score += 1
            if first == "INITIAL" or last == "FINAL":
                score -= 1
        folded = unicodedata.normalize("NFKC", word)
        if folded in _REVERSED_COMMON:
            score += 2
        elif folded in _COMMON_WORDS:
            score -= 2
    return score

The first signal uses presentation forms: a word that starts with a final-form glyph or ends with an initial-form glyph is almost certainly reversed. The second uses a short list of very frequent words (از, به, که, در, این, and so on): seeing them reversed is strong evidence. Positive means visual order.

from persian_rag.loaders import clean_extracted_text, visual_order_score

line = "دش جراخ پاچ زا 1402 لاس رد باتک نیا"
visual_order_score(line)        # 6
clean_extracted_text(line)      # 'این کتاب در سال 1402 از چاپ خارج شد'

The line is reversed and the left-to-right runs inside it, like 1402, are restored. On the ingest CLI this is --rtl-fix auto (the default), with reverse and none for when you know better than the heuristic.

The heuristic can be wrong on very short lines or lines with no frequent words and no presentation forms. Detecting per page rather than per line is the compromise: there is more evidence on a page, and a PDF rarely mixes orders within one page.

Problem 5: spelling variants that normalization must not fix

Persian has many words with more than one accepted spelling. Room can be اتاق or اطاق. Tehran is تهران or the older طهران. Board is هیئت, هیات, or هیأت. Ticket is بلیت or بلیط. These are not encoding problems, so normalization should not touch them. Rewriting documents would destroy the original text you later want to cite.

Instead, persian-rag expands the query. QueryRewriter produces a short, ordered list of variants: the original query, spelling variants from a built-in table of groups, orthographic variants (hamza seat, alef madda, ئی and یی), half-space spacing variants of prefixes and suffixes, and optionally rewrites from a language model.

from persian_rag import QueryRewriter

QueryRewriter().variants("اطاق های هتل طهران")
# ['اطاق های هتل طهران', 'اطاق های هتل تهران', 'اتاق های هتل طهران',
#  'اطاق\u200cهای هتل طهران', 'اطاقهای هتل طهران']

The spacing variants are how the fully attached form from Problem 3 gets found.

RewritingRetriever wraps any retriever, runs every variant, and fuses the rankings with reciprocal rank fusion:

weights = [self.original_weight] + [1.0] * (len(variants) - 1)
fused = reciprocal_rank_fusion(rankings, k=self.rrf_k, weights=weights)

Two decisions here matter. The original query counts double (original_weight=2.0), so expansion adds recall without pushing exact matches down. And every result records which variants found it in metadata["matched_queries"], so you can see why a document was retrieved. Debuggable expansion is much easier to trust.

The same reciprocal rank fusion merges BM25 and dense results in HybridRetriever, with rrf_k=60 and separate weights for each side. RRF works on ranks, not scores, so I never have to calibrate a BM25 score against a cosine similarity.

Measuring it: a small Persian bench

I wanted a check that fails when a change breaks Persian-specific behavior, so the repository ships a benchmark in benchmarks/: 34 short documents (several are distractors that share words with other documents) and 42 labeled questions. Twelve questions are tagged spelling: they use a different accepted spelling than the corpus. The data is also on the Hugging Face Hub as sadrmanesh/persian-rag-bench.

python benchmarks/run_benchmark.py

These are the numbers I get with the offline hashing embedder (512 dimensions, character 2 to 4 grams), chunks of 400 characters, and k=10:

| Configuration    | Hit@1 | Hit@5 | MRR@10 | nDCG@10 | Hit@1 (spelling) |
|------------------|------:|------:|-------:|--------:|-----------------:|
| sparse           | 0.881 | 0.881 |  0.881 |   0.879 |            0.583 |
| dense            | 0.952 | 0.976 |  0.957 |   0.962 |            0.917 |
| hybrid           | 0.952 | 0.976 |  0.964 |   0.967 |            0.833 |
| sparse + rewrite | 0.952 | 0.976 |  0.964 |   0.967 |            0.833 |
| dense + rewrite  | 0.952 | 0.976 |  0.957 |   0.962 |            0.917 |
| hybrid + rewrite | 0.952 | 0.976 |  0.964 |   0.967 |            0.833 |

What I read from this:

• BM25 alone misses five of the twelve misspelled queries, because a different spelling is a different token. The rewriter recovers most of them: sparse Hit@1 on the spelling subset goes from 0.583 to 0.833, and overall Hit@1 from 0.881 to 0.952. That is the effect the feature exists for.
• The dense numbers flatter the embedder. The hashing embedder compares character n-grams, so it is naturally tolerant of one-letter differences like ت versus ط. A semantic embedding model behaves differently. The script takes --embedder sentence-transformers --model ... so you can measure yours.
• Hybrid is not always best. On the spelling subset, hybrid (0.833) is below dense alone (0.917). Fusing in BM25, which misses those queries, pulls some correct dense results down. I left this in the docs on purpose. It is the kind of result that a benchmark is for.
• Rewriting is not free. Each variant is another retrieval call. In-memory it stays in the low milliseconds, but against a remote search engine it multiplies requests.

The set is small and the scores are high by design. It is a regression check for Persian behavior, not a leaderboard, and a test runs the script offline on every change so it cannot quietly rot. For decisions that matter, extend qa.jsonl with questions from your own domain.

For retrieval metrics on your own data, evaluate_retrieval reports hit rate, MRR, recall, and nDCG at several cut-offs, and persian-rag eval does the same from the CLI. If you need statistically careful comparisons between two retrieval systems, that is the problem I tackled separately in Is your new search actually better? (llm-search-eval).

What else is in the box

Briefly, since this post is about the Persian-specific parts: loaders for text, Markdown, HTML (with main-content extraction), PDF, Word, and URLs; fixed, sentence, and recursive chunking; in-memory, FAISS, Qdrant, pgvector, Elasticsearch, and OpenSearch stores (the last two also serve BM25 with the persian analyzer); incremental indexing by content hash; cross-encoder or LLM reranking; citation-grounded answers that resolve [n] markers, including Persian and Arabic-Indic digits, to source spans; conversation memory; a FastAPI server; and a CLI for ingest, query, chat, eval, and serve.

Limitations

• No morphological analysis. Attached forms, broken plurals, and Arabic-origin plural patterns are not unified by the tokenizer. Query expansion covers common spacing cases, not morphology.
• The spelling table is hand-curated. It covers common groups, and you can add your own with QueryRewriter(extra_groups=[...]), but it is not exhaustive.
• PDF order detection is a heuristic. It is good on normal prose and weaker on tables, short headings, and pages with little Persian text.
• The benchmark is small and the default embedder is not semantic. Treat its numbers as a regression check, not as evidence about your model.

What's next

More spelling groups from real query logs, an optional morphological stemmer behind an extra so the core stays dependency-free, and larger evaluation sets with semantic embedders, published with the same honesty as the current table.

Install

pip install persian-rag
pip install "persian-rag[pdf,docx,qdrant,server]"   # pick the extras you need

from persian_rag import HashingEmbedder, RAGPipeline, build_retriever

texts = [
    "حافظ در شهر شیراز زاده شد و دیوان او مشهور است.",
    "فوتبال ورزشی گروهی با دو تیم است.",
]
retriever = build_retriever(texts, HashingEmbedder())
result = RAGPipeline(retriever, lambda prompt: "...").query("حافظ در کدام شهر زاده شد؟")

The HTTP API is also published as ghcr.io/sadrmanesh/persian-rag:0.3.1.

• Repo: https://github.com/sadrmanesh/persian-rag
• Docs (English and Persian): https://sadrmanesh.github.io/persian-rag/
• Live demo: https://huggingface.co/spaces/sadrmanesh/persian-rag-demo

MIT licensed. If you work with Persian text and have a spelling group or a PDF that breaks the heuristics, please open an issue with an example.