Why Your Persian Chatbot Answers the Wrong Question

saman gh — Mon, 07 Sep 2026 11:18:24 +0000

When a Persian-language assistant answers half its questions with "I don't have information about that," the first assumption is always that its knowledge base is too small. Usually it isn't. The program is seeing one word as several unrelated strings.

The short answer

In Persian — and in Arabic script generally — a single word can be written several ways that look identical to a reader and are completely different to a computer. چت‌بات written with a zero-width non-joiner and چت بات written with an ordinary space share no byte sequence. Until text is folded into one comparable form before any comparison, a large share of real questions will miss, and nothing in the logs will indicate a fault.

One word, several spellings

None of these variants is a spelling mistake. All of them appear in formal published Persian, and all are encoded distinctly in Unicode:

The zero-width non-joiner (U+200C) is a real character with no width. It separates parts of a compound without a space, so می‌رود, می رود and میرود are three distinct strings.
Persian yeh (U+06CC) versus Arabic yeh (U+064A) render nearly identically in most fonts, and different keyboard layouts produce different ones.
Persian keheh (U+06A9) versus Arabic kaf (U+0643) — the same situation.
Heh versus teh marbuta — a writer using an Arabic layout types شرکة rather than شرکه.
Three digit sets. Persian digits (U+06F0–U+06F9), Arabic-Indic digits (U+0660–U+0669) and ASCII digits are three separate encodings of the same numbers.
Diacritics and tatweel. Optional vowel marks and the decorative elongation character survive in pasted text and break matching silently.

For a term as ordinary as "chatbot", that yields at least four common written forms, all correct. A system that recognises one of them fails three users in four.

Here is the problem in three lines:

const a = "چت‌بات";      // with U+200C
const b = "چت بات";      // with a space

a === b                  // false
a.length === b.length    // true  <-- the trap

[...a].map(c => c.codePointAt(0).toString(16))
// [ '686', '62a', '200c', '628', '627', '62a' ]
//                  ^^^^ the character you cannot see

Why nothing appears in the logs

This is the part that makes the problem persist. When matching fails the program does not crash — it falls through to a default reply. The web server records a 200, uptime monitoring stays green, and no dashboard turns red.

From outside, the only symptom is that a user asks once, receives a generic answer, and closes the window. That pattern was present in our own system and stayed invisible until a fixed set of real questions was run against it. The method is in how to actually test a Persian chatbot.

The second trap: substring matching

Suppose the system detects intent by searching for keywords inside the incoming question. It is the simplest approach and it carries a serious flaw, because Persian attaches prefixes and suffixes freely.

A real example from our own system: the keyword زمان (time) was registered to detect questions about project schedules. A user asked do you offer enterprise training? — which in Persian contains the word سازمانی (organisational). The four-letter keyword was found as a substring inside it, and the system answered with a project timeline.

"آموزش سازمانی دارید؟".includes("زمان")   // true — and wrong

The fix is to compare at whole-token level rather than by substring, and to allow a prefix match only for terms long enough that the shared opening cannot be coincidental — so آموزش still matches آموزشی, while زمان no longer matches سازمانی.

The order the folding has to happen in

Before any comparison, the incoming text and the reference text must pass through the same pipeline:

Strip diacritics and tatweel
Convert the zero-width non-joiner and directional marks to a plain space
Convert Persian and Arabic-Indic digits to ASCII
Unify letters: yeh, alef maqsura and yeh hamza to Persian yeh; Arabic kaf to keheh; teh marbuta to heh; the alef forms to plain alef
Lowercase any Latin characters
Replace every non-letter, non-digit character with a space and collapse runs of whitespace

In full — every character written as an escape, so it survives copy and paste:

const ARABIC_FORMS = {
  "ي": "ی", // Arabic yeh    -> Persian yeh
  "ى": "ی", // alef maqsura  -> Persian yeh
  "ئ": "ی", // yeh with hamza
  "ك": "ک", // Arabic kaf    -> Persian keheh
  "ة": "ه", // teh marbuta   -> heh
  "ۀ": "ه", // heh with yeh above
  "أ": "ا", // alef with hamza above
  "إ": "ا", // alef with hamza below
  "آ": "ا", // alef with madda
  "ؤ": "و", // waw with hamza -> waw
};

function normalize(input) {
  return input
    // 1. harakat, dagger alef, tatweel — decorative, never semantic
    .replace(/[ً-ْٰـ]/g, "")
    // 2. ZWNJ and the bidi control marks become a SPACE, not nothing
    .replace(/[-‏‪-‮⁦-⁩]/g, " ")
    // 3. Persian digits, then Arabic-Indic digits
    .replace(/[۰-۹]/g, (d) => String(d.charCodeAt(0) - 0x06f0))
    .replace(/[٠-٩]/g, (d) => String(d.charCodeAt(0) - 0x0660))
    // 4. one letter per sound
    .replace(
      /[يىئكةۀأإآؤ]/g,
      (c) => ARABIC_FORMS[c] ?? c
    )
    // 5 and 6. fold case, then reduce punctuation to whitespace
    .toLowerCase()
    .replace(/[^\p{L}\p{N}]+/gu, " ")
    .replace(/\s+/g, " ")
    .trim();
}

It should pass all three of these:

normalize("چت‌بات") === normalize("چت بات")   // true — the ZWNJ case
normalize("شرکة")   === normalize("شرکه")     // true — teh marbuta
normalize("۱۴۰۳")   === normalize("1403")     // true — Persian digits

The subtlety is in step two. The zero-width non-joiner must become a space, not be deleted. Delete it and چت‌بات becomes چتبات, which still does not match چت بات. Convert it to a space and both spellings arrive at the same two tokens and find each other.

"چت‌بات".replace(/‌/g, "")    // "چتبات"  — still no match
"چت‌بات".replace(/‌/g, " ")   // "چت بات" — matches

One more thing that costs nothing and catches a whole class of bugs: run the reference strings through normalize too, at startup. A keyword list typed by a developer on one keyboard layout and a question typed by a user on another will otherwise never meet, no matter how good the folding is on the input side.

Why this matters more for a local deployment

Customers of a shop, a training centre or a factory write from a range of devices and keyboard layouts. Some produce standard Persian characters; others produce the Arabic forms. Both families are therefore present in genuine inbound messages.

A system validated only against clean, uniform text performs flawlessly in a demonstration and stumbles on contact with real users. We run this evaluation against real message traffic rather than synthetic text, because the gap between those two is the gap between a successful demo and a service that holds up.

Frequently asked questions

What exactly is the zero-width non-joiner and why does it break things?

It is Unicode character U+200C. It has no visual width but is stored in the string. می‌رود and می رود are therefore identical to the eye and different to any string comparison.

Does this only affect chatbots?

No. Anywhere Persian text is compared is affected: product search, customer-name lookup, address matching, deduplication. In a database, two spellings of one company name remain two separate records.

What share of questions fail because of this?

There is no fixed figure; it depends on the vocabulary involved. In our own evaluation, before folding was introduced, three out of ten perfectly ordinary Persian questions fell through to the default reply. Afterwards, all ten reached the correct topic.

Do large language models avoid the problem?

They are more tolerant of spelling variation, but they do not remove it. Wherever string matching happens before or after the model — document retrieval, filtering, routing to a department — the same trap returns. I compare the two architectures in language model or retrieval from your own knowledge.

How expensive is the folding step?

Negligible. It is six replacements over a short string. In our measurements the entire server-side processing time, including retrieving the answer, was 10 to 14 milliseconds. The cost of not doing it is far higher.

Isn't this what `String.prototype.normalize()` is for?

No — and this is a common misread. Unicode NFC and NFKC compose and decompose characters, but Persian yeh and Arabic yeh are separate letters with separate meanings, not two encodings of one character. Unicode will not merge them for you, and it should not. Script folding is an application-level decision, and only you know how aggressive it should be.

Written while building a Persian-language assistant at Qatreh in Karaj, Iran. If you have hit a variant this pipeline misses, I would like to hear about it.

DEV Community: saman gh