DEV Community: saman gh

How to Actually Test a Chatbot (Not "The Answer Looked Fine")

saman gh — Wed, 09 Sep 2026 15:43:43 +0000

Most teams accept a chatbot the same way: they ask it a handful of questions, the answers look reasonable, and the service is signed off. That procedure proves almost nothing.

The short answer

Evaluating a Persian-language assistant properly needs three things: a fixed set of questions replayed identically every time, an explicit assertion for each one about which source the answer must come from, and response time measured separately from network time. Without all three, any change to the system can break something nobody notices.

"The answer looked fine" is not a criterion

A retrieval or generation system almost always returns something. If it does not recognise the topic it returns a default; if it recognises the wrong topic it returns an answer that is correct in itself and irrelevant to the question.

The second failure is the more deceptive one, because it reads well. In our own evaluation, the question do you offer enterprise training? was answered with the project-timeline text — entirely accurate, entirely unrelated. Anyone reading only the answer sees nothing wrong.

The fix is for each test case to assert not "the answer was good" but that the answer contains a specific marker: the path of the page it should point to, a distinctive phrase from the reference text, or the contact number. A fluent irrelevant answer then fails.

// Not this — it passes for any non-empty string:
assert(answer.length > 0);

// This — it fails a fluent answer that came from the wrong place:
assert(answer.includes("/chatbot-karaj"));

Build a question set, not a few scattered checks

What worked in practice was a list where every entry carries four fields: the question text, the language, a string that must appear in the answer, and a label saying what the case is testing.

const CASES = [
  { q: "آموزش سازمانی دارید؟",  lang: "fa", expect: "/enterprise-ai-training-karaj", why: "enterprise vs project timeline" },
  { q: "چت‌بات چقدر طول می‌کشد؟", lang: "fa", expect: "/chatbot-karaj",                why: "ZWNJ spelling"                },
  { q: "چت بات چقدر طول می‌کشد؟", lang: "fa", expect: "/chatbot-karaj",                why: "space spelling — same target" },
  { q: "Do you build chatbots?",  lang: "en", expect: "/chatbot-karaj",                why: "english routing"              },
  { q: "asdkjh qweqwe",           lang: "fa", expect: "__default__",                   why: "gibberish hits the fallback"  },
  { q: "وام می‌دهید؟",            lang: "fa", expect: "__not_offered__",               why: "must never promise this"      },
];

Ours grew to 47 cases and covers:

Ordinary questions for each service, in Persian, English and Arabic
The same question in two spellings — with a space and with a zero-width non-joiner — which must both reach the same destination. The mechanics are in why your Persian chatbot answers the wrong question
Every question that previously produced a wrong answer. A fixed defect without a test case comes back
One nonsense input, to confirm the default reply actually works
A negative assertion: something the organisation does not offer must never be promised

Do not skip that last one. If you do not provide a service, the system must say so plainly. A customer who calls on the strength of an incorrect promise costs you both time and credibility.

Three silent failures only testing reveals

1. Rate limiting that returns empty responses. Our middleware allows 20 requests per minute per address. Sending 42 questions back to back meant everything from the twenty-first onward came back empty — which looked exactly like a broken service, while the service was healthy and the test was wrong. A question set must be paced and must report the status code, not only the answer body.

// 20/min per IP, so ~3.2s between cases. Without this the suite
// reports a working service as broken from case 21 onward.
for (const c of CASES) {
  const res = await ask(c.q);
  report(c, res.status, res.body);      // status, not just the body
  await new Promise((r) => setTimeout(r, 3200));
}

2. An interface that never opens. It is possible for the interface to fail to open for every user while the API answers perfectly. No API test can see this; only loading the site in a real browser and clicking the button. Server logs answer it too: if no request from a user's browser has reached the conversation endpoint in days, the interface is not working.

3. Stale knowledge. If the knowledge base is read only at service start-up, anything added today stays invisible until the next restart. The test case is simple: publish something new and immediately ask about it.

Separate server time from network time

Measured from your own machine, response time is the sum of network and server time. Knowing which to improve takes two measurements:

A request for a static file on the same domain, giving the network floor
The same answering request issued from inside the server against its local address

In our measurements, total time from outside was around 400 milliseconds and the same request on the server itself took 10 to 14. Almost all of it was network, and further code optimisation would have changed nothing for the user. Without the second measurement, days could have gone into optimising the wrong thing.

Keep a log of unanswered questions

Every question the system could not answer should be appended to a file. It is the only reliable list of your knowledge gaps, because a user who gets no answer rarely complains — they simply leave.

Practical advice: store only the question text, a timestamp and the language; cap the file size; review it monthly. Questions that recur are next month's work list.

One caveat learned the hard way: whatever filter you use to drop keyboard-mash from that log, write it with character codes rather than a regex that has to survive several layers of shell escaping. A \s that arrives as a bare s will silently drop every question containing the letter s, and the log will look healthy while losing most of its content.

Frequently asked questions

How many test cases are enough?

There is no magic number, but three groups must be covered: ordinary questions for each service, every defect previously found, and nonsense input. Ours began at 42 cases and grew with each new defect.

How often should it run?

Before every release, and after every change to the knowledge base. A full run takes a few minutes, and those minutes move defect discovery out of production and into testing.

Can this be automated?

Yes, and it should be. A script that reads the question list, issues the requests and checks the assertions. The one technical caveat is to send Persian text from inside the program rather than from a command line — some shells corrupt the encoding, and the result looks like a system fault. A three-character word arriving as five bytes of nonsense makes every keyword lookup miss, which reads exactly like a broken retriever.

How is this different from testing an ordinary website?

On an ordinary website the correct output is unique. In an answering system several answers can be correct, so the assertion belongs on the source of the answer rather than its exact wording — "the answer must reference the chatbot page", not "the answer must be this sentence".

Does the same method work for a model-based system?

Yes, with one addition: include a case that checks the system says plainly that it does not know when no reference document exists. The two architectures and their different risks are compared in language model or retrieval from your own knowledge.

What if we have no internal technical team?

Build the question set yourselves — that takes business knowledge, not technical knowledge — and have a contractor automate the run. Ownership of the question list should stay with you, because it is what defines quality.

From building and re-building a Persian-language assistant at Qatreh in Karaj, Iran. The 47-case suite started as six cases and a wrong assumption.

Why Your Persian Chatbot Answers the Wrong Question

saman gh — Mon, 07 Sep 2026 11:18:24 +0000

When a Persian-language assistant answers half its questions with "I don't have information about that," the first assumption is always that its knowledge base is too small. Usually it isn't. The program is seeing one word as several unrelated strings.

The short answer

In Persian — and in Arabic script generally — a single word can be written several ways that look identical to a reader and are completely different to a computer. چت‌بات written with a zero-width non-joiner and چت بات written with an ordinary space share no byte sequence. Until text is folded into one comparable form before any comparison, a large share of real questions will miss, and nothing in the logs will indicate a fault.

One word, several spellings

None of these variants is a spelling mistake. All of them appear in formal published Persian, and all are encoded distinctly in Unicode:

The zero-width non-joiner (U+200C) is a real character with no width. It separates parts of a compound without a space, so می‌رود, می رود and میرود are three distinct strings.
Persian yeh (U+06CC) versus Arabic yeh (U+064A) render nearly identically in most fonts, and different keyboard layouts produce different ones.
Persian keheh (U+06A9) versus Arabic kaf (U+0643) — the same situation.
Heh versus teh marbuta — a writer using an Arabic layout types شرکة rather than شرکه.
Three digit sets. Persian digits (U+06F0–U+06F9), Arabic-Indic digits (U+0660–U+0669) and ASCII digits are three separate encodings of the same numbers.
Diacritics and tatweel. Optional vowel marks and the decorative elongation character survive in pasted text and break matching silently.

For a term as ordinary as "chatbot", that yields at least four common written forms, all correct. A system that recognises one of them fails three users in four.

Here is the problem in three lines:

const a = "چت‌بات";      // with U+200C
const b = "چت بات";      // with a space

a === b                  // false
a.length === b.length    // true  <-- the trap

[...a].map(c => c.codePointAt(0).toString(16))
// [ '686', '62a', '200c', '628', '627', '62a' ]
//                  ^^^^ the character you cannot see

Why nothing appears in the logs

This is the part that makes the problem persist. When matching fails the program does not crash — it falls through to a default reply. The web server records a 200, uptime monitoring stays green, and no dashboard turns red.

From outside, the only symptom is that a user asks once, receives a generic answer, and closes the window. That pattern was present in our own system and stayed invisible until a fixed set of real questions was run against it. The method is in how to actually test a Persian chatbot.

The second trap: substring matching

Suppose the system detects intent by searching for keywords inside the incoming question. It is the simplest approach and it carries a serious flaw, because Persian attaches prefixes and suffixes freely.

A real example from our own system: the keyword زمان (time) was registered to detect questions about project schedules. A user asked do you offer enterprise training? — which in Persian contains the word سازمانی (organisational). The four-letter keyword was found as a substring inside it, and the system answered with a project timeline.

"آموزش سازمانی دارید؟".includes("زمان")   // true — and wrong

The fix is to compare at whole-token level rather than by substring, and to allow a prefix match only for terms long enough that the shared opening cannot be coincidental — so آموزش still matches آموزشی, while زمان no longer matches سازمانی.

The order the folding has to happen in

Before any comparison, the incoming text and the reference text must pass through the same pipeline:

Strip diacritics and tatweel
Convert the zero-width non-joiner and directional marks to a plain space
Convert Persian and Arabic-Indic digits to ASCII
Unify letters: yeh, alef maqsura and yeh hamza to Persian yeh; Arabic kaf to keheh; teh marbuta to heh; the alef forms to plain alef
Lowercase any Latin characters
Replace every non-letter, non-digit character with a space and collapse runs of whitespace

In full — every character written as an escape, so it survives copy and paste:

const ARABIC_FORMS = {
  "ي": "ی", // Arabic yeh    -> Persian yeh
  "ى": "ی", // alef maqsura  -> Persian yeh
  "ئ": "ی", // yeh with hamza
  "ك": "ک", // Arabic kaf    -> Persian keheh
  "ة": "ه", // teh marbuta   -> heh
  "ۀ": "ه", // heh with yeh above
  "أ": "ا", // alef with hamza above
  "إ": "ا", // alef with hamza below
  "آ": "ا", // alef with madda
  "ؤ": "و", // waw with hamza -> waw
};

function normalize(input) {
  return input
    // 1. harakat, dagger alef, tatweel — decorative, never semantic
    .replace(/[ً-ْٰـ]/g, "")
    // 2. ZWNJ and the bidi control marks become a SPACE, not nothing
    .replace(/[-‏‪-‮⁦-⁩]/g, " ")
    // 3. Persian digits, then Arabic-Indic digits
    .replace(/[۰-۹]/g, (d) => String(d.charCodeAt(0) - 0x06f0))
    .replace(/[٠-٩]/g, (d) => String(d.charCodeAt(0) - 0x0660))
    // 4. one letter per sound
    .replace(
      /[يىئكةۀأإآؤ]/g,
      (c) => ARABIC_FORMS[c] ?? c
    )
    // 5 and 6. fold case, then reduce punctuation to whitespace
    .toLowerCase()
    .replace(/[^\p{L}\p{N}]+/gu, " ")
    .replace(/\s+/g, " ")
    .trim();
}

It should pass all three of these:

normalize("چت‌بات") === normalize("چت بات")   // true — the ZWNJ case
normalize("شرکة")   === normalize("شرکه")     // true — teh marbuta
normalize("۱۴۰۳")   === normalize("1403")     // true — Persian digits

The subtlety is in step two. The zero-width non-joiner must become a space, not be deleted. Delete it and چت‌بات becomes چتبات, which still does not match چت بات. Convert it to a space and both spellings arrive at the same two tokens and find each other.

"چت‌بات".replace(/‌/g, "")    // "چتبات"  — still no match
"چت‌بات".replace(/‌/g, " ")   // "چت بات" — matches

One more thing that costs nothing and catches a whole class of bugs: run the reference strings through normalize too, at startup. A keyword list typed by a developer on one keyboard layout and a question typed by a user on another will otherwise never meet, no matter how good the folding is on the input side.

Why this matters more for a local deployment

Customers of a shop, a training centre or a factory write from a range of devices and keyboard layouts. Some produce standard Persian characters; others produce the Arabic forms. Both families are therefore present in genuine inbound messages.

A system validated only against clean, uniform text performs flawlessly in a demonstration and stumbles on contact with real users. We run this evaluation against real message traffic rather than synthetic text, because the gap between those two is the gap between a successful demo and a service that holds up.

Frequently asked questions

What exactly is the zero-width non-joiner and why does it break things?

It is Unicode character U+200C. It has no visual width but is stored in the string. می‌رود and می رود are therefore identical to the eye and different to any string comparison.

Does this only affect chatbots?

No. Anywhere Persian text is compared is affected: product search, customer-name lookup, address matching, deduplication. In a database, two spellings of one company name remain two separate records.

What share of questions fail because of this?

There is no fixed figure; it depends on the vocabulary involved. In our own evaluation, before folding was introduced, three out of ten perfectly ordinary Persian questions fell through to the default reply. Afterwards, all ten reached the correct topic.

Do large language models avoid the problem?

They are more tolerant of spelling variation, but they do not remove it. Wherever string matching happens before or after the model — document retrieval, filtering, routing to a department — the same trap returns. I compare the two architectures in language model or retrieval from your own knowledge.

How expensive is the folding step?

Negligible. It is six replacements over a short string. In our measurements the entire server-side processing time, including retrieving the answer, was 10 to 14 milliseconds. The cost of not doing it is far higher.

Isn't this what `String.prototype.normalize()` is for?

No — and this is a common misread. Unicode NFC and NFKC compose and decompose characters, but Persian yeh and Arabic yeh are separate letters with separate meanings, not two encodings of one character. Unicode will not merge them for you, and it should not. Script folding is an application-level decision, and only you know how aggressive it should be.

Written while building a Persian-language assistant at Qatreh in Karaj, Iran. If you have hit a variant this pipeline misses, I would like to hear about it.

DEV Community: saman gh

How to Actually Test a Chatbot (Not "The Answer Looked Fine")

The short answer

"The answer looked fine" is not a criterion

Build a question set, not a few scattered checks

Three silent failures only testing reveals

Separate server time from network time

Keep a log of unanswered questions

Frequently asked questions

How many test cases are enough?

How often should it run?

Can this be automated?

How is this different from testing an ordinary website?

Does the same method work for a model-based system?

What if we have no internal technical team?

Why Your Persian Chatbot Answers the Wrong Question

The short answer

One word, several spellings

Why nothing appears in the logs

The second trap: substring matching

The order the folding has to happen in

Why this matters more for a local deployment

Frequently asked questions

What exactly is the zero-width non-joiner and why does it break things?

Does this only affect chatbots?

What share of questions fail because of this?

Do large language models avoid the problem?

How expensive is the folding step?

Isn't this what String.prototype.normalize() is for?

Isn't this what `String.prototype.normalize()` is for?