Testing Persian Text Tools with Real-World Edge Cases

Masoud Javadi — Thu, 20 Aug 2026 08:48:10 +0000

When testing text-processing tools, simple examples are usually not enough.

A sentence like "Hello world" can show that a basic counter works, but it does not reveal how a tool behaves with real content.

Real Persian text brings many edge cases that are easy to miss.

Why simple test cases fail

Many text tools are first tested with clean examples:

A few words separated by spaces
Short sentences
Simple numbers
Plain English text

These examples are useful, but real documents are much more complicated.

A Persian article may contain Persian words, English technical terms, URLs, email addresses, numbers, emojis, and invisible Unicode characters at the same time.

A tool that works perfectly on simple examples may produce unexpected results on real content.

ZWNJ and the half-space problem

One of the most common Persian text challenges is the zero-width non-joiner, also known as ZWNJ.

Persian writers use it in many words to create the correct visual structure.

For example:

می‌روم

and

خانه‌ها

look different from versions written with normal spaces.

A text tool needs to understand that these are not just ordinary spaces. Otherwise, word counting and text analysis can become inconsistent.

Mixed Persian and English content

Modern Persian content is rarely only Persian.

Technical articles often include:

Programming terms
Product names
URLs
Email addresses
File names
Numbers

For example, a single sentence may contain:

example.com

test@example.com

and a number like 2.5

A good text-processing tool needs to understand the difference between a sentence-ending period and a period that belongs to another structure.

Numbers and Unicode characters

Persian content can include Persian digits, Arabic digits, and English digits together.

For humans, these differences may look small.

For software, they are different Unicode characters.

This is why testing different number formats and Unicode cases is important when building reliable text tools.

Copying content from real sources

Another common challenge appears when content is copied from applications like Microsoft Word or web pages.

Copied content may include:

Formatting information
Hidden characters
Links
Lists
Tables
Different paragraph structures

A text analyzer should be tested against these situations, not only clean manually written text.

Building realistic test cases

This is why we started collecting more realistic Persian text analysis test cases.

Instead of testing only perfect examples, we focus on situations that actually happen during content creation and publishing workflows.

Our test cases repository on GitHub contains examples related to Persian text processing, Unicode behavior, mixed content, and edge cases.

Persian text analysis test cases

The lesson

Reliable text processing is not only about counting characters or splitting text by spaces.

The difficult part is defining expected behavior and testing it against real-world content.

Languages like Persian show why small details such as invisible characters, mixed writing systems, and formatting differences matter.

Good tools are built by understanding the problems users actually face.

DEV Community: Masoud Javadi