When testing text-processing tools, simple examples are usually not enough.
A sentence like "Hello world" can show that a basic counter works, but it does not reveal how a tool behaves with real content.
Real Persian text brings many edge cases that are easy to miss.
Why simple test cases fail
Many text tools are first tested with clean examples:
- A few words separated by spaces
- Short sentences
- Simple numbers
- Plain English text
These examples are useful, but real documents are much more complicated.
A Persian article may contain Persian words, English technical terms, URLs, email addresses, numbers, emojis, and invisible Unicode characters at the same time.
A tool that works perfectly on simple examples may produce unexpected results on real content.
ZWNJ and the half-space problem
One of the most common Persian text challenges is the zero-width non-joiner, also known as ZWNJ.
Persian writers use it in many words to create the correct visual structure.
For example:
میروم
and
خانهها
look different from versions written with normal spaces.
A text tool needs to understand that these are not just ordinary spaces. Otherwise, word counting and text analysis can become inconsistent.
Mixed Persian and English content
Modern Persian content is rarely only Persian.
Technical articles often include:
- Programming terms
- Product names
- URLs
- Email addresses
- File names
- Numbers
For example, a single sentence may contain:
example.com
and a number like 2.5
A good text-processing tool needs to understand the difference between a sentence-ending period and a period that belongs to another structure.
Numbers and Unicode characters
Persian content can include Persian digits, Arabic digits, and English digits together.
For humans, these differences may look small.
For software, they are different Unicode characters.
This is why testing different number formats and Unicode cases is important when building reliable text tools.
Copying content from real sources
Another common challenge appears when content is copied from applications like Microsoft Word or web pages.
Copied content may include:
- Formatting information
- Hidden characters
- Links
- Lists
- Tables
- Different paragraph structures
A text analyzer should be tested against these situations, not only clean manually written text.
Building realistic test cases
This is why we started collecting more realistic Persian text analysis test cases.
Instead of testing only perfect examples, we focus on situations that actually happen during content creation and publishing workflows.
Our test cases repository on GitHub contains examples related to Persian text processing, Unicode behavior, mixed content, and edge cases.
Persian text analysis test cases
The lesson
Reliable text processing is not only about counting characters or splitting text by spaces.
The difficult part is defining expected behavior and testing it against real-world content.
Languages like Persian show why small details such as invisible characters, mixed writing systems, and formatting differences matter.
Good tools are built by understanding the problems users actually face.
Top comments (0)