When a comparison system misses a passage, the matching algorithm is not the only place to look. It may be comparing a different sequence of characters from the one a reader sees. Before interpreting a similarity result, test what happens to the text on its way into the matcher.
I work on Unplag.bg as a co-founder and technical lead. The examples below are general test ideas, not a description of Unplag's internal implementation or measurements of its performance.
1. Equivalent Unicode representations
The Cyrillic letter й can be represented as a single code point or as и followed by a combining breve. They can look the same while a direct string comparison returns false. Normalizing both strings to NFC before comparison is one possible policy:
import unicodedata
composed = "й"
decomposed = "и\u0306"
assert composed != decomposed
assert unicodedata.normalize("NFC", composed) == unicodedata.normalize("NFC", decomposed)
This is a test for equivalent encodings, not a general spelling correction. The normalization policy should be documented and applied consistently to both sides. Keep the original text for display and check that highlights can be mapped back to it.
2. Lookalikes across writing systems
Latin a and Cyrillic а look similar but are different characters. The word папа written entirely in Cyrillic is not the same character sequence as пaпa, where the two a-shaped characters are Latin. A test suite should include both. A warning about mixed scripts can help a reviewer notice unexpected input; it is not evidence of deliberate manipulation. Mixed scripts occur legitimately too, for example in product names and quotations. Avoid silently replacing every lookalike: that can change meaning and create false matches. Unicode's security guidance discusses confusables and their limitations.
3. Extraction before matching
Use a small set of documents with a known reference transcription: a digital PDF, a scan requiring OCR, and a DOCX file. Compare the extracted text with the reference and inspect whitespace, punctuation, page breaks, hyphenation and Cyrillic characters. A failure here should be visible to a reviewer; it should not quietly become a low similarity score. Include examples with correctly attributed quotations and common phrases, because a match still requires contextual interpretation.
These checks cover input quality, not the accuracy of a full plagiarism or authorship assessment. A useful report should show the matching passages and sources, and leave the final judgment to people following the institution's procedure.
References
- Python: Unicode database and normalization
- Unicode: Unicode normalization forms
- Unicode: Security mechanisms and confusable characters
Disclosure: this article was generated with AI assistance for Martin Yovchevski's professional profile. Martin is co-founder of Unplag.bg. It describes general test ideas, not results from an experiment.
Top comments (0)