Why AI Detectors Fail on Non-English Text (And What I Built to Start Fixing It)

Amirali Marjani — Sat, 08 Aug 2026 13:12:10 +0000

A few months ago I ran a piece of completely human-written Persian text through one of the most popular AI-content detectors on the market. The tool was confident: 87% AI-generated. The text was a paragraph from a Persian blog post, written by a real person, years before ChatGPT existed.

That result stuck with me. Not because the tool was "buggy" — but because it revealed something more fundamental: almost every AI detector on the market was built for English, tested on English, and quietly assumes the rest of the world writes in English too.

That assumption breaks down fast outside of it. And as more of the internet's content — in every language — gets touched by LLMs, that gap is going to matter more, not less.

This post is about that gap, why it exists, and OriginLens AI, the detector I built to start closing it for Persian and English.

Why This Problem Actually Matters

AI content detection isn't just an academic curiosity. It's already load-bearing infrastructure in a few places:

Education. Universities and teachers increasingly rely on AI detectors to flag potential academic dishonesty. A false positive here isn't abstract — it can mean a real accusation against a real student.
Journalism and publishing. Editors and platforms want to know whether submitted content is human-authored, both for quality and for disclosure reasons.
Content moderation at scale. Platforms trying to limit AI-generated spam, fake reviews, or coordinated inauthentic content need some signal to work from.

All three of these use cases exist just as much in Persian, Arabic, Spanish, or Hindi as they do in English. But the tooling hasn't caught up — and when detection tools are wrong, they're not wrong in a neutral way. They're systematically wrong in the direction of flagging non-native or non-English writing as "AI," which is its own quiet form of bias.

How AI Detection Actually Works (The Short Version)

Most detectors — even the well-known commercial ones — lean on a couple of core statistical signals:

Perplexity: roughly, how "predictable" a piece of text is to a language model. Human writing tends to be less predictable — it has quirks, irregular word choices, and small imperfections. AI-generated text tends to be smoother and more statistically "expected."
Burstiness: how much sentence length and structure vary across a piece of text. Human writers naturally vary their rhythm; a lot of LLM output (especially at default settings) is more uniform.

These signals aren't crazy — they hold up reasonably well on English text, because the underlying language models were trained overwhelmingly on English data, and calibrated against English writing patterns.

The Non-English Gap

Here's where it falls apart. Persian (and many other languages) breaks the assumptions these tools quietly rely on:

Different structural baselines. Persian sentence structure, verb conjugation patterns, and word order don't map cleanly onto the "predictability" baselines these models learned from English text.
Almost no labeled training data. The datasets used to train and calibrate mainstream detectors are overwhelmingly English-first. There's very little large-scale, labeled human-vs-AI text specifically for Persian.
The false positive problem gets worse, not better. Apply an English-trained tool to a language it wasn't designed for, and the false positive rate isn't a rounding error — it's the default outcome.

What I Built

I built OriginLens AI: a hybrid AI-vs-human text detector that handles Persian and English as first-class languages, not an afterthought.

Architecture. The system combines two layers: a statistical/stylometric scorer (lexical diversity, sentence-length variance, repetition patterns, punctuation density) and a machine-learning ensemble (Logistic Regression, Random Forest, and XGBoost) trained on labeled text, with probability calibration on top. The two layers are reported separately as well as combined, so a prediction shows not just a percentage but whether the statistical and ML signals actually agree — when they don't, the system reports lower confidence or an explicit "Uncertain" verdict instead of forcing a confident-sounding number out of disagreeing evidence.

The dataset. I assembled a training set of roughly 10,000 labeled examples, combining established human-vs-AI corpora (HC3, M4) with Persian-specific sources (a Persian formal-text corpus, plus AI-generated samples across multiple LLMs and prompt styles) to avoid inheriting English-centric assumptions.

Explainability. Instead of a single opaque score, each prediction breaks down which signals pushed toward "AI" or "human" — sentence regularity, repetition, vocabulary diversity, word frequency, style markers — so the output is evidence, not a verdict from a black box.

Where It Actually Stands

I want to report this honestly rather than round up.

On a held-out test set of 600 examples (300 English, 300 Persian, never seen during training), the current model reaches 88% overall accuracy and an F1 score of 0.867. Broken down: English performs at F1 0.884, Persian at F1 0.852.

But aggregate numbers hide where the system actually struggles, and I'd rather name those directly than let a good-looking headline number stand in for them:

Informal AI-generated Persian text is still poorly detected — recall on this specific category is very low. The statistical and structural patterns that distinguish casual AI-written Persian from casual human Persian aren't well captured yet, and I don't think this is a data-volume problem alone; it may need a different kind of signal than the current feature set provides.
Formal, administrative-register Persian human text still gets over-flagged as AI in some cases, though this improved meaningfully during the last round of work (false-positive rate dropped from 19.3% to 16.0% on the held-out set).
Informal AI-generated English text is also under-detected relative to formal AI text — the same registers that trip up the Persian model turn out to be a harder problem in English too, which suggests the gap isn't purely about language, but about how differently AI models write when prompted casually versus formally.

None of these are hidden in the repo — they're documented as known limitations, along with the specific numbers, because a detector that quietly hides its weak spots is more dangerous than one that's upfront about them.

There Is No Perfect AI Detector

I don't believe any detector can reliably answer one question with 100% certainty: "Was this written by AI?" That's simply not how the problem works. Paraphrasing tools evolve. Language models improve. Human writing changes. Detection is fundamentally an arms race.

For that reason, I don't see AI detection as a system that delivers absolute truth. I see it as a decision-support tool that provides useful evidence — and the honest gaps above are part of that evidence, not something to paper over. Especially in education, no detector, including this one, should ever be treated as the final authority. Human judgment still matters.

What's Next

The next round of work is aimed squarely at the two open problems above: improving informal-register detection in both languages (which looks like it needs new features, not just more data), and closing the remaining formal-Persian false-positive gap. I'm also holding off on fine-tuning a transformer model (like ParsBERT) until the classical feature set has been pushed as far as it reasonably can — adding architecture complexity before understanding whether it's actually needed would just trade one kind of uncertainty for another.

The source code, dataset composition, and full evaluation numbers — including the categories where it still falls short — are on GitHub.

If you work with Persian or another underrepresented language and you've run into similar detector failures, I'd genuinely like to hear about it. I suspect this challenge extends well beyond Persian, and closing it will take better datasets, more transparent models, and a willingness to publish the numbers that don't look good alongside the ones that do.

🔗 Resources

GitHub: github.com/MRAVLM/OriginLens-AI
Live Demo: Coming soon

👤 Connect with Me

🔗 LinkedIn: linkedin.com/in/Amirali-Marjani
🐙 GitHub: github.com/MRAVLM
📧 Email: Amiralimarjany@gmail.com
📱 Telegram: t.me/Amirali_marjani

—

DEV Community: Amirali Marjani

Why AI Detectors Fail on Non-English Text (And What I Built to Start Fixing It)

Why This Problem Actually Matters

How AI Detection Actually Works (The Short Version)

The Non-English Gap

What I Built

Where It Actually Stands

There Is No Perfect AI Detector

What's Next

🔗 Resources

👤 Connect with Me