The problem nobody talks about
Every "AI detector" landing page promises 99% accuracy. Then you paste in a
paragraph you actually wrote yourself and it flags you as ChatGPT. I kept
seeing this in the wild — students wrongly accused, editors discarding human
copy, and a pile of tools that were really just guessing.
So I went down the rabbit hole of how AI-content detection actually works,
built a tool to test the claims, and learned that the honest answer is far more
interesting than the marketing.
Why watermarking text is genuinely hard
Image models can embed an invisible statistical watermark (the SynthID-style
approach), and that's a real, checkable signal. Text is different. A model
generates tokens probabilistically; there's no natural place to hide a bit
string that survives copy-paste. Researchers have proposed watermarking the
logit distribution (green/red token lists), but it breaks under:
- Paraphrasing (rewrite it and the signal is gone)
- Short inputs (a 20-word sentence carries almost no statistical footprint)
- Translation
- Deliberate obfuscation
If a tool claims 100% accuracy on short text, it's lying. Anyone who's actually
benchmarked one knows it.
What the models actually look at
In practice detectors lean on a few weaker, statistical signals:
- Perplexity — how "surprising" the word choices are. AI text tends to be low-perplexity, very smooth.
- Burstiness — humans vary sentence length and rhythm wildly; LLMs are flatter.
- N-gram / token-frequency patterns that leak the decoding strategy.
None of these is a watermark. They're probabilities, and they fail on edited,
mixed, or short content. That gap between "statistical likelihood" and
"provenance" is where most products quietly pretend to be something they aren't.
What I built
I got tired of the gap, so I built a detector that reports honestly — a
confidence score plus an explanation of which signals fired, rather than a
single fake certainty. You can try it here: https://detectaiwatermarks.com
The technical choices I'd highlight:
python
# A naive "is this AI?" check fails. A useful one returns evidence.
signals = {
"perplexity": score_perplexity(text),
"burstiness": sentence_variance(text),
"watermark_scan": probe_known_watermark(text), # often None, that's fine
}
# Verdict is calibrated per-length: a tweet != an essay.
verdict = calibrate(signals, length=len(text))
Top comments (1)
The single most important design rule: short text gets a "not enough signal"
answer, not a guess. Refusing to answer is more honest — and more useful —
than a confident lie.
The takeaway
There is no reliable text "AI watermark" yet. Treat any claim of one with
suspicion.
A good detector is an evidence reporter, not an oracle.
Build the "I don't know" case in from day one, or your tool will eventually
hurt a real user.
Happy to answer questions about the watermarking research or the engineering
tradeoffs in the comments.