DEV Community: João Reis

How we trained a multilingual AI-text detector (minus the secrets)

João Reis — Thu, 08 Oct 2026 13:09:49 +0000

We built our own model to detect AI-written text in six languages, and it now runs behind every check on Probator.ai. This is how we trained it.

We'll be open about the method and keep a few things to ourselves:

which models wrote our AI training text;
how much data we used;
the exact architecture;
the thresholds.

Anything that makes a detector easier to fool doesn't belong in a blog post. Everything else is here.

1. Teach "who wrote it", not "what it's about"

The easiest way to build a bad AI detector is to collect human text from one place and AI text from another. The model learns that encyclopedia articles are human and blog-style answers are AI, scores beautifully on its own test set, and fails on real documents.

So every piece of training data comes in pairs: a human text, and an AI text written on the same topic, in the same language and genre. The only consistent difference left between the two sides is who wrote them, and that's what the model has to learn.

2. Human text from before chatbots

You can't be sure a text is human-written if it was published after 2022. So the human side comes from writing that predates chatbots:

encyclopedia articles;
web pages crawled years ago;
news articles;
academic abstracts.

Every text goes through quality filters, and the material is spread across six languages: English, Portuguese, Spanish, French, German and Italian.

Variety of genre matters as much as volume. Formal, technical and translated writing is where AI detectors accuse humans most often, so that kind of text is deliberately well represented.

3. AI text from many models, written many ways

The AI side is written by a varied set of current models, with prompts that ask for different genres, lengths and tones. A detector trained on one model learns that model's habits. A detector trained on many has to learn what they share.

Two harder kinds of AI text go in as well:

AI-polished human text: a human draft that a model rewrote;
"humanized" text: text a model wrote with instructions to sound human.

4. Split by topic, never by sentence

This is the step most homemade evaluations get wrong. We score text passage by passage. If you shuffle passages into training and test sets at random, passages of the same document end up on both sides. The model then partly recognises documents it has already seen, and your accuracy looks better than it is.

We split by topic pair instead. Each pair gets a stable hash and goes, whole, to one of three sets:

train: what the model learns from;
validation: what we calibrate on;
test: what we report, and nothing else touches it.

A topic in the test set was never seen in training, by either the human or the AI text.

5. The model

Each passage becomes a multilingual sentence embedding, plus a handful of simple style features. A small neural network turns that into a probability that the passage is machine-written. The passage scores are combined into a score for the whole document.

The network is deliberately small. It adds only milliseconds to a check, and it's cheap enough to retrain whenever new models appear. The probabilities are calibrated, so 0.8 means roughly 80% of the time, not just "higher than 0.6".

6. Calibrate for the person who could be wrongly accused

An AI detector makes two kinds of mistake, and they don't cost the same. Missing an AI text is a shame. Telling a teacher that a student's own essay is AI-generated can hurt someone.

So the decision threshold is set per language, on the validation set, so that at most 1 in 100 human documents is flagged. Three more guards apply on top of the model:

Short texts: no "AI-generated" verdict under 80 words, unless there is hard evidence such as a chatbot leftover or a hidden mark.
Agreement: our model is one of several signals, alongside an expert reading by a language model, a rewrite test and forensic checks. The strongest verdict needs two of them to agree.
Transparency: when a guard changes a result, the report says so.

7. Go after your own false positives

Benchmarks don't find every weakness; users do. Early on, someone sent us a document they had written entirely themselves, and our model gave it a 9% AI likelihood. That's below any verdict, but a careful reader shouldn't see 9% on fully human writing.

We looked at what kind of writing it was: formal, structured, with few personal touches. We added more human writing of that kind to the training data, retrained, and checked again. The broader lesson: the texts most likely to be misjudged are the most valuable ones to collect.

8. Training without a GPU cluster

The whole pipeline runs on serverless infrastructure: collecting text, generating AI text, embedding and training. There's no long-running machine. That shaped the design:

Small resumable steps: each step is short and saves its state, so training survives restarts and timeouts. The model's weights are checkpointed to object storage after each step.
A queue drives the run: each finished step schedules the next.
A watchdog: it notices a step that stalled and starts it again.
Additive runs: a new run can add fresh texts on top of everything collected before, instead of starting from zero.

The first version stalled in the middle of an epoch more than once. Making every step resumable is what made training boring, which is how training should be.

9. Results

On held-out test documents, in all six languages:

Accuracy: 98.4%
AUROC: 0.998

AUROC measures how well the model ranks AI text above human text, whatever the threshold: 1.0 is perfect, 0.5 is a coin toss. Results by language are on our accuracy page, and we update it whenever a new model goes live.

A detector is never proof, though. Our results come with the reasons behind them, so a person can make the decision.

What we keep to ourselves, and why

We don't publish:

which models wrote the AI training text;
the size of the data;
the network's exact shape;
the thresholds.

With them, it would be easier to tune text until it slips under the detector. Everything that tells you whether to trust the results, the method, how the split works and the test results, is public.

Try it

Free web editor: 5,000 credits a month, no card needed.
REST API and remote MCP server: to plug the same checks into your code or your AI agent.

What would you want to know about how a detector was trained before you trusted it? Tell us in the comments.

Give your AI agent a fact-checker for text: AI detection, grammar and plagiarism over MCP

João Reis — Thu, 08 Oct 2026 12:59:16 +0000

More and more of the text that flows through our apps is written, edited or summarised by a language model. Content platforms want to know before they publish it. Schools want to know before they grade it. And under Article 50 of the EU AI Act, anyone publishing AI-generated text to inform the public will need to label it.

Probator.ai checks a text for three things in one call:

AI generation: a likelihood and a verdict, sentence by sentence, with the evidence behind it;
Grammar and style: corrections with explanations, never applied for you;
Plagiarism: matching sources from the web, academic databases and Wikipedia in every language.

It also reports hidden characters, look-alike letters and AI provenance marks (C2PA, IPTC labels, invisible Unicode tags used for hidden prompts). It works in 100+ languages, and you can call it from your code or let your AI agent call it directly.

For agents: one line of MCP config

Probator runs a remote MCP server. Add it to any client that speaks MCP over HTTP (Claude, Cursor, VS Code, your own agent):

{
  "mcpServers": {
    "probator": { "type": "http", "url": "https://probator.ai/mcp" }
  }
}

The first call returns a 401 that points the client to OAuth 2.1. The client registers itself, you sign in and approve, and that's it: no keys to copy around. If your agent prefers a key, add "headers": { "Authorization": "Bearer pb_live_…" } instead.

Your agent gets five tools:

Tool	What it does
`check_ai`	AI-text detection with evidence and provenance marks
`check_grammar`	corrections, with explanations in 6 languages
`check_plagiarism`	originality score and matching sources
`check_all`	all three in one call
`get_credits`	plan and remaining credits (free)

Agents that can't run an OAuth client can still register: they send the person's email, the person signs in, and types a 6-digit code the agent shows them. The details are in auth.md.

For your code: a plain REST API

curl https://probator.ai/v1/detect \
  -H "Authorization: Bearer $PROBATOR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text": "Paste the text you want to check here..."}'

{
  "verdict": {
    "key": "likely_ai",
    "p_ai": 0.94,
    "confidence": "high",
    "guards": []
  },
  "sentences": [ … ],
  "evidence": [ … ],
  "provenance": { "declared": null, "marks": [] },
  "credits": { "used": 142, "remaining": 499858 }
}

Endpoints: /v1/analyze runs any combination of checks, and also accepts PDF, Word, Markdown, HTML and text files. /v1/detect, /v1/grammar and /v1/plagiarism run a single check each, and /v1/usage returns your plan and credits.
Errors: every error has a stable code (insufficient_credits, too_many_words, rate_limited), and every response carries x-credits-used and x-credits-remaining headers.
Specs: the OpenAPI file is public, and so are llms.txt and llms-full.txt, so your coding assistant can read the docs itself.

How the detection works (and why you can trust a "no")

Most AI detectors give you one opaque number. Probator combines four independent signals:

Its own detection model: a classifier over multilingual sentence embeddings, trained on human and AI-written text in six languages, including AI-polished and "humanized" text.
An expert reading by a language model that quotes the passages it finds suspicious.
A rewrite test: machine text changes little when a model polishes it again.
Forensic evidence: chatbot leftovers, invisible characters, look-alike letters, file metadata.

A false positive costs much more than a miss, so the engine is built to avoid them:

Short texts: no "AI-generated" verdict under 80 words without hard evidence.
Strongest verdict: two detectors must agree.
Per-language thresholds: calibrated to keep false positives on human text at 1% or less.

When a guard changes a result, the report tells you which one, in verdict.guards.

On held-out test documents the model scores 98.4% accuracy and an AUROC of 0.998. Results by language are published on the accuracy page.

Results are probabilities with reasons, not proof. Use them to decide what a person should review, not to decide about a person.

What people build with it

Publishing pipelines: check articles before they go live and add the AI disclosure where it's needed.
Learning platforms: run submissions through detection and plagiarism, then show teachers the evidence, not just a score.
Editorial and review agents: an agent that triages incoming text, flags what needs a human, and fixes the grammar on the rest.
Moderation queues: catch generated spam and copied content across languages.
Signed certificates: turn a check into a PDF certificate with an Ed25519 signature that anyone can verify.

Privacy, briefly

Saved documents and the account database stay in the EU. Texts are never used to train models, and the language models it uses are called with no retention and no training. Probator reports hidden marks and AI labels but never removes them.

Try it

Free: 5,000 credits a month in the web editor, no card needed. AI detection costs 1 credit per word.
API and MCP: included in Pro (€12.99 a month, 500,000 credits) and Team (€39 a month for 3 seats, 2,000,000 shared credits).
Docs: probator.ai/docs/api

I'd love to hear what you'd plug it into, and what your agent would need from a tool like this. Drop it in the comments.