How to Benchmark an AI Detector on Your Own Data (Precision, Recall and False Positives in Python)

Every AI detector claims to be "99% accurate". Those numbers usually come from the vendor's own test set, written in the vendor's favourite style, and they rarely tell you the number that matters most: how often it flags real human writing.

The good news: you can measure a detector on your own data in an afternoon. In this post we'll build a small benchmark in Python that reports precision, recall and false-positive rate, and helps you pick a threshold that fits your use case.

(We build an AI detector, ProofAI, and we'd rather you test us properly than trust a marketing number. The method works for any detector.)

Step 1: Build a labelled test set

This is the part that matters most. You want two folders:

testset/
├── human/   # writing you KNOW is human
└── ai/      # writing you KNOW is AI-generated

Tips for a fair set:

• Human samples should look like your real users. If you're a university, use past essays from before 2022 (with permission). If you're in hiring, use old cover letters. Include non-native English writers, because that's where detectors are known to struggle.
• AI samples should match the same tasks. Give the same prompts to several models (for example ChatGPT, Gemini and Claude), and include some lightly edited AI text, because that's what you'll see in practice.
• Aim for at least 100 of each. Fewer than that and your numbers will swing a lot.
• Keep lengths similar. Very short texts are harder to classify, so don't compare 50-word human texts to 800-word AI ones.

Step 2: Score everything

# score_testset.py
import csv, os, time
from pathlib import Path
import requests

API_URL = "https://ai.proofai.in/api/v1/detect"

def detect(text):
    r = requests.post(API_URL,
        headers={"x-api-key": os.environ["PROOFAI_API_KEY"], "Origin": os.environ["PROOFAI_ORIGIN"]},
        data={"content": text}, timeout=30)
    r.raise_for_status()
    return r.json()["ai_probability"]

rows = []
for label in ("human", "ai"):
    for f in sorted(Path("testset", label).glob("*.txt")):
        text = f.read_text(encoding="utf-8")
        rows.append({"file": f.name, "label": 1 if label == "ai" else 0,
                     "words": len(text.split()), "score": detect(text)})
        time.sleep(0.5)

with open("scores.csv", "w", newline="") as fh:
    w = csv.DictWriter(fh, fieldnames=["file", "label", "words", "score"])
    w.writeheader(); w.writerows(rows)

Save the scores once, then experiment with thresholds offline without spending more API calls.

Step 3: Compute the metrics

# evaluate.py
import pandas as pd
from sklearn.metrics import confusion_matrix, roc_auc_score

df = pd.read_csv("scores.csv")

def report(threshold):
    pred = (df.score >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(df.label, pred, labels=[0, 1]).ravel()
    return {
        "threshold": threshold,
        "precision": tp / (tp + fp) if tp + fp else float("nan"),
        "recall": tp / (tp + fn),
        "false_positive_rate": fp / (fp + tn),
        "humans_flagged": f"{fp} of {fp + tn}",
    }

print("ROC AUC:", round(roc_auc_score(df.label, df.score), 3))
print(pd.DataFrame([report(t) for t in (50, 60, 70, 80, 85, 90, 95)]).round(3).to_string(index=False))

Reading the numbers

• False-positive rate (FPR): the share of human texts flagged as AI. For education and hiring, this is the number to watch. An FPR of 5% means 1 in 20 honest people gets flagged.
• Recall: the share of AI texts the detector catches.
• Precision: of everything flagged, how much was actually AI. This depends on how common AI text is in your real data, so don't over-trust it from a 50/50 test set.
• ROC AUC: how well scores separate the two groups across all thresholds (1.0 is perfect, 0.5 is random).

Step 4: Choose a threshold for your risk

There's no universal "right" threshold. Decide which mistake is worse for you:

Use case	Worse mistake	Pick a threshold that...
Student essays, hiring	Flagging a real person	keeps FPR very low (for example ≤1–2%), even if recall drops
Spam / content farm filtering	Letting junk through	keeps recall high, with human review on flags
Editorial quality checks	Neither is serious	sits in the middle; treat as a writing hint

Step 5: Slice the results

Overall numbers hide problems. Break them down:

df["length_bucket"] = pd.cut(df.words, [0, 150, 300, 600, 10_000])
for bucket, g in df.groupby("length_bucket", observed=True):
    human = g[g.label == 0]
    if len(human):
        print(bucket, "FPR at 85:", round((human.score >= 85).mean(), 3), f"(n={len(human)})")

Do the same by writer group (native vs non-native English), topic and AI model. If one group's false-positive rate is much higher, you've found where human review is essential.

Common mistakes

• Testing on texts the vendor might have trained on, like famous essays or public datasets. Use your own data.
• Only testing raw AI output. Real users edit, paraphrase and mix. Include those.
• Reporting one accuracy number. Always report FPR separately.
• Re-testing never. Models change. Re-run your benchmark every few months.

Wrapping up

With two folders of labelled text and about 60 lines of Python, you can replace "99% accurate" marketing with numbers that reflect your users, your tasks and your risk tolerance. Whatever detector you choose, publish your threshold and review process internally, so decisions stay consistent and fair.

Try ProofAI's free detector or the API in your own benchmark. We'd genuinely like to hear the results.

Have you benchmarked an AI detector? What false-positive rate did you see? Share in the comments. 👇

---

Disclosure: we're the team behind ProofAI at 4Born Solutions. This article was drafted with AI assistance and reviewed and edited by our team.