I built a 1,000-record synthetic benchmark for phishing detection — here's what 2 open models scored

RevealWire — Thu, 22 Oct 2026 03:46:08 +0000

I built a 1,000-record synthetic benchmark for phishing detection — here's what 2 open models scored

The problem

I run a YouTube channel where we test AI claims with real experiments. For an episode on phishing detection, I went looking for published scam/phishing-detection scores for open-weight models.

There were none. Zero. For any of them.

So we built the benchmark ourselves.

RSB-1000: what it is

1,000 records, fully synthetic (every brand, domain, name, and phone number is invented)
780 scam / 220 legitimate — the legitimate set includes deliberately tricky edge cases (urgent-but-real notifications, lookalike-but-legit senders)
Phishing bucket: 200 records = 140 classic + 60 adversary-in-the-middle (AiTM) — the AiTM subset uses AI-polished language, lookalike domains, and MFA-reauth urgency. This is the subset that breaks naive filters.
Each record: id, category, subcategory, difficulty (easy/medium/hard), text, ground_truth, plus tells on scam records and safe_action on all records
Frozen October 2026, SHA256-pinned, CC-BY-4.0

How we scored the models

Zero-shot classifier prompt, no fine-tuning, no cherry-picking: each model labeled every record as scam or legitimate, and we computed recall and false positive rate against ground truth.

Results

Model	Recall	False positive rate
Qwen 3.8-27B	98.9%	11.6%
GPT-OSS 20B	94.0%	23.1%

Both catch nearly everything. The false positive rate is the real story: at 23%, nearly 1 in 4 legitimate emails gets flagged — unusable as an autonomous blocker. The practical setup is AI as first-pass triage with human review, not a sole filter.

The AiTM subset was the hardest for both models, which tracks: AI-written lures kill the old "look for bad grammar" heuristic stone dead.

Reproduce it

The repo has the dataset, a stdlib-only eval script (eval.py — feed it a predictions JSONL, get recall/FPR/accuracy), and the full methodology:

GitHub: https://github.com/RevealWire/rsb-1000
Hugging Face: https://huggingface.co/datasets/RevealWire/rsb-1000

If you run other models against it, I'd genuinely love to see the numbers — the whole point is that nobody had published any.

Disclosure: I'm the creator of Reveal Wire, the YouTube channel this benchmark was built for. We never script the result — the evidence writes the ending.

DEV Community: RevealWire

I built a 1,000-record synthetic benchmark for phishing detection — here's what 2 open models scored

I built a 1,000-record synthetic benchmark for phishing detection — here's what 2 open models scored

The problem

RSB-1000: what it is

How we scored the models

Results

Reproduce it