We Tested 6 AIs Against a Bookmaker. The Most Accurate One Lost.

RevealWire — Sat, 24 Oct 2026 16:03:05 +0000

NOT BETTING ADVICE

This is a paper-only forecasting analysis. No wagers were placed, no stake sizing is discussed, and this story contains no betting tips.

Reveal Wire EP17 tested six AI personas against a bookmaker using English Premier League matches from January 1 to May 24, 2026. In total, the test covered 60 matches and 348 forecasts.

Watch the episode here: https://youtu.be/vWjgbjsyWsg

The result

The bookmaker's closing odds won on Brier score, with a Brier score of 0.2129 and accuracy of 43.3%.

The most accurate persona, Hot-Take Pundit, had 45.8% accuracy — higher than the bookmaker — but finished 5th on Brier score, at 0.2341.

Forecaster	Brier score	Accuracy	Brier ranking
Bookmaker closing odds	0.2129	43.3%	1st
Hot-Take Pundit	0.2341	45.8%	5th

The average vig was 5.58%.

Accuracy is not calibration

On Brier score, lower is better. Accuracy counts how often the top pick was right. Calibration asks how well the stated probabilities matched what actually happened. That distinction is the story of this test: the most accurate persona did not have the best-calibrated probabilities, so it finished 5th on Brier despite leading on accuracy.

Why this matters for evaluating AI forecasts

A leaderboard that only reports accuracy can reward confident top picks while hiding poorly calibrated probabilities. Scoring forecasts with a proper scoring rule such as the Brier score gives a fuller picture: it penalizes probabilities that are far from the outcome, not just the wrong winner.

The headline result is therefore twofold. The bookmaker's closing odds were the best-calibrated forecasts in this test, and the highest-accuracy AI persona still lost on calibration.

NOT BETTING ADVICE — paper-only analysis only.

I built a 1,000-record synthetic benchmark for phishing detection — here's what 2 open models scored

RevealWire — Thu, 22 Oct 2026 03:46:08 +0000

I built a 1,000-record synthetic benchmark for phishing detection — here's what 2 open models scored

The problem

I run a YouTube channel where we test AI claims with real experiments. For an episode on phishing detection, I went looking for published scam/phishing-detection scores for open-weight models.

There were none. Zero. For any of them.

So we built the benchmark ourselves.

RSB-1000: what it is

1,000 records, fully synthetic (every brand, domain, name, and phone number is invented)
780 scam / 220 legitimate — the legitimate set includes deliberately tricky edge cases (urgent-but-real notifications, lookalike-but-legit senders)
Phishing bucket: 200 records = 140 classic + 60 adversary-in-the-middle (AiTM) — the AiTM subset uses AI-polished language, lookalike domains, and MFA-reauth urgency. This is the subset that breaks naive filters.
Each record: id, category, subcategory, difficulty (easy/medium/hard), text, ground_truth, plus tells on scam records and safe_action on all records
Frozen October 2026, SHA256-pinned, CC-BY-4.0

How we scored the models

Zero-shot classifier prompt, no fine-tuning, no cherry-picking: each model labeled every record as scam or legitimate, and we computed recall and false positive rate against ground truth.

Results

Model	Recall	False positive rate
Qwen 3.8-27B	98.9%	11.6%
GPT-OSS 20B	94.0%	23.1%

Both catch nearly everything. The false positive rate is the real story: at 23%, nearly 1 in 4 legitimate emails gets flagged — unusable as an autonomous blocker. The practical setup is AI as first-pass triage with human review, not a sole filter.

The AiTM subset was the hardest for both models, which tracks: AI-written lures kill the old "look for bad grammar" heuristic stone dead.

Reproduce it

The repo has the dataset, a stdlib-only eval script (eval.py — feed it a predictions JSONL, get recall/FPR/accuracy), and the full methodology:

GitHub: https://github.com/RevealWire/rsb-1000
Hugging Face: https://huggingface.co/datasets/RevealWire/rsb-1000

If you run other models against it, I'd genuinely love to see the numbers — the whole point is that nobody had published any.

Disclosure: I'm the creator of Reveal Wire, the YouTube channel this benchmark was built for. We never script the result — the evidence writes the ending.

DEV Community: RevealWire

We Tested 6 AIs Against a Bookmaker. The Most Accurate One Lost.

The result

Accuracy is not calibration

Why this matters for evaluating AI forecasts

I built a 1,000-record synthetic benchmark for phishing detection — here's what 2 open models scored

I built a 1,000-record synthetic benchmark for phishing detection — here's what 2 open models scored

The problem

RSB-1000: what it is

How we scored the models

Results

Reproduce it