DEV Community: Ratnesh Saxena

I Built a Benchmark That Catches AI Models Cheating (And They All Failed)

Ratnesh Saxena — Sun, 11 Oct 2026 00:01:41 +0000

I Built a Benchmark That Catches AI Models Cheating (And They All Failed)

What task did you run?

I built TwinBench, a benchmark that tests whether AI agents actually follow rules or just pattern-match their way to plausible-looking answers.

Here's the trick: every test item comes as a twin pair. Two stimuli that are identical except for one single controlling field. A model only gets credit if it answers both twins correctly. If it's just pattern-matching surface features, the twins will trip it up because the surface looks the same but the correct answer flips.

The benchmark covers three areas:

Counterfactual Rulebooks (10 pairs): Config precedence, merge-queue ordering, timezone-aware batching, money rounding, journal replay, version ordering, authority resolution, retry gating. The kind of stuff that breaks production systems when agents get it wrong.
Global Consistency Audits (10 pairs): Deployment provenance, event replay, snapshot lineage, change-freeze scoring, release shippability. Cross-referencing multiple sources of truth and finding contradictions.
Temporal Reconciliation (6 pairs): Quota replays, FX cutovers, VOID cascades, DST payroll calculations, schema migration corrections. Time is hard, and agents need to handle it precisely.

Each pair uses a strict exact-match scorer. No partial credit. No "close enough." You either followed the rules correctly on both twins or you didn't.

Which models did you run it against?

Three frontier models, five runs each per pair:

Opus 5.5 Beta
Grok 4.6
GPT-6.1 Sol

I picked these three because they're the current top-tier reasoning models, and if any of them can handle precise rule-following under counterfactual pressure, it should be them.

The filtering protocol: a pair is retained (kept as a real benchmark item) only if at least one model fails at least 2 out of 5 runs. A run only passes if both twins get full credit. This ensures every item in the final benchmark is genuinely hard, not just hard for one model on a bad day.

What are the main insights?

Four pairs survived. All of them are brutal.

Pair	Mechanism	Opus	Grok	Sol
CR-N3	Timezone batching (DST vs fixed offset)	4/5 fail	4/5 fail	4/5 fail
CR-N5	Journal replay with ABORT semantics	5/5 fail	5/5 fail	5/5 fail
CR-N6	Version ordering (rc > release)	4/5 fail	5/5 fail	4/5 fail
CR-N8	Retry gating on idempotency registration	5/5 fail	5/5 fail	5/5 fail

CR-N5 and CR-N8: every model failed every single run. That's 30 consecutive failures across three different frontier models. These aren't edge cases, they're systematic blind spots.

What surprised me most:

The models are confident and wrong. They don't hedge or express uncertainty. They produce well-formatted, detailed answers that are just... incorrect. The formatting is perfect, the reasoning sounds plausible, but they missed the actual rule.
Twin pairs expose the cheating. On several items, a model would get twin A right and twin B wrong (or vice versa). If you're actually following the rule, the flip shouldn't matter. The fact that it does proves the model is pattern-matching, not reasoning.
Shortcut baselines score zero. I built naive heuristic solvers (ignore timezone conversion, skip ABORT logic, always say "eligible"). They got 0/10 on every retained pair. This confirms the items can't be gamed with surface tricks, you actually need to do the work.

What I'd measure next:

More models: I'd love to test Gemini, Claude Sonnet, and open-weight models to see if the failure patterns are universal or architecture-specific.
Chain-of-thought analysis: Where exactly does the reasoning break down? Is it misreading the rule, misapplying it, or losing track across the twin?
The dropped pairs: Six CR pairs were too easy (models got 4-5/5 right). Are they easy because the mechanism is simple, or because the distractors weren't strong enough?

Where can we see it?

Kaggle Writeup (full paper): https://www.kaggle.com/competitions/gemma-4-developer-agent-paper/writeups/twinbench-a-twin-pair-benchmark-for-agentic-rule
GitHub repo (code): https://github.com/rsaxena3011/twinbench (private during competition judging, will open after)

The benchmark harness, all 26 pair implementations, the filtering runner, shortcut baselines, and negative controls are all in the repo.

Built for the Kaggle Gemma 4 Developer Agent Paper Track. The twin-pair methodology is the core contribution: if your benchmark doesn't have counterfactual twins, you're measuring pattern matching, not reasoning.

I set filmmaker traps for AI "directors." The models fell for one. My rubric fell for three.

Ratnesh Saxena — Fri, 09 Oct 2026 22:06:55 +0000

I set filmmaker traps for AI "directors." The models fell for one. My rubric fell for three.

I QC frames for photorealism and anatomy before anything becomes video. I regenerate bad frames; I do not argue with them. I also direct: corporate events, mythology films, horror shorts, stock shoots.

If AI models are going to help on a real set, "usually right" is not good enough. The bar is three things: never invent a defect, show your constraints, and write prompts as if a human crew will read them. The benchmarks I have read do not measure those jobs. So I built DirectorBench: adversarial tasks for AI models doing a video director's work.

The setup

Three tasks:

PromptCraft: write a generation-ready prompt.
ShotList: plan shots from a brief with hard constraints.
QC Judge: accept or reject a frame. Some frames are decoys: correct on purpose, and must be passed.

This is a lean pilot: 24 adversarial items, two models (Anthropic Opus 5.5, Grok 4.6), both at Medium effort via Notion AI. Scoring is deterministic: regex checks plus word-overlap checks, no human judgment. Runs happened in Notion AI; the Kaggle benchmark tasks were stuck in the build queue.

Task	Opus 5.5	Grok 4.6
PromptCraft (6)	1.000	0.931
ShotList (6)	0.810	0.780
QC Judge (12)	1.000	0.917

Opus 5.5 scored higher on all three. Do not over-read that. On six-item tasks a single item is a large swing, and as you are about to see, two of the three tasks were partly measuring my keyword scorer. The useful question is not which model scored higher. It is what each lost point actually means.

What I will not claim

24 items, not the planned 70. Two models, not six. Medium effort, not max. Notion AI, not a clean API harness. One run per item, no variance data.

Two of the three findings below are as much about my rubric as about the models. I would rather say that here than have someone discover it in the comments.

Finding 1: the paranoid judge (the trap worked)

The QC task judges written descriptions of frames, not pixels. That is a real limitation; the full run needs actual images. Within that limit, the design is: 6 decoy descriptions that are correct by design and must be passed, plus 6 with genuine defects.

Opus passed all 6 decoys with the exact right codes. Zero false alarms, zero misses.

Grok caught all 6 real defects, then rejected a decoy: "the discus spins upright, balanced on the tip of his raised index finger." It marked it FAIL [PHYS].

That discus is the Sudarshana Chakra. The scene is iconography, and the instructions explicitly exempt iconography from physics review. I cannot see inside the model, so I will not claim to know why. What I can observe is that it applied a physics objection where the written instructions said not to.

One false alarm in six decoys is a single data point, not a personality. But it is exactly the failure the decoys exist to catch, and the one that matters most in my workflow. A rejection sends a usable frame back into regeneration. A judge that invents defects burns trust: once it flags something that is not wrong, I have to recheck its judgments myself.

	0 false alarms	1+ false alarms
0 misses	Opus 5.5: useful	Grok 4.6: paranoid
1+ misses	lenient	useless

This is the only finding in the pilot that is about a director's job rather than about my scorer.

Findings 2 and 3: where my rubric got trapped

ShotList is where both models scored lowest, and most of the damage came from my own checks. Each task score is the mean of six items; each item is the mean of six sub-checks. Here is where the points actually went.

Naming the exclusion gets punished. Both models lost about 14 points of ShotList score, roughly two-thirds of everything they lost, on a single sub-check: f_clean. Five of six items per model tripped it:

Opus: "no visible logo or markings" flagged for the word "logo"
Opus: "plain dial with no numbers" flagged for "dial," the brief's own word
Grok: "no signage, no readable marks" flagged for "signage"

Grok slipped past it once: "blank cream pages, no writing, no print." That dodged the trigger word; it is not evidence of a better shot list.

The models did the right directing thing: state the constraint so the crew can see it. The checker reads the mention as the violation. That is a rubric edge, not a model failure. If your eval rewards silence over clarity, you will select for models that hide their constraints.

Natural phrasing gets punished. In PromptCraft, Grok wrote "moonlight" and "moonlit" instead of "moon," and "candlelight" instead of "candle." The noun scorer wanted a whole-word match on "moon"; "moonlight" does not count. Opus wrote the bare nouns and went 6 for 6.

That does not prove Opus wrote better prompts. It proves this slice of the score measured lexical form, not prompt quality. When a benchmark tells you a model writes bad prompts, check whether it is grading craft or grep patterns.

Paraphrase scores zero. The b04 ShotList item carried a brand promise: the call is heard but never printed. Both models landed the intent in my reading. Opus wrote "coach calls out in Hindi, stopwatch face turned away." Grok wrote "Split is heard, not seen." The word-overlap sub-check scored both 0.0; it cannot see paraphrase. Those zeros sit inside the ShotList averages in the table above.

What's next

The full run is 70 items and six models, with two fixes this pilot earned: compliance scored by meaning instead of keyword presence, and QC judged on actual frames instead of descriptions. The test of DirectorBench is whether it can tell a model that breaks a rule from one that explains it, and a model that invents a defect from one that knows when the rules do not apply.

A benchmark should distinguish a bad answer from a good answer expressed differently. This pilot showed me where mine does not.

Items, answer keys, scorers: https://github.com/rsaxena3011/directorbench

Sthala: an offline audio walk that only speaks when you're standing in the story

Ratnesh Saxena — Thu, 08 Oct 2026 19:33:06 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass.

Sthala: an offline audio walk that only speaks when you're standing in the story

Walk within 35 metres of Banganga Tank and your pocket starts telling you its legend. Walk away and it stops, mid-sentence if it has to.

I make mythology videos for a living. The audio walks I tried were either generic tours read by a bored narrator or cloud apps that bill Hindi by the character. So I built Sthala (स्थल, "place"): an open-source audio storyteller that turns OpenStreetMap and Wikipedia into short Hindi or English stories, each locked to the spot it describes. Your phone stays in your pocket. The city does the talking.

Why getting it right matters

A garbled legend playing at a temple isn't a bug. It's an insult to the people who pray there. That idea drives every choice below: stories are written ahead of time, checked line by line against their sources, read by a human, then frozen. The script you hear this year is the script you hear next year. No silent model update gets to rewrite a legend.

What it feels like

Tap once. Pocket the phone. Screens off, eyes up.

A ghungroo chime speeds up as you near the next stop. Cross the 35-metre line and the story begins. Stay, and it goes deeper: history, then legend, then the detail everyone walks past. Leave, and it stops. Outside a zone, silence.

Every story ends with a small task: look at this, count that, notice something. That's my take on Touch Grass. The app's job is to make you pay attention to the place, not the app.

Demo

A 76-second run on a simulated GPS trail, driven by the same walk.py the phone runs: approach chime, zone entry, layers deepening, walk-away cutoff. This tests the mechanic, not real-world GPS reliability. A live walk is still pending. The video is in the repo.

Code

Repo: https://github.com/rsaxena3011/sthala

The Mumbai pack ships with it: all 18 MP3s, scripts, source text, and the full skeptic log. You can see exactly what the model wrote and what got cut.

How it works

The heavy lifting happens on my PC before anyone walks. The phone only does GPS and playback, so it runs offline with no API keys and no per-character fees.

harvest.py pulls places from OpenStreetMap and article text from Wikipedia. write.py drafts layered stories with Gemma 3 4B, running locally via Ollama, then runs the skeptic pass. voice.py voices them with Meta MMS-TTS, local, open weights. walk.py runs on Android through Termux and termux-api.

Mumbai is three stops, three layers, two languages: 18 tracks. Sewri Fort (its history, the flamingo season off Sewri jetty), Banganga Tank (its legend and rituals), and Walkeshwar Temple (its story and architecture).

The skeptic pass

This is the part I'm proudest of. For Mumbai, Gemma drafted 169 lines. A second pass, holding only the source article, had to defend every line against that text. It deleted 18. Each deletion is logged with its reason in skeptic_log.json.

Cut: "The fort's construction followed relentless attacks by the Siddis, African sailors allied with the Mughals, who viewed the British as enemies. [History]"

Gemma tagged that [History] with total confidence. The source never says it. That doesn't make it false, but a confident tag isn't evidence, so it went.

Survived: "This ancient Hindu temple, dedicated to Lord Shiva, was originally built in 1127 AD by Lakshman Prabhu, a minister of the Silhara dynasty. [History]"

The Wikipedia text supports it, so it stayed. The skeptic checks that a line matches its source, not that the source is right.

What it can't do yet

The skeptic is the same Gemma model checking its own draft. It catches confident inventions. It is not an independent fact-checker.

So the real last gate is me. I read every line the skeptic keeps before a pack ships. That works for three stops. It won't work for a whole city without more reviewers.

And the demo uses simulated GPS. It proves the mechanic, not real-world GPS reliability. A live walk is next.

Why open innovation matters here

For this project, open isn't a preference. It's what makes the project possible.

Stories that can't drift. Versioned packs play saved audio, so a story sounds the same on every visit. With a hosted model, someone else decides when your legend changes.

Work you can audit. Every draft, deletion and reason is public. You can't publish a black box's judgement. You can publish a log.

Hindi without a meter running. With cloud voices, a library of Hindi walks becomes a monthly bill. With open models, it's an overnight job on a home PC.

Places I'll never reach. Anyone can fork this and build a walk for their own temple, market or lane, in their own language, without asking me or paying anyone.

Make a pack for your street

Try the demo:

git clone https://github.com/rsaxena3011/sthala
cd sthala
python walk.py --demo

Building a pack is a heavier job: harvest, write, voice, all local, roughly overnight on a PC. Two ways to help this Hacktoberfest:

Open an issue proposing three stops near you and their sources.
Send a correction to the Mumbai pack. If a line is wrong, I want to know.

PRs welcome under packs/.

Then go outside and let the street talk.