This is a submission for the Kaggle Benchmarking Challenge
What I Benchmarked
Ask a person, "A bat and a ball cost $1.10, and the bat costs $1.00 more than the ball. How much is the ball?" and most will blurt out 10 cents. Ask the same person, "Is that a trick question?" and many will say yes. They can spot the trap, and they fall into it anyway.
I wanted to know whether AI models have the same gap between noticing and doing. Most benchmarks only check whether the final answer is right. Mine asks something different: when a model can clearly see a trap, does that knowledge actually protect it?
Knows It, Falls Anyway asks every problem three times, each in a separate fresh conversation so no answer can influence another:
- Detect: "Don't solve this. Is it a trick question? What's the tempting wrong answer?" A model only gets credit for spotting the trap if it names the exact tempting number, so it can't win by calling everything a trick.
- Fast: "Answer immediately with just the number," with reasoning turned off.
- Think: "Work it out step by step," with reasoning turned on.
The headline metric is the knows-but-falls rate: of the traps a model correctly identified, how many did it still get wrong?
Controls. Each of the 12 trap problems has a twin with the same math but no trap. That separates "fell for the trap" from "just sloppy at arithmetic," and it catches paranoid models: calling a plain control problem a trick counts as a false alarm.
Original problems. My first version used famous puzzles: the bat and ball, the lily pads, the widget machines. Gemini 3.7 Flash scored a perfect 100%, even on reworded versions. Famous puzzles are almost certainly in training data, and a perfect score teaches nothing. So version 2 uses harder, original problems: stacked traps (a bat-and-ball price split followed by a 50%-up, 50%-down change), multi-step traps (a clock that gains 4 minutes an hour), and classic reasoning errors in new clothing (a false-positive medical test, a tank with a leak).
Models Tested
I chose models to create contrast, since a benchmark where everyone scores 100% can't show anything:
- Gemini 3.7 Flash: a strong, fast model, and my baseline.
- GPT-5.4 nano, Gemini 3.1 Flash Lite, and Claude Haiku 4.5: the smallest models from three different labs, the most likely to answer on instinct.
- Gemma 4 26B and GPT-OSS-20B: small open-weight models.
I also ran Grok 4.20 (non-reasoning), but almost every call failed with server overload errors, so I left it out rather than report broken numbers.
Findings
| Model | Traps identified | Fast accuracy | Think accuracy | Knows-but-falls (fast) | False alarms | Control accuracy (fast) |
|---|---|---|---|---|---|---|
| Gemini 3.7 Flash | 92% | 100% | 100% | 0% | 17% | 100% |
| Gemma 4 26B | 75% | 100% | 92%* | 0% | 33% | 100% |
| Gemini 3.1 Flash Lite | 58% | 83% | 100% | 0% | 83% | 92% |
| Claude Haiku 4.5 | 58% | 58% | 100% | 29% | 50% | 75% |
| GPT-OSS-20B | 33% | 100% | 67% | 0% | 0% | 100% |
| GPT-5.4 nano | 25% | 33% | 100% | 67% | 25% | 83% |
*Gemma's one think-mode miss was a correct answer that failed JSON formatting.
1. The gap is real, but only in the smallest models. GPT-5.4 nano correctly named the trap on 3 problems and still got 2 of them wrong when answering fast. Claude Haiku 4.5 named 7 traps and fell for 2. On the false-positive medical test, nano identified the trap and then answered 50% (the correct answer is 8%). The stronger models never showed the gap at all.
2. Thinking closes the gap completely. With reasoning on, nano went from 33% to 100% on the traps, and Haiku went from 58% to 100%. These models could solve every trap; they just didn't when answering on instinct. That's the most human-like result in the whole benchmark.
3. But thinking isn't free. GPT-OSS-20B scored 100% in fast mode and only 67% when asked to think. Its think-mode mistakes weren't traps. It reached the right answer and then embellished it: 82.1 instead of 82, 29.42 instead of 29, 24.31 instead of 24, and 40 instead of 4. For some models, more reasoning adds noise to answers they already had.
4. Models don't fall for traps the way people do. The "lured" rate, meaning how often a model gave the classic human wrong answer, was nearly zero across the board. When models failed, they usually failed somewhere else. Haiku answered 99 on a problem where the tempting answer was 98 and the correct one was 82: pulled toward the trap, but not into it.
5. Spotting traps and solving them are separate skills. GPT-OSS-20B identified only a third of the traps, yet solved all of them in fast mode. Gemini 3.1 Flash Lite went the other way: it called 83% of the plain control problems tricks. A model that sees traps everywhere isn't necessarily a good trap detector.
6. The hardest traps were the least famous ones. Half of the models failed the leaky tank and the fast-running clock in fast mode. Every model solved the four traps with famous structures (the square fence, the two algae blooms, the return-speed problem, and the counting-multiples problem). Familiar trap shapes seem to be learned; unfamiliar ones still bite.
What surprised me: I expected detection and solving to go together. They didn't. Several models solved traps they couldn't name, and a few named traps they couldn't solve. Whatever "knowing" a trap means for a model, it isn't the same process that produces the answer.
Limitations: Each model saw 12 traps, so the knows-but-falls rates rest on small counts (nano's 67% is 2 out of 3). Turning reasoning "off" means the API accepted the setting, which doesn't prove no internal reasoning happened. Some models only partly accepted the "high" reasoning setting, so think mode relied mostly on the step-by-step prompt.
What I'd measure next:
- More traps per family, to turn these patterns into solid rates.
- Hinting: if I tell a model "this might be a trick" right before it solves, does the gap vanish? That would show detection and solving are separate processes that don't talk to each other.
- Beyond math: the same noticing-vs-doing gap probably shows up in code review, instruction following, and safety behavior.
My Benchmark
Knows It, Falls Anyway on Kaggle
Every problem, correct answer and tempting wrong answer is in the task code, so you can run it on any model or add your own traps.
Top comments (0)