Can an AI catch the catch? I benchmarked 14 models on bounty fine print

Hoa Nguyen — Tue, 06 Oct 2026 10:38:43 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

I'm a solo developer in Vietnam. For the last few weeks I've been hunting paid work online: bug bounties, hackathons, open-source bounties, community contests. The hardest part isn't the work. It's reading the listing.

The headline says "free", "open", "global", "deadline Oct 11". Then the fine print says:

the free $5 credit only arrives after you add a card
"no interviews, just ship" ... except section 9 says winners do a live verification video call
the issue is open ... but it's already assigned to someone
the deadline is Oct 11 PDT, which is Oct 12 afternoon where I live
a sponsor comment quietly moved the deadline a week earlier

I started asking AI assistants to read listings for me, so I wanted to know: can a model read a listing and catch the catch?

Bounty Triage is 40 short listings, each with one question and one correct answer from a fixed set. Every case is fictional (names and wording changed), but most are modelled on real listings I checked while hunting. Four categories:

Category	Question	Answers
Hidden gate	Besides submitting, what extra requirement is there?	none / live_interview / in_person / card_required / hired_first / own_cloud_account
Real deadline	When do submissions actually close, in my timezone (UTC+7)?	multiple choice
Still open?	Can I take this and expect to be the one paid?	open / claimed / contested / closed
Am I eligible?	Individual, 25, Vietnam, not a student	eligible / not_eligible / unclear

Design choices:

Controls. About half the cases have no trap. A model that shouts "trap!" at everything loses points. Some cases are reverse traps: the scary-sounding thing is optional (office hours on video, an optional city meetup, an optional card for a bonus credit).
Exact grading, no LLM judge. The model answers through a structured schema; I compare the answer to the key.
Two tiers. 25 core cases, then 15 hard ones I added after three models scored 100% on the core set.

Models Tested

14 models on Kaggle Benchmarks, from Anthropic, Google, OpenAI, DeepSeek, Alibaba (Qwen) and Zhipu (GLM), from frontier models down to the cheapest tier. I picked them in pairs on purpose: a vendor's big model next to its cheap one (Opus vs Haiku, GPT-6.1 Sol vs GPT-5.4 nano, Gemini Flash vs Flash-Lite), plus open-weights models I could run myself. The question I care about is not "which model is best" but "which model is good enough to read listings for me". Scores are the share of the 40 cases answered correctly (one run each).

Model	Overall	Core (25)	Hard (15)	Hidden gate	Deadline	Still open?	Eligible?
Claude Opus 5.5	1.00	1.00	1.00	1.00	1.00	1.00	1.00
GPT-6.1 Sol	1.00	1.00	1.00	1.00	1.00	1.00	1.00
Gemini 3.8 Flash	1.00	1.00	1.00	1.00	1.00	1.00	1.00
Gemini 3.7 Flash	1.00	1.00	1.00	1.00	1.00	1.00	1.00
Gemini 3 Flash	1.00	1.00	1.00	1.00	1.00	1.00	1.00
Gemma 4 31B (open weights)	1.00	1.00	1.00	1.00	1.00	1.00	1.00
Claude Sonnet 5	0.975	0.96	1.00	1.00	1.00	0.90	1.00
GLM-5	0.975	1.00	0.93	0.93	1.00	1.00	1.00
gpt-oss-20b (open weights)	0.975	1.00	0.93	1.00	1.00	0.90	1.00
Gemini 3.1 Flash-Lite	0.925	0.96	0.87	0.93	0.89	0.90	1.00
Qwen 3 Next 80B	0.875	0.92	0.80	0.93	0.78	0.80	1.00
Claude Haiku 4.5	0.85	0.88	0.80	0.93	0.78	0.70	1.00
GPT-5.4 nano	0.80	0.84	0.73	0.86	0.78	0.70	0.86
DeepSeek R1	0.775*	0.80	0.73	0.79	0.56	0.80	1.00

* DeepSeek R1 answered 31 of the 32 cases it formatted correctly, but 8 of its 40 replies were not valid JSON, so they count as wrong. gpt-oss-120b timed out on Kaggle's model proxy twice and is left out.

Findings

1. This benchmark does not separate the frontier. It separates the cheap tier. Six models scored 40/40. 22 of the 25 wrong answers (not counting DeepSeek's format failures, below) came from four models you would pick because they are cheap and fast: Haiku, nano, Flash-Lite and Qwen 3 Next. That is exactly the tier I would run over hundreds of listings a week, so that is where it matters. The surprise: Gemma 4 31B, an open-weights model, scored 40/40, and gpt-oss-20b missed one. Kaggle's score-vs-cost chart puts Gemma 4 31B on the efficient frontier: 40/40 for about one cent for the whole run, while the other perfect scores cost roughly 3 to 10 cents. If I want a cheap or local model to pre-filter listings, that is the one I'd try first.

2. "Free" is the most effective trap. The hardest case (H1, missed by 6 of 14) is a contest whose rule says your entry must run on one specific cloud's free tier, right under the line "No card needed for the free tier". Five models answered none: the word free closed the question for them. Qwen went the other way and answered card_required, falling for the optional card mentioned on the next line. Both mistakes come from reading the loudest line instead of the rule.

3. "Who already holds this?" is hard. The next hardest case (H10, missed by 5 of 14): a bounty issue that is open with no assignee, but a linked pull request is approved and the maintainer wrote "merging after the release freeze". The right answer is claimed, because someone is about to be paid. Models answered contested, open and even closed. A plainer version (open issue, but assigned to someone) still fooled 3 models into saying open. Status labels win over evidence for small models.

4. Timezones fail halfway. "October 11, 11:59 PM PDT" in Vietnam time is October 12, 13:59. Haiku and Flash-Lite picked 12 Oct 06:59, which is the UTC time: they converted once and stopped. The AoE deadline ("Anywhere on Earth", UTC-12) tripped 3 models. For anyone outside the US, this is the bug that makes you miss a deadline by a day.

5. One run is not a verdict. I ran the same cases on some models twice (locally and on Kaggle). Gemini 3 Flash missed H1 in one run and got it in the other; Claude Sonnet 5 missed C1 on Kaggle and not locally. At the top, 0.975 vs 1.00 is noise. The gap between 1.00 and 0.80 is not.

6. Format failures are real failures. DeepSeek R1 was nearly perfect when it answered in the requested format (31 of 32), and still came last overall, because 8 of 40 replies came back as reasoning text instead of the JSON I asked for. In an agent that reads listings for me, an answer I can't parse is the same as a wrong one.

7. A strong model found bugs in my answer key. On my first local run, one model disagreed with two of my keys, and it was right: my listing text was ambiguous. I rewrote both cases. Lesson: run a benchmark on a strong model before trusting it to judge the weak ones, and read every disagreement.

What I'd measure next: listings with three or four catches at once (a timezone and a track-specific extension and a claim that expired), real page noise (navigation, comment threads, emoji), and repeated runs per model so the top of the table gets error bars instead of ties.

What I'd tell another bounty hunter: a frontier model (or Gemma 4 31B locally) reads fine print about as well as I do on these 40 cases. Cheap models do not: on the hard tier the cheap models got 2 to 4 of 15 wrong, and their mistakes lean toward "it's open, it's free, go". Use the cheap ones to sort, never to say yes.

My Benchmark

Kaggle benchmark: https://www.kaggle.com/benchmarks/shenjun93/bounty-triage
Task: https://www.kaggle.com/benchmarks/tasks/shenjun93/bounty-triage

Example case (hidden gate, hard tier):

No interviews. No pitch. Just ship. Shipfast Weekend, $10,000 in prizes. Build anything in 48 hours and submit a repo and a 2-minute video. Judges score online. ...
Terms, section 9 (Payout): before any prize is released, each winner must complete a 20-minute live verification video call with our team, camera on.

Q: Besides submitting your work, which ONE extra requirement applies to receiving the prize money?

Expected: live_interview

I built the cases and the task code with help from an AI coding assistant; the listings, the traps and the judgment about what counts as a catch come from my own hunting.

DEV Community: Hoa Nguyen

Can an AI catch the catch? I benchmarked 14 models on bounty fine print

What I Benchmarked

Models Tested

Findings

My Benchmark