Submission for the DEV Community x Kaggle Benchmarking Challenge (#kagglechallenge)
The question
Large language models are increasingly asked to reason about probability — "what's the chance this team makes the playoffs?" is a perfectly natural thing to ask one. But do they actually reason about it, or do they just echo whatever number is floating around in the sports media they were trained on?
I already run a small sports-data business built on real Monte Carlo simulations (10,000-20,000 trials per team, cross-checked live against Kalshi prediction-market prices) for MLB and NFL playoff odds. That gave me a genuine, independently-verified ground truth to grade LLMs against — instead of grading them against each other, or against vibes.
So I built a two-task Kaggle Benchmark, "Playoff Probability Calibration: LLM vs Model", comparing 7 frontier and mid-tier models against my own simulation engine across 18 real MLB and NFL cases (5 MLB + 13 NFL).
The two tasks
Task A — Probability calibration (pure reasoning). The model gets the same plain-English context a bettor would have — team, record, games remaining, season point/run differential, a short narrative — and has to respond with a single number: its estimated playoff probability. No market price, no hints. Scored against my Monte Carlo model's own probability for that same team.
Task B — Monte Carlo code generation (agentic). No opinions allowed here. The model has to write a Python program that simulates the team's remaining games and prints a probability — and we actually execute what it writes. This isolates coding/agentic ability from sports "intuition": a model can't talk its way to a good score, it has to produce working simulation code that lands near the truth.
Both tasks share the same 18-case dataset (benchmark_dataset.csv), and 5 of those MLB cases resolve for real before the benchmark's own deadline — including a live Texas Rangers vs. Houston Astros game.
Models tested
Claude Opus 4.8, Claude Haiku 4.5, GPT-5.5, GPT-5.4 mini, Gemini 3.8 Flash, Gemini 3.7 Flash, and Qwen 3 Next 80B Instruct.
Results
(Mean score across all 18 cases, higher is better, 0-100%. See the live Kaggle benchmark for the underlying per-case breakdown.)
| Model | Task A — Calibration | Task B — Code-gen |
|---|---|---|
| GPT-5.4 mini | 98.0% | 99.6% |
| Gemini 3.7 Flash | 97.3% | 99.7% |
| Gemini 3.8 Flash | 97.1% | 99.7% |
| Claude Haiku 4.5 | 95.4% | 99.6% |
| Qwen 3 Next 80B Instruct | see note below | see note below |
| GPT-5.5 | see note below | see note below |
| Claude Opus 4.8 | see note below | see note below |
A note on the missing three. Claude Opus 4.8, GPT-5.5, and Qwen 3 Next 80B Instruct could not complete either task on Kaggle's benchmarking infrastructure — every call to them failed before it was even billed, with a 403 PermissionDeniedError: "The max estimated cost of operation exceeds your available quota (based on max_output_tokens)." I confirmed this is not simple quota exhaustion (it reproduces with most of the daily allowance still free) and not concurrency contention (it reproduces running a single model completely alone). There's no exposed way — in the task code, the model-selection UI, or the notebook itself — to lower the max_output_tokens reservation these three models apparently require. I'm treating this as a hard platform-side limitation rather than a finding about the models themselves, and reporting the results here with the four models that could actually run.
What actually stood out
Every model that could run scored strikingly well — all four sit above 95% on both tasks, and the smaller/cheaper models (GPT-5.4 mini, the Gemini Flash pair) held their own right alongside the larger Claude Haiku 4.5. The bigger separation isn't between models — it's between the two tasks: Task B (write-and-execute code) scores even higher and tighter across the board than Task A (state an opinion), which suggests these models are more reliable translating "simulate this" into working code than they are at directly reasoning their way to a well-calibrated number. That's a more interesting result than "model X beats model Y" — it says something about how to prompt a model for a probability estimate at all: ask it to write the simulation, don't ask it to guess.
Live-resolving case: Texas Rangers vs. Houston Astros
One of the 18 cases in the dataset is a real, still-upcoming MLB matchup — Texas Rangers vs. Houston Astros, around September 28, 2026. I'll update this section once the game resolves, with how each model's estimate compared to both the simulation and the actual outcome.
Methodology notes
- Ground truth: two of my own Apify-published Monte Carlo actors (
commodus67/mlb-playoff-odds-monte-carlo,commodus67/nfl-playoff-odds-api-monte-carlo-simulator), each independently checked against live Kalshi prediction-market prices before being used as the benchmark's target. - Built entirely on Kaggle's Benchmarks SDK (
kbench) — two@kbench.task-decorated aggregate functions, one per task, each running the full 18-row dataset against whichever model Kaggle passes in. - Every model got the same context, the same instructions, and the same scoring function — no reasoning traces or chain-of-thought hints, no tools beyond Python code execution for Task B.
Try it yourself
The benchmark is set up to evaluate any additional model Kaggle supports — just point "Evaluate More Models" at either task. I'd be curious whether a model with more headroom on Kaggle's token-cost reservation (once I sort that out) closes the gap on the top four, or whether ~97-99% is close to a ceiling for this kind of task.
Top comments (1)
Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support