We're excited to team up with Kaggle for a brand new challenge!
Running through October 11, the Kaggle Benchmarking Challenge asks you to build a ...
For further actions, you may consider blocking this person and/or reporting abuse
My benchmark is gonna be interesting (assuming if this goes to plan). Looking forward to see what others comes up!!
That part about benchmarks coming from specific itches really resonates. I've tried building general evaluations before and they never feel as grounded as when you're testing something you've actually struggled with.
Agreed on the re-selection point, and it bites harder than it looks: if the model picks its features inside the null, the placebo inherits the same leak and stops being a placebo. The variant I keep hitting is label-free. Subject identity sits on the top-variance direction of a frozen representation, so nothing in the test split removes it; a subject-disjoint split still leaves it there. Lin, Wu, Jung (arXiv 2606.06647) measure exactly that on EEG foundation models, 12/12 cell-model pairs, and fine-tuning amplifies it. Group-level permutation at the level the leak actually enters is the control that catches it. Concrete recipe, thanks.
Marked the dates. Proper benchmarking is honestly such an underrated discipline in modern ML—everyone wants to train heavy models, but nobody wants to spend time building clean evaluation layers. Looking forward to seeing the constraints and what telemetry metrics the community prioritizes for this challenge.
November is coming soon, friends, I wish you good luck！
I'm in.
New to Kaggle, let's try this one.
It sounds great. I can't help but want to join in now.
I will be da winners 🏆
I'm in
interesting
okay I can
Hi~everyone, i'm here
This is the kind of tooling I’d actually use. Less interested in how confidently the agent explains its work, more interested in whether I can check the result without retracing every step.
This is a submission for the Kaggle Benchmarking Challenge
Can LLMs Actually "Think" Code, or Just Fix Syntax? A Logic Bug Benchmark
Many LLMs are great at fixing missing semicolons, but how do they handle silent, destructive logical flaws? I built a targeted benchmark to find out.
🔍 What I Benchmarked
I measured Multi-step Reasoning and Code Generation/Debugging performance. Specifically, I set out to measure a model's ability to detect and fix silent logical bugs (e.g., off-by-one errors, incorrect loop boundaries, and race conditions) versus syntax errors in Python and Go.
This behavior interested me because syntax errors are caught by compilers, but logical bugs make it to production. I wanted to see if models truly understand code execution flow or just rely on pattern matching.
🤖 Models Tested
I benchmarked three models using zero-shot prompting:
📊 Findings & Real-World Meaning
The results taught me something real and surprising about current LLM limitations:
What this means practically: These results changed my view. We cannot trust LLMs to review complex backend logic autonomously yet; they are excellent syntax assistants but mediocre logical auditors.
Next Up: I plan to measure how chain-of-thought (CoT) prompting alters these scores.
🔗 My Benchmark
You can view my full dataset, prompts, and evaluation pipeline here:
👉 My Kaggle Benchmark Notebook & Dataset (Replace this with your actual Kaggle link)
I'm preparing a benchmark using kaggle-benchmarks and native tools. My verified Kaggle account receives ‘User location CN is not supported for this model/API’, while the same synthetic task runs successfully through a provider API available to me. May I use these external-model runs for this challenge, and how should I attach their accurate model IDs and results to a public Kaggle Benchmark? Would a public benchmark linking the reproducible task and external results qualify?
Update after testing the official CLI: authentication works and my private task now shows Completed. Using the SDK’s cache, I reused unchanged genuine Task/Run outputs from an external API run, preserving the actual model ID, original score and trajectories, with no new inference. However, the task page shows Results (0), and CLI status says ‘No runs yet’. I haven’t published an empty benchmark or relabelled the model. Is there a supported way to register these genuine external runs in a public Kaggle Benchmark, or must eligible scores come from the platform’s model scheduler? Would a public benchmark linking the reproducible task and clearly labelled external results qualify?
Thanks for the detailed testing and for keeping model IDs and results accurate. For this challenge, your benchmark must be built and run on Kaggle: eligible results need to come from runs on the Kaggle platform. External API runs, including cached outputs, can’t be registered as the benchmark’s results, and a public benchmark linking only to external results wouldn’t meet the requirement. For the “User location CN is not supported” error, please contact Kaggle support, as we can’t resolve platform access on our side.
The Official Deadline is 11:59pm PDT. But it showing ended at 07:00am IST even though it should be ended at 12:30pm IST. So even if i submitted it will be valid right ?. Because i had my benchmark results ready but i thought the deadline is later i will post it.
Sorry about the confusion! This was an error on our end. Your submission is still valid as long as it was submitted before 11:59pm PDT 👍
Hi, I have a question about the entry eligibility requirement stating that development of an entry must have started during the Entry Period.
I had previously developed a code-boundary evaluation task as part of a broader security and metacognition project. More recently, I've been integrating tasks into a Kaggle benchmark and running comparative evaluations across multiple models.
Would a submission focused on this newer benchmark integration and evaluation work qualify as an eligible entry, given that the underlying task and broader project predate the Entry Period?
I want to ensure that the submission complies with the rules rather than misrepresenting the project's development history. Thanks for clarifying.
Hey there! Thanks for checking. The Official Rules require that development of your entry started during the Entry Period (from September 23, 2026), and the challenge FAQ allows building on previous work as long as your changes are significant and any non-trivial prior work is clearly credited in your post.
If you submit, please describe in your post what existed before September 23 (the code-boundary task and the broader project) and what you built during the challenge (the Kaggle benchmark integration and the comparative evaluations). Eligibility is assessed at the Sponsor’s discretion under the Official Rules.
I was about to submit, the deadline was move up, by my time it should be 5 hours from now. but the challenge page says ended.
Same here
Sorry about any confusion this caused. This was an error on our end. Your projects will still be valid if you submitted before 11:59pm PDT.
@jess
I went and computed the estimator behind one of your cells, because the
[0, 0]case has a sharper edge than your paragraph gives it. No refit of your data -- this is a simulation and an exact enumeration of the percentile bootstrap for a difference in proportions and a difference in medians, at cell shapes of the kind you describe.1. The tie fraction, and its closed form. Call
tthe probability that one resample reproduces the observed difference exactly. For a cell with one event in one arm and none in the other,t = (1 - 1/n)^n, so it rises to 1/e = 0.3679 and stays there for any n: simulation gives 0.3632 at n=40, 0.3660 at n=100, 0.3674 at n=400, 0.3677 at n=1000. The number of observations stops mattering once the cell has one event in it.2.
[0, 0]is not printed by that. It requires the statistic to be constant over every resample, since the quoted 2.5th and 97.5th percentiles must both land on 0. For a difference in proportions that happens exactly at the boundary: both arms at zero events, or both at all events. I enumerated every cell with n1, n2 up to 12 -- 288 of them have tie probability exactly 1.0, and the largest tie probability over all the non-boundary cells in that range is 0.375. Simulation agrees at n=40 and n=1000 for both 0/n versus 0/n and n/n versus n/n: 1.0000.So the
[0, 0]cells in a zero-heavy table are the empty-side cells, and the phenomenon is a point mass rather than a nearly constant statistic. It is also invariant in n: 0/5 vs 0/5, 0/40 vs 0/40 and 0/1000 vs 0/1000 print the identical string, so thendisplayed beside such a row never enters the interval. The label that carries the information is not[0, 0]butno interval: the resampling distribution is a point mass (0 events in both arms)-- which is a statement about the comparison having an empty side, exactly theexclusion rulefield of your contract, not a statement about precision.3. The same zero-heaviness does not collapse the median column, which supports your rule that the interval inherits the metric beneath it. 50 observations per arm with 30 versus 28 zeros: the difference in medians ties at the observed 0 in 69.4% of resamples and its interval is still
[-1.5, +1.0], because the difference in medians has support below the observed value, while the difference in proportions with an empty arm is bounded below by 0. The amount of zeros is not what decides the collapse; the support of the statistic is.4. A smaller version of the same read is worth flagging in the non-collapsed cells. In the 1/40 versus 0/40 cell the printed interval is
[0.000, +0.075]under every seed I ran, and the 0 is not a quantile of the sampling distribution -- it is the support boundary plus the 36.3% tie mass. So a one-sided-looking interval like[0, +x]does not mean the effect was found to be non-negative; it is what the percentile method prints whenever one arm is empty, and the left end is the design, not the data.5. The three-line check, in your units. Count the resamples whose statistic equals the observed one and print that fraction beside the interval: below the quoted percentile it is a normal cell; above it, the interval is being read off the tie mass and the honest output is the tie fraction itself. That one number separates your
[0, 0]cells (tie fraction 1.0, any n) from the sparse cells that merely look like them (tie fraction 1/e), and it is cheap enough to sit in the same loop that builds the interval.This is a yummy challenge! 😋
Would love to try this one. 🥰
green ticks close the UX. settlement week still needs a tip anyone can GET after the portal flips.
which public query proves the cut without a second shadow ledger?
curious what breaks first: MDR opacity or hop finality.
marker1001h1728-dt
For the Kaggle Benchmarking Challenge that started on 23 September 2026, can an eligible Australian-resident individual receive the USD 500 cash prize into an Australian personal bank account? Please confirm the payment processor and bank/currency route, the foreign-individual tax form and any withholding, recipient/intermediary/FX fees, and expected delivery time after claim documents are accepted. No personal bank or tax details are needed for this general question.
Yes, Australian residents can receive payments to their banks. We process payments through Ramp which requires your bank IBAN account number and SWIFT codes to process. Payments arrive within 90 days.
Thank you, Jem—this is helpful. Since Australian banks do not use IBANs, could you confirm whether the Ramp Australia form has a separate BSB field, and whether its IBAN-labelled field expects the account number alone or BSB plus account number? Will winners receive USD or converted AUD, and are any sender, intermediary or FX fees deducted from the $500? Finally, which foreign-individual tax form and withholding apply? No personal banking details are needed here. Thanks for helping clarify the route.
I have already submitted a post to the challenge. That was my solo challenge. I have also another idea, I would like to submit to the challenge. Could/Should I team up with others, so technically it would be another participant submission?
Thanks for running this. The "specific itch" line is exactly how mine started: I'm building a Telegram bot my parents forward suspicious messages to, and I needed to know which model to trust with it. Public leaderboards couldn't tell me whether a model knows that Kaspi Bank never texts from a mobile number.
So I built KZ Scam Shield: 72 scenarios in Russian, Kazakh, and the Russian-Kazakh mix most people here actually text in, five tasks, and 14 models. The finding I didn't expect: every model spots an OLX prepayment scam when asked, "Is this a scam?" and all 42 replies still wrote, "I'll transfer 50% today," when asked to help reply to the seller.
dev.to/olegvdv/every-llm-i-tested-...
One tip for anyone finishing a run before the 11th: Kaggle reserves the maximum possible cost of each request against your daily quota, not the actual cost. Expensive models with a large max_tokens get refused before they ever answer. For GPT-6 Astra, that's $6.40 per request against a $10/day quota. Set max_tokens explicitly and run expensive models with low concurrency.
This is our submission for the Kaggle Benchmarking Challenge
A 0-Parameter Heuristic Tied Two Frontier Models at 84.4% and Beat 11 Others. Their Mistakes Were Completely Different.
We evaluated 45 frontier models across all major labs on Kaggle Cloud and discovered three fundamental pathologies hiding behind standard aggregate leaderboards:
Platform Artifacts:
What this challenge taught me: the benchmark is the easy half. I built Cost Per Correct — what does each correct answer cost, across models — and the write-up was finished on Oct 11. The benchmark wasn't. Since then, readers have improved it three times in public: one showed that a wrong answer is never zero-cost (price failure, and the cheapest model flips to the most expensive), one added the cost of escalation itself, and a CIO reframed the whole thing as a risk gate first and a cost comparison second. If you're judging, the discussion thread on an entry is worth reading alongside the entry — it's where the idea kept getting better. My entry: dev.to/karmendra_pandey_43ac6983/y...
This benchmark challenge is a fascinating way to test how models handle custom logic and rule-based tasks! Speaking of custom interactive logic, I recently built a lightweight, client-side tarot combination calculator to test how deterministic rule structures behave under simple web architectures without heavy APIs. Excited to see what everyone builds for this challenge!
تعادل نموذج استدلالي بدون معلمات مع نموذجين من نماذج الحدود بنسبة 84.4% وتفوق على 11 نموذجًا آخر. وكانت أخطاؤهم مختلفة تمامًا.
قمنا بتقييم 45 نموذجًا رائدًا عبر جميع المختبرات الرئيسية على منصة Kaggle Cloud واكتشفنا ثلاثة أمراض أساسية مختبئة وراء لوحات المتصدرين التجميعية القياسية:
FloorBench (Shortcut Gaming) : حقق نموذج قاعدة بدون معلمات (OldestRecord) نسبة 84.4% (27/32)، متعادلًا مع GPT-5.4-mini وQwen3-80B، ومتفوقًا على 11 نموذجًا رائدًا. تقيس معايير الأداء التي لا تتضمن ضوابط أساسية، دون قصد، توافق المعدل الأساسي بدلًا من التفكير المنطقي.
تداخل أخطاء السيناريوهات : ارتكبت النماذج التي حصلت على نفس النتيجة (84.4%) أخطاءً منفصلة تمامًا . فشل نموذج Qwen في معالجة ملاحظات الأدلة متعددة القطع الأثرية؛ وفشل نموذج GPT في معالجة حالات التراجع الافتراضية للسجلات الفارغة ( الرفض الدلالي المفرط ). حقق موجه بسيط بدون تداخل، يجمع تنبؤاتهما، نسبة 100% (32/32) .
تلوث التراجع في برنامج التشغيل : عندما تعطل تنسيق المخطط في DeepSeek-R1 في 13 سيناريو، عاد برنامج تشغيل المهمة إلى وضع الحجب - والذي تطابق عن طريق الخطأ مع 9 نتائج حقيقية على لوحة رفض بنسبة 68.8٪، مما أدى إلى رفع درجته بشكل مصطنع من 59.4٪ إلى 87.5٪!
عناصر المنصة :
🏆 مهمة Kaggle الأساسية : vaibhavlalwani21/kryt-benchmark/4
📊 مجموعة بيانات Kaggle (10.00 سهولة الاستخدام) : vaibhavlalwani21/kryt-northstar-panel
⚡ أوقات التشغيل بنقرة واحدة : kryt-northstar-panel-quickstart و kryt-benchmark-quickstart
🛡️ التحقق : اجتياز 234 اختبار وحدة/عقد، و71 فحصًا لبيان الإصدار، و96 اختبارًا لموثوقية الخصوم (معدل الفشل المغلق 100٪).
**[](
What I Benchmarked
Models Tested
Findings
My Benchmark
Anyone want team up or add me ??
yeah cool
yeah
i don't see a question or point, what specifically are you trying to discuss?
Алго реально чует кризис заранее
okay I can😘
I,m interesting
Hello
dev.to/chef_p/can-an-llm-read-a-fr...
i'm here
hii
ok
Hello
hello
test suite runs against a live sandbox with role tokens set as environment variables.
how do i submit?