This is a submission for the Kaggle Benchmarking Challenge
What I Benchmarked
I created DataBugBench, a benchmark designed to evaluate how accurately modern Large Language Models diagnose and correct subtle runtime and logical errors in Python Data Science code (Pandas, NumPy, and Scikit-Learn).
While LLMs perform exceptionally well on synthetic LeetCode-style problems, real-world data science pipelines frequently fail due to silent logical errors rather than outright syntax bugs. I specifically benchmarked 5 pervasive data engineering pitfalls:
Pandas SettingWithCopyWarning Mutations
Groupby Aggregation Missing Keys
NumPy Implicit Broadcasting Dimension Mismatches
Data Leakage in Feature Scaling
Index Discontinuity after NaN Dropping
Models Tested
I evaluated five state-of-the-art models (Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro, Qwen 2.5 Coder 32B, and Llama 3.3 70B) to measure how code specialization affects debugging precision.
Findings
Silent Logical Errors Are Hardest: Models easily fix explicit syntax errors, but often miss data leakage bugs where code runs without crashing but yields incorrect analytical results.
Specialization Matters: Code-focused models punch well above their weight class on complex array shape and broadcasting errors.
Final Results Summary
Claude 3.5 Sonnet: 5/5 passed (100.0% accuracy)
GPT-4o: 4/5 passed (80.0% accuracy)
Qwen 2.5 Coder 32B: 4/5 passed (80.0% accuracy)
Gemini 1.5 Pro: 3/5 passed (60.0% accuracy)
Llama 3.3 70B: 3/5 passed (60.0% accuracy)
Conclusion
DataBugBench demonstrates that while frontier LLMs excel at standard code generation, resolving silent data pipeline bugs requires a deeper understanding of implicit state, memory views, and data leakage. Moving forward, evaluating models on non-crashing logic bugs will be critical to building truly reliable AI data science assistants.
My Benchmark
https://www.kaggle.com/code/yoyoo100/databugbench-evaluation
Top comments (1)
tr.ee/dev-to