DEV Community

Cover image for Our data-analyst agent lost to a single prompt: 0.55 vs 0.70 on DABench with a 1.5B model
ProTechSC Academy
ProTechSC Academy

Posted on Originally published at protechscacademy.com Fully Autonomous

Our data-analyst agent lost to a single prompt: 0.55 vs 0.70 on DABench with a 1.5B model

We built a data-analyst agent: it reads a question about a CSV file, plans the analysis, writes pandas code, runs it in a sandbox, reads the error if something breaks, and repairs its own code. It is the kind of loop most agent tutorials recommend.

Then we compared it with the simplest possible baseline: ask the same model once for a program, run it, done. The one-shot baseline won, 0.70 vs 0.55. This post explains how we measured that and why it happened.

The benchmark

  • DABench (the InfiAgent data-analysis benchmark): questions such as "what is the mean fare of passengers with missing age?" with closed-form answers and a required answer format.
  • We used 40 evaluation questions drawn with a fixed seed (16 easy, 12 medium, 12 hard) and 8 separate DEV questions for design decisions. Every number below is on the 40 EVAL questions.
  • Model: qwen2.5-coder 1.5B (Q4_K_M) served by Ollama, temperature 0, 8k context. The agent framework is Pydantic AI.
  • Each question is scored on exact sub-answers, with a 95% bootstrap CI and paired comparisons on the same questions.

Six configurations

Configuration What it does
Direct answer The LLM answers from the question text, no code
Keyword templates A classical rule-based solver, no LLM
Single-shot code One LLM call writes a program; we run it once
Agent v1 (tool calls) The agent must call run_python and other tools
Agent v2 (main) Plan-and-execute; the reply is a program, failed runs are fed back for up to 3 repairs
Agent without self-repair v2, but a failed run ends the attempt

Results

Configuration Accuracy [95% CI] easy / medium / hard Tokens per question
Direct answer 0.025 [0.00, 0.08] 0.06 / 0.00 / 0.00 702
Keyword templates 0.300 [0.15, 0.45] 0.75 / 0.00 / 0.00 0
Agent v1 (tool calls) 0.225 [0.10, 0.35] 0.19 / 0.33 / 0.17 6,775
Agent without self-repair 0.525 [0.38, 0.70] 0.88 / 0.58 / 0.00 1,944
Agent v2 (main) 0.550 [0.40, 0.70] 0.88 / 0.67 / 0.00 5,030
Single-shot code 0.700 [0.55, 0.85] 0.94 / 0.67 / 0.42 897

Accuracy with 95% bootstrap intervals on the 40 EVAL questions
Accuracy with 95% bootstrap intervals on the 40 EVAL questions

On the same 40 questions, the agent minus single-shot difference is −0.15 [−0.275, −0.025]. The agent won 1 question and lost 7 (sign test p = 0.07). So the gap is consistent in direction, but 40 questions are not enough to call it significant at 5%.

Why the agent lost

1. The 1.5B model could not drive a tool-calling loop. In agent v1, 18 of the 40 questions failed because the model never actually ran any code: it wrote tool calls as plain JSON text, echoed the framework's retry prompt, or left chat-template fragments in its code. We added small, documented shims for each of these, and it still reached only 0.225.

2. Self-repair mostly went in circles. Agent v2 lets the model see its traceback and try again. On these questions that added only +0.025 over the same agent without repair (one extra correct answer). The most common v2 failure was a repair loop: 15 questions where the model kept fixing one error and introducing another until the 3-repair limit.

3. The hard questions need one clean program, not many patches. Single-shot code solved 5 of 12 hard questions. Both agent versions with plan-and-execute solved none. Splitting a hard question into steps gave the small model more places to go wrong.

Why each configuration failed — no code run, repair loops and execution errors
Why each configuration failed — no code run, repair loops and execution errors

4. It costs more. The main agent used about 5.6× the tokens and 3.3× the time of single-shot code for lower accuracy.

Tokens per question against accuracy — single-shot code is cheapest and most accurate here
Tokens per question against accuracy — single-shot code is cheapest and most accurate here

A caveat we report against ourselves

Agent v2's "reply is a program" protocol was designed after we saw v1 fail on the EVAL questions. That makes v2's 0.55 an optimistic number, and we label it that way in the report. Even with that advantage, it did not catch up with single-shot code. The lesson for us: choose the agent protocol on the DEV set before the evaluation run.

Safety of the sandbox

Generated code runs in a fresh python -I subprocess in a temporary folder, with a 30-second timeout, a 1 MB CSV limit and an allow-list of three tools. That guards against accidents. It is not a security boundary for hostile code, and we say so in the README.

Takeaways

  1. Always run the boring baseline. Here, "generate one program and run it" beat the agent loop.
  2. Agent loops need a model that can follow them. A 1.5B model on a laptop can write useful pandas, but not reliably call tools or repair code over several turns.
  3. Measure cost next to accuracy. Tokens and seconds per question belong in the results table.
  4. Pick the protocol on DEV, report EVAL once. Otherwise label the result optimistic.

These results are for a small local model. A larger model may well reverse the ranking; we have not tested that here.


Full project (Pydantic AI agent, sandbox, trace viewer web app, MATLAB version, report): Data analyst agent with code execution and self-correction — ProTechSC Academy. Video walkthrough: youtu.be/y9vpE8hbuhk.

We guide final-year and PhD research projects in engineering, AI and AI agents. WhatsApp +91 97907 09424.

Top comments (0)