<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ajay Surya Senthilrajan</title>
    <description>The latest articles on DEV Community by Ajay Surya Senthilrajan (@ajay_2112).</description>
    <link>https://dev.to/ajay_2112</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4000709%2Fd7ccb002-0d7f-4ad6-b492-91eb4d7dcad6.jpg</url>
      <title>DEV Community: Ajay Surya Senthilrajan</title>
      <link>https://dev.to/ajay_2112</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ajay_2112"/>
    <language>en</language>
    <item>
      <title>I tested whether a 4B model could replace a 12B one in a tool-using agent. My first test was wrong.</title>
      <dc:creator>Ajay Surya Senthilrajan</dc:creator>
      <pubDate>Sun, 15 Nov 2026 06:27:49 +0000</pubDate>
      <link>https://dev.to/ajay_2112/i-tested-whether-a-4b-model-could-replace-a-12b-one-in-a-tool-using-agent-my-first-test-was-wrong-3jgf</link>
      <guid>https://dev.to/ajay_2112/i-tested-whether-a-4b-model-could-replace-a-12b-one-in-a-tool-using-agent-my-first-test-was-wrong-3jgf</guid>
      <description>&lt;p&gt;&lt;em&gt;Two pre-registered runs on local models, a reader's correction between them, and the one place the small model still breaks.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I wanted to know whether a 4B local model could replace a 12B one in a small tool-using agent. I ran it as an experiment, with the prediction written down before each run. Both models ran on Ollama on one laptop, at temperature 0.7, with no API.&lt;/p&gt;

&lt;h2&gt;
  
  
  The set-up
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;24 order-desk tasks: refund an order, refund and add a note, check stock and add a note.&lt;/li&gt;
&lt;li&gt;Each task ran clean and under three injected tool faults: a timeout, a one-off error, an empty result. That makes 96 task-and-fault combinations per round.&lt;/li&gt;
&lt;li&gt;Baseline: gemma4 12B. Candidate: qwen3.5 4B. Same prompt, same tools, same retry settings.&lt;/li&gt;
&lt;li&gt;The rule for the verdict was fixed before the run: is the 4B within 10 points of the 12B?&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Run 1: my prediction lost
&lt;/h2&gt;

&lt;p&gt;I predicted the 4B would be close enough. It was not. The 12B passed 96 of 96. The 4B passed 70 of 96, and the run stopped after one round with a verdict of BLOCK.&lt;/p&gt;

&lt;p&gt;Most of the failures were one behaviour. In 21 of the 26, the 4B was asked to check stock for an order and its only call was to the stock tool, with the order id as the product code. It never looked the order up.&lt;/p&gt;

&lt;h2&gt;
  
  
  A reader caught a flaw in the test
&lt;/h2&gt;

&lt;p&gt;In my set-up that wrong call raised an exception and ended the trial. The model never saw "unknown SKU". So I had measured "makes a wrong first call, where a wrong call is fatal". I had not measured "cannot do the task". I published a correction and ran it again.&lt;/p&gt;

&lt;h2&gt;
  
  
  Run 2: the error goes back to the model
&lt;/h2&gt;

&lt;p&gt;One change, the same for both models: a tool's error is returned to the model as a message. My prediction, written first: inconclusive.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;12B: 480 of 480.&lt;/li&gt;
&lt;li&gt;4B: 428 of 480 (89%).&lt;/li&gt;
&lt;li&gt;The 4B made the wrong first call about as often as before: in 21 of 32 stock-note trials in run 1, and in 108 of 160 in run 2.&lt;/li&gt;
&lt;li&gt;Shown the error, it looked the order up next in 106 of those 108, and 68 of the 108 passed.&lt;/li&gt;
&lt;li&gt;Only 2 of 480 trials never did the lookup at all.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The test could not decide whether the 4B is within 10 points of the 12B. The gap landed at 10.8 points. So the verdict is INCONCLUSIVE: the run crossed neither the PASS nor the BLOCK threshold.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where it still breaks
&lt;/h2&gt;

&lt;p&gt;Two things going wrong in one trial. When the 4B made the wrong first call and then the lookup came back empty (the injected fault), it passed 0 of 26. In 21 of the 26 it read the error, looked the order up once, got nothing back, and stopped. The 12B retried the empty lookup every time, 120 of 120.&lt;/p&gt;

&lt;p&gt;Where the wrong first call was the only lasting problem (no injected fault, or a timeout that a retry clears), the 4B went on to pass 52 of 58.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I took from it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Know whether your tools raise their errors or return them. Here it changed what the test measured: in run 2, 77 of the 4B's 428 passes came after a tool error that would have ended the trial under run 1's set-up.&lt;/li&gt;
&lt;li&gt;In this suite, looking at one problem at a time flattered the 4B. Half of its 52 failures in run 2 (26) were the compound case above.&lt;/li&gt;
&lt;li&gt;I did not see any of this by reading sample outputs. I saw it by counting what each trial did with its tools.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;p&gt;One suite, one prompt, two models, one machine. The two runs used different seeds, so 73% against 89% is two observations, not a measured effect of returning errors. The breakdowns were chosen after seeing the data. None of this says anything about these models in general.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check it yourself
&lt;/h2&gt;

&lt;p&gt;The harness is my own open-source tool, ARCI. It runs a baseline and a candidate on the same tasks with injected tool faults and gives PASS, BLOCK or INCONCLUSIVE with a stated error bound.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/results/order-desk-study.md" rel="noopener noreferrer"&gt;Both runs, both pre-registrations and the correction&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/ajaysurya1221/agent-reliability-ci/tree/main/docs/results/order-desk-study" rel="noopener noreferrer"&gt;The raw records of every trial, with the command that re-derives both verdicts&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/ajaysurya1221/agent-reliability-ci/blob/main/docs/recipes/model-swap.md" rel="noopener noreferrer"&gt;The same comparison on your own two models&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/ajaysurya1221/agent-reliability-ci" rel="noopener noreferrer"&gt;Repository&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you have measured this on other small models, I would like to know: does returning tool errors close most of the gap, or is that specific to this pair?&lt;/p&gt;




&lt;p&gt;&lt;em&gt;The numbers were recounted from the stored trial records before publishing, and those records are public at the links above. First published on my blog, &lt;a href="https://realornoise.hashnode.dev/i-tested-whether-a-4b-model-could-replace-a-12b-one-in-a-tool-using-agent-my-first-test-was-wrong" rel="noopener noreferrer"&gt;Real or Noise&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>testing</category>
      <category>opensource</category>
    </item>
  </channel>
</rss>
