DEV Community

Debashish Ghosal
Debashish Ghosal

Posted on

We Moved to a Bigger Model to Debug the Test. Then We Went Back to the Smaller One.

#python #showdev #discuss #ai

We Moved to a Bigger Model to Debug the Test. Then We Went Back to the Smaller One.

From the agentpolicy v0.1.0 release: a local-first, Cedar-based policy decision point for AI agents (decide allow, deny, ask, rate-limit, or redact for every agent action, with shipped rule packs). Now on PyPI. This story is journal Groups 27, 29, and 30 of the v0.1.0 field test report.

AI disclosure: this article was drafted and edited with AI assistance. The field test, the defects, the fixes, and every number are real and linked below.

I did not move to a bigger model because I wanted to ship on a bigger model. I moved to a bigger model because I needed to answer a narrower question.

Was the 4-bit model actually the problem, or was my test harness lying to me?

That distinction mattered because the pinned field-test model was Qwen3-4B-Instruct-2507-4bit, and I did not want to "fix" a harness bug by quietly upgrading the hardware bill. If the small model was genuinely too weak, I wanted to know that. If the harness was wrong, I wanted to fix the harness and come back to the pinned model.

That is exactly what happened.

The failure that looked like a model failure

The failing case was FT-FWK-SMOL, one of the live framework runs driven by scripts/run_field_agents.py. Ten agents in the case were green. One agent, sm-01, failed four of its five scenario rows.

Not randomly, either. Same four rows. Two runs in a row. Same failure shape: status=not-available called=False.

That is exactly the kind of pattern that makes a weak local model look guilty. The captured responses did not help. They showed the model narrating tool calls as prose instead of emitting them. I wrote the wrong diagnosis into my own report: deterministic tool-call hallucination on the small model.

At that point I had two choices. Keep arguing with the 4-bit model, or change one variable and see whether the failure survived.

Why the 9B run mattered

I switched to Qwen3.5-9B-MLX-4bit for one reason only: falsification.

If the stronger model passed, the small model was probably the issue. If the stronger model failed in the same way, then the model was not the issue at all.

The result was clean and brutal: the same four rows failed on the 9B model too.

That instantly changed the job. I was no longer trying to make the smaller model behave. I was trying to find the structural bug both models were faithfully exposing.

That is the part I wish more agent-eval writeups said out loud: a bigger model is sometimes most valuable as a debugging instrument. It is not always the fix. Sometimes it is the control in the experiment.

The bug the model upgrade revealed

The answer was in the captured pairs under field-test/v0.1.0/results/s16-frameworks/cases/FT-FWK-SMOL/.

My runner had built one agent carrying all five scenario tools:

scn_tools=[allow, audit, escalate, deny, adversarial]

Then it drove every scenario row through that one agent. So the model was free to call a different scenario's tool than the row under test. On the failing rows, that is exactly what it did. For example, a prompt naming scn_decision_allow_01 could trigger scn_adversarial_injection_01 instead.

The policy engine then made a correct decision about the wrong tool. My RecordingEngine keys decisions by (tool, action, env, data, agent), so when the test later looked for the decision tied to the row's tool, it found nothing. Result: not-available.

That meant the bigger model had done its job. It did not rescue the test. It proved the test was asking the wrong question.

Back to the smaller model

The fix was in the harness, not the model. _run_one now builds a fresh agent per scenario row, bound only to that row's tool. One row, one tool, one decision under test.

Then came the part I care about most: we went back to the pinned 4-bit model.

No quiet upgrade. No "it only works on the 9B." Back to the original box-local model, the one the benchmark chose for the sweep.

And it passed.

The results after the fix:

  • smolagents: 14/14 rows, 29 request/response pairs
  • CrewAI: 14/14 rows, 41 pairs, up from 10/14
  • pydantic-ai: 14/14 rows, 17 pairs, up from 9/14

All three flipped green on the pinned smaller model. The 9B model also verified 5/5 on sm-01, but it was no longer the point. The point was that the release did not need a stronger model. It needed a more honest harness.

What worked

  • Using the 9B model as a control, not a crutch. It answered the question I actually had.
  • Captured evidence over intuition. The pair files showed the wrong-tool picks. The logs never did.
  • Returning to the pinned model after the fix. That is what made the result usable for the actual release.

What did not work

  • My first instinct that a repeatable failure on a small model must be the small model.
  • One agent carrying five scenario tools. That setup tests tool choice, not policy correctness.
  • Treating a stronger model as the default answer before using it as a diagnostic step.

What I took away from this run is simple: if a failure survives a model upgrade, stop negotiating with the model and inspect the test. And if the bigger model helps you find the bug, go back to the smaller pinned one and prove the real workflow is green there.

More detail and evidence

Have you ever upgraded the model just to isolate the bug, then gone back to the smaller one once you found it? That move saved this sweep, and I suspect it is underused.

Top comments (0)