DEV Community

cuculhart
cuculhart

Posted on

I thought the local LLM was weak — it was a bug in my own harness

#ollama #llm #ai #debugging

I've been benchmarking local LLMs as coding agents inside my own IDE (Teaspoon IDE) on a CPU-only machine — Ryzen 5 4500U, 32 GB RAM, no GPU, Ollama.

One symptom kept showing up across models: the agent would "re-read the same file over and over," or claim "I read it, but the contents aren't visible." Classic weak-model behavior, right?

It was a bug in my own harness. This is the record of how a 4B model exposed it — and how a control experiment proved it.

The failure that started it

The task (T3): "Running pytest shows one failure. Find the cause, fix it, and confirm all pass." I gave it to qwen3.5:4b — a small, fast 4B model — on my old harness.

What happened:

  1. READ_FILE: test_calc.py — reads it
  2. READ_FILE: calc.py — reads it
  3. READ_FILE: test_calc.py — reads it again. My dedup logic answers: "unchanged — use the content you already read"
  4. The model replies: "I read them, but the contents aren't visible here"
  5. Tries GREP, gets 0 hits → re-reads again → burns all 6 turns and fails

I was telling it "unchanged," and it was insisting it couldn't see the contents. My first assumption: the 4B just isn't smart enough to follow.

The investigation: tool results survived exactly one turn

Tracing the code revealed the flaw. The agent loop accumulated history like this:

  • User messages → kept ✓
  • Assistant replies → kept ✓
  • Tool results (READ_FILE contents, command output) → present in context for exactly ONE turn after execution, then silently dropped

So:

  • Step 1 reads the file → step 2 can see the content → everything works
  • From step 3 on, that content has already vanished from context
  • My duplicate-read suppression — "unchanged, use what you have" — was pointing at data the model did not have

The model's "I can't see the contents" was an accurate status report. It knew it had read the file (its own past READ_FILE commands were in history) but the payload was gone. It was debugging my context assembly correctly.

Bonus discovery: past evaluations needed rewriting

This changed how earlier runs looked. My 9B model (qwen3.5:9b) had been rated as "redundantly re-reading files" — 2 of 7 steps wasted on the old harness.

That wasn't weak memory. It was legitimate self-repair: fetching content that had disappeared. The smarter models were working around my bug. The weak one just reported it honestly.

The fix (and a second bug it surfaced)

  1. Persist tool results into running history — each step's tool output now stays in context for the whole run, so "unchanged" finally tells the truth
  2. Strip outer quotes from GREP patterns — models kept emitting GREP "member.*discount" and the literal " characters ended up in the regex, matching nothing. Seven turns of re-quoting, zero hits. The parser now strips surrounding quotes, and a zero-match GREP reply carries a hint to simplify the pattern or just READ the file

The control experiment

Same model, same task, same prompts — only the harness fix differs:

Condition Result
4b · old harness All 6 turns burned · re-read loop · failed
4b · fixed Completed in 5 turns (pytest → read → fix → re-verify, correct)
9b · old harness 7 steps with 2 re-reads (self-repairing)
9b · fixed 3 steps, zero re-reads, shortest path
e4b · fixed 4 turns, zero re-reads, clean

The same 4B model went from "hopeless" to "clean shortest-path completion" — just by fixing the code around it.

Lessons

  1. "Model weakness" and "harness weakness" are inseparable until you check. Every "this model is useless" article may actually be rating the harness's constraints. If you build agent tools, suspect what's in the context before you suspect the model.
  2. Dedup/context-saving must be designed against what the model can actually see. "It read it before" is only true while that content is physically in the prompt.
  3. Weak models are the best harness-testers. The strong ones paper over defects with self-repair; the 4B, with no slack, faithfully visualized my flaw. If you're testing your own agent loop, run it on a marginal model.

Data

All raw logs, task definitions, and the evaluation record are public:

The environment is deliberately modest — an old 6-core laptop CPU, no GPU. If the harness had been worse, every model would have looked broken. Check your context before you check your model.

Top comments (0)