DEV Community: Ahmad ammar

Small local models: what actually holds up when you re-run the measurement

Ahmad ammar — Fri, 21 Aug 2026 18:18:08 +0000

I run a small local model — Qwen3.5 4B via Ollama — as one deterministic-checked step inside some tooling. This is what still reproduces today, on named versions, with the scripts to check it yourself. Where a result did not survive re-running, that is here too, because that turned out to be the most useful part.

Versions, so you can pin them: Ollama 0.32.6, model qwen3.5:4b, digest 2a654d98e6fb, Q4_K_M.

None of this is a new technique. The underlying pattern is Best-of-N with verifiers, which is well established. These are measurements inside a known pattern.

`think: true` can hand you an empty answer

With thinking enabled, the chain-of-thought and the final answer share one token budget. When the thinking consumes it, response comes back empty while thinking holds the content. Code that reads only response sees "" and concludes the call failed. It didn't.

Measured today, one-line question, num_predict: 128:

think:false    response=200   thinking=0
think:true     response=0     thinking=483

Fix: for extraction and structured tasks, read both fields, or turn thinking off. Whether it triggers depends on your budget — tighten num_predict to make it likely.

(Separately: there is evidence that pushing long chain-of-thought onto small models can hurt accuracy — Luo et al., "Through the Valley: Path to Effective Long CoT Training for Small Language Models," EMNLP 2025. That is a training-time effect, not this API field-routing quirk. Different problem; noting it so the two don't get conflated.)

Self-critique made the 4B worse

I asked it to critique its own answer. It verbally retracted the wrong fix — "Remove the check proposed above" — and then re-emitted that same wrong fix in code, in the same reply.

One anecdotal example of what Huang et al. measure at scale in "Large Language Models Cannot Self-Correct Reasoning Yet" (arXiv:2310.01798, ICLR 2024): on GSM8K, GPT-4's own accuracy drops 95.5% → 91.5% → 89.0% under intrinsic self-critique.

Self-critique is not a free reliability layer. You need an external judge, not the model grading itself.

Request precision beat model depth

Same input, two prompts. Asking for output keys that match the source's naming literally moved accuracy 7/10 → 10/10. A numeric spec — "each row has exactly 3 cells and 4 dashes" — fixed a table that a descriptive spec — "a well-formed separator row" — kept producing broken.

Well-worn prompt-engineering advice. Included only because I have the before/after on the same input.

The external judge needs its own negative control

Here is the part I got wrong for a while, and it is the expensive one.

"Put a deterministic external check on the output" is half an instruction. I had that check. It passed a fabrication.

A merge task, two columns into one, 81 rows. The model invented 5 values that were never in the input. The check returned PASS, autoVerified: true, and exit code 0. It was not bypassed and it did not crash. Every invented value was drawn from a closed vocabulary — every value that could appear did appear somewhere in the input — so every token-level assertion the check ran was satisfied. The judge was asked a question that could not tell the two cases apart.

So the rule I would give someone starting this:

A judge that has never been made to fail has not been shown to pass.

Build the check, then feed it material it must reject. If it stays green, you don't have a judge — you have a second thing that agrees with the model. And when the output is a pure function of the input, compare it to the input mechanically before accepting it. An exit code is not evidence.

And one result that did not survive

An earlier note of mine reported that this model drops the space at a Latin↔Arabic boundary — ask for hello مرحبا, get back helloمرحبا. I had it in 2 of 2 runs and treated it as a real defect.

Re-running it today: 53 generations. Zero reproductions. Seven prompt shapes on the pinned model, six of those shapes repeated on two other local 4B variants, plus the original echo prompt.

I can't tell you it was fixed, and I'm not going to pretend otherwise — the model file on disk is the same one that was there when I first measured it. Prompt shape doesn't explain it either; I tried the transform-style tasks that were closer to the original context and they came back clean too.

What I can tell you is that a result I would have published as a measured defect was, six weeks later, unreproducible — and I only know that because re-running it was one command. Both scripts are in the repo; the boundary one is currently a null result, and it stays in, because a null result you can run is worth more than a claim you can't.

The check itself is four lines and still worth having if you ship bilingual text — English-centric output assertions never look at the script boundary, so nothing else in your test suite is watching that seam.

The through-line

A small model doesn't get better when you ask it to think more. It gets better when you reduce what it has to decide, put a deterministic external judge on its output — and then prove the judge can fail.

That framing isn't mine. DSPy's BestOfN with a custom reward_fn reproduces most of this shape; CodeT and AlphaCode filter samples by executing tests; Guardrails' on_fail=reask and Instructor drive retry-on-failure. Even graduated exit codes for agent steps are the Nagios convention (0 OK / 1 WARNING / 2 CRITICAL), about 25 years old.

Reproduction scripts, with versions pinned: https://github.com/ahmadyaseen35-coder/local-model-field-notes-repro

How parallel AI agents should talk to each other (and the bug that proved it)

Ahmad ammar — Fri, 21 Aug 2026 17:21:19 +0000

If you run more than one coding agent at a time, you hit a problem nobody has a settled answer for: how do two agent sessions message each other? Not the model talking to a tool — two independent sessions, running in parallel, that need to hand off a decision or a result.

The obvious channel is a human relaying copy-paste. I spent a day watching that fail in two specific ways, then replaced it, then found a bug in the replacement that is the best argument for the whole approach. Here's the pattern and the evidence.

Why the human relay fails

It fails for two reasons that have nothing to do with typos.

1. It loses provenance. When a message arrives as pasted text, the receiver cannot prove who wrote it. That matters more than it sounds. Two messages reached a session this way in one day: one asserted a state that had never happened ("you enabled X" — to a session that had done no such thing), and one reported CI green on a commit that was already two commits stale. Both were caught. They were caught only because the receiver independently checked — the message itself carried zero evidence.

2. It looks exactly like a prompt injection. "Read this and do it" is the shape of an attack. A well-behaved agent should be suspicious of instructions with no verifiable origin — which means the safe agent and the useful relay are in direct conflict.

The pattern: a committed file + a pointer + a provenance check

The fix is to stop sending content and start sending a reference to a committed artifact:

The sender writes the message to a file in a path it owns and commits it.
The sender passes one plain-text line: Read <path> and follow it.
Before acting, the receiver checks the origin:

git log -1 --format='%an %ci' -- path/to/message.md

Now the message has an author, a timestamp, and an immutable history. "Read this and do it" stops being injection-shaped because the receiver can verify the sender before it acts — and can refuse anything whose provenance it doesn't like. Refusal becomes a feature, not a failure.

Add a naming convention so a session can find its own mailbox — message-to-<recipient>-<topic>-<date>.md — and a tiny script that lists the open ones at session start. Round-trip on this channel that day: a message sent, read, acted on, and replied to in nine minutes, with a verifiable trail at every step.

The bug that proves the point

Here's the part I didn't plan. I wrote the "list my open messages" tool. It ran, and it reported 0 open — while two real messages sat unread. The tool hid its own inbox.

The cause was one line. Each message carries a status, and the closer looks like this:

// BROKEN: "done" appears in the boilerplate of every OPEN message too
function isClosed(statusLine) {
  return /\bdone\b/i.test(statusLine);
}

Every open message's status line reads: STATUS: open · Set to "done" after you consume it. The word done is right there in the instructions — so the check marked open messages closed and filtered them out. The fix reads the value, not the line:

// FIXED: read only the token right after STATUS:
function isClosed(statusLine) {
  const m = statusLine.match(/STATUS:\s*\**\s*([a-z-]+)/i);
  return (m ? m[1].toLowerCase() : "") === "done";
}

The lesson isn't "parse carefully." It's this: a green result reads as "safe" when it often means "the check can't see." Before you trust any all-clear — a gate, a linter, an inbox that says empty — prove the red can appear. One test would have caught it:

// negative control: an OPEN message whose text also contains "done" must still surface
assert(isClosed('STATUS: open · Set to "done" after you consume') === false);

My synthetic test fixture used a clean status line, so it never contained the trap. The real message did. That is the whole case for dogfooding on real inputs, and for writing the test that tries to make green turn red before you rely on it.

Two things to take

Provenance beats convenience. A message you can't attribute is a message you can't trust — build the channel so every hop is verifiable, and let the receiver refuse.
Never trust a green you haven't tried to turn red. The most dangerous failure isn't the check that fails loudly; it's the one that passes while blind. My inbox tool passed while blind to its own inbox.

Your AI agent's best-practices doc is a wish. So was the number I used to prove it.

Ahmad ammar — Fri, 21 Aug 2026 15:04:33 +0000

You wrote the AGENTS.md. Or the CLAUDE.md, or the team best-practices doc. A month later, how many of those practices does any agent actually follow? If you can't answer with a number, you don't have practices — you have wishes.

I learned that twice. The second time was worse.

Documented ≠ adopted

I run three production projects on Claude Code, and I'd written a model-routing rule into my shared config: mechanical work — inventory, renames, boilerplate — goes to the cheap model; hard reasoning — architecture, security, migrations — goes to the expensive one. I wrote it, promoted it to an auto-adopted practice, felt good about it.

Then I built a small usage analyzer to see where the tokens actually went. Zero dependencies — it just reads the session transcripts the tool already writes to disk. The cheap model came back at 0.03% of output: 18k tokens against tens of millions. Not "underused." A rounding error.

Documentation is not a control surface. The doc had not routed a single task.

The fix that doesn’t work

The reflex is to write it again, bigger. Add a reminder. Write a third markdown file about the two markdown files nobody is applying.

This is the same mistake as putting a budget in a prompt. "Be mindful of the model tier" is a wish, exactly like "max 8 searches" is a wish — a model sails past both the moment the task still feels unfinished. If your practice lives only in a doc the session may never open, you do not have a practice. You have a hope with good formatting.

Measure, then enforce below the decision

A practice that lives only in a doc fails for the same structural reason a budget in a prompt fails: nothing forces it into the path where the decision is made. So move it below that path.

Measure adoption from data you already generate. You cannot enforce what you cannot see. The transcripts already record which model ran, which tools fired, which skills were invoked. A zero-dependency reader turns that into an adoption number per practice. That is how I learned the routing rule was near zero — by counting, not guessing.

Enforce at the boundary the model cannot skip. Every session hits session-start. That is the injection surface. Instead of a doc the session may never load, the practice gets printed into the session-start context deterministically: static strings, no extra generation cost, fails open. Same principle as a budget belonging in the harness rather than the prompt — put the practice in the layer the model passes through by construction, not the one it has to choose to read.

Then I audited the counter

A month later I went to re-measure, and found the reader had been lying to me the entire time.

Subagent work is written to a separate sidechain file under the session directory. My reader was not recursive. It never opened those files. Every token a subagent spent was invisible to it.

That matters because of where the cheap model actually runs. Measured properly: 100% of the cheap model's output was subagent work. Zero on the main thread. The one surface the rule governs was the exact surface the counter could not see.

So 0.03% was never an adoption number. It was an artifact of a blind spot. The commit that fixed the reader is titled, accurately, read subagent sidechains — "Haiku=0" was a measurement artifact.

Here is the part that rescues the story: the corrected figure is 0.75% — 370k tokens out of 49.4M. Still terrible. The conclusion survived. The evidence for it did not.

That is the real lesson, and it is less comfortable than the first one: a measurement you never audit is also a wish. It just wears a number, which makes it harder to doubt. I spent a month more confident than my instrument had earned.

The honest caveat

Injection is not adoption either. Firing the reminder every session is necessary, not sufficient. The arbiter is the next measurement cycle — if the cheap-model share climbs, the injection worked; if it does not, the problem is deeper than reminding, and I will know, because I am measuring. A practice you inject but never re-measure is just a wish that loads faster.

The moves, in order

Measure adoption from data you already generate. The transcript is a telemetry stream you are not reading.
Audit the instrument. Do not ask whether it looks right — feed it a case whose answer you already know and check that the number comes back. A counter that has never been made to fail has never really passed. Mine had no such test, and when the missing surface finally showed up it moved the figure by more than twenty times.
Inject the practice where the model passes through by construction, not into a doc it may never open.
Re-measure. The doc was never the mechanism. The number is — once you have earned the right to trust it.

Step 1 usually tells you your existing docs are decorative. That is not a reason to write better docs; it is the reason to stop treating docs as controls. Step 2 occasionally tells you step 1 was decorative too.

Multi-agent fleets burn ~15x the tokens — here's the budget layer the playbooks skip

Ahmad ammar — Mon, 29 Jun 2026 19:41:26 +0000

Give ten agents a shared, metered tool — a paid search or research API where every call is real money — and you've handed ten of them the same company credit card. Each one reasons "I'll just run a quick search." You find out the total on the invoice.

Anthropic's own multi-agent write-up clocks a fleet at roughly 15x the tokens of a single chat. That's the token pool. The paid external tools are the line item the orchestration playbooks skip — and it's the one that shows up in actual dollars. I run four production projects on Claude Code with deterministic budget enforcement built into the harness, and this is the pattern that keeps that line item from surprising me.

The fix that doesn't work

You can't solve this by asking agents to be frugal. "Be mindful of the budget" in a prompt is a wish, and an LLM will sail past "max 8 searches" the moment the task still feels unfinished. If your budget lives in the prompt, you don't have a budget — you have a hope.

The split most write-ups collapse

Cost governance is two layers, and conflating them is why prompt-level "budgets" fail:

1. Enforcement is deterministic and lives below the model. A PreToolUse hook in the harness sits in the tool-call path. When an agent tries to call a metered API, the hook checks the per-session counter and the account-wide daily budget before the call reaches the network. If either ceiling is crossed, the call is denied — the harness refuses it, and the model literally cannot proceed. The model can't argue with it, jailbreak it, or sneak "just one more search" past it. This is what actually bounds the dollars — plain middleware that touches no credential, because it works by blocking the call before it gets there.

2. Judgment is the half a gate can't do. The hook can cap spend, but it can't decide whether this task deserves to spend at all. That decision is model-shaped, and it's where the agent earns its place:

A novelty-gate. Most tasks don't qualify to spend: CRUD, mechanical edits, known facts → zero research. The biggest budget win isn't a smaller cap — it's that the majority of work never reaches the paid tool.
A tier + cap per qualifying call (quick-fact vs deep-dive), declared as policy that the harness then enforces mechanically — the agent proposes the tier, the counter imposes the limit.
Honest degradation. When the paid source fails, the agent falls back to a free one and flags the reliability downgrade in its output.

So the division is clean: the gate enforces, the agent judges.

The novel part isn't "centralize the credential" — that's least-privilege, decades old. The leverage is pairing a deterministic hook that the model can't escape with model-shaped judgment about whether to spend at all — with the gate defaulting to "no."

Why two ceilings, not one

This is the detail that blew up the budget in production until I separated them:

Per-session cap on research calls (free or paid — just a count). This is the runaway catch: a single agent spinning in a loop, making 40 searches when 4 would do. Easy to detect, hard to reason about in the prompt.

Account-wide daily budget on the PAID API only. This is the actual money. Free research (web search) gets a generous ceiling — enough to do real work. Paid research (EXA, specialized APIs) gets a strict daily spend cap in dollars or calls.

If you cap both the same, you either throttle good work (the paid API becomes useless when free search hits its ceiling) or leak money (free and paid burn the same quota, paid spending happens before you notice). Separate them, and the paid tool stays protected while research quality stays real.

The distributed-counter problem, solved

If the budget is a single shared number across parallel agents and workflows, you're holding a classic distributed-counter race: two agents can both read "budget remaining," both decide they're clear, and both spend — blowing the cap. This isn't theory. It happens.

I built the account-wide budget as a single shared file written by N parallel sessions. Access goes through an atomic mutex — an exclusive create-lock that a session holds while it reads and updates the count, then releases. A crashed holder can't wedge it shut: a lock older than a few seconds is reclaimed as stale. But a live lock is never stolen — if a waiter can't acquire it in time, it fails OPEN, refusing the call rather than corrupting the count. Losing one uncounted call is safe. Corrupting a shared budget is not.

This was tested under 12 concurrent paid calls. Zero lost updates. That's not magic — it's a metered gate built right.

A surprise about fan-out

Subagents spawned from a main session, and workflow-launched agents, inherit the parent session's identity. That means they all count against the same per-session ceiling for research calls. A main agent + 3 subagents + 3 workflow calls = one shared pool. That's the runaway catch working exactly as designed.

But parallel SESSIONS (run in different harness processes or separate user sessions) each get their own per-session counter. Which is exactly why the account-wide daily budget — the paid-API cap — has to sit at the account level, not per-session. One session doesn't see another session's spending. The shared daily budget is the only thing that keeps total paid spend honest across parallel work.

What this bounds — and what it doesn't

Be precise: this harness layer controls (a) the count of external-research calls per session and (b) total daily spend on paid APIs. It is NOT a fix for general token-quota blowups from heavy agents doing non-search work. If an agent runs 12 parallel research workflows, each doing deep reasoning, the token pool scales badly — that's a different control: don't parallelize research at all, or schedule it across sessions.

This is about external tool spend and runaway calls. It's not a token limit.

The takeaway

Most "agent budget" advice optimizes the cap size. The leverage is one level up: a deterministic gate that sits below the model, makes most work never spend at all, and owns no credential because it works by refusing the call. Add a second ceiling for the paid API at the account level, not per-agent. Stop asking your fleet to be frugal. Build a harness hook that can't be escaped — and let the agent's judgment about when to spend do the rest.