Why We Use a 4B Model on a Local NPU as a 'Wind Tunnel' for 3-Tier Multi-Agent Systems

Most multi-agent frameworks are benchmarked on 70B+ cloud models. Because frontier models have massive parameter capacity, they silently mask architectural flaws—recovering from ambiguous agent descriptions, ignoring bloated system prompts, and guessing missing tool schemas.

In SpaghettiLens-AI (v0.5.0), an open-source neuro-symbolic engine I built to refactor legacy Perl/PHP monoliths into Python/FastAPI microservices, we took the opposite approach. We used a compact 4B parameter model (qwen3-4b-instruct-2507) running locally on a Snapdragon NPU (within a strict 4096-token sliding context window) as a high-pressure "wind tunnel" to harden a 3-tier smolagents CodeAgent hierarchy (L0 Supervisor -> L1 Team Leads -> L2 Specialists).

Failure Modes Exposed on 4B Edge Hardware (and How We Solved Them)

1. Multi-Tier Blindness (smolagents PR #3115)

• The Problem: In a 3-level hierarchy, the root L0 Supervisor could see the L1 Team Leads, but had zero visibility into the specialized tools nested beneath them.
• The Fix: We patched capability propagation upstream so parent agents recursively append a summary of nested sub-agents to their descriptions, giving the root supervisor deterministic routing visibility across all 3 tiers.

2. Verbose Prompt Hallucinations

• The Problem: Default managed_agent system prompts ask sub-agents to write multi-section prose reports. Under a 4096-token ceiling, this verbosity caused the 4B model to hallucinate theoretical summaries instead of emitting executable <code>...</code> blocks.
• The Fix: We engineered compact, code-first Jinja2 prompt overrides and strict per-tier constraints that force every agent to invoke its assigned tool in Step 1 and pass the output directly to final_answer.

3. Silent Token-Ceiling Loops (smolagents PR #3110)

• The Problem: When extracting large static AST/SQL dependency graphs, outputs occasionally hit max_completion_tokens mid-JSON. Because the closing } was missing, parse_json_blob raised a misleading "output does not contain any JSON blob" error, tricking the agent into regenerating the exact same oversized response in an infinite loop.
• The Fix: We added explicit truncated JSON detection so the parser immediately surfaces a token-limit truncation error, allowing the agent to compact its output on retry.

Academic Alignment: Deterministic "Expert Blocks"

Our architecture aligns with recent research on Expert Block Frameworks for Small LLMs (Naphade et al., arXiv:2509.15561).

Naphade et al. demonstrate that locally-run small LLMs can match 100B+ frontier models (within ~0.9% of GPT-4) when paired with a deterministic block that transforms raw, noisy trajectories into structured context before LLM reasoning.

SpaghettiLens-AI extends this exact principle to multi-tier software modernization on edge NPUs:

• Deterministic Context Summarizers: Instead of feeding raw, noisy legacy monoliths into a 4096-token window, our static extractor acts as a deterministic Expert Block—compressing raw ASTs and SQL DDL schemas into a structured JSON dependency graph.
• Deterministic Verification Blocks: We replace open-ended LLM self-critique with hard symbolic verification (AST parsing, parameterized SQL enforcement, and decimal.Decimal ledger checks), enabling a 4B parameter model to achieve deterministic, single-pass refactoring (iterations_used: 1) on consumer hardware.

---

I am the creator of SpaghettiLens-AI and an Open-Source ML Engineer focused on edge hardware acceleration and neuro-symbolic systems. If you are building with smolagents or local NPUs, let's connect!

Comments

[Sammi De Blas]

Using a 4B model on a local NPU as a wind tunnel is a cheap way to measure orchestration overhead before paying for a frontier model. The failure mode is that the small model passes the routing test while the larger one changes how the same tool call behaves, so the benchmark has to compare tool traces, not just answers.