DEV Community

Cover image for Small Models, Great Tools: The Engineering Behind a Local AI Agent in Production
Quentin Merle
Quentin Merle

Posted on

Small Models, Great Tools: The Engineering Behind a Local AI Agent in Production

#discuss #ai #llm #python

There is a persistent myth that to build a worthy code assistant, you absolutely must use GPT or Claude. This is false. You don't need a 1-trillion parameter model. You need a small local model and extremely rigorous engineering around it.

This is the direction history is taking for companies. As Mark Zuckerberg mentioned, the future isn't a single omniscient model, but "every company having its own specialized AI". And this specialization necessarily involves fine-tuning and local deployment (or on sovereign servers) to guarantee data security.

The thesis behind the construction of Vibrisse Agent can be summed up in one sentence: Small models, Great tools.

In this article, I will detail the technical stack and concrete engineering solutions I implemented to tame a local model and make it reliable in production: LangGraph, Ollama, FastAPI, React (no build step, with embedded custom CSS), all running on a machine with 32 GB of RAM.

For the curious who want to run the agent on their machine right now:

// MacOs / Linux
curl -sSL https://agent.vibrisse-studio.dev/install.sh | bash

// Windows
irm https://agent.vibrisse-studio.dev/install.ps1 | iex
Enter fullscreen mode Exit fullscreen mode

Architecture: Why a State Machine (LangGraph)?

At first, when building an LLM application, we tend to think in sequential chains:
Input -> Prompt -> Tool -> Output.
The problem is that if one node fails, the whole chain stops without us being able to catch the error or understand the context of the crash.

That's where LangGraph comes in. Vibrisse's architecture isn't a chain, it's a state machine. Every node in the graph has a very precise responsibility, shares a global conversation state, and uses conditional transitions to move to the next node.

I implemented the Supervisor / Worker pattern:

  • The Supervisor analyzes the user's intent. It does nothing else but route.
  • It dispatches the task to specialized Workers (the RAG Worker, the Search Worker, the Ghost Worker...).
  • If a Worker fails or needs more information, it can send the state back to the Supervisor.

The Real Fight: Taming the "Laziness" of Small Models

The most valuable part of this project — and the one very few tutorials document honestly — was the fight against the nature of small LLMs.

Choosing Weapons: The Winning Models

If you're wondering which models survived my crash tests: I built the entire core of the agent around Gemma 4 (e4b). Why? Because it natively integrates vision and "thought" management, while offering that highly structured, Google-style response format. However, for evaluation and metrics, I had to switch to Llama 3 8B. A model that is too small proves incapable of reliably evaluating its own answers.

Constraints and Thinking Out Loud

Without strict constraints, a local model will always take the path of least resistance. Concretely, if you ask it to refactor a complex file at 3 AM without firm directives, it will proudly write: // ... rest of the code here.

The solution? Ultra-structured system prompts that impose a role, a strict JSON output format, and above all, the obligation to "think out loud". Imposing the use of a <thought> tag before triggering an action is paramount for two things: debugging why the agent made a bad routing decision, and improving the UX.

Triple-Layer Robust Parsing

Forcing an LLM to answer in JSON is only half the battle. When the model "tires" or gets tangled in context, it can generate malformed JSON. To keep the agent from crashing, I had to design a 3-layer parsing system:

  1. Layer 1: Standard JSON parsing.
  2. Layer 2: Regex Fallback to extract the object if the model added text around it.
  3. Layer 3: If the JSON is completely broken, a keyword fallback guesses the intent for a fallback action. Zero crashes.

Fun Fact: This approach was born from a thought exercise late in the project: "What if we had to run this agent on a very modest machine with a highly unstable SLM (Small Language Model)?" The forced constraints gave birth to these resilience tricks that I kept. (Perhaps the starting point for a future "Vibrisse-Lite"? Stay tuned...)

Triple-Layer Retrieval: Precise RAG, Not Noisy

RAG is not meant to stuff the model with context. The more text you send to a small model, the more it hallucinates. Context must be targeted and ultra-precise. The Vibrisse agent uses a Triple-Layer Retrieval:

  1. The Deterministic Layer (Ripgrep): For exact queries (e.g., "Where is the API_KEY variable defined?"). It's 100% precise, 0% hallucination.
  2. The Semantic Layer (ChromaDB): To understand intent (e.g., "Show me how errors are handled").
  3. The Statistical Layer (BM25): A standard safety net.

We don't choose one method, we execute all three and merge the results.

The Muscles of the Agent: MCP Hub, Web Search, and Ghost Mode

An LLM alone is just a "brain in a jar". You have to realize that to graft muscles onto it, every single action must be thought out and coded, which quickly breaks the "magic" aspect. You want your agent to do a simple grep? You have to code the tool, test it alone, then test it after a Vision action, then after a Web search, to ensure LangGraph doesn't crash.

Ghost mode

The agent has local tools, but also connected tools like Web Search (Tavily API), which is vital for grabbing the most recent documentation before answering.

The MCP Hub (Model Context Protocol)

Instead of reinventing the wheel, I integrated Anthropic's MCP standard. Vibrisse acts as an MCP Client. Adding a tool is simply a matter of plugging in an external Server. The architecture is thus "future-proof", ready for evolutions like Google's webMCP.

Ghost Mode: In-File Directives

This is the workflow killer-feature. An agent's goal isn't to force you to chat in a window.

Ghost Mode

Ghost Mode

A WatcherService (based on the Python watchdog library) runs in the background. As soon as it detects the @vibrisse: tag in a saved comment, it triggers a silent Ghost Worker that generates and injects the code directly into the editor.

Architect Mode: Human-in-the-Loop and Artifacts

For complex tasks, letting an autonomous agent execute dozens of commands in a loop is architectural suicide. You need a handbrake. So I implemented a Human-in-the-Loop pattern with LangGraph.

Graph Interruption (interrupt_after)

When a structural task is detected, the router switches to a dedicated node (planning_node). This node generates an action plan formatted in Markdown, wrapped in strict XML tags (<artifact id="plan">).
The LangGraph subtlety: the graph is compiled with the interrupt_after=["planning_node"] instruction. As soon as the plan is generated, execution stops dead on the backend and the state is saved in the database.

Frontend Rendering and State Resumption

On the React side, the UI intercepts these tags with a regular expression, hides the raw XML, and generates a rich interactive component (a CodeDiff, a Mermaid diagram, or a TaskBoard). The user then has buttons to "Approve" or "Reject" the proposal.

The biggest technical trap of this feature? State resumption (Resume).
When the user clicks "Approve", the frontend calls a route that restarts the LangGraph graph. Except that if you resume the conversation without saying anything, the small LLM finds itself with its own message (AIMessage) at the end of the context history and starts hallucinating the rest of the discussion.
The ingenious solution: Silently inject a HumanMessage ("Plan approved. You may proceed with implementation") into the state before restarting inference. The model thus has a clear directive and knows exactly what is expected of it.

Persistence and Context Limits

Curing Amnesia

An agent that forgets decisions made 2 hours ago is useless. Vibrisse uses SQLite for complete thread persistence, coupled with the automatic generation of a project_map.json at each launch.

The other sworn enemy is the context window limit (often 8k tokens on these small models). If the RAG brings back too many files, the context explodes. To handle this, Vibrisse constantly monitors token consumption and displays it live in the UI's Sidebar. The developer knows exactly when the context is saturated and it's time to refresh the conversation.

Sovereign Routing: Delegating Smartly

The agent analyzes the complexity of each request before acting:

  • Simple request -> Local model (Ollama). Immediate result.
  • Complex request -> The agent proposes switching to a more powerful Cloud model, with the user's explicit consent.

Sovereign Routing

The Elephant in the Room: Latency and RAM

We have to be honest about the main flaw of local AI: latency. Combining a Vision analysis with the generation of a complex React component can take up to 3 minutes (or more) on a consumer machine. It's the price to pay for total privacy and local execution.

To make this manageable, Vibrisse integrates real hardware resource tracking:

  • A (V)RAM check at installation to finely configure the agent via the onboarding Wizard.
  • A resource gauge in the Sidebar to monitor live machine load.
  • The ThinkingConsole: The "live" streaming of thought tokens. Even when the local action is slow, seeing the text scroll drastically reduces the cognitive friction of waiting. It's "Speed Design".

Hardware Discovery

The Golden Rule: Test in Blocks

Every new feature added risks breaking an existing one. The router (which reads intents) is the most temperamental point in the entire architecture. At the slightest adjustment in a tool's description, it can derail. Everything relies on semantics.

The absolute rule: test, test, and retest. Even when you don't feel like it. But how do you test a system whose answers are random?
I set up the RAGAS framework (driven by Llama 3 8B) to evaluate the quality and relevance of the RAG in an automated way. Added to this are scenario files (rigorous manual tests) and lightweight Python test scripts to ensure the routing chains don't break before adding the next block.

Conclusion: In Praise of the Small Model

Vibrisse isn't "the best agent in the world". There will always be more powerful Cloud models. But Vibrisse is proof that a Small models, Great tools philosophy holds up in production, provided you put in the necessary engineering rigor.

The tool is open-source. The code is out there.

Your turn:

  • Vibrisse Agent on GitHub
  • The project is designed to be "plug-and-play" with installation scripts (install.sh / .bat) for Mac, Linux, and Windows.
  • The project is open to contributions. Whether it's adding a new Worker (an MCP tool), optimizing the parsing system, or just fixing a bug in Ghost Mode... Pull Requests are more than welcome! Come break the code and rebuild it with me.
  • If you had to build or adapt an agentic stack today, which part scares you the most to stabilize? Routing? Parsing? Context management?
  • Let me know in the comments.

Proudly developed in Beauce, Québec 🇨🇦. Interested in the alliance between immersive web engineering and local AI sovereignty? Let's connect via Vibrisse Studio!

Top comments (0)