DEV Community: PAWAN YADAV (AI Engineer)

Building a Production Multi-Agent System: Supervisor Routing, Tool-Calling, and Taming Hallucinations with a Small 8B LLM

PAWAN YADAV (AI Engineer) — Fri, 31 Jul 2026 13:04:56 +0000

How I designed a conversational AI system that lets end users perform structured backend operations in plain English — using a supervisor-agent architecture, tool-calling, and a lightweight 8B open-source model instead of a giant proprietary one.

The Problem
Most enterprise backend systems expose their functionality through dense, form-heavy portals. Doing something simple usually means:

Logging into a portal
Hunting through nested dropdowns and categories
Re-typing boilerplate information every time
Coming back later just to check status or add a note
Most users just want to describe what they need in plain language and be done with it.

So I built a conversational AI layer on top of an existing backend system that lets users perform three categories of operations — submitting new requests, modifying existing ones, and looking things up — entirely through natural language, in chat.

This post covers three things people keep asking me about:

How the multi-agent routing actually works
How I got reliable results out of a small 8B model instead of a large frontier model
How I reduced hallucination in a workflow where wrong answers write real records into a real system
Why Multi-Agent Instead of One Big Prompt
My first instinct was “just write one giant system prompt that handles everything.” That fell apart fast:

The prompt became enormous and slow to reason over
The model would confuse one type of operation with another
Tool-calling accuracy dropped as the number of available tools grew
Debugging why something went wrong was nearly impossible — one flow’s logic bled into another’s
The fix was to split the system into a Supervisor Agent and several specialized Worker Agents, each with a narrow, well-defined job and its own small toolset. This is the same pattern used in most production multi-agent systems: a router model decides what the user wants, and a specialist model decides how to do it.

Architecture

If you want a rendered version instead of ASCII, here’s the same flow as Mermaid (renders natively on Dev.to, GitHub, and most markdown viewers — for Medium you’d paste this into a Mermaid live editor and drop in the exported image):

How the Supervisor Routes Requests
The supervisor doesn’t call any backend tools itself — its only job is intent classification + delegation + context tracking. That single-responsibility design is what makes it reliable even on a small model: it’s not trying to reason about categories, records, and filters all at once, it’s just answering “which of 3 buckets does this go in?”

Rough logic:

Signal in user messageRoutes toWants to submit something newAgent AlphaWants to change or add to something that already existsAgent BetaWants to look up or review existing recordsAgent GammaUnclear / multiple intents in one messageSupervisor asks a clarifying question first

Each worker agent then owns its own multi-step flow. For example, Agent Alpha walks the user through: pick a category → pick a sub-category → describe what’s needed → get an AI-suggested resolution → confirm → record submitted → confirmation shown. Agent Beta and Agent Gamma follow the same “narrow tool, narrow job” philosophy, just with different backend calls (fetch existing items, apply a change, query by a filter or time window, etc.).

Doing This on a Small 8B Model (Not GPT-4-class)
The part people are most surprised by: this whole system runs on an open-source 8B parameter model, not a large frontier model. A few techniques made that viable:

Narrow the job per agent. An 8B model struggles to hold a large catalog of categories, dozens of tools, and multiple conversation flows in its head at once. Split across several agents, each model call only needs to reason about a handful of tools and a tightly scoped instruction set — well within an 8B model’s comfort zone.
Constrain tool schemas aggressively. Every tool has a strict JSON schema with enums where possible, instead of free-text parameters. Small models are much better at picking from a constrained set than generating an unconstrained one.
Force structured output, not free prose, for anything that feeds a tool call. The model never “writes” an identifier or category name from memory — it’s only ever allowed to select values that came from a prior tool response (e.g., the actual list returned by a catalog lookup). This alone eliminated most of the invented-value problem.
Multi-turn confirmation instead of one-shot execution. Nothing destructive (submitting a new record, applying a change) happens on the first pass. The flow always shows the user what it’s about to do and waits for confirmation. This isn’t just good UX — it’s a hallucination safety net: if the model got a field wrong, the user catches it before it’s written to the backend.

Overcoming Hallucination
This was the hardest part, because a hallucinated output isn’t just a wrong answer in a chat window — it’s a bad record in a real backend system. Here’s what actually moved the needle:

Ground every list in a live tool call, never in the model’s memory. Category names, sub-category names, and record identifiers are always fetched fresh from the backend right before they’re shown to the user. The model is never asked to recall or generate them.
Retrieval-augmented suggestions before action, not instead of grounding. Before a new record is submitted (or a change applied), a retrieval-augmented step generates a suggested resolution based on the user’s actual described need and a knowledge base — not the model’s raw training data. This both reduces hallucinated “answers” and often resolves the need without any record being written at all.
One category per submission, explicit selection only. The model can’t infer or guess a category — the user must explicitly pick from a list that was just returned by a tool. This removes an entire class of “close enough” hallucinations.
Validation layer between model output and tool execution. Every tool call the model proposes is validated against the tool’s schema and the last known-good state (e.g., “is this identifier actually one of the records we just fetched for this user?”) before it’s executed. If validation fails, the flow re-prompts instead of silently proceeding.
Confirm-before-commit on all write operations. As mentioned above — submissions and changes are always shown to the user in a preview state first.
Together, these turned “occasionally confidently wrong” into “reliably grounded,” even on a much smaller model than you’d normally reach for.

Managing the Server & Auto-Switching Between LLMs
In production, a single model endpoint will occasionally be busy, rate-limited, or slow to respond — especially when you’re self-hosting an 8B model instead of paying for elastic cloud capacity. Rather than let a busy model stall the whole chat, I built a small LLM router with automatic failover: if the primary model is unavailable or times out, the next request transparently goes to a backup model.

Write on Medium
Here’s a simplified, copy-paste version of that router (Python):

python

import time
import random
import requests
from dataclasses import dataclass, field
from typing import Optional
@dataclass
class LLMEndpoint:
name: str
url: str
healthy: bool = True
last_failure_ts: float = 0
cooldown_seconds: int = 30 # how long to skip this endpoint after a failure
class LLMRouter:
"""
Routes a chat/completion request across multiple LLM endpoints.
If the current endpoint is busy/slow/erroring, it automatically
fails over to the next healthy endpoint in the pool.
"""
def init(self, endpoints: list[LLMEndpoint], timeout: int = 8, max_retries: int = 3):
self.endpoints = endpoints
self.timeout = timeout
self.max_retries = max_retries
self._rr_index = 0 # round-robin pointer
def _next_endpoint(self) -> Optional[LLMEndpoint]:
"""Pick the next healthy endpoint using round-robin, respecting cooldowns."""
now = time.time()
# Re-enable endpoints whose cooldown has expired
for ep in self.endpoints:
if not ep.healthy and (now - ep.last_failure_ts) > ep.cooldown_seconds:
ep.healthy = True
healthy = [ep for ep in self.endpoints if ep.healthy]
if not healthy:
return None
self._rr_index = (self._rr_index + 1) % len(healthy)
return healthy[self._rr_index]
def _mark_failed(self, endpoint: LLMEndpoint):
endpoint.healthy = False
endpoint.last_failure_ts = time.time()
def call(self, payload: dict) -> dict:
"""
Attempts the request against healthy endpoints, retrying with
exponential backoff and failing over to a different model each attempt.
"""
last_error = None
for attempt in range(self.max_retries):
endpoint = self._next_endpoint()
if endpoint is None:
raise RuntimeError("No healthy LLM endpoints available")
try:
response = requests.post(
endpoint.url,
json=payload,
timeout=self.timeout,
)
if response.status_code == 429 or response.status_code >= 500:
# Model is busy / overloaded / erroring -> fail over
raise requests.HTTPError(f"{endpoint.name} returned {response.status_code}")
return {"endpoint": endpoint.name, "data": response.json()}
except (requests.Timeout, requests.ConnectionError, requests.HTTPError) as e:
last_error = e
self._mark_failed(endpoint)
backoff = (2 ** attempt) + random.uniform(0, 0.5)
time.sleep(backoff)
continue
raise RuntimeError(f"All retries exhausted. Last error: {last_error}")

--- Example usage ---

if name == "main":
pool = [
LLMEndpoint(name="llm-primary", url="http://localhost:8001/v1/chat/completions"),
LLMEndpoint(name="llm-secondary", url="http://localhost:8002/v1/chat/completions"),
LLMEndpoint(name="llm-tertiary", url="http://localhost:8003/v1/chat/completions"),
]
router = LLMRouter(endpoints=pool, timeout=8, max_retries=3)
payload = {
"model": "your-8b-model",
"messages": [{"role": "user", "content": "Summarize the latest status."}],
}
result = router.call(payload)
print(f"Served by: {result['endpoint']}")
print(result["data"])
How this behaves in practice:

Each model instance is treated as an interchangeable endpoint in a pool.
If an endpoint returns a 429 (busy/rate-limited), a 5xx, times out, or drops the connection, it's marked unhealthy and skipped for a cooldown window (default 30s) instead of being hammered again immediately.
The next query automatically round-robins to a different, healthy endpoint — the user never sees the failure.
Exponential backoff with jitter prevents a thundering-herd retry storm if multiple endpoints are struggling at once.
A few production notes if you adapt this:

Put a lightweight health-check ping on a background thread (e.g., every 15s) so endpoints get marked healthy again proactively, not just lazily on the next request.
If you’re running multiple different model sizes (e.g., an 8B and a 13B) as your pool, keep prompts model-agnostic — avoid model-specific prompt tricks that only one of them understands.
Log which endpoint served each request. This alone will tell you if one node is consistently overloaded and needs more replicas.
What I’d Do Differently
Add per-agent evaluation sets earlier — I only started rigorously testing routing accuracy (Supervisor → correct agent) after I’d already built all three flows, and it would’ve caught misroutes sooner.
Cache catalog/category lookups with a short TTL — they don’t change often, and re-fetching on every single message added latency for no benefit.
Instrument hallucination specifically (not just “did the tool call succeed”) — track how often the model tried to propose a value that wasn’t in the last tool response, even if validation caught it. That signal is gold for prompt tuning.
Closing Thoughts
The headline lesson: you don’t need a giant model to build a reliable agentic system — you need tight scoping. A supervisor that only routes, workers that only own one job each, tools with strict schemas, and a hard rule that nothing gets written to a backend without being grounded in a fresh tool call. That combination got an 8B open-source model to perform a task reliably that I originally assumed would require a much larger, much more expensive model.

If you’re building something similar, happy to go deeper on any piece of this — routing logic, retrieval-grounding, or the failover setup — in the comments.

Building a Centralized Vector Database Pipeline for Multi-Project RAG

PAWAN YADAV (AI Engineer) — Fri, 10 Jul 2026 09:57:10 +0000

If you've worked on more than one Retrieval-Augmented Generation (RAG) project, you've probably run into the same annoying pattern: every project ends up with its own little vector store, its own embedding logic, its own duplicate code for chunking files, and its own inconsistent way of tracking what's actually been indexed.
I got tired of that, so I built a centralized vector database service that every project in my organization can plug into. One backend, one embedding model, one clean set of endpoints - and every project gets its own isolated, admin-controlled storage space.
Here's how it works.

The Problem

When you're running multiple RAG-powered projects at once, a few pain points show up fast:
Every team reinvents chunking and embedding logic
There's no single place to see what data lives where
Deleting or updating stored documents becomes a manual, error-prone process
Nobody has a clear picture of which files are indexed in which project

I wanted one system that solves this once, for everyone.
The Idea: One Service, Many Isolated Databases
Instead of giving each project its own standalone vector database, I built a shared service where:
Each project gets its own named storage directory
All directories live under one base storage path
An admin layer controls the full lifecycle - create, insert, update, delete, list - across every project's storage
Every project only ever talks to a small set of simple endpoints to embed and retrieve its own data

This means projects don't need to know anything about vector databases, embedding models, or similarity search internals. They just send text in and get relevant results back.
Architecture Overview

Here's the high-level flow:

In plain terms:
A project sends a file or text to the service
The service breaks it into clean text chunks
Those chunks get embedded and stored (upserted) into that project's own FAISS index
Later, the project sends a query, and the service returns the most similar stored chunks
An admin layer sits on top of all of this, with full control to create, update, or delete data across any project's storage

What's Under the Hood
Embedding model: a lightweight sentence-transformer model, chosen for speed and low resource usage
Vector engine: FAISS, for fast similarity search at scale
Backend framework: a modern Python web framework built for speed and simplicity
Storage layout: every project's database lives in its own folder, containing:
the FAISS index itself
a metadata/pickle file for the index
a tracked-files registry (so you always know what's indexed)
a source map (so you always know how many chunks came from each file)

Each project is fully sandboxed. Nothing bleeds across storage directories.
The Core Workflow
Every project follows the same simple lifecycle:

Create a storage directory A new, isolated space is created for the project. Invalid or unsafe directory names are automatically rejected.
Process a file into chunks Upload a document (PDF, TXT, DOCX, or Markdown) and it comes back as clean, ready-to-embed text chunks. This step doesn't store anything yet - it just prepares the data.
Upsert the chunks The chunks get embedded and stored. If the project's index doesn't exist yet, it's created automatically. If it does exist, new vectors are simply appended. Duplicate sources are automatically prevented.
Search A query comes in, gets embedded the same way, and the service returns the top-k most similar chunks - along with which source file they came from. A similarity score threshold filters out weak matches.
Manage the data At any point, an admin can: List every database in the system, with file counts and tracked sources Delete a specific file's vectors from a project's index Have all of this triggered externally through a single webhook event, so other systems can kick off any step of this pipeline programmatically

Why This Matters
The real win here isn't the vector search itself - FAISS and sentence-transformers are well-understood tools. The win is centralization with isolation:
One codebase to maintain instead of N duplicated ones
One admin view into every project's stored data
Every project gets embedding and retrieval "for free," without owning any of the infrastructure
Adding a brand-new project to RAG-powered search takes minutes: create a directory, start upserting

It turns "set up a vector database" from an infrastructure task into a two-endpoint integration for whoever's building the next project.
What's Next
A few directions I'm considering for this pipeline:
Authentication and per-project access scoping
Async batch processing for very large files
Pluggable embedding models per project (not just one global model)
A lightweight admin dashboard on top of the existing list/delete endpoints

If you're building multiple RAG projects and keep rebuilding the same vector storage logic every time, this pattern - one shared service, isolated storage per project, centralized admin control - is worth stealing.

Have you built something similar, or run into different tradeoffs with multi-project RAG infrastructure? I'd love to hear how you approached it.

Building a Self-Hosted RAG Chatbot with a Dual-Agent LLM Pipeline (and Automatic LLM Failover)

PAWAN YADAV (AI Engineer) — Fri, 19 Jun 2026 12:50:59 +0000

Over the past few weeks I built a Retrieval-Augmented Generation (RAG) chatbot from the ground up — one that answers strictly from a knowledge base I control, supports role-based access for regular users vs. admins, and never makes things up. In this post I want to walk through the architecture, the design decisions, and one piece that took real engineering effort: automatically switching to a backup LLM when the primary one is busy or rate-limited, so a query never just fails.

This isn’t a toy demo — it’s a full pipeline with authentication, a vector database, a two-stage reasoning process, and an admin console for managing the knowledge base. Here’s how it all fits together.

The Problem I Was Solving
Most “just call an LLM API” chatbots have two issues:

They hallucinate. Ask a generic LLM a domain-specific question and it will confidently make something up if it doesn’t actually know the answer.
They have a single point of failure. If your one LLM provider is rate-limited, down, or just slow at that moment, your whole app stalls.
I wanted a system where:

Every answer is grounded in real, embedded documents — no hallucination.
Two different roles (regular users and admins) get different capabilities.
If the LLM serving a request is unavailable, the system transparently retries on a different model instead of erroring out.
High-Level Architecture
The system has three user-facing surfaces and one processing core:

Signup / Identification Page — the user enters an email; the backend checks it against a list of approved admin addresses and routes accordingly.
User View — query-only. Regular users can ask questions but cannot touch the knowledge base.
Admin View — everything a user can do, plus the ability to upload documents/URLs, view what’s been embedded, and trigger re-indexing.
Here’s the high-level flow:

flowchart TD
A[Signup Page - Enter Email] --> B{Role Check}
B -->|Regular Email| C[User View - Query Only]
B -->|Admin Email| D[Admin View - Manage + Query]
C --> E[Query Pipeline]
D --> E
E --> F[Agent 1: Tool & Argument Selection]
F --> G[Execute Retrieval Tool]
G --> H[Raw Retrieved Chunks]
H --> I[Agent 2: Refinement]
I --> J[Final Answer Returned]
If your blogging platform doesn’t render Mermaid diagrams (Medium, for instance, won’t render this natively), here’s the same flow as plain text:

Signup Page
|
v
Role Check (User or Admin?)
|---------------------|
v v
User View Admin View
(Query Only) (Manage + Query)
|---------------------|
v
Query Pipeline
v
Agent 1: Tool & Argument Selection
v
Execute Retrieval Tool
v
Raw Retrieved Chunks
v
Agent 2: Refinement
v
Final Answer Returned

Why Two Agents Instead of One?
A single LLM call trying to both decide what to retrieve and how to phrase the final answer tends to produce messier output. So I split the reasoning into two sequential agents, orchestrated as a small graph of steps rather than one big prompt:

Agent 1 — Tool Decision Takes the raw user query and decides which retrieval tool to call and with what arguments. It returns a small, strict structure like:

[{"name": "retrieval_tool", "arguments": {"query": "your parsed query here"}}]
A lightweight Python handler parses this output, extracts the tool name and arguments, and manually executes the retrieval step — no need to trust the LLM to “just run the function,” which keeps things deterministic and debuggable.

Agent 2 — Refinement Takes whatever raw chunks came back from retrieval and turns them into a clear, well-formatted, professional answer. This is also the layer that keeps responses grounded — it’s instructed to work only with what was retrieved, not to add outside knowledge.

sequence Diagram

participant U as User/Admin
participant A1 as Agent 1 (Tool Decision)
participant T as Retrieval Tool
participant V as Vector Store
participant A2 as Agent 2 (Refinement)

U->>A1: Submits query
A1->>A1: Decide tool + arguments
A1->>T: Call retrieval tool
T->>V: Similarity search
V-->>T: Top-K relevant chunks
T-->>A2: Raw chunks + sources
A2->>A2: Refine into polished answer
A2-->>U: Final answer

The Retrieval Layer
Documents and URLs go through a standard but carefully tuned pipeline:

Loading — PDFs, Word docs, plain text, and Markdown files are parsed with format-specific loaders; web pages are scraped and cleaned.
Chunking — content is split into ~400-character chunks using a recursive character splitter, with each chunk tagged with its source (filename or URL) for traceability.
Embedding — each chunk is converted into a vector using an embedding model.
Storage — vectors go into a local vector index that supports fast similarity search and incremental updates (you don’t need to rebuild the whole index every time you add a new file).
Retrieval — when a query comes in, the index returns the top-K most similar chunks, filtered by a minimum similarity score threshold so irrelevant matches get dropped before they ever reach the LLM.

Two tunable parameters matter a lot here:

score_threshold — the minimum similarity score a chunk needs to be considered relevant. Raise it for stricter, more precise retrieval; lower it if the bot is being too conservative and saying "I don't know" too often.
k — how many top chunks get passed into the refinement agent. Too low and you risk missing context; too high and you risk diluting the answer with noise (and burning more tokens).
Admin Controls
Admins get a dedicated panel to manage the knowledge base directly:

Upload new files (PDF, Word, text, Markdown) for embedding.
Submit URLs to scrape and embed.
View every currently embedded source and a running count.
Run test queries against the live index.
Trigger a manual reload of the retriever so newly embedded content becomes searchable immediately, without restarting the whole service.
Regular users never see any of this — they only get a query box. Role detection is based on a simple, server-side check against a list of approved admin emails at signup time, and every admin-only route re-checks that role before doing anything destructive or data-modifying.

Now, the Interesting Part: Automatic LLM Failover
Here’s the piece I want to focus on, since it’s the part most tutorials skip. If you’ve ever called a single LLM provider in production, you’ve hit this: rate limits, timeouts, momentary outages, or a model that’s just slow to respond under load. If your whole app depends on one model, one bad moment takes everything down with it.

Become a Medium member
The fix is a failover-aware LLM router sitting in front of both agents. Instead of hardcoding “always call Model A,” every LLM call goes through a small dispatcher that:

Tries the primary model first.
If that call fails, times out, or comes back with a rate-limit / “server busy” type error, it automatically retries the same request on the next model in a prioritized list — no user-facing error, no manual restart.
Logs which model actually served the request (useful for debugging and for tracking cost/usage per provider).
Optionally applies a short backoff before retrying the original model on the next query, so it isn’t hammered immediately after a failure.
Press enter or click to view image in full size

How to actually manage this on the server
Practically, here’s the setup that works well:

Define a prioritized list of models, not a single model.

LLM_PRIORITY = [
{"name": "primary_llm", "provider": "provider_a", "model": "model-a-large"},
{"name": "secondary_llm", "provider": "provider_b", "model": "model-b-large"},
{"name": "fallback_llm", "provider": "provider_c", "model": "model-c-small"},
]
Order them by quality/cost first, reliability second. The first entry is your “ideal” answer quality; the rest exist purely so a request never just dies.

Wrap every call in a router function that catches specific failure types.

import time
def call_llm_with_failover(prompt, models=LLM_PRIORITY, max_retries_per_model=1):
last_error = None
for model_config in models:
for attempt in range(max_retries_per_model):
try:
response = call_provider(
provider=model_config["provider"],
model=model_config["model"],
prompt=prompt,
timeout=15 # seconds — don't let one model hang the whole pipeline
)
# Success — record which model actually answered
log_model_usage(model_config["name"])
return response
except RateLimitError:
last_error = "rate_limited"
break # don't retry the same rate-limited model, move to next one
except TimeoutError:
last_error = "timeout"
continue # maybe worth one retry on the same model
except ServerBusyError:
last_error = "busy"
break
raise AllModelsUnavailableError(f"All models exhausted. Last error: {last_error}")

Use this wrapper for both agents, not just one. Agent 1 (tool decision) and Agent 2 (refinement) should each go through the same failover router independently — it’s entirely possible for the model serving Agent 1 to be busy while the model serving Agent 2 is fine, or vice versa.
Add a circuit breaker so you don’t keep hammering a model that’s clearly down. A simple in-memory counter works for small-to-medium traffic:

from collections import defaultdict
from time import time
failure_counts = defaultdict(list)
COOLDOWN_SECONDS = 60
FAILURE_THRESHOLD = 3
def is_model_in_cooldown(model_name):
now = time()
recent_failures = [t for t in failure_counts[model_name] if now - t < COOLDOWN_SECONDS]
failure_counts[model_name] = recent_failures
return len(recent_failures) >= FAILURE_THRESHOLD
def record_failure(model_name):
failure_counts[model_name].append(time())
Check is_model_in_cooldown() before attempting a model in the priority list — if it's in cooldown, skip straight to the next one instead of wasting a request and a timeout window on a model you already know is struggling.

Make the model list configurable, not hardcoded. Keep it in an environment variable or a small config file so you can reorder priority, swap providers, or add a new model without touching code:

.env

LLM_PRIORITY_ORDER=primary_llm,secondary_llm,fallback_llm
PRIMARY_LLM_PROVIDER=provider_a
SECONDARY_LLM_PROVIDER=provider_b
FALLBACK_LLM_PROVIDER=provider_c
REQUEST_TIMEOUT_SECONDS=15
COOLDOWN_SECONDS=60
This is the difference between “I have to redeploy to change providers” and “I edit one line in .env and restart the service."

Why this matters in production
With this in place, a query never just hangs or errors out because one provider happened to be under load at that exact second. The user experience stays consistent — they get an answer, possibly from a slightly different underlying model, but they’re never staring at a spinner that times out. And because every fallback event is logged, you get visibility into how often your primary model is actually struggling, which is useful data for deciding whether to renegotiate rate limits, add a fourth fallback, or just accept the current setup.

Backend: Python web framework with async support, SQL-backed database for user/role data, secure session handling, password hashing for any stored credentials.
Agent orchestration: a graph-based orchestration library to define the two-agent sequence cleanly, with a tool/document-processing framework underneath.
Embeddings & retrieval: open embedding models for vectorization, a local vector index for similarity search, recursive text chunking, and a set of format-specific document loaders (PDF, Word, text, Markdown, web pages).
Frontend: plain HTML/CSS/JS with a utility-first CSS framework, server-rendered templates.
Backups: periodic backup of the embedding store to cloud object storage.
Concurrency: thread pools and async I/O so embedding jobs and queries don’t block each other.
What’s Next
A few things on the roadmap:

Letting admins delete individual files or specific vectors from the index, instead of only adding to it.
Tightening retrieval accuracy by experimenting with chunk size and embedding model choice.
Possibly collapsing the two-agent pipeline into a single, faster agent once latency becomes a bigger priority than the clean separation of concerns.
Expanding the failover router to support weighted load balancing across models even when all of them are healthy, not just failover when one is down.
Closing Thoughts
The biggest lesson from this project: a RAG chatbot is “easy” to build to a demo-quality bar, and genuinely hard to build to a doesn’t-fall-over-in-production bar. The retrieval and refinement pipeline gets you accurate, grounded answers. The failover router is what keeps the lights on when your LLM provider has a bad five minutes. Both matter — and most tutorials only show you the first one.

If you’re building something similar, start with the dual-agent retrieval pipeline to get accuracy right, then treat your LLM calls as a resource pool rather than a single dependency from day one. Retrofitting failover later is a lot more painful than designing for it up front.

Prompt-Driven Tool-Calling for Lightweight Open Source LLMs

PAWAN YADAV (AI Engineer) — Fri, 19 Jun 2026 09:36:01 +0000

🚀 How Lightweight LLMs Can Use Tools Without Large Compute: A Prompt-Driven Tool-Calling Approach

AI #LLM #MachineLearning #AIAgents #PromptEngineering #OpenSourceAI

🚀 Introduction

Large Language Models (LLMs) like GPT-4 or Claude are extremely powerful, but they come with a major limitation:
they require huge computational resources.

But what if smaller, open-source models could also perform complex reasoning tasks—without needing massive GPUs?

This question led to my research:

“Prompt-Driven Tool-Calling for Lightweight Open Source LLMs”

🧠 The Problem

Today’s AI systems face three key challenges:

Small models lack strong reasoning ability
Tool usage (calculators, APIs, search engines) is not native
Large models are expensive and difficult to deploy everywhere

So the gap is clear:

👉 We need efficient AI agents that don’t rely on large models

⚙️ The Idea: Prompt-Driven Tool Calling

Instead of forcing a model to “learn everything,” we guide it using structured prompts that allow it to:

Decide when to use a tool
Select the correct tool
Combine outputs from multiple steps
In simple terms:

The model becomes a controller, not a knowledge storage system.

🔧 How It Works

This system enables lightweight LLMs to:

Understand user intent

The prompt helps the model break the problem into steps.

Decide tool usage

Instead of answering directly, the model selects tools such as:

Calculator
Search engine
API call
External functions

Execute multi-step reasoning

Flow:

User Question → LLM → Tool Selection → Tool Execution → Final Answer

💡 Key Benefits

This approach enables:

✅ Smaller models to behave like intelligent agents
✅ Reduced dependency on large proprietary LLMs
✅ Lower compute cost
✅ Deployment on CPUs and edge devices
✅ More practical real-world AI systems
🌍 Why This Matters

We are moving toward a future where:

Intelligence is not about model size, but about system design.

Instead of scaling parameters, we scale:

Tool integration
Reasoning workflows
System-level intelligence

This makes AI:

More accessible
More affordable
More deployable in real-world environments
📊 My Research Contribution

This work proposes a prompt-driven framework that:

Enables tool-calling in lightweight open-source LLMs
Improves multi-step reasoning capability
Reduces dependency on large models
Moves toward practical AI agent systems
📄 Publication Details

📌 Published in: AIS2C2 2025
📚 Pages: 493–497

🔗 Paper Link:
https://www.aiscindia.co.in/wp-content/uploads/2026/06/ilovepdf_merged-4.pdf

🚀 Final Thoughts

The future of AI is not just about building bigger models.

It is about building smarter systems around smaller models.

Prompt-driven tool-calling is one step toward that direction.

🤝 Let’s Connect

I’m always open to discussions around:

LLMs and AI agents
Tool-calling systems
Open-source AI development
Practical AI engineering
🔖 Tags

ArtificialIntelligence #LLM #AIAgents #PromptEngineering #MachineLearning #DeepLearning #OpenSourceAI