How I designed a conversational AI system that lets end users perform structured backend operations in plain English — using a supervisor-agent architecture, tool-calling, and a lightweight 8B open-source model instead of a giant proprietary one.
The Problem
Most enterprise backend systems expose their functionality through dense, form-heavy portals. Doing something simple usually means:
Logging into a portal
Hunting through nested dropdowns and categories
Re-typing boilerplate information every time
Coming back later just to check status or add a note
Most users just want to describe what they need in plain language and be done with it.
So I built a conversational AI layer on top of an existing backend system that lets users perform three categories of operations — submitting new requests, modifying existing ones, and looking things up — entirely through natural language, in chat.
This post covers three things people keep asking me about:
How the multi-agent routing actually works
How I got reliable results out of a small 8B model instead of a large frontier model
How I reduced hallucination in a workflow where wrong answers write real records into a real system
Why Multi-Agent Instead of One Big Prompt
My first instinct was “just write one giant system prompt that handles everything.” That fell apart fast:
The prompt became enormous and slow to reason over
The model would confuse one type of operation with another
Tool-calling accuracy dropped as the number of available tools grew
Debugging why something went wrong was nearly impossible — one flow’s logic bled into another’s
The fix was to split the system into a Supervisor Agent and several specialized Worker Agents, each with a narrow, well-defined job and its own small toolset. This is the same pattern used in most production multi-agent systems: a router model decides what the user wants, and a specialist model decides how to do it.
Architecture
If you want a rendered version instead of ASCII, here’s the same flow as Mermaid (renders natively on Dev.to, GitHub, and most markdown viewers — for Medium you’d paste this into a Mermaid live editor and drop in the exported image):
How the Supervisor Routes Requests
The supervisor doesn’t call any backend tools itself — its only job is intent classification + delegation + context tracking. That single-responsibility design is what makes it reliable even on a small model: it’s not trying to reason about categories, records, and filters all at once, it’s just answering “which of 3 buckets does this go in?”
Rough logic:
Signal in user messageRoutes toWants to submit something newAgent AlphaWants to change or add to something that already existsAgent BetaWants to look up or review existing recordsAgent GammaUnclear / multiple intents in one messageSupervisor asks a clarifying question first
Each worker agent then owns its own multi-step flow. For example, Agent Alpha walks the user through: pick a category → pick a sub-category → describe what’s needed → get an AI-suggested resolution → confirm → record submitted → confirmation shown. Agent Beta and Agent Gamma follow the same “narrow tool, narrow job” philosophy, just with different backend calls (fetch existing items, apply a change, query by a filter or time window, etc.).
Doing This on a Small 8B Model (Not GPT-4-class)
The part people are most surprised by: this whole system runs on an open-source 8B parameter model, not a large frontier model. A few techniques made that viable:
Narrow the job per agent. An 8B model struggles to hold a large catalog of categories, dozens of tools, and multiple conversation flows in its head at once. Split across several agents, each model call only needs to reason about a handful of tools and a tightly scoped instruction set — well within an 8B model’s comfort zone.
Constrain tool schemas aggressively. Every tool has a strict JSON schema with enums where possible, instead of free-text parameters. Small models are much better at picking from a constrained set than generating an unconstrained one.
Force structured output, not free prose, for anything that feeds a tool call. The model never “writes” an identifier or category name from memory — it’s only ever allowed to select values that came from a prior tool response (e.g., the actual list returned by a catalog lookup). This alone eliminated most of the invented-value problem.
Multi-turn confirmation instead of one-shot execution. Nothing destructive (submitting a new record, applying a change) happens on the first pass. The flow always shows the user what it’s about to do and waits for confirmation. This isn’t just good UX — it’s a hallucination safety net: if the model got a field wrong, the user catches it before it’s written to the backend.
Overcoming Hallucination
This was the hardest part, because a hallucinated output isn’t just a wrong answer in a chat window — it’s a bad record in a real backend system. Here’s what actually moved the needle:
Ground every list in a live tool call, never in the model’s memory. Category names, sub-category names, and record identifiers are always fetched fresh from the backend right before they’re shown to the user. The model is never asked to recall or generate them.
Retrieval-augmented suggestions before action, not instead of grounding. Before a new record is submitted (or a change applied), a retrieval-augmented step generates a suggested resolution based on the user’s actual described need and a knowledge base — not the model’s raw training data. This both reduces hallucinated “answers” and often resolves the need without any record being written at all.
One category per submission, explicit selection only. The model can’t infer or guess a category — the user must explicitly pick from a list that was just returned by a tool. This removes an entire class of “close enough” hallucinations.
Validation layer between model output and tool execution. Every tool call the model proposes is validated against the tool’s schema and the last known-good state (e.g., “is this identifier actually one of the records we just fetched for this user?”) before it’s executed. If validation fails, the flow re-prompts instead of silently proceeding.
Confirm-before-commit on all write operations. As mentioned above — submissions and changes are always shown to the user in a preview state first.
Together, these turned “occasionally confidently wrong” into “reliably grounded,” even on a much smaller model than you’d normally reach for.
Managing the Server & Auto-Switching Between LLMs
In production, a single model endpoint will occasionally be busy, rate-limited, or slow to respond — especially when you’re self-hosting an 8B model instead of paying for elastic cloud capacity. Rather than let a busy model stall the whole chat, I built a small LLM router with automatic failover: if the primary model is unavailable or times out, the next request transparently goes to a backup model.
Write on Medium
Here’s a simplified, copy-paste version of that router (Python):
python
import time
import random
import requests
from dataclasses import dataclass, field
from typing import Optional
@dataclass
class LLMEndpoint:
name: str
url: str
healthy: bool = True
last_failure_ts: float = 0
cooldown_seconds: int = 30 # how long to skip this endpoint after a failure
class LLMRouter:
"""
Routes a chat/completion request across multiple LLM endpoints.
If the current endpoint is busy/slow/erroring, it automatically
fails over to the next healthy endpoint in the pool.
"""
def init(self, endpoints: list[LLMEndpoint], timeout: int = 8, max_retries: int = 3):
self.endpoints = endpoints
self.timeout = timeout
self.max_retries = max_retries
self._rr_index = 0 # round-robin pointer
def _next_endpoint(self) -> Optional[LLMEndpoint]:
"""Pick the next healthy endpoint using round-robin, respecting cooldowns."""
now = time.time()
# Re-enable endpoints whose cooldown has expired
for ep in self.endpoints:
if not ep.healthy and (now - ep.last_failure_ts) > ep.cooldown_seconds:
ep.healthy = True
healthy = [ep for ep in self.endpoints if ep.healthy]
if not healthy:
return None
self._rr_index = (self._rr_index + 1) % len(healthy)
return healthy[self._rr_index]
def _mark_failed(self, endpoint: LLMEndpoint):
endpoint.healthy = False
endpoint.last_failure_ts = time.time()
def call(self, payload: dict) -> dict:
"""
Attempts the request against healthy endpoints, retrying with
exponential backoff and failing over to a different model each attempt.
"""
last_error = None
for attempt in range(self.max_retries):
endpoint = self._next_endpoint()
if endpoint is None:
raise RuntimeError("No healthy LLM endpoints available")
try:
response = requests.post(
endpoint.url,
json=payload,
timeout=self.timeout,
)
if response.status_code == 429 or response.status_code >= 500:
# Model is busy / overloaded / erroring -> fail over
raise requests.HTTPError(f"{endpoint.name} returned {response.status_code}")
return {"endpoint": endpoint.name, "data": response.json()}
except (requests.Timeout, requests.ConnectionError, requests.HTTPError) as e:
last_error = e
self._mark_failed(endpoint)
backoff = (2 ** attempt) + random.uniform(0, 0.5)
time.sleep(backoff)
continue
raise RuntimeError(f"All retries exhausted. Last error: {last_error}")
--- Example usage ---
if name == "main":
pool = [
LLMEndpoint(name="llm-primary", url="http://localhost:8001/v1/chat/completions"),
LLMEndpoint(name="llm-secondary", url="http://localhost:8002/v1/chat/completions"),
LLMEndpoint(name="llm-tertiary", url="http://localhost:8003/v1/chat/completions"),
]
router = LLMRouter(endpoints=pool, timeout=8, max_retries=3)
payload = {
"model": "your-8b-model",
"messages": [{"role": "user", "content": "Summarize the latest status."}],
}
result = router.call(payload)
print(f"Served by: {result['endpoint']}")
print(result["data"])
How this behaves in practice:
Each model instance is treated as an interchangeable endpoint in a pool.
If an endpoint returns a 429 (busy/rate-limited), a 5xx, times out, or drops the connection, it's marked unhealthy and skipped for a cooldown window (default 30s) instead of being hammered again immediately.
The next query automatically round-robins to a different, healthy endpoint — the user never sees the failure.
Exponential backoff with jitter prevents a thundering-herd retry storm if multiple endpoints are struggling at once.
A few production notes if you adapt this:
Put a lightweight health-check ping on a background thread (e.g., every 15s) so endpoints get marked healthy again proactively, not just lazily on the next request.
If you’re running multiple different model sizes (e.g., an 8B and a 13B) as your pool, keep prompts model-agnostic — avoid model-specific prompt tricks that only one of them understands.
Log which endpoint served each request. This alone will tell you if one node is consistently overloaded and needs more replicas.
What I’d Do Differently
Add per-agent evaluation sets earlier — I only started rigorously testing routing accuracy (Supervisor → correct agent) after I’d already built all three flows, and it would’ve caught misroutes sooner.
Cache catalog/category lookups with a short TTL — they don’t change often, and re-fetching on every single message added latency for no benefit.
Instrument hallucination specifically (not just “did the tool call succeed”) — track how often the model tried to propose a value that wasn’t in the last tool response, even if validation caught it. That signal is gold for prompt tuning.
Closing Thoughts
The headline lesson: you don’t need a giant model to build a reliable agentic system — you need tight scoping. A supervisor that only routes, workers that only own one job each, tools with strict schemas, and a hard rule that nothing gets written to a backend without being grounded in a fresh tool call. That combination got an 8B open-source model to perform a task reliably that I originally assumed would require a much larger, much more expensive model.
If you’re building something similar, happy to go deeper on any piece of this — routing logic, retrieval-grounding, or the failover setup — in the comments.
Top comments (0)