AutoRoute: Building an Agent System That Fires Its Own Employees

How I built a multi-agent system where the agents grade their own work, quit when they're not good enough, and hand off to someone stronger — all visible in a live trace viewer.

---

The Problem With Multi-Agent Systems

Every multi-agent framework ships with the same happy-path demo: a router looks at a task, picks the perfect agent, and everything works. In production it never works like that:

• You don't know which model a task actually needs until it runs.
• Routing rules you hardcode today become wrong tomorrow (models get better, workloads shift).
• Nobody checks the quality of what came out — the agent just reports "Done!" and moves on.

AutoRoute is a different bet: let the agents judge their own output, let them quit when they're not good enough, and let them hand off to the agent they believe is better. It's not a smarter router — it's self-correcting execution.

---

The Core Idea

A single entry-point agent receives a task. It completes it, then evaluates its own output against a rubric using a separate "judge" model. Based on that self-score it:

1. Accepts the output and returns it, or
2. Fires itself and hands off to a stronger/more appropriate agent, or
3. Exhausts its budget and gracefully returns the best output seen so far.

Everything runs inside the Agno framework — Team for orchestration, tool functions for behavior, and the AgentOS Playground for a zero-code trace viewer.

---

Architecture

┌───────────────┐
│  USER TASK    │
└───────┬───────┘
        ▼
┌────────────────────────────────────────────┐
│  ROUTER (Team Leader, cheap model)         │
│  classify task → pick starting agent      │
└───────┬────────────────────────────────────┘
        ▼  delegate
┌────────────────────────────────────────────┐
│  WORKER AGENT  ── runs task ──▶ output      │
└───────┬────────────────────────────────────┘
        ▼  self_evaluate(task, output)
┌────────────────────────────────────────────┐
│  JUDGE LLM  ──▶ scores 0.0–1.0             │
│  accuracy(.3) completeness(.25)            │
│  confidence(.2) task_fit(.25)              │
└───────┬────────────────────────────────────┘
        │
   score ≥ threshold          score < threshold
        │                            │
   ACCEPT & return        budget left? ──yes──▶ fire_and_handoff() → router re-delegates
        │                            │
                                  no budget
                                     │
                              return best output so far

autoroute/
├── app.py            # FastAPI + AgentOS entry point (Playground UI)
├── app_ui.py         # Self-contained Streamlit chat UI
├── config.yaml       # Provider + routing config
├── config_loader.py  # YAML + ${ENV_VAR} substitution
├── model_factory.py  # provider:model → Agno Model instance
├── team.py           # Team assembly (router + workers)
├── agents/
│   ├── profiles.py   # AGENT_PROFILES roster
│   └── worker.py     # Worker agent template factory
└── tools/
    ├── self_evaluate.py   # The judge
    └── handoff.py         # The firing mechanism

Key Design Decisions

1. Behavior lives in tools, not in agents

Every worker (from fast_drafter to creative_specialist) is the same agent template — the only things that differ are the model and metadata. The "quit when bad" behavior is injected through two tool functions:

# agents/worker.py
def make_worker_agent(profile_name: str, providers: Dict[str, Any]) -> Agent:
    profile = AGENT_PROFILES[profile_name]
    model = resolve_model_string(profile["model"], providers)
    return Agent(
        name=profile_name,
        id=profile_name,
        model=model,
        tools=[self_evaluate, fire_and_handoff],   # <-- the magic lives here
        description=f"Worker agent: {profile_name}. {profile['strength']}",
        markdown=True,
    )

Add a new agent and it automatically gets self-evaluation and self-firing. That's the whole point of the abstraction — new archetypes are data, not code.

2. The judge is a separate, cheap LLM

The agent evaluating itself is a recipe for flattery. So self_evaluate() hands the output to a separate judge model configured independently (eval_model_provider: openai, eval_model_id: gpt-4o-mini), with a deliberately harsh prompt:

"Be ruthless. 0.7 means 'good enough'. 0.9 means excellent. Do NOT inflate scores."

The judge scores four dimensions and returns structured JSON via a Pydantic model:

class EvaluationResult(BaseModel):
    task_id: str = ""
    agent_id: str = ""
    scores: Dict[str, float]   # accuracy, completeness, confidence, task_fit
    overall_score: float = 0.0       # weighted 0.3 / 0.25 / 0.2 / 0.25
    threshold_met: bool = False
    firing_reason: str = ""
    recommended_agent: str = ""
    handoff_notes: str = ""

If the judge is unreachable, a deterministic heuristic takes over (output length → completeness/confidence, static task-fit per profile) — the system degrades, it doesn't crash.

# tools/self_evaluate.py
response = eval_llm.response(prompt)
result_text = response.content if hasattr(response, "content") else response
json_match = re.search(r"\{.*\}", result_text, re.DOTALL)
result_dict = json.loads(json_match.group())

evaluation = EvaluationResult(
    agent_id=agent_id,
    scores=result_dict.get("scores", {}),
    overall_score=result_dict.get("overall_score", 0.0),
    threshold_met=result_dict.get("threshold_met", False),
    recommended_agent=result_dict.get("recommended_agent", agent_id),
    handoff_notes=result_dict.get("handoff_notes", ""),
)
state["evaluations"].append(evaluation.model_dump())   # visible in the trace

3. The firing mechanism writes to shared state

When a score misses the threshold, the worker calls fire_and_handoff(). It's not a control-flow trick — it's a tool call the router can observe, and it updates the team's session_state so the router knows exactly what to do next:

# tools/handoff.py
if handoff_count >= budget:
    return '{"status": "budget_exhausted", "action": "accept_best_output", ...}'

state["handoff_log"].append({
    "from_agent": firing_agent_id,
    "to_agent": recommended_agent,
    "handoff_count": handoff_count + 1,
    "notes": handoff_notes,
})
state["handoff_count"] = handoff_count + 1
state["next_agent"] = recommended_agent
state["best_output"] = best_output_so_far

The budget check double-guarantees termination: even if the router ignores the meta-note, the tool itself refuses once the budget is spent.

4. The router reads the smoke signals

The router (a cheap classification model) stays in the loop. After a handoff it reads the firing note — e.g. [ Agent: deep_thinker | Score: 0.89 | Handoffs: 2 | Status: ACCEPTED ] — re-delegates to the recommended agent with full context, and finally assembles a human-readable summary:

---
**AutoRoute Summary**
- Task: Explain tradeoffs between RAG and fine-tuning...
- Agents Used: fast_drafter → balanced_worker → deep_thinker
- Total Handoffs: 2
- Final Agent: deep_thinker
- Final Score: 0.89
- Status: ACCEPTED
---

---

What a Hard Task Actually Looks Like

Ask AutoRoute: "Explain the tradeoffs between RAG and fine-tuning for a production LLM system with 10M daily queries."

1. Router classifies as analytical, delegates to fast_drafter (cheapest, fastest).
2. fast_drafter writes a shallow answer and self-scores 0.51 — below the 0.75 threshold.
3. It fires itself, recommends balanced_worker, notes what's missing.
4. balanced_worker adds depth but self-scores 0.68. Fires again.
5. deep_thinker produces a nuanced answer, self-scores 0.89 → accepts.
6. Router returns it with a two-handoff summary.

Every one of those decisions is a visible tool call in the Playground trace — you can watch the model talk itself out of its own confidence. There's something genuinely fun about watching an AI decide to quit.

---

Multi-Provider by Design

No provider lock-in. model_factory.py maps provider:model strings to Agno models, so the roster can mix OpenAI, Anthropic, Google, Ollama, and LM Studio in one team:

# model_factory.py
if provider in ("openai",):
    from agno.models.openai import OpenAIChat
    return OpenAIChat(id=model_id, api_key=api_key, base_url=base_url)

elif provider in ("anthropic",):
    from agno.models.anthropic import Claude
    return Claude(id=model_id, api_key=api_key)

elif provider in ("ollama",):
    from agno.models.ollama import Ollama
    return Ollama(id=model_id, host=host, api_key=api_key)

The cost tiers on each profile (cost_tier, speed_tier) are exactly what the router uses to decide who starts — cheap and fast first, escalation only when the judge says so.

---

Two UIs, Zero Custom Web Work

AutoRoute serves the same engine through two frontends, neither hand-written:

• Agno Playground (python app.py) — AgentOS auto-wraps the team in a FastAPI app with streaming chat, session history, and a trace viewer that exposes the firing events as tool calls.
• Streamlit UI (streamlit run app_ui.py) — a self-contained dashboard with live evaluation cards, an animated handoff log, and demo-task buttons.

# app.py — the whole web layer is ~60 lines
agent_os = AgentOS(
    name="AutoRoute",
    teams=[team],
    db=SqliteDb(id="autoroute-db", db_file=db_path),
)
app = agent_os.get_app()     # ready-to-serve FastAPI app
agent_os.serve(app="app:app", host=host, port=port, reload=True)

That's the entire "backend": a team object and a database handle.

---

Lessons Learned

1. Optimal routing is unknowable in advance — quality depends on the task and the model's day-to-day mood. Runtime self-assessment beats static rules.
2. Self-scoring needs an outside judge — an agent grading its own work inflates scores. A separate cheap model fixes the incentive problem for pennies.
3. A forced-accept budget is non-negotiable — "quality at any cost" is how you get infinite loops and surprise bills. The budget is a hard guarantee, in the tool, not a hopeful instruction.
4. Traceability is a feature — when an agent changes its own mind, you need to show why. The Playground trace turns "trust me" into evidence.
5. Structure in tools, variety in data — one worker template + five profiles beats five bespoke agent classes. New archetypes are additions to a Python dict.

---

Try It / Extend It

git clone https://github.com/harishkotra/autoroute
cd autoroute
pip install -r requirements.txt
cp .env.example .env      # add your OPENAI_API_KEY / ANTHROPIC_API_KEY
python app.py             # then point app.agno.com at localhost:7777

Good next features: adaptive thresholds per task type, a multi-judge ensemble to catch overconfidence, cumulative cost/latency guardrails, code/math-focused worker archetypes, and a REST/CLI surface. Contributions welcome.

Code & more: https://www.dailybuild.xyz/project/227-autoroute