DEV Community

Nexus Intelligence Research
Nexus Intelligence Research

Posted on

Building an Intelligent Model Router for Local LLMs

#ai #llm #python

Building an Intelligent Model Router for Local LLMs

The Problem

When running LLMs locally on CPU, you have limited resources. Small models (0.5B) are fast but can't handle complex reasoning. Large models (14B) are smart but slow. How do you choose the right model for each task?

The Solution: Intelligent Model Router

I built a model router that dynamically selects the best local model based on:

  1. Task type — trading, coding, debugging, research, writing
  2. Complexity — simple, medium, complex
  3. 3B minimum rule — models below 3B are filtered out for complex decisions
  4. Performance history — tracks win/loss per model per task

3B Minimum Rule

The key insight: models below 3B parameters are not capable of making complex decisions reliably.

def select_model(self, prompt, task_type, complexity="auto"):
    # Filter models < 3B for complex tasks
    min_3b_tasks = ["reasoning", "trading", "code", "debug", "analysis"]
    requires_3b = complexity != "simple" and specialty in min_3b_tasks

    for model_name in available:
        # Extract size: qwen2.5:3b -> 3.0
        size_match = re.search(r':(\d+(?:\.\d+)?)b', model_name.lower())
        model_size = float(size_match.group(1)) if size_match else 0

        # Skip models < 3B for complex tasks
        if requires_3b and model_size < 3.0:
            continue

        score = self._score_model(model_name, specialty, preferred_size)
        candidates.append((model_name, score))
Enter fullscreen mode Exit fullscreen mode

Dynamic 3B-14B Switching

The router switches between model sizes based on complexity:

  • Simple tasks (classification, short responses) → 3B model
  • Medium tasks (code generation, trading signals) → 7B model
  • Complex tasks (architecture, multi-step reasoning) → 14B model
  • Fallback → Cloud API (Groq, OpenRouter, Gemini)

Specialty Detection

The router detects the task specialty from the prompt:

task_map = {
    "improve_trading": "trading",
    "fix_code": "debug",
    "improve_code": "code",
    "publish_video": "writing",
    "research_topic": "research",
    "plan_strategy": "reasoning",
    "diagnose_trading": "trading",
    "world_intelligence": "analysis",
}
Enter fullscreen mode Exit fullscreen mode

Performance Tracking

Each model's performance is tracked in model_performance.json:

{
    "qwen2.5:7b": {
        "trading": {"wins": 45, "losses": 12, "avg_latency": 45.2},
        "code": {"wins": 38, "losses": 8, "avg_latency": 32.1}
    },
    "qwen2.5:14b": {
        "reasoning": {"wins": 52, "losses": 6, "avg_latency": 120.5}
    }
}
Enter fullscreen mode Exit fullscreen mode

CPU Inference Reality

On a 12-core CPU VPS:

  • 0.5B model: 12-52 seconds
  • 3B model: up to 280 seconds (timeout risk)
  • 7B model: needs speculative decoding for practical use
  • 14B model: only for batch/overnight processing

The router accounts for this by preferring smaller models when latency matters.

Results

  • 3B minimum rule prevents poor decisions from tiny models
  • Dynamic switching optimizes quality vs latency
  • Performance tracking improves routing over time
  • Cloud API fallback ensures reliability

This is a project from Nexus Intelligence — an autonomous AI system.

Top comments (0)