Learn Provider-Agnostic Model Routing by Building a Tiny LLM Switchboard

Every few weeks a new model drops and my study group chat fills up with screenshots: "this one is cheaper," "this one is better at code," "switch now." I can never verify any of it quickly, because my test scripts all hard-code one provider's client. Rewriting the harness is slower than the hype cycle.

So here is the learning question: can a ~60-line, standard-library-only Python switchboard let me swap providers behind one interface, route toy tasks to different backends, and prove with a failing fixture where the routing breaks?

Run the final script and you should see:

[route=summarize] backend=local-echo   cost=0.0000  out='SUMMARY: ...'
[route=code]       backend=mock-strong cost=0.0020  out='def add(a, b): ...'
[route=summarize] backend=mock-strong cost=0.0020  out='SUMMARY: ...'  (fallback fired)

The third line is the interesting one — keep reading.

Prerequisites

• Python 3.11+ (tested on 3.12.4, macOS and WSL2 Ubuntu)
• No third-party packages. No API keys needed for the core experiment.

Why a switchboard, not a wrapper

A wrapper hides one provider's quirks. A switchboard does something different: it defines your task types (summarize, code, chat) and maps each to a backend plus a fallback. When a new model appears — whatever it is called this month — you add one entry and re-run the same fixtures. Your evaluation questions stay fixed while backends rotate under them.

That is the actual skill: separating what you ask from who answers.

The complete code

Save as switchboard.py:

"""Tiny provider-agnostic LLM switchboard (stdlib only)."""
from dataclasses import dataclass, field
from typing import Callable

# --- Backends: same signature (str) -> str, so they are interchangeable ---

def local_echo(prompt: str) -> str:
    """Free 'backend': deterministic, offline, good enough for smoke tests."""
    return f"SUMMARY: {prompt[:40]}"

def mock_strong(prompt: str) -> str:
    """Pretend paid model. Fails on empty input, like a real API rejects it."""
    if not prompt.strip():
        raise ValueError("backend rejected empty prompt")
    if prompt.startswith("write code"):
        return "def add(a, b):\n    return a + b"
    return f"SUMMARY: {prompt[:40]}"

@dataclass
class Backend:
    name: str
    fn: Callable[[str], str]
    cost_per_call: float  # USD, your own pricing notes go here

@dataclass
class Switchboard:
    routes: dict[str, list[Backend]] = field(default_factory=dict)
    spent: float = 0.0

    def register(self, task: str, backends: list[Backend]) -> None:
        self.routes[task] = backends

    def run(self, task: str, prompt: str) -> str:
        if task not in self.routes:
            raise KeyError(f"no route registered for task '{task}'")
        last_err = None
        for backend in self.routes[task]:
            try:
                out = backend.fn(prompt)
                self.spent += backend.cost_per_call
                print(f"[route={task}] backend={backend.name:<11} "
                      f"cost={backend.cost_per_call:.4f}  out={out.splitlines()[0]!r}")
                return out
            except Exception as e:  # try the fallback backend
                last_err = e
        raise RuntimeError(f"all backends failed for '{task}'") from last_err

if __name__ == "__main__":
    free = Backend("local-echo", local_echo, 0.0)
    paid = Backend("mock-strong", mock_strong, 0.002)

    sb = Switchboard()
    sb.register("summarize", [free, paid])   # cheap first, paid as fallback
    sb.register("code", [paid])              # only the 'strong' backend

    sb.run("summarize", "explain gradient descent to a first-year student")
    sb.run("code", "write code to add two numbers")
    sb.run("summarize", "   ")               # free backend accepts junk... or does it?
    print(f"total spent: ${sb.spent:.4f}")

Expected output

[route=summarize] backend=local-echo   cost=0.0000  out='SUMMARY: explain gradient descent to a f'
[route=code]       backend=mock-strong cost=0.0020  out='def add(a, b):'
[route=summarize] backend=local-echo   cost=0.0000  out='SUMMARY:     '
total spent: $0.0020

Wait — the third line did not fall back, and it returned a garbage summary of whitespace. My local_echo backend happily accepts an empty prompt. The failure I promised in the intro only fires if the primary backend raises. Before reading on: which fixture input would force the fallback line from the intro to appear? (Answer at the bottom.)

The error input that teaches the lesson

Swap the registration so the strict backend is primary:

sb.register("summarize", [paid, free])  # strict first, lenient as fallback
sb.run("summarize", "   ")

Now you get the intro's third line: mock-strong raises on the empty prompt, the switchboard catches it, and local-echo answers instead. The concept that actually matters: fallback order is a policy decision, and "free first" and "strict first" fail in opposite directions. Free-first silently returns junk; strict-first silently spends money when you expected the cheap path.

Common mistakes

1. Catching nothing or everything blindly. Catching Exception is fine for a teaching harness, but in real code you should distinguish "provider is down" (retry/fallback) from "my prompt was rejected" (fallback just re-fails expensively).
2. Comparing models on different prompts. If backend A sees a cleaned prompt and backend B sees the raw one, your cost/quality notes are meaningless. The switchboard passes the identical string to every backend — keep it that way.
3. Trusting launch-week pricing claims. When a new model is announced as "cheap and strong," verify pricing on the provider's own page before wiring it in as your default route. Announcement-week numbers change, and context-window limits quietly alter real cost per task.

Where I actually run this

I do most of these experiments inside MonkeyCode, since its free model access lets me prototype against a real LLM endpoint instead of only mocks, and the free server option means the harness runs somewhere that is not my laptop between classes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The switchboard pattern above is deliberately provider-agnostic, though — the same file works against any endpoint, including none at all.

Limitations and who should skip this

• This is a learning artifact, not production infrastructure: no retries with backoff, no rate-limit handling, no async, no token-level cost accounting (cost here is per-call fiction you fill in yourself).
• If you already use a mature router/gateway library, this teaches you nothing new operationally — though building it once is still useful for understanding what that library hides.
• Do not use fallback chains to mask evaluation problems. "It returned something" is not a quality signal.

Extension exercise

Add a third route, "chat", and a fixture file of five prompts where you predict in writing which backend should handle each one. Then make mock_strong randomly raise on 30% of calls (seed random for reproducibility) and check whether your predictions about cost still hold. If your cost estimate assumed zero failures, what does that tell you about launch-week pricing comparisons?

Answer to the question above

With summarize registered as [free, paid], only an input that makes local_echo itself raise would trigger the fallback — and local_echo never raises. The intro's third line only appears under the strict-first registration. If you predicted that, you understood the policy-ordering point; if not, run both versions and diff the output.

If you find a fixture input where the fallback makes things worse (e.g., the lenient backend returns something dangerously plausible), post it — minimal counterexamples are the best part of these threads.