Zero-Cost Prompt Routing: Stop Sending Every Task to the Most Expensive Model

You pay for tokens you never needed.

Most LLM apps send every request to one big model. Summaries, keyword extraction, and simple classification all buy the same expensive context window. That is waste. A router can split your traffic before it reaches the model.

This article builds a small, practical router in about 300 lines. It runs on a free server and uses free model access for low-risk tasks. You will learn how to route by task type, log what you saved, and catch routing mistakes before they hit production.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that offers free models and a free server for small workloads. Those two resources are enough to run this router continuously. Quotas change, so check the project README for current limits before you deploy. The code below is endpoint-agnostic; you can also point it at any OpenAI-compatible API.

Why route at all

A routing layer makes a simple decision: which model should handle this prompt?

• Short classification → small model
• Long summarization → medium model
• Complex reasoning → large model

You save tokens on the easy paths. You also protect your expensive model from noisy, repetitive traffic.

What you will build

You will create router.py, a two-layer decision system:

1. Static rules decide the model from the task type and prompt length.
2. A fallback check retries on a stronger model when the first answer looks malformed.

The whole thing runs as a background job on a free server. No credit card required.

Decision table: when to use which model

Task type	Prompt length	Suggested tier	Reason
Classification	< 200 tokens	small	Labels have stable patterns
Named-entity extraction	< 300 tokens	small	JSON output is easy to validate
Summarization	< 1500 tokens	medium	Needs some reasoning about the text
Code explanation	any	medium	Error-prone if you use the small tier
Complex debugging	any	large	High cost, but fewer false saves

The table is a starting point. You will tune it after you see real latency and failure rates.

Step 1: Define your model tiers

Create config.json:

{
  "small": "monkeycode-fast-model",
  "medium": "monkeycode-balanced-model",
  "large": "monkeycode-strong-model",
  "endpoint": "https://your-monkeycode-endpoint/v1/chat/completions"
}

Keep the model names as placeholders. Fill them from the actual names available on your account. Do not invent parameters.

Step 2: Build the router

The router reads the task type, counts tokens approximately, and picks a tier.

import json
import os
import re
import httpx
from collections import Counter


CONFIG = json.load(open("config.json"))


def count_tokens(text: str) -> int:
    # Rough heuristic: ~4 characters per token
    return max(1, len(text) // 4)


def pick_tier(task: str, text: str) -> str:
    size = count_tokens(text)
    if task in {"classify", "extract"} and size < 300:
        return "small"
    if task == "summarize" and size < 1500:
        return "medium"
    if task == "debug":
        return "large"
    return "medium"


def call_model(tier: str, messages: list) -> str:
    url = CONFIG["endpoint"]
    api_key = os.environ["ROUTER_API_KEY"]
    payload = {
        "model": CONFIG[tier],
        "messages": messages,
        "temperature": 0,
        "max_tokens": 256,
    }
    resp = httpx.post(
        url,
        headers={"Authorization": f"Bearer {api_key}"},
        json=payload,
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()["choices"][0]["message"]["content"]


def looks_broken(text: str, task: str) -> bool:
    if task == "classify":
        return len(text.split()) != 1
    if task == "extract":
        try:
            json.loads(text)
        except json.JSONDecodeError:
            return True
        return False
    return text.strip() == ""


def route(task: str, user_text: str) -> dict:
    tier = pick_tier(task, user_text)
    messages = [
        {"role": "system", "content": f"You are a {task} assistant."},
        {"role": "user", "content": user_text},
    ]
    output = call_model(tier, messages)
    if looks_broken(output, task) and tier != "large":
        tier = "large"
        output = call_model(tier, messages)
    return {"tier": tier, "output": output}


def main() -> None:
    samples = json.load(open("samples.json"))
    stats = Counter()
    for sample in samples:
        result = route(sample["task"], sample["text"])
        stats[result["tier"]] += 1
        print(f"{sample['id']}: {result['tier']} -> {result['output'][:80]}")
    print("\nTier usage:", dict(stats))


if __name__ == "__main__":
    main()

The fallback is deliberately simple. It retries only once and only when the output is missing a required shape. You can extend looks_broken with regex checks later.

Step 3: Create sample tasks

sample.json should contain realistic traffic. Here is a small set:

[
  {
    "id": 1,
    "task": "classify",
    "text": "Is a refund for a canceled flight considered taxable income?"
  },
  {
    "id": 2,
    "task": "extract",
    "text": "We tested the router with GPT-4, Claude, and Llama. Output: {"companies": ["OpenAI", "Anthropic", "Meta"]}"
  },
  {
    "id": 3,
    "task": "summarize",
    "text": "This is a long technical report about database indexing... " + "x " * 800
  },
  {
    "id": 4,
    "task": "debug",
    "text": "My code throws an IndexError only on Tuesdays. Here is the traceback..."
  }
]

Run it locally first:

export ROUTER_API_KEY="your-monkeycode-token"
python router.py

The stats line tells you that classification and extraction used the small tier, summarization used medium, and debugging used large.

Step 4: Validate your routing decisions

The router only pays off if small-model answers are correct. Build a manual audit set with expected outputs. Then compare the tier used with the answer quality.

ID	Expected tier	Actual tier	Correct?	Notes
1	small	small	yes	classic
2	small	small	yes	JSON matched
3	medium	medium	yes	summary OK
4	large	large	yes	retry avoided

If you see many retries on extraction, move that task to medium. If classifications are wrong but syntactically correct, your small model is too weak for that task.

Step 5: Deploy on a free server

The free server from MonkeyCode can run router.py on a schedule. You can also run it as a tiny HTTP service. For cron, copy your workspace and add a job:

crontab -e

Run the audit every hour:

0 * * * * cd /path/to/router && python router.py >> router.log 2>&1

Inside router.py, wrap the loop with try/except so a timeout does not kill the whole job. The free server keeps the log accessible without your laptop being awake.

Limitations and who should not use this

• Token counting is approximate. Long non-English text can be off by 30%. Use a real tokenizer if you need precision.
• The fallback only checks output shape, not semantic correctness. A wrong answer that looks right will pass undetected.
• Free models have rate limits. Do not point a real-time user request stream at this router without a queue.
• Large-model retries can burn your free allowance if the small model fails often. Track retry counts daily.

This router is not for teams that need strict cost accounting or SLA-grade reliability. It is for developers who want to stop overpaying on routine prompts and want a reproducible way to measure the savings.

A better next step

You now have a working prototype. The next improvement is to add a semantic check: ask a cheap model to compare the small model's answer with a reference answer. That turns the router into a self-correcting pipeline.

Try this setup with MonkeyCode's free models and free server. Verify the current quota in the README, then let the router run for a week. You will know exactly which tasks deserve a big model and which ones do not.

Your token budget will thank you.