My LLM Router Is a Task Policy, Not Another Agent

I keep model selection in application code whenever the task is already known. Tagging, customer replies, and policy reviews have different constraints; they do not need another LLM call just to discover that.

The useful abstraction is a service tier with a measurable objective:

• Cheap: repetitive work with inexpensive, reliable validation.
• Fast: customer-facing work with a latency target.
• Accurate: ambiguous or high-impact work with a stricter quality gate.

Those are policy labels, not universal model rankings. The router chooses a candidate, validates its output, and follows a bounded fallback list. The benchmark decides whether the model deserves its slot.

Separate the task policy from the model registry

For a unified multi-provider interface, I use CometAPI’s OpenAI-compatible endpoint at https://api.cometapi.com/v1; model selection remains my application’s responsibility.

The source catalog snapshot dated August 20, 2026 lists the following IDs and estimated consumer rates. These estimates apply the catalog’s ratio to its baseline input and output prices; confirm current availability and your account’s final rates before deployment.

Tier	Typical tasks	Model ID	Estimated USD / 1M input tokens	Estimated USD / 1M output tokens
Cheap	Tagging, extraction, deduplication	deepseek-v4-flash	$0.176	$0.528
Fast	Replies, summaries, live assistants	gemini-3.7-flash	$0.60	$3.00
Accurate	Policy review, complex reasoning, high-impact drafts	claude-opus-5	$4.00	$20.00

I would treat this mapping as a starting hypothesis. Measure task pass rate, p50 and p95 latency, and cost per accepted output on the same evaluation set before making it permanent.

Keeping the model registry separate also makes upgrades boring: change an ID, rerun the contract tests and evaluations, then promote or roll back the mapping.

Define failure before writing fallback logic

HTTP success is not application success. I want every task to have an explicit acceptance test:

• Classification: the output is an allowed label.
• Structured extraction: the result passes a JSON schema.
• Customer replies: length and prohibited-claim checks pass.

A validator is only as strong as the checks it implements. A substring check for "guarantee" is useful demonstration code, not proof that a reply contains no unsupported promises.

Transport failures need a different policy. I allow fallback for timeouts, 408, 429, and temporary 5xx failures. I do not switch models to disguise a malformed payload, invalid credentials, or unsupported parameters.

The fallback order is also an approval list. If a task cannot safely use a lower tier, that tier should not appear in its route.

A complete router with bounded retries

This needs Python 3.10 or later and the OpenAI Python package. Keep the key server-side.

pip install openai
export COMETAPI_KEY="your-key-here"

Save this as llm_task_router.py:

import os
import time

from openai import APIError, OpenAI

client = OpenAI(
    api_key=os.environ["COMETAPI_KEY"],
    base_url="https://api.cometapi.com/v1",
    max_retries=0,
    timeout=20,
)

MODELS = {
    "cheap": "deepseek-v4-flash",
    "fast": "gemini-3.7-flash",
    "accurate": "claude-opus-5",
}

# Preferred tier first; remaining entries are approved fallbacks.
ROUTES = {
    "tag": ["cheap", "fast", "accurate"],
    "reply": ["fast", "cheap", "accurate"],
    "policy_review": ["accurate", "fast", "cheap"],
}

MAX_ATTEMPTS_PER_MODEL = 2


def retryable(error):
    status = getattr(error, "status_code", None)
    return status is None or status in {408, 429} or (
        status is not None and 500 <= status < 600
    )


def call_model(model, prompt):
    for attempt in range(1, MAX_ATTEMPTS_PER_MODEL + 1):
        try:
            return client.chat.completions.create(
                model=model,
                messages=[{"role": "user", "content": prompt}],
                max_tokens=400,
            )
        except APIError as error:
            if not retryable(error) or attempt == MAX_ATTEMPTS_PER_MODEL:
                raise
            time.sleep(min(0.5 * (2 ** (attempt - 1)), 2.0))


def route(task, prompt, validate=lambda text: True):
    attempts = []
    for tier in ROUTES.get(task, ROUTES["reply"]):
        model = MODELS[tier]
        started = time.perf_counter()
        try:
            response = call_model(model, prompt)
            text = response.choices[0].message.content or ""
            attempts.append({
                "tier": tier,
                "model": model,
                "latency_ms": round((time.perf_counter() - started) * 1000),
                "accepted": validate(text),
            })
            if attempts[-1]["accepted"]:
                return {
                    "text": text,
                    "route": tier,
                    "model": model,
                    "usage": (
                        response.usage.model_dump()
                        if response.usage else None
                    ),
                    "attempts": attempts,
                }
        except APIError as error:
            attempts.append({
                "tier": tier,
                "model": model,
                "status": getattr(error, "status_code", None),
            })
            if not retryable(error):
                raise

    raise RuntimeError(f"No route passed: {attempts}")


if __name__ == "__main__":
    result = route(
        "reply",
        "Reply to a customer asking when their refund will arrive. "
        "Do not promise a date.",
        validate=lambda text: (
            30 <= len(text) <= 600
            and "guarantee" not in text.lower()
        ),
    )
    print(result)

Run it:

python3 llm_task_router.py

The client uses POST /v1/chat/completions. Check the current model entries and Chat Completions reference before adding model-specific fields.

What this code actually bounds

SDK retries are disabled. The helper permits two attempts per model, retrying an eligible API failure once before handing control back to the router.

With three tiers, that is at most six provider calls. Validation failure moves directly to the next tier; it does not regenerate repeatedly on the same model.

The 20-second client timeout is not an end-to-end request deadline. Multiple calls and backoff can accumulate, so production latency budgets need a request-wide limit too.

A few other details matter:

• Unknown task values use the reply route.
• Omitting validate accepts any returned text, including an empty string.
• attempts records tier outcomes, not every internal retry.
• Returned token usage belongs to the accepted response, not all rejected responses.
• The retry helper uses deterministic backoff; add jitter for production rate-limit handling.

I use getattr when reading an exception’s status because connection failures may not have an HTTP status code.

Test routing and model performance separately

The deterministic policy can be checked without contacting a provider:

assert ROUTES["tag"][0] == "cheap"
assert ROUTES["reply"][0] == "fast"
assert ROUTES["policy_review"][0] == "accurate"

Those assertions prove the configuration, not the quality of the chosen models.

A live smoke test should return text, the selected tier and model, available token usage, and tier-attempt records. Token counts and latency will vary; placeholder values are not benchmark results.

For the actual comparison, send the same labeled requests through all three models and collect:

• Task pass rate and error rate.
• p50 and p95 latency.
• Input and output tokens.
• Fallback and human-review rates.
• Cost per accepted output.

I care most about the last metric. A cheap call that triggers regeneration or manual review may be the expensive route.

Price the entire execution, not just the winning response

Use the same workload shape for each tier: 1 million total tokens, split into 800,000 input and 200,000 output tokens.

Using the dated rates above:

Tier	Calculation	Estimated cost
Cheap	0.8 × $0.176 + 0.2 × $0.528	$0.25
Fast	0.8 × $0.60 + 0.2 × $3.00	$1.08
Accurate	0.8 × $4.00 + 0.2 × $20.00	$7.20

At 60% cheap, 30% fast, and 10% accurate, the projected blend is approximately $1.19 per 1 million total tokens, assuming the same workload shape across routes. Sending it all to the accurate tier would cost approximately $7.20.

That is arithmetic, not evidence that the blended policy meets its quality target.

Under the same cost assumptions, a 5% one-time retry rate raises the projection to roughly $1.25. If cheap-tier output is rejected and the request is repeated on the accurate tier, both calls count.

The example needs additional instrumentation for this accounting: record usage from rejected responses and log internal retries rather than relying only on the winning response’s usage.

Keep operational errors out of the quality loop

I use this distinction during incident triage:

Signal	Response
400 or invalid request	Fix the payload; do not fall back.
401	Reload or rotate credentials; do not retry.
403	Check access and unsupported fields.
429	Back off with jitter, reduce concurrency, then use approved fallback if allowed.
Temporary 5xx or timeout	Retry within budget, then try the next compatible route; retain request IDs.
Failed quality gate	Record the reason, advance through the approved list, and stop when exhausted.

Machine-detectable failures—invalid schema, missing fields, prohibited promises—can justify bounded escalation. Vague dissatisfaction belongs in evaluation data, not an unlimited retry loop.

Decide whether fixed identities are worth maintaining

Application routing gives me fixed model identities, explicit budgets, custom validators, and an auditable fallback order. That is the right trade-off when reproducibility matters.

A managed alternative accepts model="auto" for a balanced default or model="auto-high" when quality has higher priority. It selects an eligible model dynamically from request characteristics and the current routing pool.

That reduces mapping maintenance, but the underlying model can change. I would not use it where every run must hit the same model or depend on model-specific parameters.

What I would require before production

Validate the registry during deployment

Call GET https://api.cometapi.com/api/models at deployment or startup. Fail the release if a configured model ID or required endpoint is missing. IDs, capabilities, and prices can change.

Adding another compatible model—including an OpenAI model—means updating MODELS, checking the request/response contract, evaluating it, and assigning its route position. The client configuration can stay unchanged.

Put parameter differences in adapters

A shared interface does not imply identical support for logprobs, reasoning controls, multiple candidates, or token controls. Keep those differences out of task policy and behind tested adapters.

Enforce budgets before sending requests

Cap concurrency, set an output-token ceiling, and apply exponential backoff with jitter for rate limits. A fallback chain without a latency and spend budget is still an uncontrolled workload.

Log enough to reproduce decisions

Record task type, policy version, tier, model ID, latency, token usage, validation result, retry count, fallback reason, and estimated cost. Do not log secrets or unnecessary customer content.

Promote mappings with evidence

Maintain labeled evaluations per task. Roll out changes gradually, compare against the previous policy, and preserve a quick rollback.

I change the registry when repeatable measurements show a better trade-off—not because a new model name appeared. The maintainable router is the one whose decisions I can explain, test, and reverse.