Lynkr vs LiteLLM's New `complexity_router`

Disclosure up front: this benchmark was designed, written, and run by Lynkr's
maintainer, using Lynkr's own regression harness. The scenario set was written to
test routing behaviors Lynkr considers important — that is a real selection bias
and it is not hand-waved away here (see Limitations). Mitigations:
both proxies route over identical backends, LiteLLM ran its out-of-the-box
defaults exactly as Lynkr did, every scenario ran twice, the full config and
harness are in this repo, and the results include the scenarios where LiteLLM
wins and where Lynkr shows drift. Reproduction steps are at the bottom.

Why this benchmark exists

In early August 2026, LiteLLM shipped complexity_router in core — a rule-based
tier router that classifies each request as SIMPLE / MEDIUM / COMPLEX / REASONING
(the same four-tier taxonomy Lynkr uses) via weighted keyword scoring across 7
dimensions, with zero API calls and sub-millisecond decisions. It ships alongside
adaptive_router (Thompson-sampling bandits) and a savings baseline.

This is the category leader validating tier-based cost routing as a core proxy
feature. We benchmarked it the week it shipped, on the same backends Lynkr routes
to in production.

Methodology

Systems under test

	version	routing decision mechanism
Lynkr	git working tree, 2026-08-05	embedding-anchor intent score (local, no LLM call) + risk analyzer + agentic detector + session pins + calibrated thresholds
LiteLLM	1.95.0 (PyPI; latest release)	complexity_router, heuristic classifier, default weights/boundaries, untouched

Note: LiteLLM git-main (1.97.0) could not be run — the git wheel is missing
proxy_server.py and current main is incompatible with fastapi ≥0.140
(get_flat_dependant removed). Both are being reported upstream. 1.95.0 is the
latest installable release and includes complexity_router.

Identical backends, only the decision differs

Both proxies map their four tiers to the same deployments (mirroring Lynkr's live
.env at test time):

tier	backend	cost class
SIMPLE	ollama minimax-m3:cloud	free
MEDIUM	ollama minimax-m3:cloud	free
COMPLEX	azure gpt-5.6-sol	paid
REASONING	moonshot kimi-k3	paid

LiteLLM config: litellm-autorouter-v2.yaml (in-repo), using the shipped
auto_router/complexity_router schema with tiers mapping and default_model,
nothing else configured. Each tier deployment carries model_info.id so the
x-litellm-model-id response header reveals which tier served each request.

Harness

benchmark-tier-routing.js (in-repo), MODE=routing RUNS=2. Each scenario
declares an acceptable-tier set; a proxy is routing-correct when the tier that
served the request is in the set. RUNS=2 means a scenario passes only if both
runs pass — this catches non-deterministic classifiers. Stateful scenarios embed a
per-run nonce so session pins and caches can't leak between runs.

Results

Scoreboard: 11 scenarios, acceptable-tier judging

ID	scenario (gist)	acceptable	Lynkr	LiteLLM
S1	"What does git stash do?"	SIMPLE|MEDIUM	✓ MEDIUM	✓ SIMPLE
R1	JWT security trade-offs for a banking app, step by step	COMPLEX|REASONING	✓ REASONING	✗ MEDIUM
F1	"Refactor the entire ingestion pipeline, give me the plan"	COMPLEX|REASONING	✓ COMPLEX	✗ MEDIUM
F2	"Fix the null check in src/auth/middleware.ts + tests"	COMPLEX|REASONING	✓ REASONING	✗ SIMPLE
RS1	"17+25" wrapped in injected <system-reminder> security noise	SIMPLE|MEDIUM	✓ MEDIUM	✓ SIMPLE
SR1	Suggestion-mode side request over a prior security conversation	SIMPLE|MEDIUM	✓ SIMPLE	✓ SIMPLE
A1	"Work autonomously: run tests, fix, iterate until green"	COMPLEX|REASONING	✓ REASONING	✗ SIMPLE
P1	"hi, you there?" (fresh session)	SIMPLE|MEDIUM	✓ MEDIUM	✓ SIMPLE
P2	same session, then "architecture review of the routing module"	COMPLEX|REASONING	✓ COMPLEX	✗ SIMPLE
IV1	"Review this retry helper for bugs" (bare)	MEDIUM|COMPLEX	✓ MEDIUM	✗ SIMPLE
IV2	identical ask + ~600 tokens of harness/IDE envelope noise	MEDIUM|COMPLEX	✓ MEDIUM	✗ SIMPLE

Lynkr: 11/11 · LiteLLM: 4/11.

The sharper cut: the 11 scenarios split into 4 "stay cheap" tests and 7
"escalate correctly" tests. LiteLLM went 4/4 on stay-cheap and 0/7 on
escalation. A router biased toward the cheap tier passes every stay-cheap test
by construction — the escalation tests are where routing is actually hard.

Failure analysis: three clusters, not seven random misses

1. Semantics beyond keywords (R1, F1, A1). "Security trade-offs for a banking application" and "work autonomously until the tests are green" carry complexity in meaning. The 7-dimension keyword sum doesn't reach the 0.35 COMPLEX boundary, so reasoning-grade work lands on the free tier. This is the published, predicted weakness of ex-ante lexical scoring on real traffic.
2. No risk model (F2). "Fix the null check in src/auth/middleware.ts" is lexically trivial — and it is exactly the request you never send to an uncontrolled free model. LiteLLM has no concept of protected paths; Lynkr's risk analyzer forces auth/middleware edits to the governance tier regardless of complexity score.
3. No session or envelope awareness (P2, IV1, IV2). After a "hi" opener, "now do an architecture review" stayed on SIMPLE — no pin-escape mechanism. The invariance pair is subtler: LiteLLM was technically invariant (SIMPLE both times) but invariantly wrong — a code-review ask never accumulated enough keyword weight in either form.

The cost table pathology (read before quoting)

The harness's raw cost line shows LiteLLM at $0.00 — "100% cheaper." That is
the failure, not a win: it spent nothing because it routed nearly everything —
including the 7 escalation-grade requests — to the free tier. A router that sends
everything to the cheapest model is always the "cheapest" router and always the
worst one. Cost numbers for a router are only meaningful conditional on routing
correctness. Any quotation of this benchmark that includes the cost table
without this caveat is misleading, in either direction.

Lynkr's own warts (strict-expectation view)

The harness also checks stricter per-scenario expectations than the acceptable
sets. Lynkr drifted on 3 of 11:

• R1, F2: over-routed COMPLEX → REASONING — correct-but-expensive (kimi-k3 spent where gpt-5.6-sol would do). Likely threshold drift after recent tier remapping; the nightly calibration should re-fit, being watched.
• P1: SIMPLE → MEDIUM pin drift on the trivial opener.

Over-routing burns money silently; it is the mirror image of LiteLLM's failure
mode and it deserves the same scrutiny.

Where LiteLLM is better

Symmetry matters, so plainly:

• Decision latency. Sub-millisecond, allocation-free keyword scoring vs Lynkr's embedding lookup (~10–50 ms warm, more on classifier cold start). For latency-critical single-model workloads that difference is real.
• Zero infrastructure for the decision. No embedding model, no classifier model, no state. Lynkr's anchor scorer wants a local embedding model available.
• Determinism and auditability. A weighted keyword sum is trivially explainable to a compliance reviewer; an embedding similarity is not.
• Simplicity of config. One YAML block. Lynkr's routing intelligence has meaningfully more surface to understand.
• Tunability from a known base. Weights, boundaries, and keyword lists are all exposed; a team willing to tune per-workload could close some of the gap (out-of-the-box was tested here, on both sides).
• Ecosystem and adoption. LiteLLM is the default proxy of the ecosystem, and the same release train ships adaptive_router (bandit-based) — the direction of travel is serious.

Where Lynkr is better

• Escalation correctness: 7/7 vs 0/7 — the entire hard half of the problem.
• Risk-aware routing — protected-path and instruction-risk signals route security-relevant work to governed tiers independent of complexity score.
• Session pinning with escape — sticky sessions that release when the work outgrows the pin (P2), instead of trapping or ignoring session state.
• Payload invariance by construction — the intent score is computed on cleaned user text, so harness envelopes, system-reminders, and IDE noise don't move the decision (IV1/IV2, RS1).
• A verification cascade behind the decision (not exercised in this routing- only benchmark): cheap-tier answers are structurally verified and escalated on failure, so a mis-route down is a recoverable event rather than a silent quality loss. LiteLLM's architecture is predict-then-commit.
• Third-party evaluation — RouterArena (ICLR 2026): 67.65 arena / 68.41% accuracy / $0.29 per 1K queries / 92.38 robustness, leaderboard #15.

Limitations

Every one of these is a genuine limitation of this benchmark:

• N=11, author-selected. The scenarios encode Lynkr's view of what routing should do. A LiteLLM-authored scenario set would probably look different. (Counterpoint: the escalation scenarios are not exotic — auth-file edits, refactor plans, and agentic loops are everyday coding-agent traffic.)
• Defaults vs defaults. LiteLLM's router is configurable; a tuned configuration was not tested. Neither was a tuned Lynkr.
• One day, one version pair. LiteLLM 1.95.0; main moves fast and the adaptive_router (which may compensate for complexity_router misses over time via bandit feedback) was not enabled or tested.
• Routing-only. Response quality, verification, compression, and caching were all out of scope; the cost figures are therefore not end-to-end claims.
• RUNS=2 is enough to catch gross nondeterminism, not enough for statistics. Both systems were deterministic across runs on every scenario.

Reproduction

# LiteLLM (Python ≥3.10 venv)
pip install "litellm[proxy]==1.95.0" "fastapi<0.140"
export $(grep -E '^(MOONSHOT_API_KEY|AZURE_OPENAI_API_KEY)=' .env | xargs)
litellm --port 8082 --config litellm-autorouter-v2.yaml

# Lynkr
node index.js   # serves :8081 per .env

# head-to-head, routing-only, 2 runs per scenario
MODE=routing RUNS=2 LITELLM_MASTER_KEY=sk-1234 node benchmark-tier-routing.js

Scenario definitions: benchmark-tier-routing.js. LiteLLM config:
litellm-autorouter-v2.yaml. Raw logs from this run: /tmp/litellm-bench.log,
/tmp/lynkr-bench.log.

Follow-ups queued

1. Paraphrase-pair robustness scenarios (same intent, different surface — the lexical scorer's structural weakness deserves its own measured number).
2. Re-run with LiteLLM adaptive_router enabled on top of complexity_router, with enough traffic for its bandit to update — the fairest version of this comparison over time.
3. File the two upstream LiteLLM bugs found while setting up (git wheel missing proxy_server.py; fastapi ≥0.140 incompatibility).
4. Investigate Lynkr's R1/F2 over-routing drift after tier remapping.