DEV Community: Lynkr

Lynkr vs LiteLLM's New `complexity_router`

Lynkr — Wed, 05 Aug 2026 22:17:24 +0000

Disclosure up front: this benchmark was designed, written, and run by Lynkr's
maintainer, using Lynkr's own regression harness. The scenario set was written to
test routing behaviors Lynkr considers important — that is a real selection bias
and it is not hand-waved away here (see Limitations). Mitigations:
both proxies route over identical backends, LiteLLM ran its out-of-the-box
defaults exactly as Lynkr did, every scenario ran twice, the full config and
harness are in this repo, and the results include the scenarios where LiteLLM
wins and where Lynkr shows drift. Reproduction steps are at the bottom.

Why this benchmark exists

In early August 2026, LiteLLM shipped complexity_router in core — a rule-based
tier router that classifies each request as SIMPLE / MEDIUM / COMPLEX / REASONING
(the same four-tier taxonomy Lynkr uses) via weighted keyword scoring across 7
dimensions, with zero API calls and sub-millisecond decisions. It ships alongside
adaptive_router (Thompson-sampling bandits) and a savings baseline.

This is the category leader validating tier-based cost routing as a core proxy
feature. We benchmarked it the week it shipped, on the same backends Lynkr routes
to in production.

Methodology

Systems under test

	version	routing decision mechanism
Lynkr	git working tree, 2026-08-05	embedding-anchor intent score (local, no LLM call) + risk analyzer + agentic detector + session pins + calibrated thresholds
LiteLLM	1.95.0 (PyPI; latest release)	`complexity_router`, heuristic classifier, default weights/boundaries, untouched

Note: LiteLLM git-main (1.97.0) could not be run — the git wheel is missing
proxy_server.py and current main is incompatible with fastapi ≥0.140
(get_flat_dependant removed). Both are being reported upstream. 1.95.0 is the
latest installable release and includes complexity_router.

Identical backends, only the decision differs

Both proxies map their four tiers to the same deployments (mirroring Lynkr's live
.env at test time):

tier	backend	cost class
SIMPLE	ollama `minimax-m3:cloud`	free
MEDIUM	ollama `minimax-m3:cloud`	free
COMPLEX	azure `gpt-5.6-sol`	paid
REASONING	moonshot `kimi-k3`	paid

LiteLLM config: litellm-autorouter-v2.yaml (in-repo), using the shipped
auto_router/complexity_router schema with tiers mapping and default_model,
nothing else configured. Each tier deployment carries model_info.id so the
x-litellm-model-id response header reveals which tier served each request.

Harness

benchmark-tier-routing.js (in-repo), MODE=routing RUNS=2. Each scenario
declares an acceptable-tier set; a proxy is routing-correct when the tier that
served the request is in the set. RUNS=2 means a scenario passes only if both
runs pass — this catches non-deterministic classifiers. Stateful scenarios embed a
per-run nonce so session pins and caches can't leak between runs.

Results

Scoreboard: 11 scenarios, acceptable-tier judging

ID	scenario (gist)	acceptable	Lynkr	LiteLLM
S1	"What does git stash do?"	SIMPLE\|MEDIUM	✓ MEDIUM	✓ SIMPLE
R1	JWT security trade-offs for a banking app, step by step	COMPLEX\|REASONING	✓ REASONING	✗ MEDIUM
F1	"Refactor the entire ingestion pipeline, give me the plan"	COMPLEX\|REASONING	✓ COMPLEX	✗ MEDIUM
F2	"Fix the null check in `src/auth/middleware.ts` + tests"	COMPLEX\|REASONING	✓ REASONING	✗ SIMPLE
RS1	"17+25" wrapped in injected `<system-reminder>` security noise	SIMPLE\|MEDIUM	✓ MEDIUM	✓ SIMPLE
SR1	Suggestion-mode side request over a prior security conversation	SIMPLE\|MEDIUM	✓ SIMPLE	✓ SIMPLE
A1	"Work autonomously: run tests, fix, iterate until green"	COMPLEX\|REASONING	✓ REASONING	✗ SIMPLE
P1	"hi, you there?" (fresh session)	SIMPLE\|MEDIUM	✓ MEDIUM	✓ SIMPLE
P2	same session, then "architecture review of the routing module"	COMPLEX\|REASONING	✓ COMPLEX	✗ SIMPLE
IV1	"Review this retry helper for bugs" (bare)	MEDIUM\|COMPLEX	✓ MEDIUM	✗ SIMPLE
IV2	identical ask + ~600 tokens of harness/IDE envelope noise	MEDIUM\|COMPLEX	✓ MEDIUM	✗ SIMPLE

Lynkr: 11/11 · LiteLLM: 4/11.

The sharper cut: the 11 scenarios split into 4 "stay cheap" tests and 7
"escalate correctly" tests. LiteLLM went 4/4 on stay-cheap and 0/7 on
escalation. A router biased toward the cheap tier passes every stay-cheap test
by construction — the escalation tests are where routing is actually hard.

Failure analysis: three clusters, not seven random misses

Semantics beyond keywords (R1, F1, A1). "Security trade-offs for a banking application" and "work autonomously until the tests are green" carry complexity in meaning. The 7-dimension keyword sum doesn't reach the 0.35 COMPLEX boundary, so reasoning-grade work lands on the free tier. This is the published, predicted weakness of ex-ante lexical scoring on real traffic.
No risk model (F2). "Fix the null check in src/auth/middleware.ts" is lexically trivial — and it is exactly the request you never send to an uncontrolled free model. LiteLLM has no concept of protected paths; Lynkr's risk analyzer forces auth/middleware edits to the governance tier regardless of complexity score.
No session or envelope awareness (P2, IV1, IV2). After a "hi" opener, "now do an architecture review" stayed on SIMPLE — no pin-escape mechanism. The invariance pair is subtler: LiteLLM was technically invariant (SIMPLE both times) but invariantly wrong — a code-review ask never accumulated enough keyword weight in either form.

The cost table pathology (read before quoting)

The harness's raw cost line shows LiteLLM at $0.00 — "100% cheaper." That is
the failure, not a win: it spent nothing because it routed nearly everything —
including the 7 escalation-grade requests — to the free tier. A router that sends
everything to the cheapest model is always the "cheapest" router and always the
worst one. Cost numbers for a router are only meaningful conditional on routing
correctness. Any quotation of this benchmark that includes the cost table
without this caveat is misleading, in either direction.

Lynkr's own warts (strict-expectation view)

The harness also checks stricter per-scenario expectations than the acceptable
sets. Lynkr drifted on 3 of 11:

R1, F2: over-routed COMPLEX → REASONING — correct-but-expensive (kimi-k3 spent where gpt-5.6-sol would do). Likely threshold drift after recent tier remapping; the nightly calibration should re-fit, being watched.
P1: SIMPLE → MEDIUM pin drift on the trivial opener.

Over-routing burns money silently; it is the mirror image of LiteLLM's failure
mode and it deserves the same scrutiny.

Where LiteLLM is better

Symmetry matters, so plainly:

Decision latency. Sub-millisecond, allocation-free keyword scoring vs Lynkr's embedding lookup (~10–50 ms warm, more on classifier cold start). For latency-critical single-model workloads that difference is real.
Zero infrastructure for the decision. No embedding model, no classifier model, no state. Lynkr's anchor scorer wants a local embedding model available.
Determinism and auditability. A weighted keyword sum is trivially explainable to a compliance reviewer; an embedding similarity is not.
Simplicity of config. One YAML block. Lynkr's routing intelligence has meaningfully more surface to understand.
Tunability from a known base. Weights, boundaries, and keyword lists are all exposed; a team willing to tune per-workload could close some of the gap (out-of-the-box was tested here, on both sides).
Ecosystem and adoption. LiteLLM is the default proxy of the ecosystem, and the same release train ships adaptive_router (bandit-based) — the direction of travel is serious.

Where Lynkr is better

Escalation correctness: 7/7 vs 0/7 — the entire hard half of the problem.
Risk-aware routing — protected-path and instruction-risk signals route security-relevant work to governed tiers independent of complexity score.
Session pinning with escape — sticky sessions that release when the work outgrows the pin (P2), instead of trapping or ignoring session state.
Payload invariance by construction — the intent score is computed on cleaned user text, so harness envelopes, system-reminders, and IDE noise don't move the decision (IV1/IV2, RS1).
A verification cascade behind the decision (not exercised in this routing- only benchmark): cheap-tier answers are structurally verified and escalated on failure, so a mis-route down is a recoverable event rather than a silent quality loss. LiteLLM's architecture is predict-then-commit.
Third-party evaluation — RouterArena (ICLR 2026): 67.65 arena / 68.41% accuracy / $0.29 per 1K queries / 92.38 robustness, leaderboard #15.

Limitations

Every one of these is a genuine limitation of this benchmark:

N=11, author-selected. The scenarios encode Lynkr's view of what routing should do. A LiteLLM-authored scenario set would probably look different. (Counterpoint: the escalation scenarios are not exotic — auth-file edits, refactor plans, and agentic loops are everyday coding-agent traffic.)
Defaults vs defaults. LiteLLM's router is configurable; a tuned configuration was not tested. Neither was a tuned Lynkr.
One day, one version pair. LiteLLM 1.95.0; main moves fast and the adaptive_router (which may compensate for complexity_router misses over time via bandit feedback) was not enabled or tested.
Routing-only. Response quality, verification, compression, and caching were all out of scope; the cost figures are therefore not end-to-end claims.
RUNS=2 is enough to catch gross nondeterminism, not enough for statistics. Both systems were deterministic across runs on every scenario.

Reproduction

# LiteLLM (Python ≥3.10 venv)
pip install "litellm[proxy]==1.95.0" "fastapi<0.140"
export $(grep -E '^(MOONSHOT_API_KEY|AZURE_OPENAI_API_KEY)=' .env | xargs)
litellm --port 8082 --config litellm-autorouter-v2.yaml

# Lynkr
node index.js   # serves :8081 per .env

# head-to-head, routing-only, 2 runs per scenario
MODE=routing RUNS=2 LITELLM_MASTER_KEY=sk-1234 node benchmark-tier-routing.js

Scenario definitions: benchmark-tier-routing.js. LiteLLM config:
litellm-autorouter-v2.yaml. Raw logs from this run: /tmp/litellm-bench.log,
/tmp/lynkr-bench.log.

Follow-ups queued

Paraphrase-pair robustness scenarios (same intent, different surface — the lexical scorer's structural weakness deserves its own measured number).
Re-run with LiteLLM adaptive_router enabled on top of complexity_router, with enough traffic for its bandit to update — the fairest version of this comparison over time.
File the two upstream LiteLLM bugs found while setting up (git wheel missing proxy_server.py; fastapi ≥0.140 incompatibility).
Investigate Lynkr's R1/F2 over-routing drift after tier remapping.

Loop Engineering Is Mostly Papering Over a Model That Won't Converge

Lynkr — Thu, 30 Jul 2026 07:25:49 +0000

Disclosure: I maintain Lynkr, an open-source LLM gateway with loop-guard middleware, so I have a stake in this argument. Read this as a thesis to challenge, not a neutral survey.

Agent discourse has shifted from prompts to loops: the read-act-observe cycle that keeps running until the model decides it's done. My contrarian take is that most so-called loop engineering is optimizing the wrong variable. A longer, richer loop is usually a convergence failure wearing a costume. The well-engineered loop is the short one that knows when to stop.

The consensus I'm arguing against

When an agent fails, the default move is to give the loop more:

More turns before it gives up.
More context stuffed into the window.
More scaffolding: planners, critics, reflection steps, sub-agents spawning sub-agents.
More retries with more feedback.

The implicit theory is simple: intelligence accumulates across iterations, so more turns should mean more convergence. Sometimes that works. Mostly, on real traffic, it doesn't — and the cost structure is brutal even when it does.

A concrete example from coding agents: give an underpowered model a repo-wide refactor and it usually doesn't "reason longer" into success. It misreads one dependency, anchors on the wrong file, and spends the next dozen turns elaborating the mistake. The loop isn't rescuing the model. It's financing the failure.

Why longer loops compound the wrong things

Two things compound across a loop, and neither is intelligence.

1. Loop cost grows fast. Every turn re-sends accumulated context. A 20-turn session isn't just 20 single calls back-to-back; each later turn drags earlier tokens back through the model. In many agentic workloads, the loop becomes the most expensive object in the serving budget, and "add more turns" becomes the most expensive knob you can turn.

2. Reliability collapses across dependent steps. Even with optimistic per-step reliability, multi-step chains degrade quickly. Long loops don't average out mistakes; they multiply the chance of staying on the correct trajectory. The agent that "just needs more turns" is often an agent walking deeper into the wrong branch with more conviction, because each turn conditions on its own earlier errors as if they were facts.

We wrote earlier about how stuffing the context window can make agents worse, not smarter: attention dilutes, retrieval anchors on the wrong chunk, and signal gets buried. Loop length is the same failure on the time axis. More is not more.

What a long loop is actually signaling

When an agent needs 30 turns to do something, that's usually not the harness being clever. It's a signal of one of two failures:

The task was routed to a model that can't do it. The loop is that model thrashing. Scaffolding rarely fixes an underpowered model decision; it usually just makes the failure more elaborate and more expensive.
The task genuinely has many steps. In that case, the engineering problem is not "how do I allow more turns" but "how do I verify each step cheaply so the wrong branch gets caught at turn 3 instead of compounding to turn 30?"

Both point the same direction: either allocate the step to a better model, or verify the step before proceeding. Neither says: add turns and hope.

What loop engineering should actually be

If loops are where cost and error compound, then good loop engineering should minimize compounding, not maximize capability-per-turn.

1. Cap the loop hard. Every loop needs a turn ceiling and a tool-call ceiling that it cannot exceed, full stop. Not as an emergency fallback — as a first-class design constraint. A loop that can run forever will, on some input, cost you a fortune to produce garbage.

2. Verify each step cheaply. The highest-leverage addition to a loop is usually not another reasoning stage. It's a fast, deterministic check on the step output that catches the failure modes you actually see: truncation, malformed tool calls, empty answers, degeneration loops, repeated echoes. Detection beats prediction because you already have the output in hand.

3. Escalate the failing step, not the whole session. When a step fails verification, redo that step on a better tier. Don't grant the entire loop more turns to flail. Escalation is targeted; more-turns is a blank check.

The part where this bit us too

We're not exempt. In Lynkr, we shipped a terse mode meant to save output tokens by pushing models toward shorter answers. Our verifier then misread some of those short answers as low-effort failures and escalated trivial requests up a tier into a longer, more expensive path.

The fix was one predicate. The broader lesson wasn't. A check inside a loop is itself part of the loop's cost and failure surface. "Add verification" is not free virtue; a bad verifier can extend loops as eagerly as a naive retry does. Verification has to be engineered with the same suspicion as any other turn.

The thesis, plainly

A lot of popular loop engineering optimizes capability-per-turn while treating turn count as free. That's backwards. On real traffic, turn count is often the dominant cost and a major source of compounded error.

If a task needs more intelligence, buy more intelligence for that step. If a task needs more reliability, verify the step. If a loop keeps growing, treat that as a failure signal, not sophistication.

Make the loop shorter. Make it stop. That's the engineering.

Lynkr is Apache-2.0 and self-hosted. The loop-guard middleware and step-level escalation ideas behind this post are in the repo: github.com/Fast-Editor/Lynkr. If you've seen a loop run too long and know why, I'd love to hear the war story.

Your LLM Bill Is a Misallocation Problem, Not a Model Problem

Lynkr — Thu, 30 Jul 2026 06:50:28 +0000

Disclosure: I maintain Lynkr, an open-source LLM router, so this post argues a thesis my own product embodies. The mitigation: every claim below is either published research, a July 2026 release you can verify, or a measurement you can reproduce — including the part where the obvious approach (and an earlier version of ours) doesn't work.

The dominant inefficiency in LLM serving is no longer kernel speed or quantization. It's per-request compute misallocation — and the research says you can't fix it with prediction. You fix it with verification.

The setup: the gap closed, the dispersion didn't

Three things became true more or less simultaneously in mid-2026:

Open-weight models sit within single-digit percentage points of frontier models on everyday work. GLM-5.2 (744B MoE, 40B active, MIT license) leads several long-horizon coding benchmarks outright. Kimi K3 shipped in July at 2.8T parameters with ~1.8% expert activation and a 1M-token context — open weights, hosted access.
The cost of that near-frontier capability is 4–10× lower. Hosted GLM-class models run at a fraction of frontier per-token prices; aggressive hosts (Meituan's LongCat 2.0 launched at $0.30/$1.20 per million) keep pushing the floor down.
Price-per-capability dispersion across providers has never been wider. The same request can cost $0.10 or $15 per million output tokens depending on where you send it, for a quality difference that on most traffic is inside the noise.

Put those together and the largest line item in a serving budget is no longer "the model is slow" or "the quant is fat." It's every request routed to a frontier model that a cheaper tier would have cleared. That's pure deadweight loss, and at current dispersion it usually dominates everything a kernel engineer can save you.

I say that as someone who ships kernel PRs: a +13–15% decode win on a CPU inference engine (measured, merged, byte-identical output) saves you 13% on one backend. Routing 60% of traffic down two price tiers saves ~90% across the fleet. Same engineering effort. The leverage moved up the stack.

The naive fix fails: you can't predict difficulty from the prompt

The obvious architecture is ex-ante routing: classify the prompt, pick the model, done. Every "smart routing" product demo shows this.

It doesn't survive contact with real traffic. Ex-ante difficulty prediction consistently fails to beat trivial baselines on agentic coding traffic. When we ran the deep-research pass for Lynkr's cascade design (July 2025 literature, SWE-Bench-style workloads), the pattern was blunt: prompt surface features don't carry enough signal about downstream difficulty. A one-line ask ("fix the flaky test") can require repo-wide reasoning. A 3,000-token prompt can be boilerplate a 9B model handles. The information you need to route correctly mostly doesn't exist until after a model has attempted the task.

If your router's accuracy story is a difficulty classifier — however fancy the embeddings — you've built a system that's confidently wrong in both directions: burning frontier tokens on easy requests and serving garbage on hard ones.

What works: try cheap, verify, escalate

The architecture that actually gains is the cascade: route to the cheapest plausible tier, run deterministic verification on the answer, escalate only on failure. Published cascade results show up to +14% over static assignment — and the gain lives in the verifier, because the cascade converts routing from a prediction problem (hard, information-starved) into a detection problem (tractable, information-rich: you have the actual answer in hand).

The verifier doesn't need to be an LLM judge. Ours is two layers of pure functions:

Structural checks targeting the cheap-model failure modes we actually observed live: language drift (CJK tokens appearing mid-English answer), degeneration loops, truncation, malformed tool calls, empty/echo responses. High precision — flag only what is definitely broken.
A coarse effort score: substantive ask answered with a stub, structure requested but not delivered, code requested but not delivered. Conservative threshold; only clear failures escalate.

No LLM call in the loop. The verification cost is microseconds, which matters because a router that spends tokens deciding where to send tokens is eating the margin it exists to create.

A war story about verifiers

One production bug worth sharing because it generalizes: we shipped a brevity mode that injects "answer tersely" into system prompts to cut output tokens. The verifier's effort-scorer then read the resulting short answers as low-effort and started cascading trivial greetings up to the frontier tier — burning the exact money the brevity mode saved, and then some.

The fix was one predicate (skip the effort check when the brevity instruction is present; keep the structural checks), but the lesson is bigger: a verifier must know what the model was asked to be. Any answer-quality check that ignores the instructions that produced the answer will misfire exactly when your prompt engineering works.

The second-order effect: verification failures are free training data

Here's where cascades stop being a cost trick and become a learning system.

Every verification failure is a labeled hard negative: "this request, routed to this tier, produced a detectably bad answer." Every escalation that then passes is a label: "requests like this outgrow the cheap tier." Feed those back into the routing decision — we treat it as a contextual bandit with verifier-generated rewards, logging propensities at decision time so the feedback isn't biased by the router's own choices — and the router learns the difficulty structure of your traffic, from your outcomes.

No benchmark in the loop. That mattered more than usual this month: one of the field's most-cited coding benchmarks was retracted as broken. Every router configured against it inherited the error silently, for months, with nothing in the loop that could detect it. Ground truth from your own traffic doesn't get retracted.

Receipts

This is the architecture behind Lynkr (Apache-2.0, self-hosted): deterministic embedding-anchor tier classification (zero LLM calls to route), structural verification on cheap-tier answers only (never second-guess the expensive tier — the loss is asymmetric), cascade escalation, reward pipeline with propensity logging.

On RouterArena (ICLR 2026, 8,400 queries, evaluation-only rules — no component tuned on their data), our submitted configuration measured 67.65 arena score / 68.41% accuracy at $0.29 per 1K queries, robustness 92.38 — above GPT-5's built-in router and NotDiamond at a fraction of the per-decision cost. The submission PR has the full config, prediction files, and the disclosures, including the metrics where we're mediocre (routing concentration is real; read the PR).

The thesis, restated

For a decade the way to cut inference cost was to make the model cheaper: quantize, distill, fuse kernels. That work still matters — I still do it — but it's optimizing the price of a decision that's already been made. In a world where capability is a commodity spectrum with 50× price dispersion and single-digit quality spread, the decision itself is where the money is.

The gap between open and frontier closed. Allocation is the frontier now.

Lynkr is Apache-2.0 and self-hosted: github.com/Fast-Editor/Lynkr · npm. If you route production LLM traffic and have cascade/verification war stories, I want to hear them — the failure modes section of our verifier is built from exactly those.

We Put Our Router on an Academic Benchmark. Here Are the Numbers We'd Rather Hide.

Lynkr — Mon, 27 Jul 2026 07:36:49 +0000

Disclosure: I maintain Lynkr, the router being benchmarked, so read everything here with that in mind. The mitigation: every number in this post comes from RouterArena's automated evaluation pipeline, run by their CI on their infrastructure, not by me — including the numbers that make us look mediocre.

Every LLM router's README — ours included — makes the same claim: it sends easy queries to cheap models and hard queries to good ones, and saves you money without hurting quality. Almost none of them attach evidence. The numbers that do exist are self-reported, measured on datasets the vendor picked, with baselines the vendor chose.

So when RouterArena showed up — an open, standardized benchmark for LLM routers out of the RouteWorks group (paper, leaderboard) — we submitted Lynkr to it. This post is what we learned, including the metrics where we did badly, because a benchmark result you only quote selectively is just marketing with extra steps.

What RouterArena actually measures

RouterArena evaluates a router over 8,400 queries spanning 9 domains and 44 categories at three difficulty levels, and scores it on five axes:

Arena Score — the headline number, a combined accuracy-vs-cost tradeoff.
Accuracy — did the model your router picked answer correctly?
Cost — average $ per 1K queries, at real provider prices.
Optimality — three sub-metrics against the oracle: how often you picked the cheapest correct model (Opt.Sel), how close your spend was to optimal (Opt.Cost), and how close your accuracy was to the best achievable with your pool (Opt.Acc).
Robustness — do trivial rephrasings of the same query flip your routing decision?

One rule matters more than the metrics: RouterArena is evaluation-only. Any router component trained, fitted, or tuned on their data or labels gets rejected, and violations found later get withdrawn. Keep that rule in mind — it comes back at the end of this post.

What we submitted

We submitted Lynkr's untuned production default — the same complexity scorer and the same tier boundaries that ship in the repo. The adapter is ~70 lines of Python that sends each query to a live Lynkr instance's /routing/analyze endpoint and maps the returned tier to a model. No benchmark-special code path: the endpoint runs the exact intent scorer the live proxy uses (local embeddings, no LLM call in the routing decision itself).

The model pool, all via OpenRouter:

Lynkr tier	Model
SIMPLE	gpt-oss-120b
MEDIUM	Qwen3-235B-A22B
COMPLEX / REASONING	GLM-4.7

Running all 8,400 queries cost $2.46 total. That's the entire eval bill, which says something on its own about where inference prices are in mid-2026.

The results

From RouterArena's automated evaluation on our submission PR:

Metric	Lynkr
Arena Score	67.65
Accuracy	68.41%
Cost per 1K queries	$0.29
Robustness	92.38
Opt.Acc (accuracy vs. optimal)	84.48
Opt.Cost (cost efficiency vs. optimal)	16.08
Opt.Sel (optimal model selection)	10.97

On the official leaderboard (our submission merged 2026-07-23), that lands mid-table — 15th of 27 routers.

The parts we'll be quoting

We beat GPT-5 used as a router — at 34× lower cost. The leaderboard includes GPT-5 itself as a routing baseline: Arena score 64.32 at $10.02 per 1K queries. Lynkr scores 67.65 at $0.29 per 1K. A local intent scorer making the decision without any LLM call outperforms asking a frontier model to route — which is the entire thesis Lynkr is built on, so we're relieved the benchmark agrees. Several well-known systems (NotDiamond at 57.29, RouteLLM at 48.07, both RouterBench baselines) also land below us.

Robustness 92.38 is top-five territory. Most leaderboard routers sit between 22 and 72 — meaning a rephrased query frequently flips their model choice. Lynkr's routing decision survives rephrasing 92% of the time. For an interactive tool where users iterate on prompts, decision stability is arguably worth more than a couple of accuracy points: a router that sends your reworded follow-up to a different model mid-conversation is a router you turn off.

The parts we'd rather hide (but won't)

Opt.Sel is 10.97. When multiple models in our pool could answer a query correctly, we picked the cheapest correct one about 11% of the time. Lynkr routes conservatively — when in doubt, it escalates a tier. That's a deliberate live-serving bias (a wrong cheap answer costs more user trust than an unnecessarily good one costs dollars), and $0.29/1K shows the absolute spend stays low. But the oracle comparison is unambiguous: there was money on the table we didn't pick up.

The accuracy gap to the top is real. Cross-Router leads at 78.14% accuracy; we're at 68.41%. Part of that is the pool (three self-hostable open-weight models — no frontier closed models to escalate to), but part is genuinely the router. Mid-table is where we are, not where we'd spin it.

What the benchmark taught us — and why we won't act on all of it

Reading our failure cases was the most valuable part of the exercise. The clearest pattern: our SIMPLE→MEDIUM boundary is conservative — a chunk of queries scored just past the boundary and got a mid-tier model when the cheap tier would have answered correctly.

So we prototyped a shifted boundary and ran it on a ~10% subsample locally: Arena score 69.93, accuracy 71.07%, Opt.Sel jumping from ~11 to ~66. A two-point arena gain from moving one threshold.

That config will never appear on the leaderboard. It was diagnosed from RouterArena's own failure cases, which makes it exactly what their evaluation-only rule exists to prevent — a router fitted to the eval. The leaderboard entry is the untuned default, and stays that way. The honest generalization claim is much narrower: the benchmark showed us which knob matters, and we'll validate new defaults on our own traffic, not on theirs.

If you maintain a router: this is the trap. The moment a public benchmark exists, the incentive is to tune against it, and every point you gain that way is a point of overfitting you ship to your actual users.

Caveats, so you can weigh this properly

RouterArena is single-turn Q&A-style evaluation. Lynkr's primary workload is multi-turn coding agents, which RouterArena doesn't measure — agentic routing (tool-call density, context accumulation) is a different problem than classifying one prompt.
Our score depends on our chosen pool. A pool with a frontier model on top would score differently in both directions (higher ceiling, higher cost).
Leaderboard positions move; check the live leaderboard rather than trusting this snapshot.

If you're evaluating routers — including ours — ask every vendor for their RouterArena number. It costs a few dollars and a PR to get one. "We haven't submitted" is also an answer.

Links: RouterArena paper (arXiv:2510.00202) · leaderboard · our submission PR · Lynkr

Stuffing the Context Window Is Making Your Agent Dumber: What the Research Says

Lynkr — Fri, 17 Jul 2026 01:10:15 +0000

Disclosure: I maintain Lynkr, an open-source gateway that (among other things) compresses agent tool outputs — so I have a horse in this race. This piece, though, is about the research, and every number in it is cited to its primary source.

There's an intuition almost every LLM user shares: more context = better answers. Million-token context windows are marketed as a capability. We paste in whole files "just in case." Our coding agents accumulate every grep result, every file read, every test log, on the theory that the model might need it.

The research says this intuition is not just wrong — it's backwards, and the size of the effect is embarrassing.

The result that should change how you build

The cleanest demonstration comes from the Hindsight memory system (arXiv:2512.12818, demo at ACL 2026). On LongMemEval — a benchmark of questions over long conversational histories — the same open-source 20B model scores:

39.0% when handed the full context — everything, the whole history
83.6% when handed a curated slice selected by a structured memory system

Same model. Same available information. The difference is that one setup made the model read everything, and the other selected what mattered. +44.6 points from subtraction.

It gets more uncomfortable: that 20B model with curated context also beats full-context GPT-4o, which scores 60.2% on the same benchmark. A model a fraction of the size, winning because someone cleaned its desk. As the authors put it, the memory architecture — not model scale — drives the performance.

One benchmark, one paper? No — this is a pile-on:

"Lost in the middle" (Liu et al., 2023) established the shape of the problem early: models attend well to the start and end of long contexts and poorly to the middle — exactly where your agent's fifteenth tool result lives.
Context-rot studies (notably Chroma's 2025 report) showed performance degrading as context grows even when the added tokens are relevant, and degrading faster when they're distractors.
A whole 2026 research wave now treats context as a resource to be managed, not maximized: "Agentic Context Engineering" was accepted at ICLR 2026, active context compression systems prune their own working memory (arXiv:2601.07190), and two consolidating surveys (arXiv:2512.13564, arXiv:2603.07670) formalize memory as a write–manage–read loop — with "manage" doing the heavy lifting.

The marketing said "bigger window." The research says "better librarian."

Why more context makes things worse

Three mechanisms, all well-documented:

1. Attention is a budget, not a spotlight. Every token in context competes for attention mass. Pack in 50k tokens of tool output and the three lines that matter are now competing with 49,900 tokens of noise. Needle-in-a-haystack benchmarks — the ones vendors publish — test retrieval of a planted string, which models are good at. Real tasks require reasoning over the context, which degrades much faster.

2. Distractors don't just dilute — they actively mislead. The context-rot findings show semantically-similar-but-irrelevant content is worse than random filler. Your agent's context is full of this: old versions of the file it's editing, error messages from an already-fixed bug, grep hits from a deprecated module. Each is a plausible-looking wrong answer sitting one attention head away.

3. Position effects compound over turns. Agents append. Every turn pushes the important early material (the task! the constraints!) toward the middle of the context — the attention dead zone — while burying the recent signal under boilerplate tool output. A long agent session is a machine for constructing worst-case attention layouts.

Coding agents are the pathological case of all three at once: they generate enormous, distractor-dense, structurally-repetitive context (JSON tool results, file dumps, test logs) at machine speed, across dozens of turns.

What the research says works instead

The successful systems in the literature share a shape — they spend compute deciding what the model sees instead of showing it everything:

Selection over inclusion. Hindsight's four memory networks (facts vs. experiences vs. summaries vs. beliefs) exist so retrieval pulls the right kind of memory for each question. The general lesson: retrieval into a small context beats residence in a big one.
Compression as a first-class operation. Active-context-compression agents treat "shrink my working set" as an action the agent itself takes, on par with tool calls. Summarize the resolved, drop the superseded, keep the live.
Structure beats soup. Tabular, labeled, deduplicated context consistently outperforms raw dumps of the same information — the model spends attention on content, not parsing.

What you can do about it today

You don't need a research memory system to benefit:

Treat context as a liability with interest, not an asset. Every tool result you leave in the window is re-read (and re-billed) every subsequent turn, while making each turn slightly dumber.
Compact aggressively and early. Whatever your agent's compaction/clear mechanism is (/compact, /clear, fresh sessions per task), use it before quality degrades — by the time you notice the agent going in circles, the context has been hurting you for many turns.
Scope sessions to tasks. One task, one session. The 40-turn omnibus session is the exact scenario the position-effect research warns about.
Isolate research from execution. Subagents (or separate sessions) that read a lot and report a little are context firewalls: the summary crosses over; the 30k tokens of grep output don't.
Compress tool outputs before they enter context. Test logs, JSON blobs, and directory listings compress 40–90% with zero information the model actually needs lost. Whether you do it with a proxy layer (this is the part where I mention that's what Lynkr does), a harness setting, or a wrapper script — do it somewhere.

The next time a model launch leads with context-window size, remember the 20B model with a good librarian beating GPT-4o with a hoard. Capacity isn't capability. Curation is.

Primary sources: Hindsight (arXiv:2512.12818) · benchmark data · Lost in the Middle (Liu et al.) · Active Context Compression (arXiv:2601.07190) · Memory in the Age of AI Agents — survey (arXiv:2512.13564) · Memory for Autonomous LLM Agents — survey (arXiv:2603.07670)

How We Built an Agentic-Task Detector for LLM Routing

Lynkr — Tue, 14 Jul 2026 04:18:47 +0000

Disclosure: I maintain Lynkr, the open-source LLM router whose agentic detector this post dissects. Every snippet below is real, shipping code — read the whole file here, it's 350 lines.

"Fix the auth bug in session.js."

Eight words. Every token-count heuristic on earth routes this to the small, cheap model — it's short. And every one of them is wrong, because those eight words are about to unleash a grep → read → edit → test loop with exact-string file edits, the precise workload where small models fumble tool calls and kill sessions.

The inverse request — three paragraphs asking for a detailed comparison of locking strategies — looks expensive and routes safely to a free local model, because it's pure text generation. Size and stakes are nearly uncorrelated in coding-agent traffic. So the router's real job is detecting agentic intent, and this post is a tour of how Lynkr's detector does it: the signals, the weights, the classification ladder — and the embarrassing false positive that almost made the whole thing useless.

Not "agentic: yes/no" — a ladder

The first design decision: agentic-ness isn't boolean. The detector classifies requests into four types, each with a minimum tier floor and a score boost fed into the complexity scorer:

const AGENT_TYPES = {
  SINGLE_SHOT: { minTier: 'SIMPLE',    scoreBoost: 0 },   // request-response, no tools
  TOOL_CHAIN:  { minTier: 'MEDIUM',    scoreBoost: 15 },  // read -> edit -> test
  ITERATIVE:   { minTier: 'COMPLEX',   scoreBoost: 25 },  // retry loops, debugging cycles
  AUTONOMOUS:  { minTier: 'REASONING', scoreBoost: 35 },  // "figure it out", full autonomy
};

The minTier is a floor, not a suggestion: even if every other dimension scores low, an ITERATIVE request cannot route below the COMPLEX tier. Mid-debugging-loop is the worst possible moment to hand the session to a 7B model.

The six signals

Each request accumulates a score from six independent signals. The interesting part is why each one exists:

1. Tool count (up to +25). Many tools attached usually means the client is prepared for multi-step work. Usually. This signal is also the source of the great false positive — hold that thought.

2. Agentic tools specifically (up to +25). Not all tools are equal evidence. Bash, Write, Edit, Task, git and test runners form an explicit set — these mutate state, and their presence signals mutation work. A request that can only Read/Grep/WebSearch sits in a separate read-only set and earns nothing here. Two requests with five tools each can be night and day.

3. Prior tool results (up to +30 — the heaviest signal). If the conversation already contains tool_result blocks, you're not predicting an agentic loop — you're inside one. More than five results means a deep loop with accumulated exact state (file contents, error strings); downgrading the model now throws away the context discipline keeping that loop convergent.

4. Language patterns (up to +25 each). Regexes over the last user message:

tool-chain: "then use", "after that", "step 2"
iterative: "keep trying", "until", "retry", "debug"
autonomous: "figure out", "make it work", "on your own", "whatever it takes"
multi-file: "across the codebase", "refactor entire", "everywhere"

Plus a combination rule: "implement" alone is +10-ish planning noise, but "implement" and "test/verify/make sure" in the same request is +15 — build-and-verify phrasing is a reliable tell of real work.

5. Conversation depth (up to +20). Fifteen-plus messages means established context and momentum.

6. Prompt length (+10). The weakest signal, deliberately — see the opening paragraph.

Score ≥ 25 → the request is agentic. The classification ladder then applies both thresholds and signal combinations — AUTONOMOUS needs score ≥ 60, or an explicit autonomous phrase with score ≥ 40. A phrase alone doesn't do it; a high score without autonomous language doesn't either, unless it's overwhelming.

The false positive that almost sank it

Early versions had a humiliating problem: every single Claude Code request scored agentic. Including "hello."

Why? Claude Code attaches its full tool loadout — Read, Write, Edit, Bash, Grep, Glob, Task, and friends — to every request, even a greeting. Signals 1 and 2 saw 11+ tools, four of them mutating, on everything. Every request cleared the threshold, every request routed to expensive tiers, and the router's entire value proposition — savings — evaporated. The detector was technically working and practically useless.

The fix ships as client profiles: known harnesses (Claude Code, Cursor, Codex CLI) have documented baseline loadouts, and the tool-count signals score only the tools beyond that baseline:

// Signals 1 & 2 score only tools BEYOND the harness's baseline loadout —
// Claude Code's 11 always-attached tools shouldn't count as "agentic
// intent" on their own.
toolsForScoring = clientProfiles.effectiveTools(payload, profile);

Crucially, signals 3–6 still use the full payload — prior tool results and conversational language are genuine evidence regardless of which harness sent them. Only the tool-presence signals get the subtraction, because only they are polluted by the harness's constant.

And for traffic from harnesses we've never seen? A guard: if every attached tool looks like a standard baseline and there are 10+ of them, the tool-count signals zero out rather than fire:

} else if (clientProfiles.allToolsAreBaseline(payload) && rawTools.length >= 10) {
  // Unknown harness that looks like Claude Code / Cursor / Codex —
  // zero out the tool-count signals to avoid the same trap.
  toolsForScoring = [];
  scoringNote = 'unknown_harness_guard';
}

Better to under-detect and lean on the five uncorrupted signals than to re-create the everything-is-agentic bug for unknown clients.

One subtle consequence, preserved as a comment in the source: with the baseline subtracted, tool counts rarely reach the AUTONOMOUS threshold on their own — so the autonomous phrase pattern becomes the primary path to the top classification. The signal design acknowledges its own post-fix physics.

What it still gets wrong

Honesty section. Known limitations, from the code itself:

It reads only the last user message. "Do what I described above" carries the intent of an earlier message the regexes never see. Conversation-depth and tool-result signals partially compensate — but pattern detection is myopic by design (scanning full history was too noisy).
Regexes can't tell mention from intent. "Why did the retry loop break?" trips the iterative pattern despite being a read-only question. In practice this fails safe — over-routing a question up-tier costs cents, under-routing an edit session down-tier costs the session — but it's still a false positive.
English only. The patterns are English regexes; agentic intent in other languages leans entirely on the structural signals.

Every detection returns its full evidence — score, signal list with weights, classification, and a scoringNote explaining any baseline subtraction — so when the router misjudges, the telemetry shows exactly which signal lied. Debuggability was a design requirement: a routing layer you can't interrogate is a routing layer you'll eventually rip out.

Takeaways if you're building anything similar

Subtract the constant before reading the signal. Whatever your equivalent of "the harness always attaches 11 tools" is — find it and remove it, or every request looks the same.
Separate "prepared for tools" from "already using tools." Attached tools are weak evidence; tool_result blocks in the conversation are near-proof.
Fail toward the expensive model. Asymmetric costs mean your threshold should be calibrated so mistakes over-spend pennies rather than break sessions.
Make the detector explain itself. A score without a signal list is a black box you'll never be able to tune.

The whole detector is 350 lines of dependency-free JavaScript: src/routing/agentic-detector.js. Steal it, or tell me which of your prompts it would misjudge — the failure cases are the roadmap.

Choosing a Local Tier for Your Coding Agent (July 2026 Edition)

Lynkr — Wed, 08 Jul 2026 00:14:16 +0000

Disclosure: I maintain Lynkr, the open-source router used in the config examples. The benchmark figures below are third-party or vendor-reported (flagged where vendor-only) — I haven't independently benchmarked these models yet; the point of this post is to help you match models to request classes and test on your own workload.

June 2026 was the busiest month for open-weight coding models in recent memory: GLM-5.2, MiniMax M3, Kimi K2.7 Code, Gemma 4, and NVIDIA's Nemotron 3 Ultra all landed within weeks. If you route your coding agent's simple requests to a local model — the "cloud architect, local coder" pattern — your options just changed meaningfully.

Here's how I'd map the current field onto routing tiers, by hardware budget and by what each model can safely own.

First, the trap: "best open model" ≠ "your local tier"

The headline model of the month, GLM-5.2, scores 62.1% on SWE-bench Pro — above GPT-5.5. It is also a 744B-parameter MoE whose 2-bit quant alone wants ~245 GB of memory. That's an open-weight model, not a local model; for self-hosters it's a $40k-rig proposition (one published build runs it on four RTX PRO 6000s). The same goes for DeepSeek-V4 Pro and MiniMax M3: superb models you'll realistically consume via API, where they belong in your COMPLEX/REASONING tiers, not your local one.

Your local tier is decided by a harsher question: what fits in your VRAM and still makes reliable tool calls?

The local field, by hardware budget

~16 GB RAM (ordinary laptop): Gemma 4 12B. Released June 3 as a dense 12B that genuinely fits consumer RAM (SitePoint's guide). Apache-2.0-class licensing with no usage clauses. This is a SIMPLE-tier model: explanations, one-liners, commit messages, "what does this error mean." I would not hand it an Edit tool.

24 GB GPU (RTX 3090/4090 class): Qwen3.6-27B — still the default answer. The community's consensus "local Claude" since April: within a few points of frontier models on SWE-bench Verified (77.2 reported vs Claude's 80.9 — analysis), Apache-2.0, runs quantized on a single 24 GB card or a ~$2k build. Its known weakness is exactly the one that matters for agents: tool-call reliability drifts in long contexts — fine as a supervised MEDIUM tier, risky as an unsupervised COMPLEX one.

Agentic multi-file edits on similar hardware: Devstral Small 2. Purpose-built for multi-file, tool-driven coding rather than chat (KDnuggets roundup). If your traffic is edit-heavy, it can arguably take MEDIUM-tier mutation requests that I'd keep away from general chat models.

Autocomplete-shaped work: Codestral 22B is fast and good at it — but mind the non-commercial license before using it for work.

One rule that keeps proving out (Pinggy's guide): within the same memory budget, a bigger model at Q4 usually beats a smaller one at Q8. Quantization choice matters nearly as much as model family.

Mapping to tiers

Putting that together into a routing config (Lynkr shown; the mapping logic applies to any router):

# 24 GB GPU + API keys for the hard stuff
TIER_SIMPLE=ollama:gemma4:12b            # trivia, explanations, greetings
TIER_MEDIUM=ollama:qwen3.6:27b           # code questions, supervised edits
TIER_COMPLEX=deepseek:deepseek-v4-flash  # tool-heavy mutations, via API
TIER_REASONING=deepseek:deepseek-v4      # architecture, multi-step planning

Why V4 Flash for COMPLEX: it's the first open-weight model teams report dropping into real agentic pipelines as a frontier substitute on price (OpenRouter's June analysis) — the cheapest "won't break the session" option right now. Kimi K2.7 Code (vendor-reported 58.6% SWE-bench Pro at ~30% fewer reasoning tokens) and GLM-5.2 are strong API-tier alternatives; all the June day-one numbers are vendor-reported, so treat them as directional until LiveBench catches up.

The key discipline: the boundary between MEDIUM and COMPLEX should not be "how big is the request" but "will tools mutate state." Local models in this class handle read-and-explain reliably; exact-match edits and bash execution are where they still break sessions — I wrote up those failure modes here.

What changed vs three months ago

The floor rose. A 16 GB laptop now runs a genuinely useful SIMPLE tier (Gemma 4). Six months ago that tier meant 3B models that couldn't be trusted with a paragraph.
The open-weight ceiling now beats proprietary on some coding benchmarks (GLM-5.2 > GPT-5.5 on SWE-bench Pro) — but at server scale, which strengthens the hybrid pattern: open models via cheap APIs up top, small open models on your metal below.
MoE won. Every serious June release is Mixture-of-Experts. For self-hosters this cuts both ways: better quality-per-active-param, but total memory footprints that keep the top tier out of reach.
Licensing is consolidating around MIT (DeepSeek) and Apache-2.0 (Qwen, Gemma) for the models you'd actually build on.

Test on your traffic, not on benchmarks

Every number above is someone else's workload. The honest way to pick your local tier: route a week of your real traffic through whatever candidates fit your hardware, and count session survival — how often the local model's tool calls held up — not just benchmark deltas. That's a one-line config change per candidate, and your own telemetry will contradict at least one thing this post told you.

Lynkr is Apache-2.0, self-hosted, and treats every model above as a first-class routing tier: github.com/Fast-Editor/Lynkr.

Routing Down Is Easy. Knowing When Not To Is Hard: Why Cheap Models Break Your Coding Agent

Lynkr — Wed, 08 Jul 2026 00:08:36 +0000

Disclosure: I maintain Lynkr, an open-source router whose design decisions this post explains. The failure modes described are patterns widely reported across router issue trackers and local-LLM forums — the examples are representative reconstructions, not captured transcripts. The problem is real either way; ask anyone who's routed a coding agent to a 7B model.

Everyone who gets their first LLM router working does the same thing within the hour: point the expensive coding agent at a free local model and watch the bill drop to zero.

Then the agent tries to edit a file.

The graveyard of downgraded sessions

If you browse the issue tracker of any Claude Code router — or r/LocalLLaMA on any given week — you'll find the same story in a hundred variations. The routing works perfectly. The session dies anyway. The killers, in rough order of frequency:

1. Malformed tool arguments. The agent decides to call Edit, and the model produces arguments that are almost JSON:

{"file_path": "src/auth.js", "old_string": "if (token) {", "new_string": "if (token && !expired) {"

One missing brace. The harness rejects the call, the model retries, produces a different malformation, and you're three turns deep into fixing nothing. Frontier models emit structurally valid tool calls with boring reliability; sub-10B models do it most of the time — and "most of the time," at 30 tool calls per session, means every session breaks.

2. Stale string matching. Edit-style tools require the old_string to match the file exactly. Small models paraphrase from memory instead of quoting — they'll "remember" the line as if (token) { when the file says if (accessToken) {. The edit fails, the model re-reads the file, burns 2,000 tokens, tries again with a different paraphrase. This is the single most reported failure, because it looks like the router's fault and is actually a capability cliff.

3. Hallucinated context. Ask a small model to run tests and it may confidently call Bash with npm test -- --grep "auth" in a repo that uses pytest. It's not being stupid — it's pattern-completing from training data instead of the conversation, because instruction-following degrades faster than fluency as models shrink.

4. The infinite loop. The subtlest one: the model calls Read on the same file five times in a row, or greps, reads, greps the same term again. Weak models lose the thread of what they already know in long agentic contexts. Nothing errors — the session just stops converging while tokens burn.

Here's the uncomfortable part: none of these are the router's bug, and all of them are the router's fault. The router made a judgment — "this request is cheap-model-safe" — and the judgment was wrong.

Why the obvious heuristics misjudge

Most routing setups decide with static rules: token thresholds, keyword lists, scenario slots. These fail in a specific, predictable way: they measure the request's size, not its stakes.

"Fix the auth bug in session.js" is eight words. Every token-based rule on earth routes it to the small model. But those eight words unleash a read-grep-edit-test loop — the exact workload where small models faceplant. Meanwhile, "explain the difference between optimistic and pessimistic locking, with examples" looks expensive (long answer, technical vocabulary) and is actually perfectly cheap-model-safe: it's pure text generation, no tool calls, no exact string matching, nothing to break.

Size and stakes are almost uncorrelated in agentic traffic. That's the whole problem.

What "stakes-aware" routing looks like

When I built Lynkr's router, most of the design ended up being about when not to save money. The parts that matter:

Weight the tools, not just their count. A request where Grep and Read are in play is research — paraphrase-tolerant, failure-tolerant, ideal for a local model. A request where Bash, Write, or Edit will fire is a mutation with exact-match requirements. Lynkr assigns each tool a risk weight (Bash 0.9, Write 0.8, Edit 0.7 … Grep 0.2) and scores the request's effective toolset. Two requests with five tools each can land tiers apart.

Treat mid-session as a signal. If the conversation already contains three tool results, you're inside an agentic flow with accumulated exact-state (file contents, error strings). Downgrading the model mid-flow throws away the one thing that was keeping the loop convergent. Prior tool usage and conversation depth push requests up-tier even when the latest message is short.

Subtract the harness baseline. Claude Code ships ~14 tool schemas with every request — including "hello." Count them naively and everything looks agentic, so nothing ever routes local and you save nothing. Score only the tools the request could plausibly use, and the safe majority routes down while the risky minority stays up.

Some patterns override everything. Greetings and "what does X do" questions force-route local, always. Security-sensitive analysis force-routes to the strong tier, always — a JWT architecture question is short, toolless, and precisely the wrong place to save four cents.

The result on my own traffic: 70–90% of requests route to free local models — but they're the right 70–90%, which is the entire difference between "my bill dropped" and "my agent broke."

Takeaways, router-agnostic

Route research down, mutations up. If your router can't tell a Grep request from an Edit request, it isn't routing — it's gambling on which sessions break.
Never downgrade mid-loop. Model consistency across an agentic sequence is worth more than the marginal savings of one cheap turn.
Measure session survival, not just cost. A routing setup that saves 60% and breaks one session in five is more expensive than the bill it replaced — you're paying in re-runs and rage.
The ceiling is rising. Local models' tool-calling improves every quarter; the set of safely-downgradable requests grows with it. A router with per-tool judgment gets to expand that set gradually. A token threshold has to guess again from scratch.

The router's job was never "pick the cheapest model." It's "pick the cheapest model that won't break the session" — and those five extra words are where all the engineering lives.

The scorer described here is ~1,000 lines of readable Apache-2.0 JavaScript: src/routing/complexity-analyzer.js. Steal the design, or file an issue telling me where it misjudges — the failure cases are the interesting part.

The 5% Router Tax: What Hosted LLM Gateways Charge For (and How to Self-Host It)

Lynkr — Sun, 05 Jul 2026 08:48:56 +0000

Disclosure: I maintain Lynkr, the self-hosted gateway discussed in the second half. OpenRouter and Requesty are good products — this post is about understanding what you're paying for so you can decide whether you need to.

Hosted LLM routers had a huge 2026 — OpenRouter alone pushes 25 trillion tokens a week. The pitch is real: one API key, 400+ models, automatic failover. The price is a ~5% fee on every token you route (5.5% on OpenRouter credits, 5% on Requesty), plus a subtler cost: every prompt, every file your coding agent reads, every secret that leaks into a context window transits their infrastructure.

For a hobby project, 5% of a small bill is nothing and the convenience wins. For an agentic coding workload — where teams routinely spend $500–$2,000 per engineer per month — 5% is real money, and the data-transit question stops being academic. So it's worth asking precisely: what does the hosted router actually do for that fee, and which parts can you self-host?

What the fee buys

Unified API across providers — one format in, translated per-provider out.
Failover — a provider 500s, your request retries elsewhere.
Model marketplace — new models available the day they launch.
Consolidated billing — one invoice instead of six provider accounts.
(Sometimes) smart routing — OpenRouter's auto router picks a model per-request.

Items 1, 2, and 5 are software. Items 3 and 4 are genuinely hard to self-host — if you want day-one access to every new model with zero account setup, the marketplace earns its fee. But most coding workloads use a handful of models, not four hundred.

The parts a hosted router structurally can't give you

Local models as a tier. No hosted router will route your easy requests to the Ollama instance on your own machine — free, private, zero latency to first byte on cached weights. For coding traffic, where (in my instrumented sessions) 70–90% of requests are simple enough for a good local model, this is the single biggest cost lever, and it's only available to something running on your side of the wire.
Your data staying home. Self-hosted means prompts, code, and keys never transit a third party. For anyone with a compliance requirement — or code they'd rather not ship to a router's logs — this isn't a preference, it's a prerequisite.
Token optimization before the bill. A hosted router bills you for the tokens you send it — it has no incentive to shrink them. A self-hosted proxy can strip unusable tool schemas (measured: −53% on tool-heavy requests) and compress JSON tool results (measured: 3,458 → 427 tokens on a grep result) before any provider bills you. That's not a routing saving; it stacks on top of routing.
No availability dependency. Hosted routers go down (OpenRouter's outages have their own HN threads) and offer no SLA at consumer tiers. A local proxy fails independently of anyone's status page.

What self-hosting costs you

Honesty cuts both ways:

You run a process. npm install -g lynkr && lynkr init && lynkr start — but it's yours now: updates, logs, the works.
You manage provider accounts. Two or three API keys instead of one. The consolidated invoice is genuinely gone.
Model lag. A new provider means waiting for support (or a PR) instead of it appearing in a dropdown.
Nobody to email. Self-hosted support is a GitHub issue tracker.

If those trade-offs read as "fine," the math is straightforward: the 5% fee disappears, the local-tier routing removes the easy majority of requests from your bill entirely, and compression shrinks what's left.

The hybrid that actually makes sense

This isn't either/or. A pattern I see working:

Coding tool → self-hosted proxy (Lynkr)
                ├─ SIMPLE/MEDIUM  → local Ollama/llama.cpp   (free)
                ├─ COMPLEX        → direct provider API keys  (no fee)
                └─ exotic models  → OpenRouter               (5% on the long tail only)

Keep a hosted router as one backend for the long tail of models you rarely need, route the bulk directly or locally, and let the proxy's classifier decide per-request. You get the marketplace when you want it without paying the tax on your entire volume.

Lynkr is Apache-2.0, self-hosted, supports 13 providers including Ollama, llama.cpp, LM Studio, Bedrock, Azure, Databricks — and OpenRouter itself as a tier: github.com/Fast-Editor/Lynkr. Benchmarks with methodology are in the repo; run them on your own workload before believing anyone's percentages, including mine.

How a 13-Dimension Complexity Scorer Decides Which Model Gets Your Request

Lynkr — Sun, 05 Jul 2026 08:48:40 +0000

Disclosure: I'm the author of Lynkr, the open-source proxy whose internals this post walks through. All code shown is real and Apache-2.0 — read it here.

The most expensive default in AI coding tools is that model choice is a setting, not a decision. You pick a model once; every request — "what does git stash do?" and "refactor this auth module" alike — goes there. Routing each request to the cheapest model that can actually handle it is worth 50%+ of most bills, but it only works if the "can actually handle it" judgment is reliable. Get it wrong downward and a small model fumbles your file edits; get it wrong upward and you've saved nothing.

Here's how Lynkr makes that judgment, in enough detail that you could reimplement it.

Why not just count tokens?

The obvious heuristics fail in both directions:

"Long request → big model" fails on a 60k-token context that's mostly grep output around a trivial question.
"Short request → small model" fails catastrophically on "fix the auth bug in session.js" — eight words that unleash a tool-heavy agentic session a 7B model will faceplant on.

Token count is one signal. The failure cases all come from treating it as the only one.

The architecture: weighted dimensions, then overrides

Every request gets a 0–100 score from 13 dimensions in four groups. The weights are configurable; these are the defaults:

const DIMENSION_WEIGHTS = {
  // Content Analysis (35%)
  tokenCount: 0.08,
  promptComplexity: 0.10,   // avg sentence length/structure
  technicalDepth: 0.10,     // technical keyword density
  domainSpecificity: 0.07,  // security/ML/distributed/db/frontend/devops
  // Tool Analysis (25%)
  toolCount: 0.08,
  toolComplexity: 0.10,     // which tools, not how many
  toolChainPotential: 0.07, // "first...then", "step 2", sequencing language
  // Reasoning Requirements (25%)
  multiStepReasoning: 0.10,
  codeGeneration: 0.08,
  analysisDepth: 0.07,      // trade-off/comparison markers
  // Context Factors (15%)
  conversationDepth: 0.05,
  priorToolUsage: 0.05,     // tool_results already in the conversation
  ambiguity: 0.05,
};

A few design decisions worth stealing:

Not all tools are equal. A request that can Grep is not like a request that can Bash. Each tool carries a hand-tuned risk weight — Bash 0.9, Write 0.8, Edit 0.7, down to Grep at 0.2. A request whose available toolset averages 0.8 is an agentic mutation session; one averaging 0.25 is read-only research. Same tool count, completely different stakes.

Subtract the harness baseline. Claude Code ships ~14 tool schemas with every request, including "hello". If you count them naively, everything looks agentic and nothing routes local. The scorer subtracts the client's constant baseline and scores only the effective tools the request could plausibly use — one of those fixes that sounds trivial and changed everything.

Conversation history is a signal. Three tool_result blocks already in the conversation means you're mid-agentic-flow — this is not the moment to downgrade models and break the session's momentum. priorToolUsage and conversationDepth push mid-session requests up-tier.

Ambiguity cuts the other way. "file X, line 42, this error" is specific — a small model can act on it. "Something feels slow sometimes" needs interpretation before action. Specificity markers (paths, line numbers, error strings) lower the score.

Overrides: the classifier knows what it can't know

Two pattern lists short-circuit the whole scoring pipeline:

Force-local: greetings, acknowledgments, "what does X do" one-liners. Score 0, never leave the machine, no cloud tokens ever.
Force-cloud: security-critical analysis, architecture decisions, anything matching high-risk patterns. Straight to the top tier regardless of how cheap it looks. A JWT-vs-cookies security question is short and toolless — every naive heuristic routes it local. This is the wrong request to save $0.004 on.

On top of the regex dimensions, an AST pass (tree-sitter) scores actual code structure in the payload — cyclomatic signals beat keyword counting when real code is present.

From score to model

score < threshold        → SIMPLE   (e.g. ollama:qwen2.5:7b, free)
threshold..~65           → MEDIUM   (e.g. ollama:qwen2.5-coder, free)
above                    → COMPLEX  (your API key: Sonnet, GPT-4o...)
reasoning markers heavy  → REASONING (o3, DeepSeek R1...)

The threshold moves with a single mode switch — aggressive (60) routes more local, conservative (25) routes more to the cloud, default is 40. Multi-turn conversations score with a recency-weighted sliding window, so a short follow-up ("now add tests") inherits the complexity of the work it refers to instead of scoring as a trivial one-liner.

Crucially, the classifier only chooses among models you listed. It's not an autonomous agent picking providers — you define the tiers, it picks the tier.

Does it work?

In my instrumented sessions, 70–90% of requests score SIMPLE or MEDIUM and run free on local models, while tool-heavy and security-flagged requests reliably escalate. The failure mode everyone fears — cheap model breaking an agentic session — is exactly what the tool weights, baseline subtraction, and prior-tool-usage dimensions exist to prevent.

Is 13 hand-weighted dimensions the optimal design? Almost certainly not — a learned router trained on outcome data would beat it eventually. But it's transparent (every routing decision logs its per-dimension breakdown), it's tunable, it runs in-process in microseconds, and it never sends your prompts to a third-party classifier API.

The whole thing is readable in one sitting: src/routing/complexity-analyzer.js. Steal the design or use the proxy — either outcome means fewer frontier-model tokens spent on git stash questions.

The 21,000-Token Typo: Where Agentic Coding Budgets Actually Die

Lynkr — Sun, 05 Jul 2026 08:48:24 +0000

Disclosure: I maintain Lynkr, an open-source proxy mentioned at the end. The first 80% of this post is tool-agnostic and the takeaways apply whether or not you ever use it.

There's a documented case of a coding agent burning 21,000+ input tokens to fix a one-line README typo. Not a bug. Not a runaway loop. That's the normal cost structure of agentic coding, and once you see why, you can't unsee it on your own bill.

Stanford's Digital Economy Lab measured it: agentic tasks consume on the order of 1000x the tokens of ordinary code chat, and the same task with the same agent can vary 30x in cost depending on how the session unfolds. Teams running heavy automation report $500–$2,000 per engineer per month. So where does it go?

The anatomy of one "small" agentic task

Say you ask your agent to fix a typo. Here's what actually crosses the wire:

Turn 1: Your one-line prompt... plus the system prompt, plus ~14 tool schemas (Write, Edit, Bash, Grep, Git — a couple thousand tokens before anyone thinks).

Turn 2: The agent greps for the file. The result comes back as JSON — paths, line numbers, match context, metadata. A modest grep is easily 1,000–3,000 tokens. It's now in the context and gets re-sent on every subsequent turn.

Turn 3: The agent reads the file. Add the full file contents to the context. Re-sent every turn from now on.

Turn 4: The edit itself — the cheapest part of the entire session.

Turn 5: Verification: re-read, maybe run a linter, another JSON blob of output.

Five turns, and your one-line fix carried: 5x the tool schemas, 4x the grep results, 3x the file contents. Input tokens dominate output roughly 25:1 in typical sessions. You're not paying for intelligence — you're paying for cargo.

The three structural leaks

Leak 1: Tool schemas on every request. The agent might use two tools this session. You ship fourteen schemas every turn anyway, because the client doesn't know which ones matter. Measured on a realistic Claude Code request: schemas the request couldn't use accounted for 53% of billed input tokens.

Leak 2: Raw JSON in the context window. JSON is the least token-efficient format your context will ever hold — keys repeated per element, quotes, braces, whitespace. A 60-match grep result: ~3,400 tokens raw, 427 after conversion to a tabular token-oriented format with redundant fields stripped. Nothing lost that the model needed.

Leak 3: Frontier models on non-frontier requests. "What does git stash do?" does not need the same model as "refactor this auth module." But your client sends both to the same place, because model choice is a config setting, not a per-request decision. In my instrumented sessions, 70–90% of requests scored as simple or medium complexity — they'd be fine (and free) on a local model.

What to do about it — tool-agnostic

Instrument before optimizing. Log tokens per request by category (schemas / tool results / conversation). You cannot fix a leak you haven't sized. Most people find their intuition about their own spend is wrong.
Never let raw JSON accumulate in a context window. Compact it, tabularize it, or summarize it. Tabular JSON is nearly free compression — same information, a fraction of the tokens.
Keep sessions short and contexts clean. Every tool result you leave in the context is a recurring charge, billed again on every turn until the session ends.
Match model to request, not to workflow. Route the easy 80% somewhere cheap or local; reserve the frontier model for the requests that actually exercise it. Bring your own API keys and the routing is entirely within your control.

The plumbing version

Everything above can be done manually. I got tired of doing it manually, so I built it into a proxy: Lynkr sits between your coding tool (Claude Code, Cursor, Codex CLI) and your providers, strips unusable tool schemas, compresses JSON tool results, caches semantically, and scores each request on 13 dimensions to route it to a tier you define — local Ollama for the easy stuff, your API keys for the hard stuff. Self-hosted, Apache-2.0, no markup, zero client changes.

But the numbers above aren't about my tool. They're about a cost structure every agentic workflow shares. The 21,000-token typo isn't an outlier — it's the default. Measure yours.

Lynkr vs claude-code-router: Static Rules vs a Complexity Classifier

Lynkr — Sun, 05 Jul 2026 08:48:08 +0000

Disclosure: I'm the author of Lynkr. claude-code-router is a genuinely good project that pioneered this category — this is a technical comparison of two different approaches, not a takedown. Where CCR is the better choice, I say so.

If you want to keep the Claude Code harness but route requests to other models, you have two main self-hosted options today: claude-code-router (CCR, ~35k stars, the incumbent) and Lynkr. They solve the same problem with fundamentally different architectures, and which one fits you depends on how much you want to configure versus delegate.

The core difference in one paragraph

CCR routes by scenario rules you write. It has slots — default, background, think, longContext (triggered above a token threshold), webSearch, image — and you assign a model to each. It's predictable, transparent, and entirely under your control.

Lynkr routes by scoring the request itself. Every request gets a 0–100 complexity score computed from 13 weighted dimensions — token count, technical keyword density, tool complexity, multi-step reasoning markers, conversation depth, ambiguity, and so on — and lands in a tier (SIMPLE/MEDIUM/COMPLEX/REASONING) you've mapped to models. You configure the tiers once; the classifier decides per-request.

Where CCR wins

Maturity and ecosystem. 35k stars, ~730k monthly npm downloads, 20+ provider transformers, custom JS plugins, a web UI, and in-session /model switching. If you hit a weird provider quirk, someone has already hit it.
Predictability. A rule is a rule. If you want "long contexts always go to Gemini", CCR expresses that in one line and never surprises you.
Claude Code specialization. CCR does one client deeply. Lynkr supports Claude Code, Cursor, Codex CLI, Cline, and Continue — breadth costs some depth.

Where the rule-based approach breaks down

Browse CCR's issue tracker (~1,000 open issues) and one complaint dominates: tool-calling breakage on downgraded models — failed file edits, broken git operations, agents going in circles. The root cause usually isn't CCR's code. It's that static rules can't see what the request needs:

A short prompt ("fix the auth bug in session.js") looks cheap by token count — but it's an agentic, tool-heavy task that a small local model will fumble.
A long context triggers the longContext rule — but if it's 60k tokens of grep output around a trivial question, an expensive long-context model is wasted money.

Token counts and scenario names are proxies. The thing you actually care about — can a cheap model handle this without breaking the session? — requires looking at the request's structure.

What Lynkr does differently

Three things, all absent from CCR by design (it aims to be a lean router):

1. The complexity classifier. Requests with agentic signals (write/edit/bash tool availability, prior tool results in the conversation, sequential-step language) score into higher tiers even when they're short. Trivia stays local even when the context is long. Force-patterns short-circuit both ways: greetings never hit the cloud; security-critical analysis never gets downgraded. The design goal is exactly the failure mode above — route down only when the answer will still work.

2. Token optimization on the wire. Lynkr strips tool schemas the request can't use (measured: 53% fewer tokens on a realistic 14-tool Claude Code request) and compresses large JSON tool results before they hit the model (measured: 3,458 → 427 tokens on a 60-match grep result). CCR forwards requests as-is.

3. Semantic caching. Paraphrased repeat questions are served from an embedding cache in ~171ms with zero tokens billed.

Honest comparison table

	claude-code-router	Lynkr
Routing logic	Scenario rules + token threshold	13-dimension complexity score → tiers
Configuration	Per-scenario, per-provider (flexible, verbose)	Pick 4 tier models via `lynkr init` wizard
Tool-schema stripping	No	Yes (−53% measured)
JSON tool-result compression	No	Yes (TOON + field stripping)
Semantic cache	No	Yes
Clients	Claude Code (deep)	Claude Code, Cursor, Codex CLI, Cline, Continue
Provider transformers/plugins	20+, custom JS	13 providers built-in
Ecosystem maturity	~35k stars, huge community	Young (~500 stars), one maintainer
In-session model switching	Yes (`/model`)	No (automatic per-request)
License	MIT	Apache-2.0

Which should you use?

You want explicit control and a battle-tested ecosystem → CCR. It's the safe default and its community is unmatched.
You're tired of tuning rules, or your cheap-model sessions keep breaking → try Lynkr. The classifier exists precisely because static rules degrade on agentic workloads.
Your bill is dominated by tool output and repeated context → Lynkr, regardless of routing preference; the compression and caching layers work even if you route everything to one model.

Both are self-hosted, free, and take five minutes to try. Run your own workload through each and compare the token logs — that's the only benchmark that matters. Mine are reproducible here: github.com/Fast-Editor/Lynkr.