DEV Community: Lynkr

LLM Inference Is Reinventing the Operating System

Lynkr — Fri, 04 Sep 2026 22:54:56 +0000

Operating systems solved a problem decades ago:

How do you make a limited amount of fast memory appear much larger to applications?

LLM inference is now facing a surprisingly similar problem.

Except instead of CPU processes and memory pages, we have inference requests and KV-cache blocks.

Instead of:

CPU → RAM → Disk

we increasingly have:

GPU HBM → CPU DRAM → NVMe → Remote Memory

And instead of simply executing instructions, the system must continuously decide:

What state should stay in GPU memory?
What can be shared?
What should be evicted?
What should be prefetched?
Which request should run next?
Where should its state live?

Here is how LLM inference is turning into an operating-system problem.

1. The KV Cache Turns Inference Into a Memory-Management Problem

To understand everything else, we first need to understand the KV cache.

Transformers use self-attention.

For every token, attention produces:

a Query (Q)
a Key (K)
a Value (V)

During autoregressive generation, previously generated tokens don't need to have their Keys and Values recomputed every time.

So inference engines store them.

That's the KV cache.

Conceptually:

Token 1 → K1, V1
Token 2 → K2, V2
Token 3 → K3, V3
Token 4 → K4, V4
...
Token N → KN, VN

When generating token N+1, the model can reuse:

K1...KN
V1...VN

instead of recomputing them.

This dramatically reduces computation.

But it creates a new problem:

The longer the context, the more memory the KV cache consumes.

A simplified KV-cache memory calculation is:

Mₖᵥ = 2 × L × T × Hₖᵥ × D × B

where:

(L) = number of transformer layers
(T) = number of tokens
(H_{KV}) = number of KV heads
(D) = head dimension
(B) = bytes per element
2 = Key + Value

Notice the important variable:

Double the context length and you roughly double the KV memory.

Now imagine thousands of simultaneous requests.

Request A → 10K tokens
Request B → 50K tokens
Request C → 2K tokens
Request D → 100K tokens
...

The model weights may be static.

The KV cache is dynamic memory.

And that is where the operating-system analogy begins.

2. PagedAttention Turns KV Cache Into Pages

A naive inference engine might allocate one large contiguous region of memory for every request.

For example:

GPU Memory

┌─────────┬─────────┬─────────┬─────────┐
│ Request │ Request │ Request │ Request │
│    A    │    B    │    C    │    D    │
└─────────┴─────────┴─────────┴─────────┘

But requests don't behave nicely.

Request A may finish.

Request C may grow.

Request D may be paused.

A new request may arrive.

Eventually memory becomes fragmented.

You can have plenty of free memory but not enough contiguous memory for a large allocation.

Operating systems solved a similar problem with paging.

Instead of requiring a process to occupy one contiguous region:

Process A:

[A][A][A][A][A]

the OS divides it into pages:

[A0][A1][A2][A3][A4]

and stores those pages wherever free physical frames are available:

Physical memory:

[A0][C2][B1][A3][D4][A1][FREE][A2]

A page table maps:

Virtual page → Physical frame

Modern LLM serving systems use a similar idea for KV cache.

The KV cache is divided into blocks:

Request A:

[A0][A1][A2][A3][A4]

Those blocks can be placed independently in GPU memory.

Conceptually:

A0 → GPU block 17
A1 → GPU block 42
A2 → GPU block 5
A3 → GPU block 91

The original vLLM/PagedAttention work explicitly uses the virtual-memory analogy to manage KV cache efficiently.

The important idea is:

Logical token state doesn't need to correspond to one contiguous physical memory region.

That single change makes KV memory dramatically easier to manage.

3. KV Cache Sharing Turns Into Shared Memory

Now consider multiple requests with the same prefix.

For example, thousands of users might send requests beginning with:

You are an expert financial analyst.

Here is the company's annual report:
...

The beginning of every request is identical.

Without sharing:

Request A:
[P][P][P][A]

Request B:
[P][P][P][B]

Request C:
[P][P][P][C]

The same prefix is stored three times.

That's wasteful.

Instead, we can share the prefix:

              ┌── Request A
              │
[P][P][P] ────┼── Request B
              │
              └── Request C

One physical set of KV blocks.

Multiple logical requests.

This is conceptually similar to shared memory.

It also resembles copy-on-write:

Shared prefix
      │
 ┌────┼────┐
 A    B    C

As long as requests only read the shared blocks, they can reuse them.

If they diverge, new blocks can be allocated.

Systems such as SGLang's RadixAttention exploit prefix reuse to avoid recomputing common KV state.

Now the inference runtime isn't simply asking:

"How much memory does this request need?"

It can also ask:

"How much of this request can I reuse from memory I already have?"

That is a much more sophisticated memory-management problem.

4. KV Eviction Turns Into a Cache-Management Problem

GPU HBM is limited.

Eventually:

HBM

████████████████████
100% FULL

Something must leave.

This introduces KV-cache eviction.

Suppose we have:

[A][B][C][D][E][F][G][H]

and a new request needs another block.

Which one should we remove?

Possible policies include:

LRU
LFU
FIFO
TTL
Priority
Reuse probability
Cost-aware eviction

The simplest approach is something like LRU:

Evict the least recently used block.

But LLM workloads are more complicated.

Imagine:

Request A → active
Request B → paused
Request C → active
Request D → paused

Request B might not have been used recently.

But perhaps B is an agent waiting for a database query.

In two seconds it may resume and need its entire KV history.

So blindly evicting B could be expensive.

There are now multiple possible costs:

KEEP
 ↓
consume valuable HBM

EVICT
 ↓
reload later

RECOMPUTE
 ↓
use GPU compute

TRANSFER
 ↓
move from another memory tier

The runtime is therefore solving something like:

Bottleneck = min(Cₘₑₘₒᵣᵧ, Cᵣₑₗₒₐd, Cᵣₑcₒₘₚᵤₜₑ, Cₜᵣₐₙₛfₑᵣ)

This is no longer simply "run the neural network."

It is cache management.

5. Hierarchical KV Cache Creates a Memory Hierarchy

Once HBM becomes insufficient, we need somewhere else to put KV state.

Modern systems can increasingly use multiple memory tiers.

Conceptually:

                  GPU
                   │
             ┌─────▼─────┐
             │    HBM    │
             │   HOT     │
             └─────┬─────┘
                   │
             ┌─────▼─────┐
             │   DRAM    │
             │   WARM    │
             └─────┬─────┘
                   │
             ┌─────▼─────┐
             │   NVMe    │
             │   COLD    │
             └─────┬─────┘
                   │
             ┌─────▼──────┐
             │   Remote    │
             │   Storage   │
             └─────────────┘

This looks remarkably similar to a computer memory hierarchy:

CPU cache
   ↓
RAM
   ↓
SSD

Except now:

GPU HBM
   ↓
CPU DRAM
   ↓
NVMe
   ↓
Network

Each level has different:

latency
bandwidth
capacity
cost

So the runtime needs to decide:

Where should this KV block live?

Hot blocks:

HBM

Warm blocks:

DRAM

Cold blocks:

NVMe / remote storage

SGLang's HiCache is an example of a system explicitly exploring multi-level KV-cache management.

At this point, LLM inference is starting to look less like a simple GPU program and more like a memory hierarchy manager.

6. KV Prefetching Turns Into an I/O Problem

Moving KV state between memory tiers can be expensive.

Suppose a request needs a block currently stored in CPU memory.

A naive implementation does:

Need KV
   ↓
Request transfer
   ↓
Wait
   ↓
KV arrives
   ↓
Compute

The GPU is waiting.

A better system tries to predict what will be needed next.

               COMPUTE
                  │
                  │
                  ▼
             ┌─────────┐
             │   GPU   │
             └────┬────┘
                  ▲
                  │
              PREFETCH
                  │
                  │
             ┌────┴────┐
             │  DRAM   │
             └─────────┘

While the GPU is computing, the runtime asynchronously moves future KV blocks closer.

This is classic prefetching.

The objective is to hide:

T_{memory}

behind:

T_{compute}

Ideally:

T_effective ≈ max(T_compute, T_memory)

rather than:

T_{effective}= T_{compute} + T_{memory}

Modern inference research is actively exploring asynchronous KV-cache prefetching because memory movement increasingly becomes a bottleneck.

The inference engine now needs to answer:

What will this request need next?

That's a prediction problem.

And prediction determines memory performance.

7. Cache-Aware Scheduling Turns Into an OS Scheduler Problem

Traditional request scheduling might look like:

Queue:

A
B
C
D
E

The scheduler chooses which request runs next.

But with KV caching, requests aren't independent.

Suppose:

A shares prefix with B
A shares nothing with C
A shares nothing with D

Running:

A → B

might be much more efficient than:

A → C → D → B

because A and B can reuse the same KV state.

So the scheduler now needs to consider:

GPU utilization
+
request latency
+
batch size
+
KV residency
+
prefix reuse
+
memory pressure

The scheduling problem becomes:

``
Choose(request)= f(latency,compute,memory,cache reuse)

This is very similar to an OS scheduler making decisions based on resource availability.

But now the scheduler is also trying to maximize memory locality.

That's a major conceptual shift.

The inference scheduler isn't just scheduling compute. It's scheduling data locality.

8. Prefill/Decode Disaggregation Turns Into Resource Scheduling

LLM inference has two very different phases.

Prefill

The model processes the user's prompt.

For example:

10,000 input tokens
        ↓
parallel processing
        ↓
KV cache generated

This tends to be relatively compute-intensive.

Decode

The model generates output one token at a time:

Token 1
   ↓
Token 2
   ↓
Token 3
   ↓
Token 4

Decode is much more sensitive to memory bandwidth and latency.

So the same GPU is being asked to serve two fundamentally different workloads.

This leads to prefill/decode disaggregation.

Conceptually:

                Requests
                    │
             ┌──────┴──────┐
             │             │
          PREFILL        DECODE
             │             │
             ▼             ▼
        GPU cluster    GPU cluster
             │             │
             └──────┬──────┘
                    │
                 KV state

Now the system can optimize the two phases separately.

You can dedicate resources to:

Prefill:
compute throughput

and:

Decode:
memory bandwidth
latency

This is another classic systems principle:

Different workloads should not necessarily compete for the same resource pool.

It is similar to how operating systems and distributed systems isolate different classes of work.

9. Agentic Workloads Turn KV Management Into a Process-Lifetime Problem

This is where things get really interesting.

A normal chatbot looks like:

Request
   ↓
LLM
   ↓
Response
   ↓
Done

An agent looks more like:

LLM
 ↓
Tool call
 ↓
Wait
 ↓
LLM
 ↓
Database
 ↓
Wait
 ↓
LLM
 ↓
Browser
 ↓
LLM

The request can be inactive for seconds or minutes.

But inactive does not mean finished.

Suppose the agent has accumulated:

50,000 tokens

and is now waiting for a tool.

Should we keep all 50K tokens in HBM?

Maybe.

But if thousands of agents are simultaneously waiting:

Agent A → waiting
Agent B → waiting
Agent C → waiting
...
Agent 10,000 → waiting

we cannot keep everything in HBM.

So we need to understand request lifetime.

This starts looking like process management.

Traditional OS:

RUNNING
   ↓
WAITING
   ↓
RUNNABLE
   ↓
RUNNING
   ↓
TERMINATED

Agent:

GENERATING
   ↓
TOOL WAIT
   ↓
READY
   ↓
GENERATING
   ↓
TOOL WAIT

And memory policy now needs to understand those states.

A request that is waiting might have a high probability of becoming active again.

Therefore:

P(\text{reuse soon})

becomes relevant to eviction.

This is fundamentally different from simple LRU.

Agentic AI is therefore pushing inference toward application-aware memory management.

10. Distributed KV Turns Inference Into a Distributed Operating System

Now scale beyond one GPU.

A production system may have:

GPU 0 ─── GPU 1
 │          │
 │          │
GPU 2 ─── GPU 3
      │
      │
   CPU RAM
      │
      │
   Network
      │
      ▼
Distributed storage

Now KV cache might exist:

GPU HBM
CPU DRAM
another GPU
another machine
NVMe
remote storage

The runtime needs to understand:

capacity
latency
bandwidth
topology
locality
contention

Moving a KV block from local HBM might be cheap.

Moving it from another machine could be much more expensive.

So:

``
Cost(block) =f(distance,bandwidth,latency,contention)

Now memory management and network scheduling become connected.

This resembles distributed operating systems and NUMA systems, where memory isn't uniformly accessible.

The runtime must answer:

Where should this state live, and where should computation happen?

That is a distributed-systems problem.

The Bigger Picture

Put all ten layers together:

                  LLM APPLICATION
                         │
                       AGENT
                         │
                INFERENCE RUNTIME
                         │
          ┌──────────────┴──────────────┐
          │                             │
      SCHEDULER                    MEMORY MANAGER
          │                             │
          │                     ┌───────┴───────┐
          │                     │               │
          │                    HBM             DRAM
          │                     │               │
          │                    NVMe          Network
          │                     │               │
          └──────────────┬──────┴───────────────┘
                         │
                      GPU

And the ten transformations are:

1. KV Cache
   ↓
   Dynamic memory

2. PagedAttention
   ↓
   Paging

3. Prefix sharing
   ↓
   Shared memory

4. KV eviction
   ↓
   Cache management

5. Hierarchical KV
   ↓
   Memory hierarchy

6. KV prefetching
   ↓
   I/O management

7. Cache-aware scheduling
   ↓
   OS-style scheduling

8. Prefill/decode disaggregation
   ↓
   Resource scheduling

9. Agentic workloads
   ↓
   Process lifetime management

10. Distributed KV
    ↓
    Distributed memory management

This is why I think the phrase “inference OS” isn't as crazy as it initially sounds.

We are building increasingly sophisticated software to manage:

compute
memory
state
scheduling
locality
data movement
sharing
eviction
prefetching
distributed resources

The model is only one component.

The Most Important Difference

There is one major difference between traditional operating systems and LLM inference.

Operating systems mostly manage passive data.

LLMs generate their own working state.

Every token can create more KV cache:

Token
  ↓
KV state
  ↓
More memory
  ↓
More pressure
  ↓
Eviction / transfer
  ↓
Potential recomputation

So the workload itself continuously changes the memory requirements.

That's unusual.

And it means the inference runtime must understand both:

computation

and

memory.

Conclusion

We spent decades optimizing:

CPU → Memory → Storage

Then AI gave us:

GPU → HBM → CPU Memory → Storage → Network

And now the most interesting part of LLM infrastructure may not be the model itself.

It may be the system managing the model's working state.

Paging.

Caching.

Prefetching.

Eviction.

Scheduling.

Memory hierarchies.

Shared state.

Resource allocation.

Distributed memory.

These are all classic operating-system and systems concepts.

But they're being applied to something new:

the runtime state of a neural network.

The model generates tokens.

The tokens generate state.

The state consumes memory.

The memory determines scheduling.

The scheduler determines throughput.

And the entire system feeds back into the model's performance.

So perhaps the next generation of AI infrastructure won't just be:

better models + faster GPUs

It will be:

better inference operating systems.

The model is the intelligence.

The runtime is the machine that makes that intelligence usable.

How to run Claude Desktop and ChatGPT desktop app for free with llama.cpp and other local LLMs.

Lynkr — Sun, 30 Aug 2026 07:11:41 +0000

Most desktop AI apps still assume the backend is fixed.

Claude Desktop assumes Anthropic. ChatGPT.app assumes OpenAI.

That works until you want to route the same desktop app across local and cloud models, keep one gateway for caching and fallback, and pin stronger or cheaper models without changing frontend UX.

That is the problem we just solved in Lynkr.

Lynkr is an open-source LLM gateway. In the latest code changes, we added native support for two desktop surfaces that normally do not give you much backend control: Claude Desktop and ChatGPT.app (Codex Desktop).

Claude Desktop: using the hidden third-party gateway path

Claude Desktop does not expose a normal bring-your-own-provider flow. The new Lynkr support works by using Claude Desktop's undocumented third-party gateway mode — the same general mechanism Ollama's Claude Desktop launcher relies on.

The new install flow is:

lynkr desktop-token <oauth-token>

Under the hood, Lynkr edits config in ~/Library/Application Support/ and manages:

Claude/claude_desktop_config.json
Claude-3p/claude_desktop_config.json
Claude-3p/configLibrary/_meta.json
Claude-3p/configLibrary/<uuid>.json

The installer also writes Claude-3p/configLibrary/.lynkr-backup.json, which means restore is not a blind delete. Lynkr saves the original deployment mode and applied profile, then restores them exactly later.

lynkr desktop-token --restore

That restore command removes the Lynkr profile, resets deployment bookkeeping, and puts Claude Desktop back into its previous mode.

Why Claude Desktop needs a token

Once Desktop is switched into third-party gateway mode (deploymentMode: "3p"), it is no longer talking directly to Anthropic. Lynkr becomes the API layer receiving Desktop traffic.

That is why the install command expects a Claude OAuth token. The current implementation explicitly validates that the token starts with sk-ant-oat.

Lynkr does not mint or refresh that token. It installs the token into Claude Desktop's gateway profile so Desktop can send it to Lynkr, and Lynkr can forward it when a request routes to an Anthropic-backed tier.

Claude Desktop's model picker becomes a tier selector

Instead of treating the picker as cosmetic, Lynkr uses it as a routing control surface.

The docs describe fixed picker entries that map to Lynkr tiers:

Auto -> no pin, normal content-based scoring
claude-opus-5 -> REASONING
claude-sonnet-5 -> COMPLEX
claude-sonnet-4-6 -> MEDIUM
claude-haiku-4-5-20251001 -> SIMPLE

Because Claude Desktop validates model ids against its own catalog, Lynkr cannot advertise made-up ids like lynkr-medium. Instead, it reuses real Claude-family ids and treats them as tier pins.

So the contract becomes:

Auto = gateway decides
specific model = user decides

When the user picks anything other than Auto, Lynkr bypasses classifier-based routing for that request and pins the tier directly.

Claude Desktop detection is gated, not global

Lynkr does not force this behavior onto every Anthropic-compatible caller. The Claude Desktop gateway logic only intercepts callers that look like Desktop specifically — for example requests carrying an anthropic-version header or an explicit ?format=anthropic path.

Other Anthropic-format clients still fall through to the normal model list behavior.

ChatGPT.app: one backend for Desktop and CLI

ChatGPT.app required a different strategy.

The key detail is that Codex Desktop and Codex CLI share the same config surface:

~/.codex/config.toml

That means Lynkr can be added once as a provider block and then back both interfaces with the same gateway configuration.

ChatGPT.app exposes a real routing signal: `reasoning.effort`

The most interesting part of the ChatGPT.app work is that the model picker sends structured routing hints on the wire.

The latest code adds a dedicated mapping module in src/routing/openai-model-slots.js.

That file documents the behavior Lynkr now relies on:

Codex sends a real OpenAI model id
it also sends reasoning.effort
Lynkr maps that pair into a routing tier

The current effort-to-tier mapping is:

minimal -> SIMPLE
low -> MEDIUM
medium -> COMPLEX
high -> REASONING

This was confirmed live from a captured request.

One verified path from the implementation:

UI label: Light
wire value: reasoning.effort = "low"
Lynkr routing tier: MEDIUM
classifier path: bypassed

If the app already tells the gateway how much reasoning the user wants, the gateway should respect it.

The pin is intentionally conservative

If Lynkr sees an unrecognized model or effort value, it falls back to normal content-based routing instead of guessing.

There is also an important scope note in the code: the model/effort pin only governs the first model call of a turn. If the turn becomes multi-step — tool calls, follow-up steps, more agent iterations — later steps can still be rescored by content.

Why these integrations matter

The reason these integrations matter is not just that Lynkr now supports two more apps.

It is that both apps normally own the backend relationship:

Claude Desktop assumes Anthropic
ChatGPT.app assumes OpenAI

Once both route through Lynkr, the control plane moves out of the app and into the gateway:

Claude Desktop / ChatGPT.app
-> Lynkr
-> local + cloud models
-> tier routing + token optimization + fallback

At that point:

the app owns the UX
Lynkr owns routing logic
providers become interchangeable

That is the real story in these latest Lynkr changes.

Lynkr vs LiteLLM's New `complexity_router`

Lynkr — Wed, 05 Aug 2026 22:17:24 +0000

Disclosure up front: this benchmark was designed, written, and run by Lynkr's
maintainer, using Lynkr's own regression harness. The scenario set was written to
test routing behaviors Lynkr considers important — that is a real selection bias
and it is not hand-waved away here (see Limitations). Mitigations:
both proxies route over identical backends, LiteLLM ran its out-of-the-box
defaults exactly as Lynkr did, every scenario ran twice, the full config and
harness are in this repo, and the results include the scenarios where LiteLLM
wins and where Lynkr shows drift. Reproduction steps are at the bottom.

Why this benchmark exists

In early August 2026, LiteLLM shipped complexity_router in core — a rule-based
tier router that classifies each request as SIMPLE / MEDIUM / COMPLEX / REASONING
(the same four-tier taxonomy Lynkr uses) via weighted keyword scoring across 7
dimensions, with zero API calls and sub-millisecond decisions. It ships alongside
adaptive_router (Thompson-sampling bandits) and a savings baseline.

This is the category leader validating tier-based cost routing as a core proxy
feature. We benchmarked it the week it shipped, on the same backends Lynkr routes
to in production.

Methodology

Systems under test

	version	routing decision mechanism
Lynkr	git working tree, 2026-08-05	embedding-anchor intent score (local, no LLM call) + risk analyzer + agentic detector + session pins + calibrated thresholds
LiteLLM	1.95.0 (PyPI; latest release)	`complexity_router`, heuristic classifier, default weights/boundaries, untouched

Note: LiteLLM git-main (1.97.0) could not be run — the git wheel is missing
proxy_server.py and current main is incompatible with fastapi ≥0.140
(get_flat_dependant removed). Both are being reported upstream. 1.95.0 is the
latest installable release and includes complexity_router.

Identical backends, only the decision differs

Both proxies map their four tiers to the same deployments (mirroring Lynkr's live
.env at test time):

tier	backend	cost class
SIMPLE	ollama `minimax-m3:cloud`	free
MEDIUM	ollama `minimax-m3:cloud`	free
COMPLEX	azure `gpt-5.6-sol`	paid
REASONING	moonshot `kimi-k3`	paid

LiteLLM config: litellm-autorouter-v2.yaml (in-repo), using the shipped
auto_router/complexity_router schema with tiers mapping and default_model,
nothing else configured. Each tier deployment carries model_info.id so the
x-litellm-model-id response header reveals which tier served each request.

Harness

benchmark-tier-routing.js (in-repo), MODE=routing RUNS=2. Each scenario
declares an acceptable-tier set; a proxy is routing-correct when the tier that
served the request is in the set. RUNS=2 means a scenario passes only if both
runs pass — this catches non-deterministic classifiers. Stateful scenarios embed a
per-run nonce so session pins and caches can't leak between runs.

Results

Scoreboard: 11 scenarios, acceptable-tier judging

ID	scenario (gist)	acceptable	Lynkr	LiteLLM
S1	"What does git stash do?"	SIMPLE\|MEDIUM	✓ MEDIUM	✓ SIMPLE
R1	JWT security trade-offs for a banking app, step by step	COMPLEX\|REASONING	✓ REASONING	✗ MEDIUM
F1	"Refactor the entire ingestion pipeline, give me the plan"	COMPLEX\|REASONING	✓ COMPLEX	✗ MEDIUM
F2	"Fix the null check in `src/auth/middleware.ts` + tests"	COMPLEX\|REASONING	✓ REASONING	✗ SIMPLE
RS1	"17+25" wrapped in injected `<system-reminder>` security noise	SIMPLE\|MEDIUM	✓ MEDIUM	✓ SIMPLE
SR1	Suggestion-mode side request over a prior security conversation	SIMPLE\|MEDIUM	✓ SIMPLE	✓ SIMPLE
A1	"Work autonomously: run tests, fix, iterate until green"	COMPLEX\|REASONING	✓ REASONING	✗ SIMPLE
P1	"hi, you there?" (fresh session)	SIMPLE\|MEDIUM	✓ MEDIUM	✓ SIMPLE
P2	same session, then "architecture review of the routing module"	COMPLEX\|REASONING	✓ COMPLEX	✗ SIMPLE
IV1	"Review this retry helper for bugs" (bare)	MEDIUM\|COMPLEX	✓ MEDIUM	✗ SIMPLE
IV2	identical ask + ~600 tokens of harness/IDE envelope noise	MEDIUM\|COMPLEX	✓ MEDIUM	✗ SIMPLE

Lynkr: 11/11 · LiteLLM: 4/11.

The sharper cut: the 11 scenarios split into 4 "stay cheap" tests and 7
"escalate correctly" tests. LiteLLM went 4/4 on stay-cheap and 0/7 on
escalation. A router biased toward the cheap tier passes every stay-cheap test
by construction — the escalation tests are where routing is actually hard.

Failure analysis: three clusters, not seven random misses

Semantics beyond keywords (R1, F1, A1). "Security trade-offs for a banking application" and "work autonomously until the tests are green" carry complexity in meaning. The 7-dimension keyword sum doesn't reach the 0.35 COMPLEX boundary, so reasoning-grade work lands on the free tier. This is the published, predicted weakness of ex-ante lexical scoring on real traffic.
No risk model (F2). "Fix the null check in src/auth/middleware.ts" is lexically trivial — and it is exactly the request you never send to an uncontrolled free model. LiteLLM has no concept of protected paths; Lynkr's risk analyzer forces auth/middleware edits to the governance tier regardless of complexity score.
No session or envelope awareness (P2, IV1, IV2). After a "hi" opener, "now do an architecture review" stayed on SIMPLE — no pin-escape mechanism. The invariance pair is subtler: LiteLLM was technically invariant (SIMPLE both times) but invariantly wrong — a code-review ask never accumulated enough keyword weight in either form.

The cost table pathology (read before quoting)

The harness's raw cost line shows LiteLLM at $0.00 — "100% cheaper." That is
the failure, not a win: it spent nothing because it routed nearly everything —
including the 7 escalation-grade requests — to the free tier. A router that sends
everything to the cheapest model is always the "cheapest" router and always the
worst one. Cost numbers for a router are only meaningful conditional on routing
correctness. Any quotation of this benchmark that includes the cost table
without this caveat is misleading, in either direction.

Lynkr's own warts (strict-expectation view)

The harness also checks stricter per-scenario expectations than the acceptable
sets. Lynkr drifted on 3 of 11:

R1, F2: over-routed COMPLEX → REASONING — correct-but-expensive (kimi-k3 spent where gpt-5.6-sol would do). Likely threshold drift after recent tier remapping; the nightly calibration should re-fit, being watched.
P1: SIMPLE → MEDIUM pin drift on the trivial opener.

Over-routing burns money silently; it is the mirror image of LiteLLM's failure
mode and it deserves the same scrutiny.

Where LiteLLM is better

Symmetry matters, so plainly:

Decision latency. Sub-millisecond, allocation-free keyword scoring vs Lynkr's embedding lookup (~10–50 ms warm, more on classifier cold start). For latency-critical single-model workloads that difference is real.
Zero infrastructure for the decision. No embedding model, no classifier model, no state. Lynkr's anchor scorer wants a local embedding model available.
Determinism and auditability. A weighted keyword sum is trivially explainable to a compliance reviewer; an embedding similarity is not.
Simplicity of config. One YAML block. Lynkr's routing intelligence has meaningfully more surface to understand.
Tunability from a known base. Weights, boundaries, and keyword lists are all exposed; a team willing to tune per-workload could close some of the gap (out-of-the-box was tested here, on both sides).
Ecosystem and adoption. LiteLLM is the default proxy of the ecosystem, and the same release train ships adaptive_router (bandit-based) — the direction of travel is serious.

Where Lynkr is better

Escalation correctness: 7/7 vs 0/7 — the entire hard half of the problem.
Risk-aware routing — protected-path and instruction-risk signals route security-relevant work to governed tiers independent of complexity score.
Session pinning with escape — sticky sessions that release when the work outgrows the pin (P2), instead of trapping or ignoring session state.
Payload invariance by construction — the intent score is computed on cleaned user text, so harness envelopes, system-reminders, and IDE noise don't move the decision (IV1/IV2, RS1).
A verification cascade behind the decision (not exercised in this routing- only benchmark): cheap-tier answers are structurally verified and escalated on failure, so a mis-route down is a recoverable event rather than a silent quality loss. LiteLLM's architecture is predict-then-commit.
Third-party evaluation — RouterArena (ICLR 2026): 67.65 arena / 68.41% accuracy / $0.29 per 1K queries / 92.38 robustness, leaderboard #15.

Limitations

Every one of these is a genuine limitation of this benchmark:

N=11, author-selected. The scenarios encode Lynkr's view of what routing should do. A LiteLLM-authored scenario set would probably look different. (Counterpoint: the escalation scenarios are not exotic — auth-file edits, refactor plans, and agentic loops are everyday coding-agent traffic.)
Defaults vs defaults. LiteLLM's router is configurable; a tuned configuration was not tested. Neither was a tuned Lynkr.
One day, one version pair. LiteLLM 1.95.0; main moves fast and the adaptive_router (which may compensate for complexity_router misses over time via bandit feedback) was not enabled or tested.
Routing-only. Response quality, verification, compression, and caching were all out of scope; the cost figures are therefore not end-to-end claims.
RUNS=2 is enough to catch gross nondeterminism, not enough for statistics. Both systems were deterministic across runs on every scenario.

Reproduction

# LiteLLM (Python ≥3.10 venv)
pip install "litellm[proxy]==1.95.0" "fastapi<0.140"
export $(grep -E '^(MOONSHOT_API_KEY|AZURE_OPENAI_API_KEY)=' .env | xargs)
litellm --port 8082 --config litellm-autorouter-v2.yaml

# Lynkr
node index.js   # serves :8081 per .env

# head-to-head, routing-only, 2 runs per scenario
MODE=routing RUNS=2 LITELLM_MASTER_KEY=sk-1234 node benchmark-tier-routing.js

Scenario definitions: benchmark-tier-routing.js. LiteLLM config:
litellm-autorouter-v2.yaml. Raw logs from this run: /tmp/litellm-bench.log,
/tmp/lynkr-bench.log.

Follow-ups queued

Paraphrase-pair robustness scenarios (same intent, different surface — the lexical scorer's structural weakness deserves its own measured number).
Re-run with LiteLLM adaptive_router enabled on top of complexity_router, with enough traffic for its bandit to update — the fairest version of this comparison over time.
File the two upstream LiteLLM bugs found while setting up (git wheel missing proxy_server.py; fastapi ≥0.140 incompatibility).
Investigate Lynkr's R1/F2 over-routing drift after tier remapping.

Loop Engineering Is Mostly Papering Over a Model That Won't Converge

Lynkr — Thu, 30 Jul 2026 07:25:49 +0000

Disclosure: I maintain Lynkr, an open-source LLM gateway with loop-guard middleware, so I have a stake in this argument. Read this as a thesis to challenge, not a neutral survey.

Agent discourse has shifted from prompts to loops: the read-act-observe cycle that keeps running until the model decides it's done. My contrarian take is that most so-called loop engineering is optimizing the wrong variable. A longer, richer loop is usually a convergence failure wearing a costume. The well-engineered loop is the short one that knows when to stop.

The consensus I'm arguing against

When an agent fails, the default move is to give the loop more:

More turns before it gives up.
More context stuffed into the window.
More scaffolding: planners, critics, reflection steps, sub-agents spawning sub-agents.
More retries with more feedback.

The implicit theory is simple: intelligence accumulates across iterations, so more turns should mean more convergence. Sometimes that works. Mostly, on real traffic, it doesn't — and the cost structure is brutal even when it does.

A concrete example from coding agents: give an underpowered model a repo-wide refactor and it usually doesn't "reason longer" into success. It misreads one dependency, anchors on the wrong file, and spends the next dozen turns elaborating the mistake. The loop isn't rescuing the model. It's financing the failure.

Why longer loops compound the wrong things

Two things compound across a loop, and neither is intelligence.

1. Loop cost grows fast. Every turn re-sends accumulated context. A 20-turn session isn't just 20 single calls back-to-back; each later turn drags earlier tokens back through the model. In many agentic workloads, the loop becomes the most expensive object in the serving budget, and "add more turns" becomes the most expensive knob you can turn.

2. Reliability collapses across dependent steps. Even with optimistic per-step reliability, multi-step chains degrade quickly. Long loops don't average out mistakes; they multiply the chance of staying on the correct trajectory. The agent that "just needs more turns" is often an agent walking deeper into the wrong branch with more conviction, because each turn conditions on its own earlier errors as if they were facts.

We wrote earlier about how stuffing the context window can make agents worse, not smarter: attention dilutes, retrieval anchors on the wrong chunk, and signal gets buried. Loop length is the same failure on the time axis. More is not more.

What a long loop is actually signaling

When an agent needs 30 turns to do something, that's usually not the harness being clever. It's a signal of one of two failures:

The task was routed to a model that can't do it. The loop is that model thrashing. Scaffolding rarely fixes an underpowered model decision; it usually just makes the failure more elaborate and more expensive.
The task genuinely has many steps. In that case, the engineering problem is not "how do I allow more turns" but "how do I verify each step cheaply so the wrong branch gets caught at turn 3 instead of compounding to turn 30?"

Both point the same direction: either allocate the step to a better model, or verify the step before proceeding. Neither says: add turns and hope.

What loop engineering should actually be

If loops are where cost and error compound, then good loop engineering should minimize compounding, not maximize capability-per-turn.

1. Cap the loop hard. Every loop needs a turn ceiling and a tool-call ceiling that it cannot exceed, full stop. Not as an emergency fallback — as a first-class design constraint. A loop that can run forever will, on some input, cost you a fortune to produce garbage.

2. Verify each step cheaply. The highest-leverage addition to a loop is usually not another reasoning stage. It's a fast, deterministic check on the step output that catches the failure modes you actually see: truncation, malformed tool calls, empty answers, degeneration loops, repeated echoes. Detection beats prediction because you already have the output in hand.

3. Escalate the failing step, not the whole session. When a step fails verification, redo that step on a better tier. Don't grant the entire loop more turns to flail. Escalation is targeted; more-turns is a blank check.

The part where this bit us too

We're not exempt. In Lynkr, we shipped a terse mode meant to save output tokens by pushing models toward shorter answers. Our verifier then misread some of those short answers as low-effort failures and escalated trivial requests up a tier into a longer, more expensive path.

The fix was one predicate. The broader lesson wasn't. A check inside a loop is itself part of the loop's cost and failure surface. "Add verification" is not free virtue; a bad verifier can extend loops as eagerly as a naive retry does. Verification has to be engineered with the same suspicion as any other turn.

The thesis, plainly

A lot of popular loop engineering optimizes capability-per-turn while treating turn count as free. That's backwards. On real traffic, turn count is often the dominant cost and a major source of compounded error.

If a task needs more intelligence, buy more intelligence for that step. If a task needs more reliability, verify the step. If a loop keeps growing, treat that as a failure signal, not sophistication.

Make the loop shorter. Make it stop. That's the engineering.

Lynkr is Apache-2.0 and self-hosted. The loop-guard middleware and step-level escalation ideas behind this post are in the repo: github.com/Fast-Editor/Lynkr. If you've seen a loop run too long and know why, I'd love to hear the war story.

Your LLM Bill Is a Misallocation Problem, Not a Model Problem

Lynkr — Thu, 30 Jul 2026 06:50:28 +0000

Disclosure: I maintain Lynkr, an open-source LLM router, so this post argues a thesis my own product embodies. The mitigation: every claim below is either published research, a July 2026 release you can verify, or a measurement you can reproduce — including the part where the obvious approach (and an earlier version of ours) doesn't work.

The dominant inefficiency in LLM serving is no longer kernel speed or quantization. It's per-request compute misallocation — and the research says you can't fix it with prediction. You fix it with verification.

The setup: the gap closed, the dispersion didn't

Three things became true more or less simultaneously in mid-2026:

Open-weight models sit within single-digit percentage points of frontier models on everyday work. GLM-5.2 (744B MoE, 40B active, MIT license) leads several long-horizon coding benchmarks outright. Kimi K3 shipped in July at 2.8T parameters with ~1.8% expert activation and a 1M-token context — open weights, hosted access.
The cost of that near-frontier capability is 4–10× lower. Hosted GLM-class models run at a fraction of frontier per-token prices; aggressive hosts (Meituan's LongCat 2.0 launched at $0.30/$1.20 per million) keep pushing the floor down.
Price-per-capability dispersion across providers has never been wider. The same request can cost $0.10 or $15 per million output tokens depending on where you send it, for a quality difference that on most traffic is inside the noise.

Put those together and the largest line item in a serving budget is no longer "the model is slow" or "the quant is fat." It's every request routed to a frontier model that a cheaper tier would have cleared. That's pure deadweight loss, and at current dispersion it usually dominates everything a kernel engineer can save you.

I say that as someone who ships kernel PRs: a +13–15% decode win on a CPU inference engine (measured, merged, byte-identical output) saves you 13% on one backend. Routing 60% of traffic down two price tiers saves ~90% across the fleet. Same engineering effort. The leverage moved up the stack.

The naive fix fails: you can't predict difficulty from the prompt

The obvious architecture is ex-ante routing: classify the prompt, pick the model, done. Every "smart routing" product demo shows this.

It doesn't survive contact with real traffic. Ex-ante difficulty prediction consistently fails to beat trivial baselines on agentic coding traffic. When we ran the deep-research pass for Lynkr's cascade design (July 2025 literature, SWE-Bench-style workloads), the pattern was blunt: prompt surface features don't carry enough signal about downstream difficulty. A one-line ask ("fix the flaky test") can require repo-wide reasoning. A 3,000-token prompt can be boilerplate a 9B model handles. The information you need to route correctly mostly doesn't exist until after a model has attempted the task.

If your router's accuracy story is a difficulty classifier — however fancy the embeddings — you've built a system that's confidently wrong in both directions: burning frontier tokens on easy requests and serving garbage on hard ones.

What works: try cheap, verify, escalate

The architecture that actually gains is the cascade: route to the cheapest plausible tier, run deterministic verification on the answer, escalate only on failure. Published cascade results show up to +14% over static assignment — and the gain lives in the verifier, because the cascade converts routing from a prediction problem (hard, information-starved) into a detection problem (tractable, information-rich: you have the actual answer in hand).

The verifier doesn't need to be an LLM judge. Ours is two layers of pure functions:

Structural checks targeting the cheap-model failure modes we actually observed live: language drift (CJK tokens appearing mid-English answer), degeneration loops, truncation, malformed tool calls, empty/echo responses. High precision — flag only what is definitely broken.
A coarse effort score: substantive ask answered with a stub, structure requested but not delivered, code requested but not delivered. Conservative threshold; only clear failures escalate.

No LLM call in the loop. The verification cost is microseconds, which matters because a router that spends tokens deciding where to send tokens is eating the margin it exists to create.

A war story about verifiers

One production bug worth sharing because it generalizes: we shipped a brevity mode that injects "answer tersely" into system prompts to cut output tokens. The verifier's effort-scorer then read the resulting short answers as low-effort and started cascading trivial greetings up to the frontier tier — burning the exact money the brevity mode saved, and then some.

The fix was one predicate (skip the effort check when the brevity instruction is present; keep the structural checks), but the lesson is bigger: a verifier must know what the model was asked to be. Any answer-quality check that ignores the instructions that produced the answer will misfire exactly when your prompt engineering works.

The second-order effect: verification failures are free training data

Here's where cascades stop being a cost trick and become a learning system.

Every verification failure is a labeled hard negative: "this request, routed to this tier, produced a detectably bad answer." Every escalation that then passes is a label: "requests like this outgrow the cheap tier." Feed those back into the routing decision — we treat it as a contextual bandit with verifier-generated rewards, logging propensities at decision time so the feedback isn't biased by the router's own choices — and the router learns the difficulty structure of your traffic, from your outcomes.

No benchmark in the loop. That mattered more than usual this month: one of the field's most-cited coding benchmarks was retracted as broken. Every router configured against it inherited the error silently, for months, with nothing in the loop that could detect it. Ground truth from your own traffic doesn't get retracted.

Receipts

This is the architecture behind Lynkr (Apache-2.0, self-hosted): deterministic embedding-anchor tier classification (zero LLM calls to route), structural verification on cheap-tier answers only (never second-guess the expensive tier — the loss is asymmetric), cascade escalation, reward pipeline with propensity logging.

On RouterArena (ICLR 2026, 8,400 queries, evaluation-only rules — no component tuned on their data), our submitted configuration measured 67.65 arena score / 68.41% accuracy at $0.29 per 1K queries, robustness 92.38 — above GPT-5's built-in router and NotDiamond at a fraction of the per-decision cost. The submission PR has the full config, prediction files, and the disclosures, including the metrics where we're mediocre (routing concentration is real; read the PR).

The thesis, restated

For a decade the way to cut inference cost was to make the model cheaper: quantize, distill, fuse kernels. That work still matters — I still do it — but it's optimizing the price of a decision that's already been made. In a world where capability is a commodity spectrum with 50× price dispersion and single-digit quality spread, the decision itself is where the money is.

The gap between open and frontier closed. Allocation is the frontier now.

Lynkr is Apache-2.0 and self-hosted: github.com/Fast-Editor/Lynkr · npm. If you route production LLM traffic and have cascade/verification war stories, I want to hear them — the failure modes section of our verifier is built from exactly those.

We Put Our Router on an Academic Benchmark. Here Are the Numbers We'd Rather Hide.

Lynkr — Mon, 27 Jul 2026 07:36:49 +0000

Disclosure: I maintain Lynkr, the router being benchmarked, so read everything here with that in mind. The mitigation: every number in this post comes from RouterArena's automated evaluation pipeline, run by their CI on their infrastructure, not by me — including the numbers that make us look mediocre.

Every LLM router's README — ours included — makes the same claim: it sends easy queries to cheap models and hard queries to good ones, and saves you money without hurting quality. Almost none of them attach evidence. The numbers that do exist are self-reported, measured on datasets the vendor picked, with baselines the vendor chose.

So when RouterArena showed up — an open, standardized benchmark for LLM routers out of the RouteWorks group (paper, leaderboard) — we submitted Lynkr to it. This post is what we learned, including the metrics where we did badly, because a benchmark result you only quote selectively is just marketing with extra steps.

What RouterArena actually measures

RouterArena evaluates a router over 8,400 queries spanning 9 domains and 44 categories at three difficulty levels, and scores it on five axes:

Arena Score — the headline number, a combined accuracy-vs-cost tradeoff.
Accuracy — did the model your router picked answer correctly?
Cost — average $ per 1K queries, at real provider prices.
Optimality — three sub-metrics against the oracle: how often you picked the cheapest correct model (Opt.Sel), how close your spend was to optimal (Opt.Cost), and how close your accuracy was to the best achievable with your pool (Opt.Acc).
Robustness — do trivial rephrasings of the same query flip your routing decision?

One rule matters more than the metrics: RouterArena is evaluation-only. Any router component trained, fitted, or tuned on their data or labels gets rejected, and violations found later get withdrawn. Keep that rule in mind — it comes back at the end of this post.

What we submitted

We submitted Lynkr's untuned production default — the same complexity scorer and the same tier boundaries that ship in the repo. The adapter is ~70 lines of Python that sends each query to a live Lynkr instance's /routing/analyze endpoint and maps the returned tier to a model. No benchmark-special code path: the endpoint runs the exact intent scorer the live proxy uses (local embeddings, no LLM call in the routing decision itself).

The model pool, all via OpenRouter:

Lynkr tier	Model
SIMPLE	gpt-oss-120b
MEDIUM	Qwen3-235B-A22B
COMPLEX / REASONING	GLM-4.7

Running all 8,400 queries cost $2.46 total. That's the entire eval bill, which says something on its own about where inference prices are in mid-2026.

The results

From RouterArena's automated evaluation on our submission PR:

Metric	Lynkr
Arena Score	67.65
Accuracy	68.41%
Cost per 1K queries	$0.29
Robustness	92.38
Opt.Acc (accuracy vs. optimal)	84.48
Opt.Cost (cost efficiency vs. optimal)	16.08
Opt.Sel (optimal model selection)	10.97

On the official leaderboard (our submission merged 2026-07-23), that lands mid-table — 15th of 27 routers.

The parts we'll be quoting

We beat GPT-5 used as a router — at 34× lower cost. The leaderboard includes GPT-5 itself as a routing baseline: Arena score 64.32 at $10.02 per 1K queries. Lynkr scores 67.65 at $0.29 per 1K. A local intent scorer making the decision without any LLM call outperforms asking a frontier model to route — which is the entire thesis Lynkr is built on, so we're relieved the benchmark agrees. Several well-known systems (NotDiamond at 57.29, RouteLLM at 48.07, both RouterBench baselines) also land below us.

Robustness 92.38 is top-five territory. Most leaderboard routers sit between 22 and 72 — meaning a rephrased query frequently flips their model choice. Lynkr's routing decision survives rephrasing 92% of the time. For an interactive tool where users iterate on prompts, decision stability is arguably worth more than a couple of accuracy points: a router that sends your reworded follow-up to a different model mid-conversation is a router you turn off.

The parts we'd rather hide (but won't)

Opt.Sel is 10.97. When multiple models in our pool could answer a query correctly, we picked the cheapest correct one about 11% of the time. Lynkr routes conservatively — when in doubt, it escalates a tier. That's a deliberate live-serving bias (a wrong cheap answer costs more user trust than an unnecessarily good one costs dollars), and $0.29/1K shows the absolute spend stays low. But the oracle comparison is unambiguous: there was money on the table we didn't pick up.

The accuracy gap to the top is real. Cross-Router leads at 78.14% accuracy; we're at 68.41%. Part of that is the pool (three self-hostable open-weight models — no frontier closed models to escalate to), but part is genuinely the router. Mid-table is where we are, not where we'd spin it.

What the benchmark taught us — and why we won't act on all of it

Reading our failure cases was the most valuable part of the exercise. The clearest pattern: our SIMPLE→MEDIUM boundary is conservative — a chunk of queries scored just past the boundary and got a mid-tier model when the cheap tier would have answered correctly.

So we prototyped a shifted boundary and ran it on a ~10% subsample locally: Arena score 69.93, accuracy 71.07%, Opt.Sel jumping from ~11 to ~66. A two-point arena gain from moving one threshold.

That config will never appear on the leaderboard. It was diagnosed from RouterArena's own failure cases, which makes it exactly what their evaluation-only rule exists to prevent — a router fitted to the eval. The leaderboard entry is the untuned default, and stays that way. The honest generalization claim is much narrower: the benchmark showed us which knob matters, and we'll validate new defaults on our own traffic, not on theirs.

If you maintain a router: this is the trap. The moment a public benchmark exists, the incentive is to tune against it, and every point you gain that way is a point of overfitting you ship to your actual users.

Caveats, so you can weigh this properly

RouterArena is single-turn Q&A-style evaluation. Lynkr's primary workload is multi-turn coding agents, which RouterArena doesn't measure — agentic routing (tool-call density, context accumulation) is a different problem than classifying one prompt.
Our score depends on our chosen pool. A pool with a frontier model on top would score differently in both directions (higher ceiling, higher cost).
Leaderboard positions move; check the live leaderboard rather than trusting this snapshot.

If you're evaluating routers — including ours — ask every vendor for their RouterArena number. It costs a few dollars and a PR to get one. "We haven't submitted" is also an answer.

Links: RouterArena paper (arXiv:2510.00202) · leaderboard · our submission PR · Lynkr

Stuffing the Context Window Is Making Your Agent Dumber: What the Research Says

Lynkr — Fri, 17 Jul 2026 01:10:15 +0000

Disclosure: I maintain Lynkr, an open-source gateway that (among other things) compresses agent tool outputs — so I have a horse in this race. This piece, though, is about the research, and every number in it is cited to its primary source.

There's an intuition almost every LLM user shares: more context = better answers. Million-token context windows are marketed as a capability. We paste in whole files "just in case." Our coding agents accumulate every grep result, every file read, every test log, on the theory that the model might need it.

The research says this intuition is not just wrong — it's backwards, and the size of the effect is embarrassing.

The result that should change how you build

The cleanest demonstration comes from the Hindsight memory system (arXiv:2512.12818, demo at ACL 2026). On LongMemEval — a benchmark of questions over long conversational histories — the same open-source 20B model scores:

39.0% when handed the full context — everything, the whole history
83.6% when handed a curated slice selected by a structured memory system

Same model. Same available information. The difference is that one setup made the model read everything, and the other selected what mattered. +44.6 points from subtraction.

It gets more uncomfortable: that 20B model with curated context also beats full-context GPT-4o, which scores 60.2% on the same benchmark. A model a fraction of the size, winning because someone cleaned its desk. As the authors put it, the memory architecture — not model scale — drives the performance.

One benchmark, one paper? No — this is a pile-on:

"Lost in the middle" (Liu et al., 2023) established the shape of the problem early: models attend well to the start and end of long contexts and poorly to the middle — exactly where your agent's fifteenth tool result lives.
Context-rot studies (notably Chroma's 2025 report) showed performance degrading as context grows even when the added tokens are relevant, and degrading faster when they're distractors.
A whole 2026 research wave now treats context as a resource to be managed, not maximized: "Agentic Context Engineering" was accepted at ICLR 2026, active context compression systems prune their own working memory (arXiv:2601.07190), and two consolidating surveys (arXiv:2512.13564, arXiv:2603.07670) formalize memory as a write–manage–read loop — with "manage" doing the heavy lifting.

The marketing said "bigger window." The research says "better librarian."

Why more context makes things worse

Three mechanisms, all well-documented:

1. Attention is a budget, not a spotlight. Every token in context competes for attention mass. Pack in 50k tokens of tool output and the three lines that matter are now competing with 49,900 tokens of noise. Needle-in-a-haystack benchmarks — the ones vendors publish — test retrieval of a planted string, which models are good at. Real tasks require reasoning over the context, which degrades much faster.

2. Distractors don't just dilute — they actively mislead. The context-rot findings show semantically-similar-but-irrelevant content is worse than random filler. Your agent's context is full of this: old versions of the file it's editing, error messages from an already-fixed bug, grep hits from a deprecated module. Each is a plausible-looking wrong answer sitting one attention head away.

3. Position effects compound over turns. Agents append. Every turn pushes the important early material (the task! the constraints!) toward the middle of the context — the attention dead zone — while burying the recent signal under boilerplate tool output. A long agent session is a machine for constructing worst-case attention layouts.

Coding agents are the pathological case of all three at once: they generate enormous, distractor-dense, structurally-repetitive context (JSON tool results, file dumps, test logs) at machine speed, across dozens of turns.

What the research says works instead

The successful systems in the literature share a shape — they spend compute deciding what the model sees instead of showing it everything:

Selection over inclusion. Hindsight's four memory networks (facts vs. experiences vs. summaries vs. beliefs) exist so retrieval pulls the right kind of memory for each question. The general lesson: retrieval into a small context beats residence in a big one.
Compression as a first-class operation. Active-context-compression agents treat "shrink my working set" as an action the agent itself takes, on par with tool calls. Summarize the resolved, drop the superseded, keep the live.
Structure beats soup. Tabular, labeled, deduplicated context consistently outperforms raw dumps of the same information — the model spends attention on content, not parsing.

What you can do about it today

You don't need a research memory system to benefit:

Treat context as a liability with interest, not an asset. Every tool result you leave in the window is re-read (and re-billed) every subsequent turn, while making each turn slightly dumber.
Compact aggressively and early. Whatever your agent's compaction/clear mechanism is (/compact, /clear, fresh sessions per task), use it before quality degrades — by the time you notice the agent going in circles, the context has been hurting you for many turns.
Scope sessions to tasks. One task, one session. The 40-turn omnibus session is the exact scenario the position-effect research warns about.
Isolate research from execution. Subagents (or separate sessions) that read a lot and report a little are context firewalls: the summary crosses over; the 30k tokens of grep output don't.
Compress tool outputs before they enter context. Test logs, JSON blobs, and directory listings compress 40–90% with zero information the model actually needs lost. Whether you do it with a proxy layer (this is the part where I mention that's what Lynkr does), a harness setting, or a wrapper script — do it somewhere.

The next time a model launch leads with context-window size, remember the 20B model with a good librarian beating GPT-4o with a hoard. Capacity isn't capability. Curation is.

Primary sources: Hindsight (arXiv:2512.12818) · benchmark data · Lost in the Middle (Liu et al.) · Active Context Compression (arXiv:2601.07190) · Memory in the Age of AI Agents — survey (arXiv:2512.13564) · Memory for Autonomous LLM Agents — survey (arXiv:2603.07670)

How We Built an Agentic-Task Detector for LLM Routing

Lynkr — Tue, 14 Jul 2026 04:18:47 +0000

Disclosure: I maintain Lynkr, the open-source LLM router whose agentic detector this post dissects. Every snippet below is real, shipping code — read the whole file here, it's 350 lines.

"Fix the auth bug in session.js."

Eight words. Every token-count heuristic on earth routes this to the small, cheap model — it's short. And every one of them is wrong, because those eight words are about to unleash a grep → read → edit → test loop with exact-string file edits, the precise workload where small models fumble tool calls and kill sessions.

The inverse request — three paragraphs asking for a detailed comparison of locking strategies — looks expensive and routes safely to a free local model, because it's pure text generation. Size and stakes are nearly uncorrelated in coding-agent traffic. So the router's real job is detecting agentic intent, and this post is a tour of how Lynkr's detector does it: the signals, the weights, the classification ladder — and the embarrassing false positive that almost made the whole thing useless.

Not "agentic: yes/no" — a ladder

The first design decision: agentic-ness isn't boolean. The detector classifies requests into four types, each with a minimum tier floor and a score boost fed into the complexity scorer:

const AGENT_TYPES = {
  SINGLE_SHOT: { minTier: 'SIMPLE',    scoreBoost: 0 },   // request-response, no tools
  TOOL_CHAIN:  { minTier: 'MEDIUM',    scoreBoost: 15 },  // read -> edit -> test
  ITERATIVE:   { minTier: 'COMPLEX',   scoreBoost: 25 },  // retry loops, debugging cycles
  AUTONOMOUS:  { minTier: 'REASONING', scoreBoost: 35 },  // "figure it out", full autonomy
};

The minTier is a floor, not a suggestion: even if every other dimension scores low, an ITERATIVE request cannot route below the COMPLEX tier. Mid-debugging-loop is the worst possible moment to hand the session to a 7B model.

The six signals

Each request accumulates a score from six independent signals. The interesting part is why each one exists:

1. Tool count (up to +25). Many tools attached usually means the client is prepared for multi-step work. Usually. This signal is also the source of the great false positive — hold that thought.

2. Agentic tools specifically (up to +25). Not all tools are equal evidence. Bash, Write, Edit, Task, git and test runners form an explicit set — these mutate state, and their presence signals mutation work. A request that can only Read/Grep/WebSearch sits in a separate read-only set and earns nothing here. Two requests with five tools each can be night and day.

3. Prior tool results (up to +30 — the heaviest signal). If the conversation already contains tool_result blocks, you're not predicting an agentic loop — you're inside one. More than five results means a deep loop with accumulated exact state (file contents, error strings); downgrading the model now throws away the context discipline keeping that loop convergent.

4. Language patterns (up to +25 each). Regexes over the last user message:

tool-chain: "then use", "after that", "step 2"
iterative: "keep trying", "until", "retry", "debug"
autonomous: "figure out", "make it work", "on your own", "whatever it takes"
multi-file: "across the codebase", "refactor entire", "everywhere"

Plus a combination rule: "implement" alone is +10-ish planning noise, but "implement" and "test/verify/make sure" in the same request is +15 — build-and-verify phrasing is a reliable tell of real work.

5. Conversation depth (up to +20). Fifteen-plus messages means established context and momentum.

6. Prompt length (+10). The weakest signal, deliberately — see the opening paragraph.

Score ≥ 25 → the request is agentic. The classification ladder then applies both thresholds and signal combinations — AUTONOMOUS needs score ≥ 60, or an explicit autonomous phrase with score ≥ 40. A phrase alone doesn't do it; a high score without autonomous language doesn't either, unless it's overwhelming.

The false positive that almost sank it

Early versions had a humiliating problem: every single Claude Code request scored agentic. Including "hello."

Why? Claude Code attaches its full tool loadout — Read, Write, Edit, Bash, Grep, Glob, Task, and friends — to every request, even a greeting. Signals 1 and 2 saw 11+ tools, four of them mutating, on everything. Every request cleared the threshold, every request routed to expensive tiers, and the router's entire value proposition — savings — evaporated. The detector was technically working and practically useless.

The fix ships as client profiles: known harnesses (Claude Code, Cursor, Codex CLI) have documented baseline loadouts, and the tool-count signals score only the tools beyond that baseline:

// Signals 1 & 2 score only tools BEYOND the harness's baseline loadout —
// Claude Code's 11 always-attached tools shouldn't count as "agentic
// intent" on their own.
toolsForScoring = clientProfiles.effectiveTools(payload, profile);

Crucially, signals 3–6 still use the full payload — prior tool results and conversational language are genuine evidence regardless of which harness sent them. Only the tool-presence signals get the subtraction, because only they are polluted by the harness's constant.

And for traffic from harnesses we've never seen? A guard: if every attached tool looks like a standard baseline and there are 10+ of them, the tool-count signals zero out rather than fire:

} else if (clientProfiles.allToolsAreBaseline(payload) && rawTools.length >= 10) {
  // Unknown harness that looks like Claude Code / Cursor / Codex —
  // zero out the tool-count signals to avoid the same trap.
  toolsForScoring = [];
  scoringNote = 'unknown_harness_guard';
}

Better to under-detect and lean on the five uncorrupted signals than to re-create the everything-is-agentic bug for unknown clients.

One subtle consequence, preserved as a comment in the source: with the baseline subtracted, tool counts rarely reach the AUTONOMOUS threshold on their own — so the autonomous phrase pattern becomes the primary path to the top classification. The signal design acknowledges its own post-fix physics.

What it still gets wrong

Honesty section. Known limitations, from the code itself:

It reads only the last user message. "Do what I described above" carries the intent of an earlier message the regexes never see. Conversation-depth and tool-result signals partially compensate — but pattern detection is myopic by design (scanning full history was too noisy).
Regexes can't tell mention from intent. "Why did the retry loop break?" trips the iterative pattern despite being a read-only question. In practice this fails safe — over-routing a question up-tier costs cents, under-routing an edit session down-tier costs the session — but it's still a false positive.
English only. The patterns are English regexes; agentic intent in other languages leans entirely on the structural signals.

Every detection returns its full evidence — score, signal list with weights, classification, and a scoringNote explaining any baseline subtraction — so when the router misjudges, the telemetry shows exactly which signal lied. Debuggability was a design requirement: a routing layer you can't interrogate is a routing layer you'll eventually rip out.

Takeaways if you're building anything similar

Subtract the constant before reading the signal. Whatever your equivalent of "the harness always attaches 11 tools" is — find it and remove it, or every request looks the same.
Separate "prepared for tools" from "already using tools." Attached tools are weak evidence; tool_result blocks in the conversation are near-proof.
Fail toward the expensive model. Asymmetric costs mean your threshold should be calibrated so mistakes over-spend pennies rather than break sessions.
Make the detector explain itself. A score without a signal list is a black box you'll never be able to tune.

The whole detector is 350 lines of dependency-free JavaScript: src/routing/agentic-detector.js. Steal it, or tell me which of your prompts it would misjudge — the failure cases are the roadmap.

Choosing a Local Tier for Your Coding Agent (July 2026 Edition)

Lynkr — Wed, 08 Jul 2026 00:14:16 +0000

Disclosure: I maintain Lynkr, the open-source router used in the config examples. The benchmark figures below are third-party or vendor-reported (flagged where vendor-only) — I haven't independently benchmarked these models yet; the point of this post is to help you match models to request classes and test on your own workload.

June 2026 was the busiest month for open-weight coding models in recent memory: GLM-5.2, MiniMax M3, Kimi K2.7 Code, Gemma 4, and NVIDIA's Nemotron 3 Ultra all landed within weeks. If you route your coding agent's simple requests to a local model — the "cloud architect, local coder" pattern — your options just changed meaningfully.

Here's how I'd map the current field onto routing tiers, by hardware budget and by what each model can safely own.

First, the trap: "best open model" ≠ "your local tier"

The headline model of the month, GLM-5.2, scores 62.1% on SWE-bench Pro — above GPT-5.5. It is also a 744B-parameter MoE whose 2-bit quant alone wants ~245 GB of memory. That's an open-weight model, not a local model; for self-hosters it's a $40k-rig proposition (one published build runs it on four RTX PRO 6000s). The same goes for DeepSeek-V4 Pro and MiniMax M3: superb models you'll realistically consume via API, where they belong in your COMPLEX/REASONING tiers, not your local one.

Your local tier is decided by a harsher question: what fits in your VRAM and still makes reliable tool calls?

The local field, by hardware budget

~16 GB RAM (ordinary laptop): Gemma 4 12B. Released June 3 as a dense 12B that genuinely fits consumer RAM (SitePoint's guide). Apache-2.0-class licensing with no usage clauses. This is a SIMPLE-tier model: explanations, one-liners, commit messages, "what does this error mean." I would not hand it an Edit tool.

24 GB GPU (RTX 3090/4090 class): Qwen3.6-27B — still the default answer. The community's consensus "local Claude" since April: within a few points of frontier models on SWE-bench Verified (77.2 reported vs Claude's 80.9 — analysis), Apache-2.0, runs quantized on a single 24 GB card or a ~$2k build. Its known weakness is exactly the one that matters for agents: tool-call reliability drifts in long contexts — fine as a supervised MEDIUM tier, risky as an unsupervised COMPLEX one.

Agentic multi-file edits on similar hardware: Devstral Small 2. Purpose-built for multi-file, tool-driven coding rather than chat (KDnuggets roundup). If your traffic is edit-heavy, it can arguably take MEDIUM-tier mutation requests that I'd keep away from general chat models.

Autocomplete-shaped work: Codestral 22B is fast and good at it — but mind the non-commercial license before using it for work.

One rule that keeps proving out (Pinggy's guide): within the same memory budget, a bigger model at Q4 usually beats a smaller one at Q8. Quantization choice matters nearly as much as model family.

Mapping to tiers

Putting that together into a routing config (Lynkr shown; the mapping logic applies to any router):

# 24 GB GPU + API keys for the hard stuff
TIER_SIMPLE=ollama:gemma4:12b            # trivia, explanations, greetings
TIER_MEDIUM=ollama:qwen3.6:27b           # code questions, supervised edits
TIER_COMPLEX=deepseek:deepseek-v4-flash  # tool-heavy mutations, via API
TIER_REASONING=deepseek:deepseek-v4      # architecture, multi-step planning

Why V4 Flash for COMPLEX: it's the first open-weight model teams report dropping into real agentic pipelines as a frontier substitute on price (OpenRouter's June analysis) — the cheapest "won't break the session" option right now. Kimi K2.7 Code (vendor-reported 58.6% SWE-bench Pro at ~30% fewer reasoning tokens) and GLM-5.2 are strong API-tier alternatives; all the June day-one numbers are vendor-reported, so treat them as directional until LiveBench catches up.

The key discipline: the boundary between MEDIUM and COMPLEX should not be "how big is the request" but "will tools mutate state." Local models in this class handle read-and-explain reliably; exact-match edits and bash execution are where they still break sessions — I wrote up those failure modes here.

What changed vs three months ago

The floor rose. A 16 GB laptop now runs a genuinely useful SIMPLE tier (Gemma 4). Six months ago that tier meant 3B models that couldn't be trusted with a paragraph.
The open-weight ceiling now beats proprietary on some coding benchmarks (GLM-5.2 > GPT-5.5 on SWE-bench Pro) — but at server scale, which strengthens the hybrid pattern: open models via cheap APIs up top, small open models on your metal below.
MoE won. Every serious June release is Mixture-of-Experts. For self-hosters this cuts both ways: better quality-per-active-param, but total memory footprints that keep the top tier out of reach.
Licensing is consolidating around MIT (DeepSeek) and Apache-2.0 (Qwen, Gemma) for the models you'd actually build on.

Test on your traffic, not on benchmarks

Every number above is someone else's workload. The honest way to pick your local tier: route a week of your real traffic through whatever candidates fit your hardware, and count session survival — how often the local model's tool calls held up — not just benchmark deltas. That's a one-line config change per candidate, and your own telemetry will contradict at least one thing this post told you.

Lynkr is Apache-2.0, self-hosted, and treats every model above as a first-class routing tier: github.com/Fast-Editor/Lynkr.

Routing Down Is Easy. Knowing When Not To Is Hard: Why Cheap Models Break Your Coding Agent

Lynkr — Wed, 08 Jul 2026 00:08:36 +0000

Disclosure: I maintain Lynkr, an open-source router whose design decisions this post explains. The failure modes described are patterns widely reported across router issue trackers and local-LLM forums — the examples are representative reconstructions, not captured transcripts. The problem is real either way; ask anyone who's routed a coding agent to a 7B model.

Everyone who gets their first LLM router working does the same thing within the hour: point the expensive coding agent at a free local model and watch the bill drop to zero.

Then the agent tries to edit a file.

The graveyard of downgraded sessions

If you browse the issue tracker of any Claude Code router — or r/LocalLLaMA on any given week — you'll find the same story in a hundred variations. The routing works perfectly. The session dies anyway. The killers, in rough order of frequency:

1. Malformed tool arguments. The agent decides to call Edit, and the model produces arguments that are almost JSON:

{"file_path": "src/auth.js", "old_string": "if (token) {", "new_string": "if (token && !expired) {"

One missing brace. The harness rejects the call, the model retries, produces a different malformation, and you're three turns deep into fixing nothing. Frontier models emit structurally valid tool calls with boring reliability; sub-10B models do it most of the time — and "most of the time," at 30 tool calls per session, means every session breaks.

2. Stale string matching. Edit-style tools require the old_string to match the file exactly. Small models paraphrase from memory instead of quoting — they'll "remember" the line as if (token) { when the file says if (accessToken) {. The edit fails, the model re-reads the file, burns 2,000 tokens, tries again with a different paraphrase. This is the single most reported failure, because it looks like the router's fault and is actually a capability cliff.

3. Hallucinated context. Ask a small model to run tests and it may confidently call Bash with npm test -- --grep "auth" in a repo that uses pytest. It's not being stupid — it's pattern-completing from training data instead of the conversation, because instruction-following degrades faster than fluency as models shrink.

4. The infinite loop. The subtlest one: the model calls Read on the same file five times in a row, or greps, reads, greps the same term again. Weak models lose the thread of what they already know in long agentic contexts. Nothing errors — the session just stops converging while tokens burn.

Here's the uncomfortable part: none of these are the router's bug, and all of them are the router's fault. The router made a judgment — "this request is cheap-model-safe" — and the judgment was wrong.

Why the obvious heuristics misjudge

Most routing setups decide with static rules: token thresholds, keyword lists, scenario slots. These fail in a specific, predictable way: they measure the request's size, not its stakes.

"Fix the auth bug in session.js" is eight words. Every token-based rule on earth routes it to the small model. But those eight words unleash a read-grep-edit-test loop — the exact workload where small models faceplant. Meanwhile, "explain the difference between optimistic and pessimistic locking, with examples" looks expensive (long answer, technical vocabulary) and is actually perfectly cheap-model-safe: it's pure text generation, no tool calls, no exact string matching, nothing to break.

Size and stakes are almost uncorrelated in agentic traffic. That's the whole problem.

What "stakes-aware" routing looks like

When I built Lynkr's router, most of the design ended up being about when not to save money. The parts that matter:

Weight the tools, not just their count. A request where Grep and Read are in play is research — paraphrase-tolerant, failure-tolerant, ideal for a local model. A request where Bash, Write, or Edit will fire is a mutation with exact-match requirements. Lynkr assigns each tool a risk weight (Bash 0.9, Write 0.8, Edit 0.7 … Grep 0.2) and scores the request's effective toolset. Two requests with five tools each can land tiers apart.

Treat mid-session as a signal. If the conversation already contains three tool results, you're inside an agentic flow with accumulated exact-state (file contents, error strings). Downgrading the model mid-flow throws away the one thing that was keeping the loop convergent. Prior tool usage and conversation depth push requests up-tier even when the latest message is short.

Subtract the harness baseline. Claude Code ships ~14 tool schemas with every request — including "hello." Count them naively and everything looks agentic, so nothing ever routes local and you save nothing. Score only the tools the request could plausibly use, and the safe majority routes down while the risky minority stays up.

Some patterns override everything. Greetings and "what does X do" questions force-route local, always. Security-sensitive analysis force-routes to the strong tier, always — a JWT architecture question is short, toolless, and precisely the wrong place to save four cents.

The result on my own traffic: 70–90% of requests route to free local models — but they're the right 70–90%, which is the entire difference between "my bill dropped" and "my agent broke."

Takeaways, router-agnostic

Route research down, mutations up. If your router can't tell a Grep request from an Edit request, it isn't routing — it's gambling on which sessions break.
Never downgrade mid-loop. Model consistency across an agentic sequence is worth more than the marginal savings of one cheap turn.
Measure session survival, not just cost. A routing setup that saves 60% and breaks one session in five is more expensive than the bill it replaced — you're paying in re-runs and rage.
The ceiling is rising. Local models' tool-calling improves every quarter; the set of safely-downgradable requests grows with it. A router with per-tool judgment gets to expand that set gradually. A token threshold has to guess again from scratch.

The router's job was never "pick the cheapest model." It's "pick the cheapest model that won't break the session" — and those five extra words are where all the engineering lives.

The scorer described here is ~1,000 lines of readable Apache-2.0 JavaScript: src/routing/complexity-analyzer.js. Steal the design, or file an issue telling me where it misjudges — the failure cases are the interesting part.

The 5% Router Tax: What Hosted LLM Gateways Charge For (and How to Self-Host It)

Lynkr — Sun, 05 Jul 2026 08:48:56 +0000

Disclosure: I maintain Lynkr, the self-hosted gateway discussed in the second half. OpenRouter and Requesty are good products — this post is about understanding what you're paying for so you can decide whether you need to.

Hosted LLM routers had a huge 2026 — OpenRouter alone pushes 25 trillion tokens a week. The pitch is real: one API key, 400+ models, automatic failover. The price is a ~5% fee on every token you route (5.5% on OpenRouter credits, 5% on Requesty), plus a subtler cost: every prompt, every file your coding agent reads, every secret that leaks into a context window transits their infrastructure.

For a hobby project, 5% of a small bill is nothing and the convenience wins. For an agentic coding workload — where teams routinely spend $500–$2,000 per engineer per month — 5% is real money, and the data-transit question stops being academic. So it's worth asking precisely: what does the hosted router actually do for that fee, and which parts can you self-host?

What the fee buys

Unified API across providers — one format in, translated per-provider out.
Failover — a provider 500s, your request retries elsewhere.
Model marketplace — new models available the day they launch.
Consolidated billing — one invoice instead of six provider accounts.
(Sometimes) smart routing — OpenRouter's auto router picks a model per-request.

Items 1, 2, and 5 are software. Items 3 and 4 are genuinely hard to self-host — if you want day-one access to every new model with zero account setup, the marketplace earns its fee. But most coding workloads use a handful of models, not four hundred.

The parts a hosted router structurally can't give you

Local models as a tier. No hosted router will route your easy requests to the Ollama instance on your own machine — free, private, zero latency to first byte on cached weights. For coding traffic, where (in my instrumented sessions) 70–90% of requests are simple enough for a good local model, this is the single biggest cost lever, and it's only available to something running on your side of the wire.
Your data staying home. Self-hosted means prompts, code, and keys never transit a third party. For anyone with a compliance requirement — or code they'd rather not ship to a router's logs — this isn't a preference, it's a prerequisite.
Token optimization before the bill. A hosted router bills you for the tokens you send it — it has no incentive to shrink them. A self-hosted proxy can strip unusable tool schemas (measured: −53% on tool-heavy requests) and compress JSON tool results (measured: 3,458 → 427 tokens on a grep result) before any provider bills you. That's not a routing saving; it stacks on top of routing.
No availability dependency. Hosted routers go down (OpenRouter's outages have their own HN threads) and offer no SLA at consumer tiers. A local proxy fails independently of anyone's status page.

What self-hosting costs you

Honesty cuts both ways:

You run a process. npm install -g lynkr && lynkr init && lynkr start — but it's yours now: updates, logs, the works.
You manage provider accounts. Two or three API keys instead of one. The consolidated invoice is genuinely gone.
Model lag. A new provider means waiting for support (or a PR) instead of it appearing in a dropdown.
Nobody to email. Self-hosted support is a GitHub issue tracker.

If those trade-offs read as "fine," the math is straightforward: the 5% fee disappears, the local-tier routing removes the easy majority of requests from your bill entirely, and compression shrinks what's left.

The hybrid that actually makes sense

This isn't either/or. A pattern I see working:

Coding tool → self-hosted proxy (Lynkr)
                ├─ SIMPLE/MEDIUM  → local Ollama/llama.cpp   (free)
                ├─ COMPLEX        → direct provider API keys  (no fee)
                └─ exotic models  → OpenRouter               (5% on the long tail only)

Keep a hosted router as one backend for the long tail of models you rarely need, route the bulk directly or locally, and let the proxy's classifier decide per-request. You get the marketplace when you want it without paying the tax on your entire volume.

Lynkr is Apache-2.0, self-hosted, supports 13 providers including Ollama, llama.cpp, LM Studio, Bedrock, Azure, Databricks — and OpenRouter itself as a tier: github.com/Fast-Editor/Lynkr. Benchmarks with methodology are in the repo; run them on your own workload before believing anyone's percentages, including mine.

How a 13-Dimension Complexity Scorer Decides Which Model Gets Your Request

Lynkr — Sun, 05 Jul 2026 08:48:40 +0000

Disclosure: I'm the author of Lynkr, the open-source proxy whose internals this post walks through. All code shown is real and Apache-2.0 — read it here.

The most expensive default in AI coding tools is that model choice is a setting, not a decision. You pick a model once; every request — "what does git stash do?" and "refactor this auth module" alike — goes there. Routing each request to the cheapest model that can actually handle it is worth 50%+ of most bills, but it only works if the "can actually handle it" judgment is reliable. Get it wrong downward and a small model fumbles your file edits; get it wrong upward and you've saved nothing.

Here's how Lynkr makes that judgment, in enough detail that you could reimplement it.

Why not just count tokens?

The obvious heuristics fail in both directions:

"Long request → big model" fails on a 60k-token context that's mostly grep output around a trivial question.
"Short request → small model" fails catastrophically on "fix the auth bug in session.js" — eight words that unleash a tool-heavy agentic session a 7B model will faceplant on.

Token count is one signal. The failure cases all come from treating it as the only one.

The architecture: weighted dimensions, then overrides

Every request gets a 0–100 score from 13 dimensions in four groups. The weights are configurable; these are the defaults:

const DIMENSION_WEIGHTS = {
  // Content Analysis (35%)
  tokenCount: 0.08,
  promptComplexity: 0.10,   // avg sentence length/structure
  technicalDepth: 0.10,     // technical keyword density
  domainSpecificity: 0.07,  // security/ML/distributed/db/frontend/devops
  // Tool Analysis (25%)
  toolCount: 0.08,
  toolComplexity: 0.10,     // which tools, not how many
  toolChainPotential: 0.07, // "first...then", "step 2", sequencing language
  // Reasoning Requirements (25%)
  multiStepReasoning: 0.10,
  codeGeneration: 0.08,
  analysisDepth: 0.07,      // trade-off/comparison markers
  // Context Factors (15%)
  conversationDepth: 0.05,
  priorToolUsage: 0.05,     // tool_results already in the conversation
  ambiguity: 0.05,
};

A few design decisions worth stealing:

Not all tools are equal. A request that can Grep is not like a request that can Bash. Each tool carries a hand-tuned risk weight — Bash 0.9, Write 0.8, Edit 0.7, down to Grep at 0.2. A request whose available toolset averages 0.8 is an agentic mutation session; one averaging 0.25 is read-only research. Same tool count, completely different stakes.

Subtract the harness baseline. Claude Code ships ~14 tool schemas with every request, including "hello". If you count them naively, everything looks agentic and nothing routes local. The scorer subtracts the client's constant baseline and scores only the effective tools the request could plausibly use — one of those fixes that sounds trivial and changed everything.

Conversation history is a signal. Three tool_result blocks already in the conversation means you're mid-agentic-flow — this is not the moment to downgrade models and break the session's momentum. priorToolUsage and conversationDepth push mid-session requests up-tier.

Ambiguity cuts the other way. "file X, line 42, this error" is specific — a small model can act on it. "Something feels slow sometimes" needs interpretation before action. Specificity markers (paths, line numbers, error strings) lower the score.

Overrides: the classifier knows what it can't know

Two pattern lists short-circuit the whole scoring pipeline:

Force-local: greetings, acknowledgments, "what does X do" one-liners. Score 0, never leave the machine, no cloud tokens ever.
Force-cloud: security-critical analysis, architecture decisions, anything matching high-risk patterns. Straight to the top tier regardless of how cheap it looks. A JWT-vs-cookies security question is short and toolless — every naive heuristic routes it local. This is the wrong request to save $0.004 on.

On top of the regex dimensions, an AST pass (tree-sitter) scores actual code structure in the payload — cyclomatic signals beat keyword counting when real code is present.

From score to model

score < threshold        → SIMPLE   (e.g. ollama:qwen2.5:7b, free)
threshold..~65           → MEDIUM   (e.g. ollama:qwen2.5-coder, free)
above                    → COMPLEX  (your API key: Sonnet, GPT-4o...)
reasoning markers heavy  → REASONING (o3, DeepSeek R1...)

The threshold moves with a single mode switch — aggressive (60) routes more local, conservative (25) routes more to the cloud, default is 40. Multi-turn conversations score with a recency-weighted sliding window, so a short follow-up ("now add tests") inherits the complexity of the work it refers to instead of scoring as a trivial one-liner.

Crucially, the classifier only chooses among models you listed. It's not an autonomous agent picking providers — you define the tiers, it picks the tier.

Does it work?

In my instrumented sessions, 70–90% of requests score SIMPLE or MEDIUM and run free on local models, while tool-heavy and security-flagged requests reliably escalate. The failure mode everyone fears — cheap model breaking an agentic session — is exactly what the tool weights, baseline subtraction, and prior-tool-usage dimensions exist to prevent.

Is 13 hand-weighted dimensions the optimal design? Almost certainly not — a learned router trained on outcome data would beat it eventually. But it's transparent (every routing decision logs its per-dimension breakdown), it's tunable, it runs in-process in microseconds, and it never sends your prompts to a third-party classifier API.

The whole thing is readable in one sitting: src/routing/complexity-analyzer.js. Steal the design or use the proxy — either outcome means fewer frontier-model tokens spent on git stash questions.

DEV Community: Lynkr

LLM Inference Is Reinventing the Operating System

1. The KV Cache Turns Inference Into a Memory-Management Problem

2. PagedAttention Turns KV Cache Into Pages

3. KV Cache Sharing Turns Into Shared Memory

4. KV Eviction Turns Into a Cache-Management Problem

5. Hierarchical KV Cache Creates a Memory Hierarchy

6. KV Prefetching Turns Into an I/O Problem

7. Cache-Aware Scheduling Turns Into an OS Scheduler Problem

8. Prefill/Decode Disaggregation Turns Into Resource Scheduling

Prefill

Decode

9. Agentic Workloads Turn KV Management Into a Process-Lifetime Problem

10. Distributed KV Turns Inference Into a Distributed Operating System

The Bigger Picture

The Most Important Difference

Conclusion

How to run Claude Desktop and ChatGPT desktop app for free with llama.cpp and other local LLMs.

Claude Desktop: using the hidden third-party gateway path

Why Claude Desktop needs a token

Claude Desktop's model picker becomes a tier selector

Claude Desktop detection is gated, not global

ChatGPT.app: one backend for Desktop and CLI

ChatGPT.app exposes a real routing signal: reasoning.effort

The pin is intentionally conservative

Why these integrations matter

Lynkr vs LiteLLM's New `complexity_router`

Why this benchmark exists

Methodology

Systems under test

Identical backends, only the decision differs

Harness

Results

Scoreboard: 11 scenarios, acceptable-tier judging

Failure analysis: three clusters, not seven random misses

The cost table pathology (read before quoting)

Lynkr's own warts (strict-expectation view)

Where LiteLLM is better

Where Lynkr is better

Limitations

Reproduction

Follow-ups queued

Loop Engineering Is Mostly Papering Over a Model That Won't Converge

The consensus I'm arguing against

Why longer loops compound the wrong things

What a long loop is actually signaling

What loop engineering should actually be

The part where this bit us too

The thesis, plainly

Your LLM Bill Is a Misallocation Problem, Not a Model Problem

The setup: the gap closed, the dispersion didn't

The naive fix fails: you can't predict difficulty from the prompt

What works: try cheap, verify, escalate

A war story about verifiers

The second-order effect: verification failures are free training data

Receipts

The thesis, restated

We Put Our Router on an Academic Benchmark. Here Are the Numbers We'd Rather Hide.

What RouterArena actually measures

What we submitted

The results

The parts we'll be quoting

The parts we'd rather hide (but won't)

What the benchmark taught us — and why we won't act on all of it

Caveats, so you can weigh this properly

Stuffing the Context Window Is Making Your Agent Dumber: What the Research Says

The result that should change how you build

Why more context makes things worse

What the research says works instead

What you can do about it today

How We Built an Agentic-Task Detector for LLM Routing

Not "agentic: yes/no" — a ladder

The six signals

The false positive that almost sank it

What it still gets wrong

Takeaways if you're building anything similar

Choosing a Local Tier for Your Coding Agent (July 2026 Edition)

First, the trap: "best open model" ≠ "your local tier"

The local field, by hardware budget

Mapping to tiers

ChatGPT.app exposes a real routing signal: `reasoning.effort`