An LLM router that tells you why it chose local, and what "why" has to mean

The setup a lot of us end up with: Ollama on the machine, and a key for a paid API when the local model is not up to it. The decision between them is either made by hand every time or hidden inside a gateway.

I got tired of the second one. It usually chose fine, but when it chose something I disagreed with, the only way to find out why was reading a config file and guessing.

So the design goal was narrow: every routing decision carries the reasons that produced it.

What "explainable" has to mean

A log line saying routed to ollama only reports the outcome you already observed.

What you need is the losing candidates and why each one lost, because eliminations are the part you can argue with.

So routing accumulates a trace as it goes:

const reason = rejectionFor(backend, ctx, health);
if (reason) {
  trace.push({ backend: backend.name, decision: 'skipped', detail: reason });
  continue;
}

The reasons are written as sentences:

const REJECTIONS = {
  unhealthy:       (b)      => `${b.name} is not answering health checks`,
  noModel:         (b, ctx) => `${b.name} does not serve ${ctx.model}`,
  contextTooSmall: (b, ctx) =>
    `${b.name} has a ${b.contextWindow} token window and this request needs about ${ctx.tokens}`,
  noTools:         (b)      => `${b.name} does not support tool calling`,
  overBudget:      (b, ctx) => `${b.name} would cost more than the ${ctx.maxCostPerRequest} budget`,
};

contextTooSmall earns its keep. When a long document goes to the cloud, the trace says the local backend has an 8192 token window and the request needs about 31,000. You either raise the window or accept the cost.

Full sentences instead of codes was deliberate. A code like E_CONTEXT needs a lookup table that will drift from the code. The sentence is generated from the same values the decision used, so it cannot disagree with it.

nearcall route "$(cat long-document.txt)" --model gpt-4o

The ordering, in four tiers

export function rank(backends, ctx, { preferLocal = true, health = {} } = {}) {
  return [...backends].sort((a, b) => {
    if (preferLocal && a.local !== b.local) return a.local ? -1 : 1;

    const byPriority = (a.priority ?? 100) - (b.priority ?? 100);
    if (byPriority !== 0) return byPriority;

    const costDelta = estimateCost(a, ctx.tokens) - estimateCost(b, ctx.tokens);
    if (Math.abs(costDelta) > 1e-9) return costDelta;

    const latencyA = health[a.name]?.latencyMs ?? Number.MAX_SAFE_INTEGER;
    const latencyB = health[b.name]?.latencyMs ?? Number.MAX_SAFE_INTEGER;
    return latencyA - latencyB;
  });
}

Local first, then your configured priority, then cost, then observed latency.

Local first is an explicit branch and not a consequence of local being free, which is worth being clear about because the two produce different behaviour. A local model taking two seconds still beats a cloud model taking one and charging for it, and that preference survives even when the cost difference is negligible. Set preferLocal: false and the ordering falls back to cost, which is the right choice if you care more about wall clock than about where the tokens go.

Latency is last on purpose. It is measured by the health checks rather than configured, so a local model on a loaded machine can lose to a faster sibling, but only after cost has already been settled.

Token estimation is roughly four characters per token, Math.ceil(text.length / 4). That is crude on purpose. It is picking a route, not billing you, and being exact would mean running a tokeniser per backend family for a decision that only needs to be approximately right.

Not competing with LiteLLM

LiteLLM is good and it is built for a platform team: a Python service, a large dependency tree, and a config file before it does anything useful. For one developer with Ollama on a laptop and a key for the hard cases, that is a lot of machinery.

This is one command, zero dependencies, and it works before you write any config. It also serves an OpenAI-compatible endpoint, so anything already speaking to OpenAI can point at it unchanged:

npx nearcall serve

lightport on npm covers similar ground, and Portkey ships a Node client for its hosted gateway. For a managed gateway with a dashboard and spend controls, take one of those. This is for the local case with no service to run.

The command I use most is nearcall doctor, which reports what it can reach right now. "Why is everything going to the cloud" is nearly always "the local backend is not running".

41 tests, zero dependencies, MIT. nearcall.

One design decision in the health checking that I got wrong first has its own article.