The hardest problem in my local AI agent was never the model

How tools get picked in Eris: from grep to embeddings to grammar.

Or: how a local 14B model decides which of ~50 tools to call, without me writing a giant if-else on the user's message.

When I started this I knew basically nothing. How you even get an LLM to call a tool, let alone pick the right one, was not obvious to me at all, and every step was me poking at it to see what breaks. So this post is not a "here is the clean way" post. It is the honest version of that learning curve, all the way from the first clumsy version to what runs now. Tool selection in Eris went through three whole lives to get here: grepping the model's own text output, then keyword lists on the user message, then embeddings with a policy layer sitting on top. Each life existed because the one before it kept failing in a new way, and none of them made the model smarter, they just stopped me handing a small model the wrong tool.

Where it landed today: the user text gets embedded, compared against one precomputed vector per tool, and a small policy layer decides whether to offer one tool, a cluster of related ones, or the whole roster. That offered list then feeds two things at once, the slim tool prompt and the GBNF grammar, so prompt and grammar can never disagree about what is callable.

But it did not start there. It started with grep.

The grep era

Before Eris was Eris it was FUCKUP. Named after Hagbard Celine's supercomputer in the Illuminatus! trilogy, the First Universal Cybernetic-Kinetic Ultramicro-Programmer that answers questions from inside a golden submarine. The name doubling as a status report on the code was a happy accident, or an easterfuck. The FcpError taxonomy from back then still is alive in Eris until today. Ollama plus an Obsidian vault plus Redis plus a bash-ish tool bridge. The relevant line from that README, which I keep around as a monument:

The agent has no native tools. It outputs API: <command> lines. A loop parses them, calls the bridge, and feeds results back.

That is the real grep era, spelled out. The model printed text like API: healthcheck, a shell loop grepped the output for API: lines, matched the command against an allowlist, ran it through a bridge on the host, and pasted the result back into the next turn. Tool "selection" was string parsing on both ends: regex on the model output to find the call, keyword lists to decide what was even relevant to offer. It worked well enough to be dangerous. Means, it worked in the demo. In real use it did not, and you find that out live.

Later it grew a second path: a patched fork of an existing orchestrator doing structured tool calling in OpenAI function-call format, because text API: parsing was too brittle. And that is where the lesson landed that made me throw the whole thing away and rewrite in Rust. Local models through Ollama were awful at structured tool calls. Streaming broke tool_calls and handed back raw text, custom tools did not get passed through, the system prompt sometimes never reached the model. You can patch a TypeScript orchestrator around all of that, and I did. But the deeper problem stayed.

My first fix was the obvious one. If the model does not get the tool contract, then push the whole tool contract into the system prompt, hard. And it kind of worked. But the system prompt got bloated fast. Every tool with its full schema, all the time, every turn. On sparse VRAM that is exactly the wrong place to spend your tokens. The context is your budget, and I was burning it on tool definitions the model did not even need this turn.

So the real fix was somewhere else. Move to llama.cpp, stop describing the tools in prose, and constrain the sampler directly instead. Hand it only the tools that matter for this turn. JIT the structured part instead of front-loading all of it. Reliability has to come from constraining the sampler, not from politely asking a streaming API to behave. That is the moment Eris starts.

But back to selection. Even the keyword-matching half of the old approach, the "which tools are relevant" part, took the user line, lowercased it, and ran a pile of contains checks. "weather" in the string, offer the weather tool. "remind" in the string, offer the reminder. It is the honest first thing everybody builds.

Then real usage happens and it falls apart in the boring ways:

• Synonyms. User says "how warm is it outside". No "weather" token anywhere. Miss.
• Other language. I chat with Eris half in German. "wie spät ist es" does not contain "time".
• Pointing. "do that again", "same as before", "open it". None of these words live in any keyword list.
• Collisions. "open" matched "open the file" and "you will open the way for future AIs" with equal confidence. The second one is not a tool call, it is me being poetic at my own agent at 2am.

You can patch each of these with more keywords, and then you have a 400 line lexical matcher that nobody dares touch and that still misses the next phrasing. I know because I wrote it and lived with it.

Two things survived that era on purpose, and I want to be honest that they survived, because a lot of routing writeups pretend they went fully semantic and they did not.

The first survivor is a set of lexical guards for cases where a keyword is actually a hard signal. If the text has a real URL or a domain-looking token, I do not want to hope the embedder feels like offering web:fetch. I force it:

if Self::has_web_lexical_intent(thought) && !hits.iter().any(|(t, _)| t == "web:fetch") {
    tracing::info!(
        event = "LEXICAL_TOOL_GUARD",
        forced_tool = "web:fetch",
        thought_preview = %thought.chars().take(120).collect::<String>(),
        "Forcing web:fetch due to lexical URL/page intent"
    );
    hits.push(("web:fetch".to_string(), 1.0));
}

Note the score 1.0. That is not a real cosine, it is a "do not argue with me" marker, and later the policy layer treats anything at ~0.99 and above as a forced hit that never gets demoted. The lexical checks themselves learned some manners over time. Bare open used to match figurative English, so now it is phrases like open page, open the website, visit the page, not the lone verb.

The second survivor is the keyword lists themselves. They did not get deleted. They got repurposed.

Routing phrases: keywords that grew up

The old keyword lists became routing phrases. Same strings, roughly, but now they do a different job. Instead of being matched with contains, they are the text you embed to describe a tool.

Every tool has a line of "this is what people say when they want me". For tools without a richer descriptor there is a compile-time fallback in routing_phrases.rs:

"clock:now" => "what time is it, current time, timezone, date now, local time",
"weather:current" => {
    "weather now, temperature outside, is it raining, rainfall, sunny or cloudy, conditions today, current conditions, what's the weather like"
}

And for the tools I cared about more, the phrases live in the TOML descriptor next to when_to_use / when_not_to_use, as routing_hints:

tool_name = "agenda:remind_at"
routing_hints = [
    "remind me at",
    "remind me in",
    "remind me about",
    # ...
]

Descriptor hints win when present, otherwise the fallback string, otherwise just the tool description. That resolution order lives in one place so nothing drifts:

fn enrich_for_routing(
    name: &str,
    description: &str,
    descriptors: Option<&ToolDescriptorRegistry>,
) -> String {
    if let Some(registry) = descriptors
        && let Some(desc) = registry.get(name)
        && !desc.routing_hints.is_empty()
    {
        return format!(
            "{}: {}. Common triggers: {}",
            name, description, desc.routing_hints.join(", ")
        );
    }
    let hints = crate::tools::routing_phrases::fallback_triggers(name);
    if hints.is_empty() {
        format!("{}: {}", name, description)
    } else {
        format!("{}: {}. Common triggers: {}", name, description, hints)
    }
}

So the exact same phrases that used to be a brittle contains list are now the seed text for a vector. Synonyms and paraphrase stop being my problem and become the embedder's problem, which is the whole point, because that is a thing embedders are actually good at.

Embeddings now

At startup the ToolRouter embeds each tool's enriched text once and keeps the vectors in memory:

pub async fn new(
    embed: Arc<dyn EmbeddingProvider>,
    tool_descriptions: Vec<(String, String)>,
    descriptors: Option<Arc<ToolDescriptorRegistry>>,
    threshold: f32,
) -> Result<Self> {
    let mut tool_embeddings = Vec::with_capacity(tool_descriptions.len());
    for (name, description) in &tool_descriptions {
        let text = Self::enrich_for_routing(name, description, descriptors.as_deref());
        let embedding = embed.embed(&text).await?;
        tool_embeddings.push((name.clone(), embedding));
    }
    Ok(Self { embed, tool_embeddings, threshold })
}

Same embedding model as the vector memory (nomic-embed-text by default), so I am not shipping a second model just for routing. One embed model does prefetch, memory recall, and tool routing.

Per turn, embed the user text, cosine against every tool vector, keep whatever clears the threshold (tool_match_threshold, default 0.50), sort descending:

let thought_vec = self.embed_text(thought).await?;
let mut hits: Vec<(String, f32)> = self
    .tool_embeddings
    .iter()
    .filter_map(|(name, emb)| {
        let sim = cosine_similarity(&thought_vec, emb);
        if sim >= self.threshold { Some((name.clone(), sim)) } else { None }
    })
    .collect();
hits.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));

Before any of this even runs there is a cheap short-input guard: greetings and tiny utterances ("hey", "thanks", "test") are treated as conversation and skip embedding entirely, unless they carry an obvious tool token like a URL or "list my documents". No point paying for an embed call and a grammar recompile to answer "hi".

There is also one embarrassingly specific semantic floor worth mentioning, because it is the kind of thing you only learn from running the thing daily. Moltbook tools (my little federated social thing) kept getting weakly matched by general chat and especially by memory-recall phrasing, they all sit around 0.50 to 0.56 against each other. So unless the user actually said "moltbook" or "submolt", moltbook hits below 0.58 get dropped. Not elegant. Correct.

More than top-k: the policy layer

Here is the part I am actually a bit proud of, and the part most "just embed it" writeups skip.

Naive routing is: take top-k by cosine, offer those, done. That is fine until you remember what happens downstream. Downstream, on the llama.cpp backend, the offered tools get compiled into a GBNF grammar that forces the model's output. If the router confidently picks one wrong tool and hands the grammar exactly that one tool, the model is now structurally locked into calling the wrong thing. Top-1 with a hard grammar behind it is not "confident", it is "unrecoverable".

So there is a policy layer between the raw cosine hits and the offer. It does not replace embeddings, it just vetoes, widens, or pairs. The entry point reads like a little rulebook:

pub fn decide(
    signals: &RoutingSignals,
    registered: &[String],
    knobs: RoutingPolicyKnobs,
) -> RoutingDecision {
    // Rule 1 — dialog pairing (agenda → mail → calendar → gated doc).
    if let Some(decision) = try_dialog_pairing(signals, registered) {
        return decision;
    }
    let (forced, embed): (Vec<_>, Vec<_>) = signals
        .embed_hits
        .iter().cloned()
        .partition(|(_, score)| *score >= FORCED_HIT_FLOOR);

    // Rule 2 — lone weak embed hit demotion (+ unsure fallback).
    // Rule 3 — near-tie across related domains → affinity cluster union.
    // ...
}

Three moves matter here.

Demote the lone weak hit. If the only thing that cleared the threshold is a single tool sitting below tool_single_hit_floor (0.58), that is not confidence, that is a coin flip. Offering it alone would GBNF-lock the model onto a guess. So a lone weak hit gets demoted and we fall through to the full roster (or, configurable, to that tool's domain cluster):

fn demote_lone_weak_embed(
    embed: Vec<(String, f32)>,
    single_hit_floor: f32,
) -> (Vec<(String, f32)>, Option<(String, f32)>) {
    if embed.len() == 1 && embed[0].1 < single_hit_floor {
        tracing::info!(
            tool = %embed[0].0, score = embed[0].1, floor = single_hit_floor,
            event = "routing.policy.single_hit_demoted",
            "Lone weak semantic hit demoted (avoid GBNF lock-in)"
        );
        return (Vec::new(), Some((embed[0].0.clone(), embed[0].1)));
    }
    (embed, None)
}

Widen a near-tie into a cluster, but only for related domains. When the top hits are within match_margin (0.05) of each other, that is the model being torn between neighbours. "remind me tomorrow at 10" scores clock:alarm and agenda:remind_at almost identically, because honestly they are almost the same intent. In that case I do not want to pick one and lock the grammar to it. I widen to the union of the related domain clusters and let the model choose inside the grammar:

// clock + agenda + calendar share the "time" affinity bucket.
pub fn affinity_group(domain: &str) -> Option<&'static str> {
    match domain {
        "agenda" | "clock" | "calendar" => Some("time"),
        "web" | "news" | "wiki" => Some("web"),
        "doc" | "vault" | "memory" | "media" => Some("knowledge"),
        // ...
        _ => None,
    }
}

The "only related domains" bit matters. A near-tie across moltbook + web + db + doc is not a coherent cluster, that is just mush where everything landed around 0.55. Dumping all four families' tools into the grammar would be worse than useless. So unrelated near-ties stay a plain cosine-ranked subset, no cluster dump. This one rule killed a whole class of "why did it offer the train schedule tool for a memory question" bugs.

Keep the ordering honest. Whatever comes out, forced or cluster-widened or plain, gets re-ranked by the original cosine at the end, so the tool the embedder actually liked stays near the top of the phrase map the model reads. Cluster siblings that never got a real score inherit the best score from their domain, minus a hair, so they sort right after the seed instead of jumping the queue.

There is also a small dialog memory: the last handful of successful tools is kept (capped, session-scoped) so "delete that email" after a mail:check can pair to mail:delete instead of being read as a bare "delete" with no object. Agenda wins over mail and calendar in those pairings, because that is the order that matched how I actually talk to it.

The slim tool offer

Okay, so the policy layer produced an offered list. Now, what does the model actually see?

The naive answer is: all the tools, full JSON Schema for every one, dumped into the system prompt. For ~50 tools that is a wall of schema that eats context and, worse, gives a small model 50 ways to be wrong. My earlier GBNF post already argued that fewer choices means fewer mistakes, and this is the prompt-side half of that.

So by default (slim_tool_prompt = true) the tool-mode prompt is not the schema wall. It is two smaller things:

1. A phrase map, a little markdown table of tool name, short description, and the same routing phrases from earlier, so the model has natural-language hooks for each offered tool.
2. The tool definitions with parameters stripped. The model sees that vault:write exists and roughly what it is for, but not the full argument schema. The full schema is not needed to decide to call a tool, and if the model later gets the args shape wrong, gatekeeper schema recovery supplies the real schema on demand.

The phrase map is generated at runtime from the registered tools plus their descriptors:

"| Tool | Description (short) | Typical phrasing / triggers |".to_string(),
"| ---- | ------------------- | --------------------------- |".to_string(),
// one row per offered tool, phrases = descriptor hints, else fallback, else description

Before that list is finalized there are a few offer overlays, the pragmatic pairings that experience forced on me:

// web:fetch or web:search in the offer? then web:find has to come too,
// because you always want to query what you just fetched.
let needs_web_find = offered.iter().any(|n| n == "web:fetch" || n == "web:search");
// doc:read in the offer usually means the model will want to write a summary,
// so pair vault:write if the state allows it.
if offered.iter().any(|n| n == "doc:read")
    && !offered.iter().any(|n| n == "vault:write")
    && Gatekeeper::state_allows_tool(state, "vault:write")
{
    offered.push("vault:write".to_string());
}

There is also a cap (tool_map_offer_cap) if you want to hard-limit how many router hits make it into the map, and a "moltbook latch" so that once a turn is clearly about moltbook, the whole moltbook family comes along instead of one lonely endpoint.

One list, two consumers, no drift

Here is the bit that ties the whole thing together and is the reason I bothered writing overlays.rs as its own module instead of inlining it twice.

The offered-tool list feeds two consumers on every tool turn: the slim prompt assembly, and the GBNF grammar. If those two ever disagree, you get the worst kind of bug, the prompt tells the model tool X is available, the grammar forbids tool X's tokens, and the model wedges. So both call the exact same function:

pub(crate) fn slim_offered_tool_names(
    pre_llm_matched_tools: &[String],
    tool_map_offer_cap: usize,
    moltbook_overlay_latched: bool,
    gatekeeper: &Gatekeeper,
    state: &AgentState,
) -> Vec<String> {
    crate::orchestrator::routing::apply_offer_overlays(
        pre_llm_matched_tools, tool_map_offer_cap,
        moltbook_overlay_latched, gatekeeper, state,
    )
}

Single source of truth. The prompt and the grammar are computed from the same offered, in the same order, with the same overlays. They physically cannot drift, because there is only one list.

Why the grammar is the whole reason this is careful

If you only remember one thing: the reason routing is this defensive is that on llama.cpp the offer is not a suggestion, it is a hard constraint at sample time.

The offered list gets compiled into a GBNF grammar. GBNF constrains which tokens the sampler is even allowed to emit, before sampling, so anything that cannot extend into a valid parse gets masked out. The model does not "try to" produce a tool call in the right shape, it cannot produce anything else. Full detail is in the GBNF post, but the shape is: a fixed protocol envelope (thought, status, message, tool_calls), and a tool-call rule whose allowed names are exactly this turn's offered tools, with each tool's args compiled from its JSON Schema.

Which is exactly why I do not let the router hand over a single confident guess. Under a hard grammar:

• Offer one wrong tool, the model is forced to call the wrong tool. No recovery inside the turn.
• Offer an empty tool list when the router says "this is just chat", and tool_calls can only be []. The model cannot hallucinate a call. That is a feature, that is the conversational path.
• Offer a cluster on a near-tie, and the model gets to pick the right neighbour, but only from a small, related, structurally valid set.

So the grammar is what turns "structured output" from a prompt-time please into a real guarantee. The policy layer only makes sure that guarantee points at roughly the right tools. In the end there are four layers stacked on each other. Embeddings find the candidates. The policy decides how wide to open the door. The slim map tells the model what is behind it. The grammar makes sure whatever comes back actually parses. None of these layers I planned upfront. Each one is there because the one before it burned me first.

What I would tell someone building this

If you route tools for a local model with grammar-constrained output: the thing I got wrong the longest was to treat routing as a ranking problem. It is not really about ranking. The real question is how much you trust that ranking before you let a grammar act on it.

• Embeddings beat keyword matching for finding candidates. Keep your old keyword lists though, they are perfectly good embedding seeds and decent hard-override floors.
• Top-1 plus a hard grammar is a trap. Confidence has to be earned with a score floor and a margin check, and when it is not there, widen instead of guessing.
• Widen along real relationships (time-ish tools together, knowledge tools together), not along whatever happened to score near 0.55.
• Compute the offer once and feed both the prompt and the grammar from it. Two derivations of "what is callable" will drift, and the failure mode is nasty.

None of this made the model smarter. It just stopped me from handing a small model a loaded grammar pointed at the wrong tool. Which, on a 14B, is most of the battle.

Code is at github.com/janpauldahlke/eris, Apache 2.0. The full post lives on my site at eris-system.dev.

Routing lives under src/orchestrator/routing/, the router itself in src/orchestrator/tool_router.rs. Poke at it, break it, tell me where it is dumb ;-) !!

Jan