DEV Community: jan paul

The hardest problem in my local AI agent was never the model

jan paul — Wed, 09 Sep 2026 11:00:34 +0000

How tools get picked in Eris: from grep to embeddings to grammar.

Or: how a local 14B model decides which of ~50 tools to call, without me writing a giant if-else on the user's message.

When I started this I knew basically nothing. How you even get an LLM to call a tool, let alone pick the right one, was not obvious to me at all, and every step was me poking at it to see what breaks. So this post is not a "here is the clean way" post. It is the honest version of that learning curve, all the way from the first clumsy version to what runs now. Tool selection in Eris went through three whole lives to get here: grepping the model's own text output, then keyword lists on the user message, then embeddings with a policy layer sitting on top. Each life existed because the one before it kept failing in a new way, and none of them made the model smarter, they just stopped me handing a small model the wrong tool.

Where it landed today: the user text gets embedded, compared against one precomputed vector per tool, and a small policy layer decides whether to offer one tool, a cluster of related ones, or the whole roster. That offered list then feeds two things at once, the slim tool prompt and the GBNF grammar, so prompt and grammar can never disagree about what is callable.

But it did not start there. It started with grep.

The grep era

Before Eris was Eris it was FUCKUP. Named after Hagbard Celine's supercomputer in the Illuminatus! trilogy, the First Universal Cybernetic-Kinetic Ultramicro-Programmer that answers questions from inside a golden submarine. The name doubling as a status report on the code was a happy accident, or an easterfuck. The FcpError taxonomy from back then still is alive in Eris until today. Ollama plus an Obsidian vault plus Redis plus a bash-ish tool bridge. The relevant line from that README, which I keep around as a monument:

The agent has no native tools. It outputs API: <command> lines. A loop parses them, calls the bridge, and feeds results back.

That is the real grep era, spelled out. The model printed text like API: healthcheck, a shell loop grepped the output for API: lines, matched the command against an allowlist, ran it through a bridge on the host, and pasted the result back into the next turn. Tool "selection" was string parsing on both ends: regex on the model output to find the call, keyword lists to decide what was even relevant to offer. It worked well enough to be dangerous. Means, it worked in the demo. In real use it did not, and you find that out live.

Later it grew a second path: a patched fork of an existing orchestrator doing structured tool calling in OpenAI function-call format, because text API: parsing was too brittle. And that is where the lesson landed that made me throw the whole thing away and rewrite in Rust. Local models through Ollama were awful at structured tool calls. Streaming broke tool_calls and handed back raw text, custom tools did not get passed through, the system prompt sometimes never reached the model. You can patch a TypeScript orchestrator around all of that, and I did. But the deeper problem stayed.

My first fix was the obvious one. If the model does not get the tool contract, then push the whole tool contract into the system prompt, hard. And it kind of worked. But the system prompt got bloated fast. Every tool with its full schema, all the time, every turn. On sparse VRAM that is exactly the wrong place to spend your tokens. The context is your budget, and I was burning it on tool definitions the model did not even need this turn.

So the real fix was somewhere else. Move to llama.cpp, stop describing the tools in prose, and constrain the sampler directly instead. Hand it only the tools that matter for this turn. JIT the structured part instead of front-loading all of it. Reliability has to come from constraining the sampler, not from politely asking a streaming API to behave. That is the moment Eris starts.

But back to selection. Even the keyword-matching half of the old approach, the "which tools are relevant" part, took the user line, lowercased it, and ran a pile of contains checks. "weather" in the string, offer the weather tool. "remind" in the string, offer the reminder. It is the honest first thing everybody builds.

Then real usage happens and it falls apart in the boring ways:

Synonyms. User says "how warm is it outside". No "weather" token anywhere. Miss.
Other language. I chat with Eris half in German. "wie spät ist es" does not contain "time".
Pointing. "do that again", "same as before", "open it". None of these words live in any keyword list.
Collisions. "open" matched "open the file" and "you will open the way for future AIs" with equal confidence. The second one is not a tool call, it is me being poetic at my own agent at 2am.

You can patch each of these with more keywords, and then you have a 400 line lexical matcher that nobody dares touch and that still misses the next phrasing. I know because I wrote it and lived with it.

Two things survived that era on purpose, and I want to be honest that they survived, because a lot of routing writeups pretend they went fully semantic and they did not.

The first survivor is a set of lexical guards for cases where a keyword is actually a hard signal. If the text has a real URL or a domain-looking token, I do not want to hope the embedder feels like offering web:fetch. I force it:

if Self::has_web_lexical_intent(thought) && !hits.iter().any(|(t, _)| t == "web:fetch") {
    tracing::info!(
        event = "LEXICAL_TOOL_GUARD",
        forced_tool = "web:fetch",
        thought_preview = %thought.chars().take(120).collect::<String>(),
        "Forcing web:fetch due to lexical URL/page intent"
    );
    hits.push(("web:fetch".to_string(), 1.0));
}

Note the score 1.0. That is not a real cosine, it is a "do not argue with me" marker, and later the policy layer treats anything at ~0.99 and above as a forced hit that never gets demoted. The lexical checks themselves learned some manners over time. Bare open used to match figurative English, so now it is phrases like open page, open the website, visit the page, not the lone verb.

The second survivor is the keyword lists themselves. They did not get deleted. They got repurposed.

Routing phrases: keywords that grew up

The old keyword lists became routing phrases. Same strings, roughly, but now they do a different job. Instead of being matched with contains, they are the text you embed to describe a tool.

Every tool has a line of "this is what people say when they want me". For tools without a richer descriptor there is a compile-time fallback in routing_phrases.rs:

"clock:now" => "what time is it, current time, timezone, date now, local time",
"weather:current" => {
    "weather now, temperature outside, is it raining, rainfall, sunny or cloudy, conditions today, current conditions, what's the weather like"
}

And for the tools I cared about more, the phrases live in the TOML descriptor next to when_to_use / when_not_to_use, as routing_hints:

tool_name = "agenda:remind_at"
routing_hints = [
    "remind me at",
    "remind me in",
    "remind me about",
    # ...
]

Descriptor hints win when present, otherwise the fallback string, otherwise just the tool description. That resolution order lives in one place so nothing drifts:

fn enrich_for_routing(
    name: &str,
    description: &str,
    descriptors: Option<&ToolDescriptorRegistry>,
) -> String {
    if let Some(registry) = descriptors
        && let Some(desc) = registry.get(name)
        && !desc.routing_hints.is_empty()
    {
        return format!(
            "{}: {}. Common triggers: {}",
            name, description, desc.routing_hints.join(", ")
        );
    }
    let hints = crate::tools::routing_phrases::fallback_triggers(name);
    if hints.is_empty() {
        format!("{}: {}", name, description)
    } else {
        format!("{}: {}. Common triggers: {}", name, description, hints)
    }
}

So the exact same phrases that used to be a brittle contains list are now the seed text for a vector. Synonyms and paraphrase stop being my problem and become the embedder's problem, which is the whole point, because that is a thing embedders are actually good at.

Embeddings now

At startup the ToolRouter embeds each tool's enriched text once and keeps the vectors in memory:

pub async fn new(
    embed: Arc<dyn EmbeddingProvider>,
    tool_descriptions: Vec<(String, String)>,
    descriptors: Option<Arc<ToolDescriptorRegistry>>,
    threshold: f32,
) -> Result<Self> {
    let mut tool_embeddings = Vec::with_capacity(tool_descriptions.len());
    for (name, description) in &tool_descriptions {
        let text = Self::enrich_for_routing(name, description, descriptors.as_deref());
        let embedding = embed.embed(&text).await?;
        tool_embeddings.push((name.clone(), embedding));
    }
    Ok(Self { embed, tool_embeddings, threshold })
}

Same embedding model as the vector memory (nomic-embed-text by default), so I am not shipping a second model just for routing. One embed model does prefetch, memory recall, and tool routing.

Per turn, embed the user text, cosine against every tool vector, keep whatever clears the threshold (tool_match_threshold, default 0.50), sort descending:

let thought_vec = self.embed_text(thought).await?;
let mut hits: Vec<(String, f32)> = self
    .tool_embeddings
    .iter()
    .filter_map(|(name, emb)| {
        let sim = cosine_similarity(&thought_vec, emb);
        if sim >= self.threshold { Some((name.clone(), sim)) } else { None }
    })
    .collect();
hits.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));

Before any of this even runs there is a cheap short-input guard: greetings and tiny utterances ("hey", "thanks", "test") are treated as conversation and skip embedding entirely, unless they carry an obvious tool token like a URL or "list my documents". No point paying for an embed call and a grammar recompile to answer "hi".

There is also one embarrassingly specific semantic floor worth mentioning, because it is the kind of thing you only learn from running the thing daily. Moltbook tools (my little federated social thing) kept getting weakly matched by general chat and especially by memory-recall phrasing, they all sit around 0.50 to 0.56 against each other. So unless the user actually said "moltbook" or "submolt", moltbook hits below 0.58 get dropped. Not elegant. Correct.

More than top-k: the policy layer

Here is the part I am actually a bit proud of, and the part most "just embed it" writeups skip.

Naive routing is: take top-k by cosine, offer those, done. That is fine until you remember what happens downstream. Downstream, on the llama.cpp backend, the offered tools get compiled into a GBNF grammar that forces the model's output. If the router confidently picks one wrong tool and hands the grammar exactly that one tool, the model is now structurally locked into calling the wrong thing. Top-1 with a hard grammar behind it is not "confident", it is "unrecoverable".

So there is a policy layer between the raw cosine hits and the offer. It does not replace embeddings, it just vetoes, widens, or pairs. The entry point reads like a little rulebook:

pub fn decide(
    signals: &RoutingSignals,
    registered: &[String],
    knobs: RoutingPolicyKnobs,
) -> RoutingDecision {
    // Rule 1 — dialog pairing (agenda → mail → calendar → gated doc).
    if let Some(decision) = try_dialog_pairing(signals, registered) {
        return decision;
    }
    let (forced, embed): (Vec<_>, Vec<_>) = signals
        .embed_hits
        .iter().cloned()
        .partition(|(_, score)| *score >= FORCED_HIT_FLOOR);

    // Rule 2 — lone weak embed hit demotion (+ unsure fallback).
    // Rule 3 — near-tie across related domains → affinity cluster union.
    // ...
}

Three moves matter here.

Demote the lone weak hit. If the only thing that cleared the threshold is a single tool sitting below tool_single_hit_floor (0.58), that is not confidence, that is a coin flip. Offering it alone would GBNF-lock the model onto a guess. So a lone weak hit gets demoted and we fall through to the full roster (or, configurable, to that tool's domain cluster):

fn demote_lone_weak_embed(
    embed: Vec<(String, f32)>,
    single_hit_floor: f32,
) -> (Vec<(String, f32)>, Option<(String, f32)>) {
    if embed.len() == 1 && embed[0].1 < single_hit_floor {
        tracing::info!(
            tool = %embed[0].0, score = embed[0].1, floor = single_hit_floor,
            event = "routing.policy.single_hit_demoted",
            "Lone weak semantic hit demoted (avoid GBNF lock-in)"
        );
        return (Vec::new(), Some((embed[0].0.clone(), embed[0].1)));
    }
    (embed, None)
}

Widen a near-tie into a cluster, but only for related domains. When the top hits are within match_margin (0.05) of each other, that is the model being torn between neighbours. "remind me tomorrow at 10" scores clock:alarm and agenda:remind_at almost identically, because honestly they are almost the same intent. In that case I do not want to pick one and lock the grammar to it. I widen to the union of the related domain clusters and let the model choose inside the grammar:

// clock + agenda + calendar share the "time" affinity bucket.
pub fn affinity_group(domain: &str) -> Option<&'static str> {
    match domain {
        "agenda" | "clock" | "calendar" => Some("time"),
        "web" | "news" | "wiki" => Some("web"),
        "doc" | "vault" | "memory" | "media" => Some("knowledge"),
        // ...
        _ => None,
    }
}

The "only related domains" bit matters. A near-tie across moltbook + web + db + doc is not a coherent cluster, that is just mush where everything landed around 0.55. Dumping all four families' tools into the grammar would be worse than useless. So unrelated near-ties stay a plain cosine-ranked subset, no cluster dump. This one rule killed a whole class of "why did it offer the train schedule tool for a memory question" bugs.

Keep the ordering honest. Whatever comes out, forced or cluster-widened or plain, gets re-ranked by the original cosine at the end, so the tool the embedder actually liked stays near the top of the phrase map the model reads. Cluster siblings that never got a real score inherit the best score from their domain, minus a hair, so they sort right after the seed instead of jumping the queue.

There is also a small dialog memory: the last handful of successful tools is kept (capped, session-scoped) so "delete that email" after a mail:check can pair to mail:delete instead of being read as a bare "delete" with no object. Agenda wins over mail and calendar in those pairings, because that is the order that matched how I actually talk to it.

The slim tool offer

Okay, so the policy layer produced an offered list. Now, what does the model actually see?

The naive answer is: all the tools, full JSON Schema for every one, dumped into the system prompt. For ~50 tools that is a wall of schema that eats context and, worse, gives a small model 50 ways to be wrong. My earlier GBNF post already argued that fewer choices means fewer mistakes, and this is the prompt-side half of that.

So by default (slim_tool_prompt = true) the tool-mode prompt is not the schema wall. It is two smaller things:

A phrase map, a little markdown table of tool name, short description, and the same routing phrases from earlier, so the model has natural-language hooks for each offered tool.
The tool definitions with parameters stripped. The model sees that vault:write exists and roughly what it is for, but not the full argument schema. The full schema is not needed to decide to call a tool, and if the model later gets the args shape wrong, gatekeeper schema recovery supplies the real schema on demand.

The phrase map is generated at runtime from the registered tools plus their descriptors:

"| Tool | Description (short) | Typical phrasing / triggers |".to_string(),
"| ---- | ------------------- | --------------------------- |".to_string(),
// one row per offered tool, phrases = descriptor hints, else fallback, else description

Before that list is finalized there are a few offer overlays, the pragmatic pairings that experience forced on me:

// web:fetch or web:search in the offer? then web:find has to come too,
// because you always want to query what you just fetched.
let needs_web_find = offered.iter().any(|n| n == "web:fetch" || n == "web:search");
// doc:read in the offer usually means the model will want to write a summary,
// so pair vault:write if the state allows it.
if offered.iter().any(|n| n == "doc:read")
    && !offered.iter().any(|n| n == "vault:write")
    && Gatekeeper::state_allows_tool(state, "vault:write")
{
    offered.push("vault:write".to_string());
}

There is also a cap (tool_map_offer_cap) if you want to hard-limit how many router hits make it into the map, and a "moltbook latch" so that once a turn is clearly about moltbook, the whole moltbook family comes along instead of one lonely endpoint.

One list, two consumers, no drift

Here is the bit that ties the whole thing together and is the reason I bothered writing overlays.rs as its own module instead of inlining it twice.

The offered-tool list feeds two consumers on every tool turn: the slim prompt assembly, and the GBNF grammar. If those two ever disagree, you get the worst kind of bug, the prompt tells the model tool X is available, the grammar forbids tool X's tokens, and the model wedges. So both call the exact same function:

pub(crate) fn slim_offered_tool_names(
    pre_llm_matched_tools: &[String],
    tool_map_offer_cap: usize,
    moltbook_overlay_latched: bool,
    gatekeeper: &Gatekeeper,
    state: &AgentState,
) -> Vec<String> {
    crate::orchestrator::routing::apply_offer_overlays(
        pre_llm_matched_tools, tool_map_offer_cap,
        moltbook_overlay_latched, gatekeeper, state,
    )
}

Single source of truth. The prompt and the grammar are computed from the same offered, in the same order, with the same overlays. They physically cannot drift, because there is only one list.

Why the grammar is the whole reason this is careful

If you only remember one thing: the reason routing is this defensive is that on llama.cpp the offer is not a suggestion, it is a hard constraint at sample time.

The offered list gets compiled into a GBNF grammar. GBNF constrains which tokens the sampler is even allowed to emit, before sampling, so anything that cannot extend into a valid parse gets masked out. The model does not "try to" produce a tool call in the right shape, it cannot produce anything else. Full detail is in the GBNF post, but the shape is: a fixed protocol envelope (thought, status, message, tool_calls), and a tool-call rule whose allowed names are exactly this turn's offered tools, with each tool's args compiled from its JSON Schema.

Which is exactly why I do not let the router hand over a single confident guess. Under a hard grammar:

Offer one wrong tool, the model is forced to call the wrong tool. No recovery inside the turn.
Offer an empty tool list when the router says "this is just chat", and tool_calls can only be []. The model cannot hallucinate a call. That is a feature, that is the conversational path.
Offer a cluster on a near-tie, and the model gets to pick the right neighbour, but only from a small, related, structurally valid set.

So the grammar is what turns "structured output" from a prompt-time please into a real guarantee. The policy layer only makes sure that guarantee points at roughly the right tools. In the end there are four layers stacked on each other. Embeddings find the candidates. The policy decides how wide to open the door. The slim map tells the model what is behind it. The grammar makes sure whatever comes back actually parses. None of these layers I planned upfront. Each one is there because the one before it burned me first.

What I would tell someone building this

If you route tools for a local model with grammar-constrained output: the thing I got wrong the longest was to treat routing as a ranking problem. It is not really about ranking. The real question is how much you trust that ranking before you let a grammar act on it.

Embeddings beat keyword matching for finding candidates. Keep your old keyword lists though, they are perfectly good embedding seeds and decent hard-override floors.
Top-1 plus a hard grammar is a trap. Confidence has to be earned with a score floor and a margin check, and when it is not there, widen instead of guessing.
Widen along real relationships (time-ish tools together, knowledge tools together), not along whatever happened to score near 0.55.
Compute the offer once and feed both the prompt and the grammar from it. Two derivations of "what is callable" will drift, and the failure mode is nasty.

None of this made the model smarter. It just stopped me from handing a small model a loaded grammar pointed at the wrong tool. Which, on a 14B, is most of the battle.

Code is at github.com/janpauldahlke/eris, Apache 2.0. The full post lives on my site at eris-system.dev.

Routing lives under src/orchestrator/routing/, the router itself in src/orchestrator/tool_router.rs. Poke at it, break it, tell me where it is dumb ;-) !!

Jan

I built a forgetting curve for an agent with one user

jan paul — Tue, 25 Aug 2026 10:54:40 +0000

What I learned, mostly: I overbuilt the clever part, and then kept it anyway.

In Greek myth the dead drink from the Lethe before they cross into the underworld. River of forgetting. The water washes the old life out so the soul can move on, and the part I always liked is that forgetting there is not a flaw in the design. It is the point. Without it nothing ever gets lighter.

I liked that picture a bit too much when I built memory for Eris.

There is a second thing I never believed in: keeping the whole chat as "memory". Dumping every single turn into long term storage, that is not remembering, that is hoarding. Somebody has to sit down and decide, this one stays, this one can go. Human or agent, does not matter, same job. A vault should stay a place you curated and not a landfill of old transcripts.

So I went and modelled human memory on purpose. Not as a cute comment somewhere in the code. As actual behaviour. Things you touch again should rise, things nobody touches should slowly fade. Decay was my Lethe, a slow drain on a float, so the agent would not drown in every little staging decision it ever made. A mention in the chat pushed the value back up a bit. For a personal agent that sits on a Markdown vault this felt serious. Grown up, even.

And then the serious thing turned into a small economics model.

Every staged memory carries a promotion_score: f64. There is a background pass, it wakes up every two minutes, takes a fixed bite out of the score (that is the decay), and then maybe it pushes the entry one rung up the ladder, Session to Scratch to Promote, or one rung back down when the score got too low. Staging something explicitly with memory:stage gives a bigger bump. Get over the threshold, longer TTL. Fall under half of the threshold that got you into your current tier, and back down you go. My own little forgetting curve. I was pretty proud of it, honestly.

It compiles. Tests green. And after running this on my own vault since months, here is the summary in the short version:

I never once looked at a Scratch tier score and made any decision from it.

Which sounds like I forgot why the numbers were there in the first place. I did not forget. The idea was never that I watch a dashboard. The idea was a memory curation mode for later: the agent, when idle or in a dedicated pass, walks its own staged memory, looks at score and tier, then decides what to keep, what to merge, what to demote, what to finally commit. The hippocampus theatre, that was the metaphor. The real reader of those floats was supposed to be the model itself, some gardener style loop that, well, does not exist yet.

So right now the river runs for an observer I never got around to building. No curation mode. No walk. Thresholds are still pure gut feeling. One user on the chat path, and that user is me. So the scaffolding for a future curator already sits on top of the live agent and slows it down. That is the mess.

This post is about why I should probably delete all of that, and about one simple inversion: decide less when you write, rank more when you read. Spoiler, so nobody feels cheated at the end: I did not delete it. I keep the ladder for now. More on that later. Forgetting stays good either way. I just put it in the wrong corner. If you build agent memory these days, maybe check quickly whether you are digging yourself the same hole.

What the ladder looks like

Eris is a local agent living on a Markdown vault. Working memory is a moka cache. The long term stuff, the notes a human should actually read, that is plain Markdown on disk. Semantic recall goes over a vector store. All fine so far. The clever bit was this one here:

pub enum EphemeralTier {
    Session,  // just staged, short TTL
    Scratch,  // "getting interesting"
    Promote,  // eligible for memory:commit_all
}

And the ladder itself:

impl EphemeralTier {
    pub fn next(self) -> Option<Self> {
        match self {
            Self::Session => Some(Self::Scratch),
            Self::Scratch => Some(Self::Promote),
            Self::Promote => None,
        }
    }

    pub fn prev(self) -> Option<Self> {
        match self {
            Self::Session => None,
            Self::Scratch => Some(Self::Session),
            Self::Promote => Some(Self::Scratch),
        }
    }
}

commit_all flushes only the Promote entries. Session and Scratch count as "not ready yet".

And this is exactly where it got annoying for the agent. The flow was clunky to begin with, first you memory:stage, then later memory:commit or memory:commit_all. Small local models, they are not great at this multi step bookkeeping. So the model stages something genuinely useful, then tries to flush everything with commit_all, and every entry that did not reach Promote yet just gets skipped, quietly. From the model's side the tool call looked like a success. Nothing landed in the vault though. So it tries again, stages again, lists the staged rows, gets confused, burns turns. I saw this happen so many times. The ladder should have kept junk out of the vault. In practice it mostly kept the agent from getting the job done.

Scoring on each pass, roughly:

// every promotion tick
updated.promotion_score =
    (updated.promotion_score - config.promotion_decay_per_tick).max(0.0);

if updated.promotion_score >= threshold && !updated.needs_review {
    if let Some(next_tier) = updated.tier.next() {
        updated.tier = next_tier;
        updated.expires_at = now + config.ttl_for_tier(next_tier);
    }
}

// demotion: score fell below 0.5 x the threshold that got us here
if updated.promotion_score < threshold_to_current * 0.5 {
    updated.tier = prev_tier;
}

And then the config kept growing. Three TTLs. Two thresholds. Decay per tick, an eval interval, a mention boost, a stage boost. Six knobs, more actually, for a subsystem I was never even looking at. Should have been the smell already, right there.

There is one detail that is almost a bit funny. The stage boost is 3.5, the threshold from Session to Scratch is 3.0. Which means one single memory:stage already gets you over the first rung on the next pass. Scratch happened more or less by itself. I had built a step that nobody ever really stands on.

How scores go up, and why it breaks

Decay was the Lethe half. The remembering half was, in the end, two bumps:

Explicit memory:stage adds promotion_stage_boost.
After a user turn a mention hook walks over the staged rows. When the user text "covers" the row's canonical_key, it adds promotion_mention_boost, bumps mention_count, refreshes the TTL.

Now the second one sounds semantic. It really is not. No nomic-embed in there, no cosine, nothing. The code takes the user message, cuts it into words of length three and up, and checks how many overlap with the tokens from the key. hagbard_loves_piano becomes hagbard, loves, piano. Two token hits, or one that is long enough on its own, and the score climbs. The whole thing fits on one screen:

/// True if user message words match this row strongly enough (deterministic, no embeddings).
fn user_covers_canonical_key(
    user_words: &HashSet<String>,
    canonical_key: &str,
) -> bool {
    let tokens: Vec<&str> = canonical_key.split('_').filter(|t| t.len() >= 3).collect();
    if tokens.is_empty() {
        return false;
    }
    let mut hits = 0usize;
    let mut longest_hit = 0usize;
    for t in &tokens {
        if user_words.contains(*t) {
            hits += 1;
            longest_hit = longest_hit.max(t.len());
        }
    }
    if hits >= 2 {
        return true;
    }
    hits >= 1 && longest_hit >= 5
}

So, as long as you keep repeating the words that happen to live in the title slug, the row goes up. In that one narrow case it works. It breaks the second real language shows up:

Synonyms. You staged coffee_preference, and the user says "espresso order" or "my usual drink". Overlap zero. No boost. The thing you are so obviously still talking about decays like you dropped it completely.
Pointing. Real chat is full of this, "keep that", "commit it", "same as before", "write this into the vault". None of these words ever live in a canonical_key, so the hook basically cannot fire. Exactly in the moments where you point at a staged row most clearly, you miss the bump. Brittle is the friendly word for it.
Paraphrase. Same topic, other words, score stays flat, the pass keeps eating. The tier can demote while the conversation is literally still about that note.

So this "forgetting curve" did not measure importance at all. It measured string overlap against a slug that I myself invented at stage time. Repeat the slug, score up. Talk like a normal person, with pointers and synonyms, and the bumps never arrive. And here is the annoying part: I already had nomic-embed in the stack, for vault prefetch and for tool routing. I left it out of promotion on purpose, for determinism. That call by itself was fine. What was not fine is that I then used this brittle string thing as if it were a hippocampus.

The curation mode was supposed to read these scores later on. I am honestly glad it never shipped on top of them.

The one concurrency exception

I have a hard rule in this codebase. No shared mutable state across threads. Orchestrator and UI talk over mpsc. No Arc<Mutex<T>> just because it happens to be convenient.

Except for one single AtomicBool.

Snapshot daemon and orchestrator share promotion_suppressed_during_step. While step() is running, the LLM round trip, the tool batch, all the slow parts, the daemon is not allowed to decay or to promote. Otherwise you get these lovely races, "the model is about to commit this row" versus "decay just demoted it". So the orchestrator flips a flag, the daemon reads it and skips its tick.

/// When true, skip evaluate_promotions_and_decay so decay/tier moves
/// do not race a slow LLM or tool batch.
promotion_suppressed_during_step: Arc<AtomicBool>,

I made an exception to my own concurrency rule, so that a forgetting curve I do not even look at can interleave "safely" with the chat. When you arrive at a point like that, the design is usually already wrong somewhere, and throwing more locking at it will not save you.

What it was actually for

Honestly? Two jobs got glued into one.

Short term the ladder was a write time gate. Do not let junk flow into the vault by itself, only the "important" things should turn into Markdown, and important means the score, built up over time.

Long term those same scores and tiers were meant as the input for the curation mode from above. Agent walks the memory, reads the ladder state, chooses. I built the sensors first and the walker never came. Either way it sounds scientific. Mostly it is a prior with extra code around it, plus a futures contract I never once cashed in.

And without the ladder I already had:

explicit memory:commit and memory:commit_all, the model or me decides
a needs_review flag that stops auto promotion, which for now I flip by hand, there is no detector yet
turn end mention handling, a key shows up again, interest bumps

That was already enough to stop the vault from becoming a dump. The score ladder was a third mechanism on a door that had two locks on it already. Plus it pretended to know the future.

And that is the part that bugs me now. At write time you simply do not know if a thing will matter in three weeks. You guess. Embeddings are far better at a different question: given what the user just said right now, what from the past is relevant? That is a read time question. And that one you can actually compute.

So in function I need two states, not three tiers:

State	Meaning
Staged	working memory, has a TTL, listable, can be committed
Committed	Markdown in the vault, indexed for semantic recall

Scratch was not needed. The Promote only gate on commit_all was worse than not needed, because it stalled the agent.

What retrieval was already doing anyway

While the daemon was busy decaying its floats, the turn start prefetch was already ranking the vault memory by similarity and dropping a small block into the system prompt:

[RELEVANT_LEARNED_MEMORY]
...snippets the embedder thinks match this turn...
[/RELEVANT_LEARNED_MEMORY]

No tiers. No ladder scores. Query now, rank now, stay inside a character budget, done. Junk that never matches any query just sits there and hurts nobody. I should have trusted this a lot earlier.

So if I ever pull the trigger, the delete list is fairly obvious. Call it one big TODO:

Scratch tier, next, prev, index, the whole ladder
promotion_score, the decay pass, the demotion logic
most of the config knobs, keep one staged_ttl_secs and be done
the AtomicBool, it only exists because of the ladder anyway
the Promote only gate on commit_all, so it would commit all staged rows minus the contradicted ones

What would stay: the tools the model already knows (stage, commit, commit_all, staged_list, query), the needs_review flag, the snapshot daemon but only as persistence, and the mention handling, probably just refreshing the TTL instead of feeding a fake score.

A couple hundred lines gone. One concurrency exception gone. Same capabilities. And a lot less to explain to myself the next time I open that file. On paper it is a clean win. Keep that "on paper" in mind, we come back to it.

The deeper inversion, and Markdown stays

Once you stop over curating at write time, another design gets kind of obvious.

You can append cheap episodic records, turn digests, tool outcomes, mentions, without first deciding whether they are "Promote material" or not. At this scale storage is basically free. You filter at retrieval instead, similarity times recency, maybe some BM25 on top. The curated layer stays what it is supposed to be, human readable Markdown in the vault. The index is a derived thing. Delete the index, rebuild it from the vault on boot. That property is not up for discussion, not for me. A binary agent memory file, fine, let it hold the index. It must never become the source of truth.

Small note to myself: do not go and tune the ladder instead of deleting it. Making machinery that nobody looks at more precise, that is just polishing the overengineering.

Zettelkasten, a graph I prepared and then never really walked

There is a second ghost in this whole design, Niklas Luhmann's Zettelkasten.

Short version for the people who do not live in that world. Luhmann, a German sociologist, built this massive slip box of notes. Every note small, atomic, linked by hand to other notes. And the power was never "save everything". It was a growing graph of ideas that you could walk through. A new note had to find its neighbours first. The thinking happened inside the links, as much as inside the cards. People in the PKM and Obsidian corner still chase exactly this shape, and for good reasons.

I wanted the vault to become that, for the agent. Stable node_ids, revisioned Markdown commits, folders like Topology, Discourse, Synthesis, contradiction as a thing you mark instead of silently overwriting it. The layout is more or less ready to turn into a graph.

We are just not there yet.

What I actually run day to day is flatter than all this: stage, sometimes commit, semantic search over the chunks, a handful of snippets prefetched into the prompt. The graph dream sat right next to the forgetting curve dream. Both of them were "human memory, but engineered". Both made the agent path heavier, before the simple loop was even boringly reliable.

Why this is an Eris story, and an overambition story

Eris is my local first Rust agent. Vault as memory, tools behind a gatekeeper, llama.cpp with GBNF so the JSON protocol is already constrained at sample time, a TUI and a web frontend hanging off the same orchestrator.

I put a lot of hours into making small local models behave like they had a real protocol. Grammars, recovery budgets, an LLM only view of the transcript for the llama-server templates. And memory was the exact spot where I quietly got soft again. A float, a threshold, a story about importance that sounded like neuroscience, and on top a Zettelkasten shaped vault for a graph I never really walked. The agent did not get smarter from any of it. The code got heavier. The AtomicBool was the receipt for that.

If you build agent memory and you catch yourself doing Session, Scratch, long term, with decay "like a human hippocampus", stop for a second and ask the boring question. Who reads the scores? If the answer is nobody, then you are not building a forgetting curve. You are building config debt and telling yourself a nice myth about it.

Here comes the uncomfortable bit, plainly. I overengineered. I wanted too much at once, Lethe style decay, Promote gates, a future agent that curates by score, Luhmann links, the whole cathedral. And I did not even seriously try the things other people already shipped, or at least sketched out, in this space. Memvid. Mempalace. Karpathy's wiki style memory. Probably three more approaches I bookmarked and then ignored, while I polished my own knobs. For somebody who keeps claiming he cares about deletion, that is a weak look.

So the sane next move would be simple. Delete the ladder. Keep staged versus committed. Keep choosing what to remember instead of dumping the whole chat into a file. Keep Markdown as the source of truth. If the agent later walks its own memory it can lean on retrieval, contradictions, plain note text. It does not need my homemade hippocampus floats for that.

Would. And here is the honest part I owe you as a solo dev: I am not doing it. Not this week anyway. There is now a fat // TODO: kill the ladder sitting in the repo, and I fully expect to walk past it every day and leave it alone. Eris is a hobby. I build it for the pure joy of building it, and the ladder is somehow my little darling even while I write a whole post about why it is wrong. So it stays, on probation.

And honestly the ladder is not even the loudest voice in my head. The real solo dev problem is that I cannot decide where to go next, because everything sounds more fun than removing a f64. Put the chat sessions into SQLite and let the agent query its own history with real SQL. A proper sandbox so it can write and run code without me hovering over the process. Actual builds I can hand to other people, instead of "clone the repo and pray". A web based LlmEngine backend, OpenRouter and the like, so Eris is not chained to my local llama.cpp forever. Every single one of these is more exciting than a cleanup, and that is exactly why the ladder is still alive.

Forgetting stays good. I still like the Lethe image a lot. I only stop pretending that a f64 on a daemon tick ever was that river. The scores can keep ticking in the background for now, I just do not tell myself a neuroscience story about them anymore. TTL expiry, an explicit commit, ranking at read time, that already forgets plenty on its own, without a fake observer watching over it.

Two states would be enough. Explicit commit. Rank when you read. Notes in plain text. Ambition very much not on a leash, if we are being honest.

One day I will delete the ladder. Nobody would miss the scores, nobody reads them. But not today. Today there is a SQLite branch calling, and that is simply more fun.

Repo: github.com/janpauldahlke/eris · site: eris-system.dev

How I use GBNF grammars to make small local models stop hallucinating JSON

jan paul — Wed, 29 Jul 2026 09:22:43 +0000

Or: why your 8B model keeps breaking your agent's tool protocol, and what you can do about it at the token level.

Originally published on eris-system.dev/blog/gbnf-grammars

I've been building Eris, a local-first AI agent written in Rust. It sits on your machine, reads and writes an Obsidian-compatible Markdown vault, and calls tools (memory, reminders, web fetch, email, calendar) through a structured JSON protocol. The LLM backend is llama.cpp, the models are 8-26B GGUFs, and nothing leaves your machine unless you explicitly opt in.

The single hardest engineering problem in the whole project wasn't the orchestrator, the TUI, or the semantic memory layer. It was this:

Getting a small local model to reliably emit valid, schema-conformant JSON.

This post is about how I solved it with GBNF grammars (compiled per-session and narrowed per-turn) and why I think this approach generalises to anyone building tool-calling agents on local models.

The problem: small models are sloppy with JSON

If you've tried to build an agent on a local 8B or even 12B model, you know the drill. You tell the model to respond in JSON. It does, mostly. Then on turn 47 it:

Forgets a closing brace
Adds a trailing comma in an array
Invents a tool name that doesn't exist (vault_read instead of vault:read)
Wraps its JSON in a markdown code fence
Emits valid JSON followed by a paragraph of conversational text
Returns "args": "path=notes/today.md" instead of "args": {"path": "notes/today.md"}

Ollama's format: json helps a bit. It constrains output to some valid JSON, but it says nothing about which JSON. Your tool protocol schema isn't enforced. The model can return {"sure": "here is your file"} and that's perfectly valid JSON. It just isn't a valid tool call.

OpenAI and Anthropic solve this with server-side function-calling APIs. But we're running locally. We don't have that luxury. We have something better.

GBNF: constraining generation at the token level

llama.cpp supports GBNF grammars, a BNF-like format that constrains which tokens the sampler is allowed to emit. If a token would produce a string that can't be extended into a valid parse of the grammar, it gets masked out before sampling. The model literally cannot generate malformed output. Not 'usually doesn't'. Cannot.

This is a fundamentally different thing from prompt-engineering the model to 'please respond in JSON'. It's a hard constraint on the sampler, enforced at every single token position.

Eris compiles a GBNF grammar at session start that defines the exact shape of its protocol envelope. Here's the skeleton from src/engine/grammar/envelope.rs:

const STATIC_GRAMMAR: &str = r#"root ::= "{" ws thought-kv "," ws status-kv "," ws message-kv "," ws toolcalls-kv ws "}"

ws ::= [ \t\n]*

thought-kv ::= "\"thought\"" ws ":" ws json-string
status-kv  ::= "\"status\"" ws ":" ws status-enum
message-kv ::= "\"message_to_user\"" ws ":" ws (json-string | "null")
toolcalls-kv ::= "\"tool_calls\"" ws ":" ws "[" ws tool-call-list ws "]"

status-enum ::= "\"Task\"" | "\"Reflect\"" | "\"Idle\"" | "\"Process\""
"#;

Every LLM response must be a JSON object with exactly four keys: thought (chain-of-thought, always present), status (one of four enum values), message_to_user (string or null), and tool_calls (array of name+args objects). The model can't deviate. It can't add a fifth key. It can't misspell tool_calls. It can't emit a status value that doesn't exist in the enum.

The tool call shape itself gets appended depending on whether we're in static mode (any JSON object for args) or dynamic mode (per-tool typed args):

/// Per-tool dynamic-args version: tool-call dispatches into `tool-with-args`.
const TOOL_CALL_RULES_DYNAMIC: &str = r#"tool-call-list ::= "" | tool-call ("," ws tool-call)*
tool-call ::= "{" ws "\"name\"" ws ":" ws tool-with-args ws "}"
"#;

Notice tool-with-args here. That's where it gets interesting.

Compiling JSON Schema to GBNF rules

Tool arguments are where small models really struggle. A vault:read call needs {"relative_path": "some/file.md"}. A web:fetch call needs {"url": "https://..."}. Without constraints, the model will happily call vault:read with {"file": "notes.md"} (wrong key name, schema validation fails, recovery turn wasted).

Eris goes further: it compiles each tool's JSON Schema into GBNF rules at startup. The schemas come from Rust structs via schemars, so the types are the source of truth. Here's the compiler entry point in src/engine/grammar/schema_to_gbnf.rs:

pub fn schema_to_gbnf_rule(
    tool_name: &str,
    schema: &RootSchema,
) -> Option<(String, Vec<GbnfRule>)> {
    let rule_name = tool_name_to_rule_name(tool_name);
    let mut ctx = CompileCtx {
        definitions: &schema.definitions,
        extra_rules: Vec::new(),
        depth: 0,
    };

    let body = compile_schema_object(&schema.schema, &rule_name, &mut ctx)?;
    let mut rules = vec![(rule_name.clone(), body)];
    rules.extend(ctx.extra_rules);
    Some((rule_name, rules))
}

A GbnfRule is just a (String, String) tuple: rule name and rule body. The compiler walks the JSON Schema tree and emits GBNF productions for each type it finds. Objects become key-value sequences with required fields unconditional and optional fields wrapped in (...)?. Strings with enums become GBNF alternations. Arrays get a helper list rule. $ref pointers get resolved against the definitions map.

The object compiler is the meatiest part. It sorts properties (required first, then optional, alphabetically within each group) and emits them with proper comma handling:

fn compile_object(
    validation: Option<&ObjectValidation>,
    parent_rule_name: &str,
    ctx: &mut CompileCtx<'_>,
) -> Option<String> {
    // ... validation checks, depth guard (MAX_DEPTH = 2) ...

    let required_set: std::collections::HashSet<&str> = validation
        .required.iter().map(|s| s.as_str()).collect();

    // Sort: required first (alphabetically), then optional (alphabetically).
    let all_fields: Vec<&FieldInfo> = required_fields
        .iter().chain(optional_fields.iter()).copied().collect();

    let mut parts = Vec::new();
    for (i, field) in all_fields.iter().enumerate() {
        let kv = format!(
            "\"\\\"{}\\\"\" ws \":\" ws {}",
            field.key, field.expr
        );
        if i == 0 {
            if field.required {
                parts.push(kv);
            } else {
                parts.push(format!("({kv} )?"));
            }
        } else if field.required {
            parts.push(format!("ws \",\" ws {kv}"));
        } else {
            parts.push(format!("(ws \",\" ws {kv} )?"));
        }
    }

    let body = format!("\"{{\" ws {} ws \"}}\"", parts.join(" "));
    Some(body)
}

So for a Rust struct like this:

#[derive(JsonSchema, Deserialize)]
struct VaultWriteArgs {
    relative_path: String,
    content: String,
    mode: WriteMode,  // enum: "overwrite" | "append"
}

The compiler produces GBNF rules roughly like:

vault-write-args ::= "{" ws "\"content\"" ws ":" ws json-string
                      ws "," ws "\"mode\"" ws ":" ws ("\"overwrite\"" | "\"append\"")
                      ws "," ws "\"relative_path\"" ws ":" ws json-string ws "}"

If the model emits "vault:write" as the tool name, the sampler forces the args to conform to vault-write-args. The model can't use the wrong key. It can't pass a number where a string is required. It can't use a mode value that isn't overwrite or append. The schema is literally part of the grammar.

Arrays work too. For a tool with tags: Vec<String>, the compiler emits a helper list rule:

fn compile_array(
    validation: Option<&ArrayValidation>,
    parent_rule_name: &str,
    ctx: &mut CompileCtx<'_>,
) -> Option<String> {
    let items_expr = match validation.and_then(|v| v.items.as_ref()) {
        Some(SingleOrVec::Single(item_schema)) => match item_schema.as_ref() {
            Schema::Object(obj) => {
                let item_rule = format!("{parent_rule_name}-item");
                compile_schema_object(obj, &item_rule, ctx)
                    .unwrap_or("json-value".into())
            }
            Schema::Bool(true) => "json-value".into(),
            Schema::Bool(false) => return Some("\"[\" ws \"]\"".into()),
        },
        // ...
        None => "json-value".into(),
    };

    let list_rule_name = format!("{parent_rule_name}-list");
    let list_body = format!("{items_expr} (\",\" ws {items_expr})*");
    ctx.extra_rules.push((list_rule_name.clone(), list_body));

    Some(format!("\"[\" ws ({list_rule_name})? ws \"]\""))
}

Which produces something like:

test-array-args-tags-list ::= json-string ("," ws json-string)*

The depth guard (MAX_DEPTH = 2) prevents the compiler from chasing recursive schemas forever. If a schema is too complex (deeply nested oneOf, recursive types, free-form additionalProperties with no fixed keys), the compiler returns None and the tool falls back to a generic json-object rule. Graceful degradation, not a crash.

Per-turn grammar narrowing: the real trick

Here's the thing that really moves the needle for small models: Eris doesn't give the model the full 50-tool grammar on every turn.

Before each LLM call, the orchestrator runs a ToolRouter, a semantic similarity search using the same embedding model as vector memory. The user's message gets embedded and compared against precomputed vectors for each tool (built from tool names, descriptions, and routing hints). Only tools above a cosine similarity threshold make the cut.

The GBNF grammar is then recompiled to include only those tools. If the user says 'what time is it', the grammar's tool-with-args alternation might contain only clock:now. The model can't call vault:write even if it wanted to. Those tokens simply aren't available to the sampler.

The subset cache lives in src/orchestrator/core/llama_gbnf_subset.rs:

pub(crate) fn get_or_compile_subset(
    &self,
    gatekeeper: &Gatekeeper,
    tool_names: &[String],
) -> Result<Arc<str>> {
    let mut sorted: Vec<String> = tool_names.to_vec();
    sorted.sort();
    let key: String = if sorted.is_empty() {
        CACHE_KEY_NO_TOOLS.to_string()
    } else {
        sorted.join("\x1e")  // ASCII record separator as cache key
    };

    let mut guard = self.inner.lock().map_err(|_| {
        FcpError::EngineFault("GBNF subset cache mutex poisoned".to_string())
    })?;

    if let Some(hit) = guard.get(&key) {
        return Ok(Arc::clone(hit));
    }

    // Cache miss: compile grammar for exactly these tools
    let entries: Vec<ToolGrammarEntry> = sorted
        .iter()
        .map(|name| {
            let per_tool_rules = gatekeeper
                .parameters_root_schema_for(name)
                .and_then(|schema| schema_to_gbnf_rule(name, &schema))
                .map(|(_rule_name, rules)| rules);

            ToolGrammarEntry {
                name: name.clone(),
                per_tool_rules,
            }
        })
        .collect();

    let compiled = compile_fcp_envelope_grammar_dynamic(&entries);
    let arc: Arc<str> = Arc::from(compiled.into_boxed_str());
    guard.insert(key, arc.clone());
    Ok(arc)
}

Same tool set on a different turn? Cache hit, no recompilation. The Arc<str> keeps the compiled grammar around cheaply.

And here's how the orchestrator picks which grammar to use on each LLM call, from the main cognitive loop in src/orchestrator/core/step.rs:

let (grammar_override, attach_session_grammar) = if !self.config.is_llamacpp() {
    (None, true)  // Ollama: no grammar support, skip
} else if !tools_needed {
    // No tools matched this turn: compile empty grammar (forces tool_calls: [])
    let g = self.gbnf_subset_cache
        .get_or_compile_subset(&self.gatekeeper, &[])?;
    (Some(g), true)
} else if !targeted_tools.is_empty() {
    // Recovery/retry: only the specific tools we're retrying
    let names: Vec<String> = targeted_tools.iter().cloned().collect();
    let g = self.gbnf_subset_cache
        .get_or_compile_subset(&self.gatekeeper, &names)?;
    (Some(g), true)
} else if slim_assembly {
    // Normal path: pre-LLM router's top-K matched tools
    let offered = slim_offered_tool_names(
        &pre_llm_matched_tools,
        self.tool_map_offer_cap,
        moltbook_overlay_latched,
        &self.gatekeeper,
        &self.state,
    );
    if offered.is_empty() {
        (None, true)  // fall back to full session grammar
    } else {
        let g = self.gbnf_subset_cache
            .get_or_compile_subset(&self.gatekeeper, &offered)?;
        (Some(g), true)
    }
} else {
    (None, true)  // full tool roster, session grammar
};

Four branches, four different grammar scopes. The !tools_needed case is fun: when the router decides the user is just chatting (no tools relevant), the grammar still enforces the protocol envelope but compiles with an empty tool list. That means tool_calls can only be []. The model simply cannot hallucinate a tool call when the router says 'this is just conversation'.

This has two effects:

Reliability goes way up. Fewer choices means fewer mistakes. An 8B model choosing between 3 tools is dramatically more reliable than one choosing between 50.
Generation is faster. A smaller grammar means less work for the constrained sampler at each token position.

The gatekeeper: defence in depth

Grammar enforcement is necessary but not sufficient. Even with a perfect grammar, the model can still call a tool with semantically wrong arguments (a valid path that doesn't exist, or an action that's not appropriate in the current agent state).

Eris layers a Gatekeeper on top (src/tools/gatekeeper.rs). Every tool call passes through it before execution:

pub async fn execute_tool(
    &self,
    state: &AgentState,
    name: &str,
    args: Value,
) -> Result<String> {
    // 1. State-based authorization
    if !Self::state_allows_tool(state, name) {
        return Err(FcpError::ToolFault {
            tool_name: name.to_string(),
            reason: format!("Tool not authorized in state {:?}", state),
        });
    }

    // 2. Normalize common LLM arg-naming mistakes before validation
    let args = normalize_tool_args(name, args);

    // 3. Runtime JSON Schema validation (belt and suspenders with grammar)
    let compiled_schema = JSONSchema::options()
        .compile(&schema_value)
        .map_err(|e| FcpError::Config(...))?;
    if let Err(errors) = compiled_schema.validate(&args) {
        return Err(FcpError::SchemaViolation(...));
    }

    // 4. Execute
    let result = tool.execute(args).await?;

    // 5. Semantic guard: empty results in Recover state are suspicious
    if state == &AgentState::Recover && result.trim().is_empty() {
        return Err(FcpError::ToolFault {
            tool_name: name.to_string(),
            reason: "Semantic Guard: Tool returned zero logic results during recovery"
                .to_string(),
        });
    }
    Ok(result)
}

The arg normalization is worth a mention. Small models frequently use slightly wrong key names: q instead of query, path instead of relative_path, top_n instead of max_headlines. Rather than wasting a recovery turn, the gatekeeper silently maps common aliases before validation:

fn normalize_tool_args(tool_name: &str, mut args: Value) -> Value {
    let Some(obj) = args.as_object_mut() else { return args };

    if tool_name == "web:search" {
        for alias in ["q", "search_query", "search"] {
            if let Some(val) = obj.remove(alias) {
                if !obj.contains_key("query") {
                    obj.insert("query".to_string(), val);
                }
            }
        }
        obj.retain(|k, _| WEB_SEARCH_KEYS.contains(&k.as_str()));
    }
    // ... similar blocks for web:fetch, web:find, news:today,
    //     vision:see, media:catalog (path -> relative_path) ...
    args
}

This is a pragmatic concession. The grammar enforces structure. The normalizer handles naming drift. The schema validator catches whatever slips through both.

What this looks like in practice

On the Ollama backend (no grammar), a long session with a 12B model hits about 3-5% protocol failures per turn. Malformed JSON, invented tool names, trailing prose after the JSON object. Each failure costs a recovery turn (re-prompt with the parse error), which burns context window and adds latency.

On the llama.cpp backend with GBNF, protocol failures drop dramatically. Short to medium sessions are essentially clean. But I won't claim zero failures: on long contexts where the model starts losing coherence, even grammar-constrained output can degrade. The grammar guarantees syntactic validity (the JSON will parse, the keys will be right, the enum values will exist), but it can't fix a model that fills every string field with garbage or repeats the same tool call in a loop because it's lost the plot at 28k tokens.

That's why Eris doesn't rely on the grammar alone. There's a bounded recovery loop that catches schema violations, empty tool results, and semantic nonsense, feeds the model a concrete correction, and retries with a capped budget. The grammar killed the easy failures (malformed JSON, invented tool names). The recovery loop handles the rest. I'll write more about that mechanism in a follow-up post.

The difference in session stability is still massive. A 100-turn session on Ollama reliably degrades; the same session on llama.cpp with grammar stays usable far longer. Just don't let anyone tell you it's a silver bullet.

Bounded generation: the other half

Even with a perfect grammar, generation can go wrong if the model fills its context window mid-response. llama.cpp will truncate, and your valid-so-far JSON becomes invalid because the closing braces never got emitted.

Eris sets n_predict (max tokens to generate) per call, calibrated to leave room in the context window for the response to actually finish. The grammar ensures structure; bounded generation ensures the structure has room to close.

The architecture, briefly

For context on where the grammar fits in the bigger picture:

Single Rust binary. ~270 source files, ~50 tools, three presentation surfaces (ratatui TUI, localhost web UI via Axum+SSE, optional Discord sidecar), all sharing one orchestrator.
Markdown vault as memory. Notes in an Obsidian-compatible directory tree. Tiered memory: ephemeral staging (in-process), committed (written to vault), semantic recall (Qdrant vector search over vault chunks). Turn-start prefetch embeds the user message and injects relevant memories into the system prompt before the LLM even runs.
No unsafe, no panics, no println. Production code uses ? propagation through a typed error taxonomy (FcpError). Diagnostics go through tracing to rotating log files under .fcp/telemetry/. Never to the TUI buffer. println! would corrupt the ratatui render.
Actor model concurrency. No Arc<Mutex<T>> shared state between threads (well, except the grammar cache, which is a cold path). Communication via tokio::sync::mpsc channels.

Should you do this?

If you're building an agent on local models and you need structured output: yes, absolutely. GBNF grammar enforcement is the single biggest reliability win I've made on this project. More impactful than prompt engineering, more impactful than model selection, more impactful than retry logic.

The key insight is that the grammar is not a prompt. It's not a suggestion. It's a constraint on the sampler. The model can't violate it any more than it can output a token that doesn't exist in its vocabulary. That's a qualitative difference from anything you can do with system prompts.

The tradeoff: you're coupled to llama.cpp (or any sampler that supports grammars; VLLM has experimental support, Outlines does it differently). You need to write or compile the grammar. And very complex schemas (recursive types, polymorphic unions) may not compile cleanly. But for the typical agent tool protocol (an envelope with a discriminated union of tool calls) it works beautifully.

Try it

git clone https://github.com/janpauldahlke/eris
cd eris && cargo build --release
docker run -d -p 6333:6333 -p 6334:6334 \
  -v eris-qdrant-data:/qdrant/storage qdrant/qdrant
./target/release/eris chat   # first run launches the setup wizard

The code is at github.com/janpauldahlke/eris. The project site is eris-system.dev. Apache 2.0. Stars and contributions welcome.

If you're working on local AI agents, structured generation, or Rust+LLM integration, reach out or open an issue.*

thinking to myself when setting up a project boilerplate

jan paul — Tue, 04 May 2021 17:33:39 +0000

this is a thinking out loud to myself.

how to prepare a SPA in different frameworks/libraries but also in angular

ng new create-react-app vue cli
enable router
enable i18n strategy
enable store / reduxy stuff
bind router into store
ensure your API with smthng like or curls / postmany stugff

test GET / POST/ UPDATE
brief moment of think: do we need to auth? if so handle auth token with like setLocalStorage

create an entityfolder for selected endpoint, (if angular, also create a module here!!)
create model/type declartion (only if typescript, which we prefer) of entities
create service to consume any http-client (eg axios) // consider auth process

test GET / POST/ UPDATE loadAll(), oneOneById(), postNew(), update(), delete() for selected entity

test servicemethods in bare minimum but routable component, most liekly in lifecycle method of given framework
from now on we would need to distungiush from framework to framework/library which should be more articles,

in angular -which we atm work with- we go with:

create routeresolvers
prepare actions / reducer / selectors from ngrx, naivly bind http service into the loadaction (we like ngrx-entity here)
access store in template and trigger action for udf to ensure actions work
create effect for entity -> loadAllEntity -> http-service -> dispatch loadAllEntitySucess, bind result into store
we forget about, register router, effect, store, store-router in module, considering it silently given from now on, knowing forRoot() and forFeature().
remove 3 from template
consume loadEntity from action in routeresolver -> which should also will trigger the effect -> which will trigger the http -> which will trigger store binding
bind observable/subscriber to store in template list view
prepare route for single entity
access route, write resolver that gains single entity from store / write actions..... from here on
rinse repaet

quod erat demonstrandum.

looking for a good vpn provider

jan paul — Sat, 09 Jan 2021 16:14:05 +0000

I just asked this on the Typescript Discord Channel -- Offtopic, but tought why not enhance my question radius.

[17:08] Paulquapper: could somone suggest my a VPN provider with

1. a documented (javascripty) API and
2. the feature torenew ip, for given city/country. Ideally non US, no US-Server provider.
3. "cost efficient" (paying annualy)

Need help to improve my usage of gatsby-image

jan paul — Thu, 19 Nov 2020 10:26:11 +0000

Hello fellow Gatsby'ers. Today i am comming with a question to you. I need to improve my bundle size, and therefore i want to rewrite my gatsby-image wrapper to be more efficient.

Consider my bundle

BUT: the 150 images itself are only 14MB in total, which means gatsby bloats it like crazy.

I know my code smell in terms of BIG-O, but, my poor excuse, i didn't knew another way than mapping over the result. Can you guys help me to improve here, considering:

const ImageLarge = props => {
  return (
    <StaticQuery
      query={graphql`
        query {
          allFile(filter: { sourceInstanceName: { eq: "assets" } }) {
            edges {
              node {
                relativePath
                name
                childImageSharp {
                  fluid(maxWidth: 600, quality: 100) {
                    ...GatsbyImageSharpFluid
                  }
                }
              }
            }
          }
        }
      `}

and here now my pile of shame:

 const image = data.allFile.edges.find(n => {
          return n.node.relativePath.includes(`${path + props.src}`)
        })

        if (!image) return null

Btw, my problem are comming from not beeing able to parametrize my static gqlquery with arguments, so i took all images, mapped over and returned only the image that i needed. horrible inifficnient.

Why cant i do something like this?

const ImageLarge = props => {
  return (
    <StaticQuery
      query={graphql`
        query {
          allFile(filter: { sourceInstanceName: { eq: `assets/${name}.png` } }) {
            edges {
              node {
                relativePath
                name
                childImageSharp {
                  fluid(maxWidth: 600, quality: 100) {
                    ...GatsbyImageSharpFluid
                  }
                }
              }
            }
          }
        }
      `}

I asked google and found this, which describes my problem perfectly, but offers no solution

"Believe it or not you cannot create a fully dynamic image component using a gastby-image without risking a (possibly very large) bloat in your bundle size. The problem is that static queries in Gatsby do not support string interpolation in it's template literal."

what-is-the-best-way-to-display-all-pictures-of-a-directory-in-a-gatsby-project

If one you could help me to solve this problem, i'd be glad and i also think, that we should write about it here, so others can find this googeling.

May the code be with you 🙏

Javascript Homeoffice Stream

jan paul — Thu, 12 Nov 2020 09:46:46 +0000

Hey fellow nerds,

out of boredom in the homeOffice era, i started my very "unique" frontend stream. Feel free to join and chit chat.

https://www.twitch.tv/paulquappe :-)

How to deal with real imposters?

jan paul — Thu, 12 Nov 2020 09:43:26 +0000

Since we all read a lot about imposter syndrom, we tend to see an unrecognized genius in our collegues.

BUT? How to deal with an actual real imposter in your job?

I have been hired for a nice project, JAVA Backend + REACT (SSR) Frontend, Webassembly, webGL and Azure, just to bring you some buzzwords. The run is planed for like 5 years, so to say, it is today already planned, what they want to do with the product in the future.

After 10 years of beeing a developer, mostly Frontend, but also JSP/jinja/smartie and other templating stuff, i decided to take this position for the following reasons:

the stack looks familiar, but also there are some new things
the lifetime of the project (to comparsion, most agency project last 3 months in coding)
there will be a new team formed, since we take the project over and make it an internal company project
my desire to become a senior developer (not only moneywise)

And when they hired me, they said, there will be an IT Team soon in our company. So far, so good.

Then "the corana" happened and it was decided to hire only the Senior Architect, the HR found a promising candidate. No, they don't liked the indian guys, sadly. Still i bet he was good at JAVA but bad in german (since iam german).
They offered the job to someone else. It is a handsome, very charming, young and old, 10 years of experience as team-leader "senior software architect".

Just after stalking him on xing (our german linkedin) i realized, that his buzzwords never fit to each other. he has 4 star experience in C++/C#!!

??
!!

WTF, in either one, or in both? i mean for me, its been worlds between microsoft .NET and C++, but ok...

Then he has 4 Stars in JAVA/Android Studio? Hu? Java OR android or both or what? Iam confused.

But, i am the smallest lightbulb in the world, so they hired him, and he gets paid well, since i know my salary, which is pretty ok, but it is not the money i am about.

After talking to him for an hour, i realized he mixes terminology up. From my experience, if i dont develop daily, skills decay. I forget things, even the any type in Hypescript (joke is on me).

I wonder if it is reasonable to think, that someone who has not been in coding for 10 years, since his resumee is saying "PROJEKT MANAGER" and not "DEVELOPER", simply can not code.

We talked. He didn't know that JAVA servers are configurable for the MEM they are allowed to use.
He has no ****ing clue about JAVA changing it's business model and versioning, openjdk vs oracle java.

He messes up PUT and POST and sends the same JSON-body on the same route, and asks me after two hours "why is this not working" (we use a swagger).

So i told him: "please be honest, how good is your JAVA?"
He replied: "i can learn fast, iam just learning spring boot on udemy right now."
And then he said, and this made me really listen up. "In the job description the where looking for a nodejs guy", So his resume has 4 of 5 stars in JAVA/android, but he applied for nodejs?

JEZ. I get confused about all the wrong buzzwords and project manager gibber he is talking.

My boss asked me, what i think about my new collegue. What would you say in this moment? I tend to be diplomatic. "He is really nice. I like him. I hope he can show me things in backend, that will make me senior in 4-5 years too."

I know this differs to the usual dev.to stuff, it's more a writeup. But let me get to the point.

I think he is an imposter, since he knows what is paid in IT.

How would you react if you find out your colleque is a liar about his skillset, wirite some tipps in the comments.

TL:DR
Among all the smart guys with imposter-syndrome, are some real imposters, for the lulz and the $$$$

console.fun() or how to mess up the DOM

jan paul — Wed, 13 Mar 2019 19:15:11 +0000

I stumbled upon this little gem in the vue-land #random channel and it made me smile. Posted by xan#2554.

Thought maybe it can make you smile too and decided to share it to you. Have fun.

Instructions

copy script

visit your most preferred website -- ..most likely dev.to 😜

open console

paste script and press enter

(function funnyFunction() {
  var screenwidth = window.innerWidth;
  var screenheight = window.innerHeight;
  document.querySelectorAll('body *').forEach(function(node) {
    randT = Math.floor(Math.random() * screenheight);
    randL = Math.floor(Math.random() * screenwidth);
    node.setAttribute('style', 'transition-duration:500ms; transition-property: left, top; position: fixed; top:'+randT+'px; left:'+randL+'px');
  });
  setInterval(funnyFunction, 500);
})();

Be honest to yourself, did you knew what will happen?

cheers

i installed a 9 month old mini project again

jan paul — Fri, 09 Nov 2018 09:08:29 +0000

I installed a npm project again after 9 months...
The package.json contained less then 10 entries.

This feels insecure, somehow.... but iam not an expert ;-)

How to improve the build speed in React-Typescript, when using material ui

jan paul — Mon, 22 Oct 2018 09:44:47 +0000

Hi,
you are using material - ui in react typescript and having trouble with the performance? Well i stumpled upon the same issue.
The build takes like forever, since i started to use material-ui and i was really no fun to code against such a lame project.

But lets go in medias res and describe the scenario:

I wrote a header component like this
Header. Pretty staright forward, right?

While coding i went really unhappy, because the build and the livereolad grew pretty slow. So i played around with the Typescript compiler and found this little command
tsc --diagnostics --listFiles which i made an alias healtcheck in my package.json.

Running it was painfull and ended in results like

Files:          5598
Lines:        108833
Nodes:        453997
Identifiers:  146625
Symbols:      217251
Types:         56743
Memory used: 300772K
I/O read:      4.66s
I/O write:     0.48s
Parse time:   26.90s
Bind time:     1.62s
Check time:    9.72s
Emit time:     1.64s
Total time:   39.87s

As you guys can easily see, it takes quite some time and runs to a TON of files.

I imported the two icons for the header like this:

import {
  AccountCircle,
  Language
} from '@material-ui/icons';

And i was wondering about the mass of files that --diagnostics considered when building, especially the trillions of /node_modules/@material-ui/icons/**.d.ts.

Yes iam a typescript user, and althought i had some troubles to get into it, i really like it now. As you might know, if a package does not bring its own typings, there is a huge community in DefinitlyTyped on github, that provides typings for large packages.

For example: one would get the typings for material-ui like this: npm install --save-dev @types/material-ui, with the result of beeing able to consume material-ui in the typescript world.

But a build time of nearly 40seconds? I wanted to improve this and did some research at big brother.
Found this github issue that point to the same problem. And i was really happy to read this comment with a major improvement.

Instead of importing like above, i simply rewrote my import of the icons to:

import AccountCircle from '@material-ui/icons/AccountCircle';
import  Language from '@material-ui/icons/Language';

Lets see how much this impacts our healtcheck now:

Files:           382
Lines:         98396
Nodes:        386176
Identifiers:  130972
Symbols:      201601
Types:         56743
Memory used: 245687K
I/O read:      0.35s
I/O write:     0.37s
Parse time:    4.29s
Bind time:     0.79s
Check time:    5.16s
Emit time:     1.44s
Total time:   11.69s

Wow! Iam was speechless by such a little change, but with an impact like this. We gained nearly 30seconds by altering 2 lines, i mean it could be more lines, but by NOT IMPORTING VIA THE INDEX.

Iam glad i could share this to you, and it will maybe help the one or another.

My name is JanPaul and iam working for instant-data. We are small company, that provides internal tooling for large agencies and also data-exchange platforms.

May the code be with you :pray:
Thanks for the title image to Emil Bruckner @unsplash.

reducks, a boilerplate

jan paul — Tue, 11 Sep 2018 18:27:54 +0000

img by Jase Ess

Hello dear dev.to,

Iam working for instant-data. a small company that provides solutions in the environment of a larger advertising agency. mostly internal toolings.

After playing round a little with the create-react-app, i wanted to share my latest boilerplate with you and ask for your opinions, or improvements. You can visit me here on github.
In this i follow the ducks pattern, in which you try to keep all your redux and store reducer stuff in one single file, but for sakes of DRY, i will present you the README.MD. I appreciate any feedback ;-)

It is created via the CLI-Tool

create-react-app <your-appname> --scripts-version=react-scripts-ts

and customized to my needs, by running

npm eject

Prequisites

Installed node and npm Version 8 or higher.

Available Scripts

In the project directory, you can run:

`npm start`

Open http://localhost:3000 to view it in the browser.

`npm run build`

Builds the app for production to the build folder.

`npm run seed` Authentication seed

To make the boilerplate work from scratch, and to simulate the flow of Auth, we put a JSON-Server.
You should remove the complete seed folder, or better add it to your .gitignore, when you implement your own workflow. To make use of it simply npm run seed. You will get served at json-server --port 4000 --watch -- seed/db.json.

Folder Structure

After creation, your project should look like this:

my-app/
  README.md
  node_modules/
  package.json
  public/
    index.html
    favicon.ico
  seed/
    /* a helper fake backend for firsttimers */  
  src/
    index.tsx
    components/
      App.tsx
    containers/
      /* your connected containers here */
    ducks/
      /* your ducks here */
    services/
      /* put all your global helpers inside here */
    store/
      history.ts
      index.ts
      localStorage.ts
  types/
    /*your types go here */

Debugging and DevTools

I strongly suggest you to install and use theses BrowserExtensions.

Material-UI and Decorators

The experimental feature for Decorators is activated in the tsconfig.json. By this it is possible to use the withStyle decorator in combination with compose of the recompose package.
Example usage is:

import { compose } from 'recompose';
import { withStyles } from '@material-ui/core/styles';

@(compose(withStyles(styles)) as any)
class Example extends React.Component<IExampleProps, IExampleState> {...}

To make use of themes in material-ui we need to wrap the content of App inside a

<MuiThemeProvider theme={createMuiTheme()}>
  //any content here
</MuiThemeProvider>

Environment

There is the possibility to provide ENV variables. use the .env file for this.
Please note, they need to follow the naming convention of REACT_APP_ EXAMPLE=FOOBAR.
Inside the project one can consume them by proccess.env.REACT_APP_EXAMPLE.
At a new start its set to the seedings path, so one might want to change it there.

HTTP Requests

Please use axios for your requests. It supports Promises and also async/await.
https://github.com/axios/axios
Also make sure the allow crossorigin-access on your server.

Ducks

We follow the 'Ducks'-Pattern to structure our application. Get insights on this specc here
https://github.com/erikras/ducks-modular-redux
The pattern is to have all these inside one duck:

action enum
action types
action creators
reducer functions
mainreducer
thunk
initialStore

Message Component

There is a rudimental example prepared in this boilerplate, that shows usage and consumption of a high level message or notification-component. Import the NofictationDuck inside the Ducks that you want the NotificationStore to be catched.

import NotificationDuck from '~/src/ducks/notification';

your.action()
  .then(() => /* success */)
  .catch((err) => {
      dispatch(NotificationDuck.throwNotificationWithError({text: 'your text', title: 'your title'}))
  })

But you can also connect them directly to components. Please keep in mind to reset the NotificationStore, with the RESET_NOTIFICATION_STORE action!

Store

Use the folder src/store to configure the store of redux according to your needs. There is also the possibilty to subscribe parts of the store to your localStorage. To do so check the index.ts and have a brief look at the store subscribe pattern.

store.subscribe(throttle(() => {
  const storage = store.getState() as RootState;
  const AuthStore = storage.AuthStore;

  saveState({
    AuthStore
  } as RootState);
}, 300));

Please make use of the throttle lodash-helper. JSON.parse()/JSON.stringyfy are expensive in Javascript.

Services

Anything you want to consume repetitive belongs to the services. Another word could be globalHelpers.

apicontroller -> an axois instance the can be configured. import it into your ducks to make http-verbs.
authenticationService -> can be used in conjunction with apicontroller inside of ducks to verify auth-Status

inside your duck - an example usage in thunk

import axios from '../../services/apicontroller';

public static getAuth() {
    return function (dispatch: any, getState:() => RootState): Promise<void> {
      dispatch(AuthDuck.getAuthAction());

      //return the axois object with header, pass the state by `getState()` to auth and GET the route with `get('/yourRoute')`

      return axios(getState()).get('/Authentication')
        .then((res) => {
          dispatch(AuthDuck.getAuthSuccessAction(res.data));
         })
        .catch((err: Error) => {
          /**
        });
    };
  }

There is also a whole react-router v4 part, i will explain at 50 :heart: :-)