How I built deferred tool discovery for my desktop AI assistant (no embeddings needed)

I'm building Ankita, an open-source desktop AI assistant (Electron + terminal CLI) that can run shell commands, edit files with approval diffs, search the live web, manage scheduled routines, and talk hands-free. It's powered by GitHub Copilot models, and one design constraint has shaped almost everything in its architecture:

The CLI has zero runtime npm dependencies.

That constraint is really about context, not packaging. When your assistant calls web_search or git_diff, the model needs the full parameter schema for every tool it might pick — and those schemas are context you pay for on every single request. Ship dozens of tools with everything loaded upfront and you burn tokens before the user has typed anything.

So I built a deferred tool-discovery system. Here's how it works, and the three small decisions that made it feel invisible.

The problem: schemas are context

Ankita's tool catalogue covers web search/fetch/scrape, Git, filesystem, process management, scheduling, page watches, GitHub notifications, project memory, MCP servers, image generation, voice… Each tool is one ESM module under tools/, exporting name, description, parameters, and run(). If I loaded all of them into every request, the model would see hundreds of lines of JSON schema before the conversation even starts.

Decision 1: one find_tools tool, not fifty

The assistant's default toolset is deliberately small. When it needs something outside that set, it calls a single tool:

export const name = "find_tools";
export const description =
  "Load extra tools that are not in your default set: searching the internet and scraping pages, " +
  "Git, port/process management, scheduled routines and page watches, project management and memory, GitHub notifications, and " +
  "directory creation. Call this first whenever a task needs one of those; the tools become callable " +
  "immediately afterwards.";

It takes a plain-language query — "search the web", "remind me daily", "where does this project stand" — and returns the matched tool schemas. Those tools then become callable in the same session. The rest stay unloaded.

Decision 2: categories with keyword summaries, no embeddings

The matching lives in tools/catalog.mjs. Tools are grouped into categories, and each category carries a short summary plus a keyword list:

{
  id: 'process',
  summary: 'find port owners and terminate an approved process or port listener',
  keywords: ['port', 'process', 'pid', 'address in use', 'eaddrinuse', 'kill', 'listener', 'taskkill'],
  tools: [portStatus, killProcess],
},

matchCategories() in tools/find-tools.mjs then matches the user's query against category ids, keywords, and even tool names — with word-boundary regexes, not naive substring checks (so "port" doesn't fire on "transport"). It's pure and synchronous, which makes it trivially testable with Node's built-in test runner.

No embeddings, no vector index, no extra dependency. This is a deliberate trade-off: embeddings would handle paraphrase better, but a curated keyword list is predictable, debuggable, and costs nothing at runtime. For a desktop app where I control both ends, predictability wins.

There are also pragmatic disambiguation rules baked in — e.g. a "github notifications" query loads the built-in GitHub inbox category and not the connectors category, so one request doesn't pull in an unrelated schema. Real usage is full of these collisions, and a comment in the code documenting why beats cleverness every time.

Decision 3: skills are just markdown, loaded on demand

Tools cover capabilities; skills cover procedures. A skill is a SKILL.md file with frontmatter (name, description, suggested-tools) and markdown instructions below. The skill tool loads one by name and returns the body, capped at 8000 characters:

export function run(args = {}, ctx = {}) {
  const requested = String(args.name ?? '').trim().toLowerCase();
  // ...
  return renderSkill(found); // "# Skill: commit-review\n...---\n" + body, capped
}

Suggested tools are hints only — the skill never forces a tool call. This keeps procedural knowledge (like the repo's own conventions in ankita-dev: read before editing, zero runtime deps, run tests before reporting) out of the system prompt and loaded only when the task actually matches the skill's description.

What I'd do differently next

The keyword lists are hand-maintained, and they drift: every time I add a tool I have to think about which synonyms users will type. I've considered generating candidate keywords at build time from tool descriptions and reviewing the diff — human curation with machine assistance, rather than either extreme. And the always-on categories (personal memory, skills) deserve a periodic review, because "always loaded" is a cost I should keep auditing.

Try it and tell me what breaks

Ankita is open source — akyourowngames/A.N.K.I.T.A — with portable Windows builds in every release. The code discussed here is in tools/catalog.mjs, tools/find-tools.mjs, and tools/skills/skill.mjs.

If you've built lazy tool-loading for an agent of your own — embeddings, keyword maps, something weirder — I'd genuinely like to hear what worked and what bit you. I'm 16, building this in public, and the feedback loop is the best part.

Comments

[AI Frontier Post]

Skipping embeddings for a zero-dep CLI is the right call, but the real cost is keyword drift: once a keyword list goes stale, the agent never even gets offered the tool it needs. One middle ground could be generating the keyword lists from each tool's name and parameter descriptions at build time instead of hand-maintaining them.

[Krish Verma]

You're right about the drift — and you're half describing what's already there. Tool names are matched automatically: a query containing web_search loads the web category with no keyword needed, and MCP server matching tokenizes server ids and tool names at runtime, so renaming things updates matching for free.

The hand-maintained part is only the per-category keyword lists in catalog.mjs — and yes, those have usage-driven entries (the browser list literally contains flight, hotel, youtube) that will rot as features change. Parameter descriptions aren't consulted at all right now, which is the real gap. Deriving keywords from name + description at startup (there's no build step, it's a CLI) is a genuinely good next step — might steal that.

[Mihai Perdum]

This is a clean pattern, and the word-boundary regex catching "port" without misfiring on "transport" is the kind of detail that only shows up after real usage, not design.

One thing I'm curious about: when find_tools gets called a second time in the same session for a different category, do the previously loaded tool schemas stay in context alongside the new ones, or does each call replace the prior set? If they accumulate across a long session, you'd eventually converge back toward the "everything loaded" cost this design is meant to avoid, just spread across turns instead of the first request.

[Krish Verma]

Great question — they stay. find_tools adds tool names to a per-session set (activatedTools) that only grows within the session; nothing ever unloads. On every model round, specParts() re-attaches the schemas of everything in that set, grouped by category, alongside the always-on core set. So a second call for a different category just appends.

The one caveat is the schema byte budget: if the context window gets tight (long memory recall, small model), a whole group gets dropped for that round rather than half-sent — the model was told the group is callable as a unit, so partial schemas would lie to it. Under pressure you can temporarily lose a group, but it's never replaced by the newer call.

[Edward Izgorodin]

The growing set has a second cost besides size, if the provider behind the client caches prompt prefixes. Anthropic's caching docs put tools first in the cached prefix and say that modifying tool definitions invalidates the entire cache, and OpenAI keys its cache on the initial tokens, tool definitions included. So the round after a find_tools call that adds a group pays the uncached rate, or the higher write rate, for nearly the whole prompt instead of reading it from cache.

The cost of an activation therefore grows with the length of the prompt at that moment. A group found on the second turn reprocesses a short prompt, and the same group found on the fortieth reprocesses forty turns. The budget rule falls on the expensive end, because it fires when the prompt is closest to the window, so the round that drops a group pays for nearly the whole prompt again at its longest. Whether the round that brings the group back can still read the cache from before the drop depends on how far back the provider looks for an earlier prefix and on the cache lifetime, and if the set that fits changes from round to round, each change is another miss. Logging cached tokens per round, where the provider reports them, shows which of these is happening.

[Krish Verma]

You're right about the second cost, and it's one I only started thinking about properly after writing this post. There's a detail in my implementation that works in the cache's favor on re-entry: the activated set (state.activatedTools in tools/find-tools.mjs) only grows. A group dropped by the budget rule isn't unloaded — currentSpecs() in src/core/agent.mjs just skips it for that turn and re-evaluates next turn. So when the group fits again, the emitted spec list is byte-identical to the pre-drop round, which is the best case for a prefix cache hit: the tool-definitions prefix matches exactly.

The honest caveats: the history between those rounds kept growing anyway, so whether the provider finds the older prefix depends on its prefix-matching and cache lifetime — and as you say, Anthropic's ordering (tools first in the cached prefix) means the activation itself invalidates everything after it. Also, I have to admit I can't confirm any of this from real runs: /usage shows turn/session token totals and estimated cost only (src/core/cli.mjs), with no cached-token breakdown. Your suggestion of logging cached tokens per round is exactly the measurement I'd need before I make any claim about which effect dominates here. Noted as the next instrumentation to add.