DEV Community

udaysaai
udaysaai

Posted on

Mycelium – Sub-10ms Semantic Tool Routing for AI Agents (No LLM overhead)

#agents #ai #architecture #performance

Hey dev,

I’m Uday, a solo engineering student building US Neural.

Over the past few months, as agentic workflows scaled from 5 to 50+ tools, I kept hitting the same architectural wall: The Tool Routing Bottleneck.

Today, developers route agent capabilities using two flawed methods:

  1. Hardcoded if/else chains or keyword regex (breaks on synonyms like "currency" vs "forex").
  2. Passing all tool schemas into a frontier LLM prompt (adds 1,500ms – 3,000ms latency and burns recurring API tokens just to pick a tool).

I built Mycelium to decouple routing from prompt engineering. It is an open-source, edge-native semantic registry that maps natural language user intent directly to agent endpoints using a local ChromaDB vector-mesh and compact embeddings (all-MiniLM-L6-v2).

📊 Audited Benchmarks (100k Agent Corpus)

We benchmarked Mycelium against BM25 and lexical search across a 100,000-agent synthetic corpus with 441 unique task-oriented queries (cold cache, embedding time included on commodity CPU):

  • Intent Accuracy: 70.7% Top-1 (vs 40.4% BM25, +30.3% gain)
  • Cold Discovery Latency: 9.56ms (vs 194.0ms BM25, 20x faster)
  • Multi-Agent Chain (2-hop): 37.6ms end-to-end (Weather -> Translation)
  • Single-Node Throughput: 130+ req/sec with 0.0% error rate under 100 concurrent workers.

🛡️ Anthropic MCP & Zero-Trust Safety (HOL Guard)

We also added a native bridge for Anthropic's Model Context Protocol (MCP) with Human-On-The-Loop (HOL) Guard:

  • READ_ONLY intents (weather queries, wiki lookups) auto-execute in <15ms.
  • MUTATING intents (fund transfers, SQL table drops) are intercepted at the network layer and quarantined until human cryptographic authorization.

🔗 Open-Source & Live Demos

Would love your brutal feedback on our evaluation methodology and architecture. Tear it apart!

Top comments (4)

Collapse
 
max_quimby profile image
Max Quimby •

This is the right layer to attack — once you're past ~30 tools, stuffing every schema into the prompt taxes both latency and the model's own selection accuracy, so decoupling routing is a genuine win. The number I'd push on is the 70.7% Top-1. For read-only intents that auto-execute in <15ms that's fine, but a ~30% miss rate on the routing decision is a lot to hand to something that then acts. Two questions: (1) is that 70.7% measured against a single "correct" tool per query, or do you score a hit when any acceptable tool in the top-1 fires? Multi-tool-valid queries make Top-1 look artificially harsh. (2) How does the MUTATING-vs-READ_ONLY classification itself get made — is that a property of the registered endpoint, or inferred from the query? If it's inferred by the same embedding step, a misroute could also misclassify safety tier, which is the failure mode I'd want bounded before trusting the auto-execute path. Nice to see the negative-latency case (BM25 at 194ms) included rather than a cherry-picked win.

Collapse
 
udaysaai profile image
udaysaai •

Thanks for the thoughtful pushback — these are exactly the two core edge-cases we had to design around.

To answer your questions directly:

  1. On the 70.7% Top-1 metric:
    It is evaluated via Family-Level Intent Correctness, not exact-ID matching. In our earlier v1 benchmark with 100k agents, exact-ID scoring produced an artificial 0.0% hit rate because the synthetic corpus had ~12,500 interchangeable agents per capability domain (weather, forex, ticketing). Exact-ID was purely rewarding tie-breaking order in lexical search rather than functional suitability. We restructured the evaluation across 441 unique task-phrased queries where selecting any capable agent in the target domain counts as a Top-1 hit.

  2. On Safety Gating (MUTATING vs READ_ONLY):
    Great catch — we deliberately avoid relying solely on the semantic vector step for safety classification. The safety policy is defense-in-depth:

  3. Static Declaration: Each agent/tool endpoint statically declares its permission boundary at registration (policy: "READ_ONLY" vs "MUTATING").

  4. Deterministic Intent Heuristics (HOL Guard): The HOL Guard intercepts before the execution phase. Even if a user query semantically resolves to a tool, if either the tool scope is marked mutating OR the intent targets state-altering actions (financial transfers, writes, table deletions), the execution pipeline is strictly quarantined and diverted to a human cryptographic approval hook.

We don't trust the embedding model to be the safety boundary. The vector-mesh only handles discovery; the network gate enforces quarantine.

Would love to hear your thoughts on this architecture or any edge cases you've seen in production multi-agent setups! Repo & reproducible scripts are here: github.com/udaysaai/mycelium

Collapse
 
udaysaai profile image
udaysaai •

Sharp observations — particularly on the abstention necessity for open-world routing.

Here is how the current implementation handles these constraints, along with what’s planned:

  1. Abstention & Out-of-Domain Filtering:
    You hit on a critical vector-search pitfall: pure k-NN will always return the nearest neighbor even for irrelevant prompts (e.g., asking for pizza delivery in a registry with only forex and weather tools).
    To prevent hallucinated routing, Mycelium enforces a calibrated similarity threshold (min_score, default 0.15–0.30 cosine bound). Queries falling below the threshold trigger an explicit ABSTAIN / NO_TOOL_FOUND status rather than dispatching to an arbitrary nearest neighbor. We are currently formalizing the Accuracy-vs-Abstention ROC curves across out-of-domain query injections for our upcoming v0.4.0 benchmark slice.

  2. Multilingual & Code-Switched Queries:
    Right now, all-MiniLM-L6-v2 is primarily English-optimized. In our technical report's Threats to Validity / Limitations section, we explicitly flagged this: while it handles minor semantic drift and common synonyms cleanly, highly code-switched or purely non-English intent requires multilingual dense backends (e.g., paraphrase-multilingual-MiniLM-L12-v2). The tradeoff there is higher memory footprint (~470MB vs ~90MB) and slightly higher embedding latency on CPU.

  3. ARM / Edge Profiling & Index Cost:
    On local CPU execution:

  4. Model Footprint: all-MiniLM-L6-v2 sits at ~80–90MB peak RAM in float32, dropping to ~30MB with INT8 ONNX runtime quantization.

  5. Index Update Cost: Single agent upserts into local ChromaDB average 3–5ms on modern cores.
    We haven't yet run a dedicated thermal and memory ceiling benchmark on bare-metal ARM (e.g., RK3588 or Apple Silicon) for the 100k corpus, but that is on our roadmap for the edge-deployment paper.

Really appreciate the rigor here. If you're interested in reviewing our raw benchmark scripts or contributing to the edge profiling test harness, the repo is open-source: github.com/udaysaai/mycelium