DEV Community

udaysaai
udaysaai

Posted on

Mycelium – Sub-10ms Semantic Tool Routing for AI Agents (No LLM overhead)

#agents #ai #architecture #performance

Hey dev,

I’m Uday, a solo engineering student building US Neural.

Over the past few months, as agentic workflows scaled from 5 to 50+ tools, I kept hitting the same architectural wall: The Tool Routing Bottleneck.

Today, developers route agent capabilities using two flawed methods:

  1. Hardcoded if/else chains or keyword regex (breaks on synonyms like "currency" vs "forex").
  2. Passing all tool schemas into a frontier LLM prompt (adds 1,500ms – 3,000ms latency and burns recurring API tokens just to pick a tool).

I built Mycelium to decouple routing from prompt engineering. It is an open-source, edge-native semantic registry that maps natural language user intent directly to agent endpoints using a local ChromaDB vector-mesh and compact embeddings (all-MiniLM-L6-v2).

📊 Audited Benchmarks (100k Agent Corpus)

We benchmarked Mycelium against BM25 and lexical search across a 100,000-agent synthetic corpus with 441 unique task-oriented queries (cold cache, embedding time included on commodity CPU):

  • Intent Accuracy: 70.7% Top-1 (vs 40.4% BM25, +30.3% gain)
  • Cold Discovery Latency: 9.56ms (vs 194.0ms BM25, 20x faster)
  • Multi-Agent Chain (2-hop): 37.6ms end-to-end (Weather -> Translation)
  • Single-Node Throughput: 130+ req/sec with 0.0% error rate under 100 concurrent workers.

🛡️ Anthropic MCP & Zero-Trust Safety (HOL Guard)

We also added a native bridge for Anthropic's Model Context Protocol (MCP) with Human-On-The-Loop (HOL) Guard:

  • READ_ONLY intents (weather queries, wiki lookups) auto-execute in <15ms.
  • MUTATING intents (fund transfers, SQL table drops) are intercepted at the network layer and quarantined until human cryptographic authorization.

🔗 Open-Source & Live Demos

Would love your brutal feedback on our evaluation methodology and architecture. Tear it apart!

Top comments (0)