DEV Community

XYG-LUNA
XYG-LUNA

Posted on

Building a Multi-Skill MCP Server: Lessons from Serving 284 AI Tools

#mcp #ai #architecture #opensource

When you're serving 284 different AI tools through a single MCP (Model Context Protocol) server, the architecture challenges are... not what you'd expect.

I've been running CHROMATIC-MCP for about a year now. Here's what I've learned about building a multi-skill AI platform that actually works in production.

The Context Window Problem

The first thing that breaks when you have 284 tools: the tools/list response. Most AI clients have a context window limit. If you dump all 284 tool descriptions at once, you've burned through 40% of the available context before the user even asks a question.

Our solution: Two-layer directory

{
  "tools": [
    {"name": "discover_skills", "description": "Search available skills by category or keyword"},
    {"name": "lianzhu_ziwei", "description": "Purple Star Astrology chart analysis"},
    // ... 9 more curated "featured" tools
  ]
}
Enter fullscreen mode Exit fullscreen mode

The client sees ~10 tools initially. When it needs something specific, it calls discover_skills to search the full catalog. This dropped our context overhead from 12K tokens to under 2K.

Description Engineering > Prompt Engineering

Here's the counterintuitive lesson: the tool descriptions matter more than the prompts inside the tools.

AI models decide whether to call your tool based solely on the description. Bad description = invisible tool.

What didn't work:

"description": "Analyzes personality"
Enter fullscreen mode Exit fullscreen mode

What works:

"description": "MBTI personality coaching - Given a user's MBTI type (e.g. INFJ, ENTP), provides targeted career advice, relationship insights, and personal growth strategies. Input: {type: string, question: string}. Returns structured coaching response."
Enter fullscreen mode Exit fullscreen mode

Key elements that improved our call rate:

  1. Start with what it IS (noun phrase)
  2. Explain the input format explicitly
  3. Describe what the output looks like
  4. Include 2-3 example use cases in the description itself

We A/B tested descriptions across 50 skills. Good descriptions increased call rates by 3-4x.

The Execute-Locally Pattern

Not every skill needs server-side LLM execution. For our 50 free skills, we use an "execute-locally" pattern:

  1. Client calls tools/call with the skill name
  2. Server returns... a SKILL.md document (not an LLM response)
  3. The client's own LLM uses that document as instructions
  4. Execution happens entirely on the client side

This means:

  • Zero server-side compute cost for free skills
  • Users can run them with any local model (Ollama, ChatGLM, etc.)
  • No API key required for the free tier
  • Complete privacy - nothing leaves the user's device

Billing: Per-Task, Not Per-Token

We deliberately chose per-task billing over per-token:

  • Task succeeds → charge (¥1.29-6.99 depending on complexity)
  • Task fails, times out, or produces garbage → no charge

The implementation uses a "completion certificate" pattern:

Request → Processing → Validation Gate → Certificate Issued → Billing Triggered
                         ↓
                    Failure → No Certificate → No Charge
Enter fullscreen mode Exit fullscreen mode

This aligns incentives: we want tasks to succeed (that's how we get paid), and users don't fear experimenting (failures are free).

Monitoring in Production

With 284 skills, you need observability. Our setup:

  • JSONL structured logs per skill invocation
  • Key metrics: call_count, success_rate, p95_latency, certificate_issued
  • Weekly Pareto analysis: top 20 skills account for 80% of traffic

Current numbers (honest):

  • ~500 daily requests
  • 85% task success rate
  • P95 latency: <2s
  • Most traffic from US (working on domestic CN growth)

What I'd Do Differently

  1. Start with fewer skills. 284 is maintenance hell. Start with 20, nail those, then expand.
  2. Invest in description testing early. We wasted 3 months with poor descriptions before realizing this was the bottleneck.
  3. Build the billing system before launch. Retrofitting billing into a running system is painful.

Try It

Questions welcome. Happy to dive deeper into any of these topics.

Top comments (0)