I built a Lighthouse for MCP tools — it scores your tool definitions on every PR

The problem

AI agents choose between tools based on one thing: the quality of their descriptions.

Research shows 97% of MCP tool descriptions have quality defects (arXiv 2602.14878), and optimized tools get selected 3.6x more often (arXiv 2602.18914).

Most MCP developers don't know their tool definitions are broken until an agent silently ignores them.

What I built

ToolRank scores MCP tool definitions across 4 dimensions:

• Findability (25pts) — Can agents discover your tool?
• Clarity (35pts) — Can agents understand what it does?
• Precision (25pts) — Is the input schema complete?
• Efficiency (15pts) — Is it token-efficient?

It's like Lighthouse, but for MCP tools.

GitHub Action — score on every PR

Today I published ToolRank Score on GitHub Marketplace.

Add this to your repo:

name: ToolRank Score
on:
  pull_request:
    paths: ['**/*.json']

permissions:
  pull-requests: write

jobs:
  score:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: imhiroki/toolrank-action@v1

On every PR that touches tool definitions, you get a comment like:

🟣 mcp.json — 95/100 (Dominant, top 3%)

Tool	Score	F	C	P	E
search_web	95	25	33	23	14

Top issues:

• search_web: No usage context — Add 'Use this when...' (+5pt)

Scored by ToolRank v1.0.0

You can also set a minimum score and fail PRs that don't meet it:

- uses: imhiroki/toolrank-action@v1
  with:
    min-score: 70
    fail-on-low: true

REST API

If you want to integrate scoring into your own workflow:

curl -X POST https://mcp.toolrank.dev/api/score \
  -H "Content-Type: application/json" \
  -d '{"tools": [{"name": "search", "description": "Searches things"}]}'

Returns score, level, percentile, dimensions, and specific fix suggestions.

Ecosystem data

We scan 4,000+ MCP servers daily from Smithery and the Official MCP Registry. Some findings:

• 374 servers have scorable tool definitions
• 73% of MCP servers have zero tool definitions (invisible to agents)
• Average score: 85.7/100
• Servers scoring 85+ get selected ~85% of the time in competitive scenarios

Check where your server ranks: toolrank.dev/ranking

What's next

• Layer 2: Real LLM selection testing (Q2 2026)
• Agent framework integrations (LangChain, CrewAI)
• Registry partnerships

The scoring engine is fully open source. Star it if this is useful.

Try it

1. Score your tools — paste JSON or enter your Smithery server name
2. Add the GitHub Action — 30 seconds to set up
3. Check the ranking — see where you stand

---

I'm building ToolRank as the quality standard for the MCP ecosystem. If you're building MCP tools, I'd love to hear what you think.