I Built a Security Scorer for 213 MCP Servers — Here's What Made the Difference

Ismail Khouya — Sun, 26 Jul 2026 16:52:35 +0000

Every time I wanted to add an MCP server to Claude Desktop or Cursor, I ended up
with a dozen browser tabs open, trying to answer one question the README never
answers: is this actually safe to give tool-call access to my machine?

MCP (Model Context Protocol) directories exist, but most of them just mirror
GitHub — stars, a README excerpt, a copy-paste config. None of them score
risk. So I built RepoAI, and I want to share the exact
methodology, because I think it's more interesting than the product pitch.

The problem with "just check the stars"

Stars measure popularity, not safety. A server with 5,000 stars can still:

have no read-only mode (every tool call can write/delete)
ship with no auth method at all
be an unmaintained fork nobody's patched in a year
be archived — meaning whatever's wrong with it stays wrong, forever

None of that shows up in a star count.

The 15 signals

computeSecurityScore() runs 15 checks against each server, sums signed point
values, and clamps the result to 0–100. Two categories:

Repo health (max +101):

Check	Points	Why
Official maintainer	+18 / 0	Vendor-published > random fork
Editorially verified	+5 / 0	A human actually reviewed it
Actively maintained	+12 / +6 / 0	Last commit ≤90d / ≤365d / older
Established contributor base	+10 / +5 / 0	Bus factor: ≥5 / ≥2 / 1
Community adoption	+10 / +5 / 0	Stars: ≥1000 / ≥100 / fewer
Has license	+8 / 0	Unlicensed = unclear usage terms
Reasonable dependency footprint	+5 / +2 / 0	Smaller supply-chain surface
Has homepage/docs	+3 / 0
Substantial README	+5 / 0	≥500 chars
Archived	0 / −40	No future fixes. Ever.

MCP-specific:

Check	Points	Why
Read-only mode available	+15 / 0	A safer way to run it should exist
Supports OAuth	+10 / 0	Stronger than static tokens
Dangerous tools present	0 / −10 / −25	Share of tools flagged execute/delete/write
Dangerous tools, no safe mode	0 / −20	The double penalty — risky and no escape hatch
Weak or missing auth	0 / −15	Anyone with the config file can use it

Scores land in three bands: 80–100 safe, 60–79 medium, 0–59 high risk.

The part I almost got wrong

My first instinct was to let an LLM write the verdict and set the score.
That's a mistake — an AI can be convinced by good marketing copy in a README
just as easily as a human can. So the pipeline splits it:

The score comes only from verifiable facts: GitHub metadata, license, commit history, tool risk classification. No AI in that loop at all.
The verdict (the human-readable "should you use this" paragraph) starts as an AI draft grounded strictly in those same facts — then a human reads it before it ever goes live. Nothing gets published on the strength of an AI's opinion alone.

What it looks like end to end

For a server like qdrant/mcp-server-qdrant, the pipeline:

Scrapes the repo (README, package.json, releases, commit history)
Runs the 15 checks → a score (58, in this case — medium risk, mostly held back by no OAuth support and a moderate dependency footprint)
Generates a ready-to-paste config for Claude Desktop / Cursor / VS Code / Windsurf
Extracts its actual tool list and FAQs from the README
Waits for a human to confirm the AI's draft summary before anything ships

Where it stands today

213 servers, 36 categories, all re-scraped and re-scored on a rolling basis
so a score reflects last week's commit activity, not last year's.

It's open at repoai.io — MCP is the first layer, Claude
Skills and the rest of the AI-tool ecosystem are next, same trust model.