DEV Community: ranadhir ghosh

Know What Your Agent's Tools Can Do to You: Scanning MCP Servers Before You Install Them

ranadhir ghosh — Tue, 03 Nov 2026 19:07:00 +0000

In an April 2026 audit, 27 detections from MCP security scanners boiled down to 6 genuine concerns. The scanners weren't missing attacks — they were crying wolf, which in a CI pipeline is the same failure wearing a different coat.

So I built a scanner whose first feature is its own error profile: every rule publishes a measured precision from a labelled corpus of planted positives and near-miss traps. This is how it works, what it catches — tool poisoning, rug pulls, dangerous sinks, hardcoded secrets — and where its numbers say it still gets things wrong.

One command before you install

Sentinel Scan is a security and conformance scanner for MCP servers and SKILL.md agent skills. One command scans a server or skill for tool poisoning, rug pulls, dangerous code sinks, and hardcoded secrets — and reports each rule's measured precision rather than just asserting it.

pip install sentinel-scan
sentinel-scan demo

The demo scans the bundled 12-tool corpus in about a second: 6 benign tools — two of them near-miss traps — and 6 with planted findings, producing 7 findings. The two traps produce no findings: a tool that "ignores previous cached results" and a security tool that quotes an injection as an example. That restraint is the product.

Pure Python, no API keys, no GPU, no network. Python ≥ 3.10.

What it catches

Tool poisoning: hidden instructions in tool descriptions — markup directives, "note to the AI" framing, ignore-instruction phrasing — plus descriptions that direct credentials, files, or conversation contents out, and covert actions a tool takes "without telling the user."
Rug pulls: a tool's description changing after you pinned it. Sentinel Scan stores a SHA-256 pin of the normalized description and flags quiet changes (whitespace-only edits ignored).
Dangerous sinks: command execution (shell=True, os.system, string commands — fixed argv with shell=False does not fire), dynamic code (eval/exec — ast.literal_eval does not fire), and sensitive-file access (~/.ssh, ~/.aws, /etc/passwd, key files, full environment dumps — a comment merely mentioning a path does not fire).
Hardcoded secrets: AWS key ids, GitHub tokens, OpenAI-style keys, private-key blocks, assigned credential literals (${VAR} references do not fire).
Skill rules: hidden instructions in a SKILL.md body, capabilities the body uses but the frontmatter does not declare, and filesystem servers scoped to /, ~, or a home directory instead of a project directory.

Measured, not asserted

Each rule is deliberately narrow: it fires on a specific, explainable signal shape. The rates are measured with scripts/measure.py over the labelled corpus in fixtures/corpus/ — 32 cases of planted positives and benign near-miss negatives for every rule. Across all rules: 18 true positives, 0 false positives, 0 false negatives.

Those are corpus rates, not a promise about the open internet. Re-run the measurement yourself:

python3 scripts/measure.py

Using it

Scan an MCP config, a tools manifest, server sources, or skill folders:

sentinel-scan scan ~/.config/...

Exit codes are CI-shaped: the scan exits 1 when any finding is at or above --fail-on (default high). Outputs are terminal text, JSON, and SARIF, which renders in the GitHub Security tab. There is also a GitHub Action (ranadhir19/sentinel-scan@v0.1.0) that runs the scan, writes SARIF, and uploads it to the Security tab.

What it does not do: no dynamic execution of scanned servers, no LLM judgement calls, no network calls. It is static analysis. There is also a local web demo in demo/ (React front end, FastAPI backend) if you want to paste a description, a SKILL.md body, or source and watch it scan.

Repo: https://github.com/ranadhir19/sentinel-scan

Prove It Works: Building an Eval That Can Catch a Broken Benchmark

ranadhir ghosh — Tue, 03 Nov 2026 18:57:15 +0000

A benchmark with flawed tests measures the tests. So I built two suites — one clean, one with 12 flaws planted in the tests — and watched a do-nothing agent start scoring points while my reference agent lost 8.3 points to the tests themselves.

This is the full experiment: calibrating the graders first, pre-registering the audit bar, catching 11 of 12 flaws — and being honest about the one that got away.

The setup

Local, deterministic, no API keys — a lab for auditing your eval before your eval audits your agent. The setup is a clean 60-task suite (arithmetic, extraction and classification; gold answers cross-checked by an independent second implementation) and an identical twin with 12 flaws planted in the tests.

What the flaws did

Null agent (outputs nothing): 0/60 on the clean suite, 2/60 on the flawed suite
Echo agent: 0/60 on the clean suite, 4/60 on the flawed suite
Reference agent: 60/60 on the clean suite, 55/60 on the flawed suite — 8.3 points lost to the tests, not the tasks

Calibrate the graders first

Three graders were validated on 150 labelled items before any audit run: exact-match 80.0%, normalized 100%, rubric 99.3%. The point is stated plainly in the lab: an uncalibrated grader is just another flawed test.

The audit

The audit bar was registered before the run: at least 10 of the 12 flaws caught, at most 3 false flags. The audit caught 11 of 12 planted flaws with zero false flags. The one miss, flaw-09, had its gold and its source document disagreeing (300 vs 315) from the same table — shared-source flaws are the hard class, and that is exactly what to test for next.

If you build or trust agent benchmarks, audit the proof itself, not just the score.

Repo: https://github.com/ranadhir19/agent-eval-lab

Build an MCP Server a Model Can Actually Use

ranadhir ghosh — Tue, 03 Nov 2026 18:51:52 +0000

Most MCP tutorials stop at "it connects." But a model never sees your server code — it sees tool names, descriptions and JSON schemas. An MCP server is a user interface designed for a language model, and interfaces can be measured.

So I built one server twice, attacked both designs with the ten worst calls a model could plausibly make, and ran a model against each. The loose design rejected 0 of 10. The strict redesign rejected 10 of 10. Here is the full experiment — including the metric that pointed the wrong way.

Two designs, one notes store

The lab is a notes MCP server in pure Python stdlib, speaking JSON-RPC 2.0 over stdio, with initialize, tools/list and tools/call. The same notes store is exposed through two schema designs:

v1, deliberately loose: three vague tools, a free-form action string, no required fields, and bloated results.
v2, the strict redesign: one verb per tool, descriptions that say when not to use the tool, strict schemas, compact results, and structured errors.

The question was stated before any run: does schema redesign change how often models pick the wrong tool, produce malformed calls, or fail tasks? The pre-stated bar was that v2 counts as a win only if wrong-tool calls fall by at least 30% relative to v1 and task completion rises.

The adversarial audit

The deterministic audit sends 10 adversarial call classes — unknown ids, missing fields, wrong types, and invented actions. v1 rejected none of them and let bad calls change its data; v2 rejected all 10.

The model trials

Then a small local model (Qwen2.5-0.5B) ran 16 tasks against each design, with a chooser that sees only the tools/list output and a request, answers with one JSON tool call, and has that call executed for real against a fresh seeded store. Wrong-tool calls fell from 7 to 1, malformed calls from 4 to 1, and completed tasks rose from 6 to 13 of 16.

One honest failure is kept in the repo: a word-overlap "confusability" metric ranked the designs backwards (0.074 vs 0.289). Longer descriptions share more vocabulary; shared words are not shared meaning.

The point is not the notes app. It is that an MCP server is a UI for a model, and that interface can be measured — with an audit suite, a fixed task set, and a grading rubric — before you trust it in production.

Repo (server, both schemas, harness, audits — all runnable): https://github.com/ranadhir19/mcp-server-lab