DEV Community: William Weishuhn

I Turned My Obsession With Trustworthy AI Agents Into MCP Observatory

William Weishuhn — Sun, 12 Jul 2026 19:56:33 +0000

This is a submission for Weekend Challenge: Passion Edition.

What I Built

I built MCP Observatory, an open-source security and reliability toolkit for Model Context Protocol servers.

AI agents are only as trustworthy as the tools they can call. As MCP servers move from demos into real workflows, teams need a way to answer practical questions before depending on them:

What capabilities does this server expose?
Did its tool schemas or behavior change?
Can we reproduce and inspect a failed session?
Will a risky regression be caught before deployment?
Is there evidence an operator or another agent can verify?

MCP Observatory turns those questions into repeatable checks. It can discover servers, scan them, run security tests and attack simulations, detect schema and response drift, record and replay sessions, generate health scores and badges, and emit CI-friendly Markdown, JSON, and SARIF artifacts.

The project is my response to a subject I keep returning to: how do we make increasingly capable AI agents safer without making them less useful? I wanted the answer to be practical, inspectable, and open source—not just another policy document.

Demo

Landing page

👉 mcp-observatory.com

Try the CLI

npx @kryptosai/mcp-observatory@latest

Run a deep security scan:

npx @kryptosai/mcp-observatory@latest scan deep --security

Audit an MCP server and produce a readable report:

npx @kryptosai/mcp-observatory@latest audit   npx -y @modelcontextprotocol/server-everything   --profile nsa-mcp   --format markdown   --output mcp-audit.md

Set up CI and SARIF output:

npx @kryptosai/mcp-observatory@latest setup-ci   --all   --command "npx -y your-mcp-server"   --sarif

The same toolkit can run as an MCP server, which means an agent can use Observatory to inspect another MCP server:

claude mcp add mcp-observatory --   npx -y @kryptosai/mcp-observatory serve

Code

The core CLI and MCP server are MIT licensed for local open-source use.

How I Built It

MCP Observatory is built in TypeScript and organized around a simple idea: a test should leave behind evidence.

The workflow is roughly:

Discover MCP servers from supported configurations.
Connect over the server's transport and inspect its advertised capabilities.
Exercise tools and prompts through targeted checks or deeper security campaigns.
Record schemas, responses, timing, and failures as structured artifacts.
Compare current behavior with earlier runs to detect drift.
Report the result as Markdown for humans, JSON for automation, or SARIF for code scanning.
Enforce the checks in CI so regressions are caught before agents depend on them.

Some of the most interesting engineering work has been making the output useful beyond a single terminal session. A pass/fail result is not enough when behavior changes over time, so Observatory supports receipts, run artifacts, record/replay cassettes, risk graphs, badges, and regression reports.

I also built a GitHub Action and CI setup flow because security tooling only matters when teams can adopt it without rebuilding their workflow around the tool.

Why This Is My Passion Project

The “passion” here is not just enthusiasm for AI. It is a stubborn belief that powerful developer tools should earn trust through evidence.

I love building systems that make invisible behavior visible: a schema change that would otherwise surprise an agent, a response drift that quietly breaks automation, or a risky capability that deserves review before production.

MCP is becoming connective tissue for agentic software. I want the safety layer around it to be open, reproducible, and accessible to individual maintainers—not reserved for teams with enterprise security budgets.

That is what keeps me working on MCP Observatory: every new test, artifact, and integration makes it a little easier to scan before you trust.

What I Learned

The biggest lesson is that reliability and security are not separate features for agent tools. Both depend on understanding what a server promised, what it actually did, and whether that behavior changed.

I also learned that adoption is part of the engineering problem. A sophisticated security check that cannot run locally or in CI will be ignored. The best workflow is one command to try, structured evidence to inspect, and a clear path to automation.

If you work with MCP servers, I would love feedback on the risks or failure modes you most want Observatory to catch next.

MCP Observatory: Scan, Test, and Secure MCP Servers Before Agents Depend on Them

William Weishuhn — Sun, 12 Jul 2026 19:50:02 +0000

MCP servers are becoming production dependencies.

If an agent depends on a server, that server needs the same basic safety net as any other production integration: CI checks, security review, regression reports, and artifacts that teams can inspect when behavior changes.

MCP Observatory helps teams test, secure, and monitor MCP servers before agents depend on them.

It can:

scan MCP servers from local configs
run in CI through the GitHub Action
detect capability, schema, and response drift
record, replay, and verify MCP sessions
produce health scores, badges, and production/security reports
run as an MCP server so agents can test other MCP servers

The open source CLI and MCP server remain free for local OSS use under MIT.

For teams running MCP in private repos or production workflows, paid pilots are available for hosted reporting, private repo CI history, security reports, production monitoring, certification, support, and fleet visibility.

Try it

npx @kryptosai/mcp-observatory@latest

Deep scan:

npx @kryptosai/mcp-observatory@latest scan deep --security

Running MCP in production? Contact william@banksey.com.

mcp-observatory：一行命令扫描 MCP 服务器安全漏洞

William Weishuhn — Tue, 07 Jul 2026 19:07:19 +0000

你的 AI Agent 正在盲连 MCP 服务器。没有人检查过它们是否安全——直到现在。

你在盲连，而你不知道

2026 年，MCP（Model Context Protocol）已经成了 AI Agent 的事实标准协议。Claude Code、Cursor、Codex、以及字节跳动的 UI-TARS，都在通过 MCP 连接外部工具：文件系统、数据库、浏览器、Kubernetes 集群。

问题在于，绝大多数 MCP 服务器从未被安全审计过。

你给 Agent 接了一个 Kubernetes MCP 服务器。Agent 能执行什么命令？它有没有 shell 注入面？上游更新后 Schema 是否悄悄变了？答案在今天是：没人知道。

我们做了一个实验。对 mcp-server-kubernetes——一个 3,000+ star 的官方 MCP 服务器——跑了安全扫描。结果是 72/100 分，安全维度直接 0 分，发现 3 个 HIGH 级别问题：

工具	风险等级	问题
`kubectl_create`	HIGH	`command` 参数允许任意命令执行（shell 注入）
`exec_in_pod`	HIGH	`command` 参数允许在 Pod 内执行任意命令
`kubectl_generic`	HIGH	通用 `command` 参数绕过所有安全约束

这不是漏洞指控——而是在你把它接上生产集群之前，你需要知道的事。

mcp-observatory：MCP 服务器的安全望远镜

mcp-observatory 就是为回答这类问题而生的开源工具。一条命令，完成自动化安全审计、攻击模拟、Schema 漂移检测和合规检查。

截至发文：142 GitHub Stars、21 Forks、npm 月下载 2,300+。已在 Gitee 建立镜像，国内可直接访问。


bash
npx @kryptosai/mcp-observatory