This is a submission for Weekend Challenge: Passion Edition.
What I Built
I built MCP Observatory, an open-source security and reliability toolkit for Model Context Protocol servers.
AI agents are only as trustworthy as the tools they can call. As MCP servers move from demos into real workflows, teams need a way to answer practical questions before depending on them:
- What capabilities does this server expose?
- Did its tool schemas or behavior change?
- Can we reproduce and inspect a failed session?
- Will a risky regression be caught before deployment?
- Is there evidence an operator or another agent can verify?
MCP Observatory turns those questions into repeatable checks. It can discover servers, scan them, run security tests and attack simulations, detect schema and response drift, record and replay sessions, generate health scores and badges, and emit CI-friendly Markdown, JSON, and SARIF artifacts.
The project is my response to a subject I keep returning to: how do we make increasingly capable AI agents safer without making them less useful? I wanted the answer to be practical, inspectable, and open source—not just another policy document.
Demo
Landing page
Try the CLI
npx @kryptosai/mcp-observatory@latest
Run a deep security scan:
npx @kryptosai/mcp-observatory@latest scan deep --security
Audit an MCP server and produce a readable report:
npx @kryptosai/mcp-observatory@latest audit npx -y @modelcontextprotocol/server-everything --profile nsa-mcp --format markdown --output mcp-audit.md
Set up CI and SARIF output:
npx @kryptosai/mcp-observatory@latest setup-ci --all --command "npx -y your-mcp-server" --sarif
The same toolkit can run as an MCP server, which means an agent can use Observatory to inspect another MCP server:
claude mcp add mcp-observatory -- npx -y @kryptosai/mcp-observatory serve
Code
The core CLI and MCP server are MIT licensed for local open-source use.
How I Built It
MCP Observatory is built in TypeScript and organized around a simple idea: a test should leave behind evidence.
The workflow is roughly:
- Discover MCP servers from supported configurations.
- Connect over the server's transport and inspect its advertised capabilities.
- Exercise tools and prompts through targeted checks or deeper security campaigns.
- Record schemas, responses, timing, and failures as structured artifacts.
- Compare current behavior with earlier runs to detect drift.
- Report the result as Markdown for humans, JSON for automation, or SARIF for code scanning.
- Enforce the checks in CI so regressions are caught before agents depend on them.
Some of the most interesting engineering work has been making the output useful beyond a single terminal session. A pass/fail result is not enough when behavior changes over time, so Observatory supports receipts, run artifacts, record/replay cassettes, risk graphs, badges, and regression reports.
I also built a GitHub Action and CI setup flow because security tooling only matters when teams can adopt it without rebuilding their workflow around the tool.
Why This Is My Passion Project
The “passion” here is not just enthusiasm for AI. It is a stubborn belief that powerful developer tools should earn trust through evidence.
I love building systems that make invisible behavior visible: a schema change that would otherwise surprise an agent, a response drift that quietly breaks automation, or a risky capability that deserves review before production.
MCP is becoming connective tissue for agentic software. I want the safety layer around it to be open, reproducible, and accessible to individual maintainers—not reserved for teams with enterprise security budgets.
That is what keeps me working on MCP Observatory: every new test, artifact, and integration makes it a little easier to scan before you trust.
What I Learned
The biggest lesson is that reliability and security are not separate features for agent tools. Both depend on understanding what a server promised, what it actually did, and whether that behavior changed.
I also learned that adoption is part of the engineering problem. A sophisticated security check that cannot run locally or in CI will be ignored. The best workflow is one command to try, structured evidence to inspect, and a clear path to automation.
If you work with MCP servers, I would love feedback on the risks or failure modes you most want Observatory to catch next.
Top comments (0)