It started as a personal need
A while back I was working on a project with a few MCP servers wired into it. Every so often something would go subtly wrong. The server hadn't crashed, the logs were clean, but the AI client on the other end just behaved oddly. It would skip a tool, misread a result, or wander off. And every time, I'd lose an hour or two bisecting commits and re-reading the spec, trying to figure out which small thing my server had quietly stopped doing correctly.
After the third or fourth time, I got tired of debugging the same class of problem by hand. What I actually wanted was something that would connect to the server, check it against the spec, and just tell me plainly what broke and where, before it ever reached a client. I couldn't find a tool that did that the way I wanted, so I built one.
What vexyo does
vexyo is a CI-native testing harness for MCP servers. It does two things.
Conformance. It connects to your server exactly like an AI client would and runs 16 rules across initialization, discovery, error semantics, and transport, checking it against the MCP spec (2025-11-25). Every finding cites the exact section of the spec it enforces, so you're never guessing:
✗ discovery/tools-list-available
tools/list failed despite the tools capability being advertised
spec: Server Features §Tools / Listing Tools
15 pass · 1 fail · exit 1
Regression. It records how your tools actually behave (golden sets), then flags three kinds of drift on every commit (schema, behavioral, and coverage), so a change that quietly breaks a working tool fails your build instead of shipping.
It exits non-zero on failure, so it gates CI by default. The GitHub Action prints a job summary you can act on, with the rule, severity, spec citation, and fix, right in the PR. That job summary is exactly the thing I wished I'd had during those debugging sessions.
It works with any language
vexyo talks to your server over its transport (stdio or Streamable HTTP), not your runtime. So it tests MCP servers written in Python, TypeScript, Go, PHP, anything. Point it at a command or an HTTP URL:
import { defineConfig } from '@vexyo/cli/config';
export default defineConfig({
specVersion: '2025-11-25',
target: { transport: 'stdio', command: 'python', args: ['-m', 'my_server'] },
});
Try it
npm i -D @vexyo/cli
npx vexyo init
npx vexyo run
That scaffolds a config, points it at your server, and gives you a first result in a couple of minutes.
- Docs: https://vexyo.dev
- Repo: https://github.com/vexyohq/vexyo
- It's free and open source (Apache-2.0).
It solved my own problem first, and I'm genuinely curious what breaks first for other people. If you run it against your server, I'd love to hear what it caught (or got wrong).
PS. If you try it, tell me what it caught, or what it missed. That's how it gets better. Questions and issues are very welcome.
Top comments (3)
Nice split between conformance and regression. For behavioral golden sets, I’d avoid snapshotting raw results as the primary oracle because timestamps, IDs, ordering, and live data make them brittle. Normalize volatile fields, then assert invariants and metamorphic behavior: pagination completeness, stable error classes, idempotency, schema/result compatibility, and authorization-dependent tool visibility. Add an identity/tenant matrix and planted negative cases, including semantic false-success where the transport returns success but the payload violates the tool contract. Finally bind every baseline to the server artifact, spec version, tool-list digest, policy version, and fixture version. That makes a regression failure explain which contract moved, not merely that JSON changed.
@mads_hansen_27b33ebfee4c9 this is a great breakdown, thank you!! you're right that raw snapshotting is brittle (!), that's why record runs everything through normalizers (timestmps, uuids, hex ids, paths) before writing the golden, and the fix for a noisy diff is adding a normalizer, never regenerating. ordering i only handle partially: goldens are stable-Sorted on write, but there's no normalizer yet for unstable ordering inside the results.
The invariant/metamorphic angle is a level past what i have… today i assert normalized - output equality, not properties like pagination completeness or idempotency. the “semantic false-success" is exactly what i want to catch and don't fully yet, and the identity/tenant matrix + planted negatives… i hadn't thought about at all :// ..
and on binding baselines: partially there tbh. goldens carry the spec version, server name+version, and a full tool-definition snapshot, and failures are classified as schema vs behavioral vs coverage drift … but no tool-list digest, policy version, or fixture version yet, so your "which contract moved" framing is sharper than what i report atm.
mind if i open an issue and quote some of this? it's basically a roadmap :))) thanks!
@mads_hansen_27b33ebfee4c9 fyi github.com/vexyohq/vexyo/issues/4