Here is a failure no dashboard will show you.
An API your agent calls every morning answers 200 OK. The JSON schema has not changed. Every field is still there. But 42 of the 50 products now have "price": null, and updated_at has not moved in months.
Your uptime check is green. Your schema diff is green. A person looking at this data would notice the empty column in two seconds. Your agent does not stop: it writes a confident report from what it received, and nobody finds out until a customer does.
Status codes say the server answered. Schemas say the shape is valid. Neither says the data your agent depends on is still there.
Test the data, from the consumer's side
So I built a small tool that treats this as a testing problem. A contract says what your agent needs from a source, and a check tells you when the source stops delivering it:
-
Coverage: at least 90% of items carry a usable price.
null,"",[]and{}do not count. -
Freshness: the newest
updated_atis less than 3 days old. - Volume: the list has at least 25 items, so an empty array is a failure and not a success.
- Distinct values: the column is not stuck on one value, like a price of 0 everywhere.
You do not have to write that by hand, and you do not need the provider's cooperation. You put a local proxy between your agent and the source, and it learns the contract from the real calls. It stores shapes, never values.
Try it in a minute
git clone https://github.com/driftprobe/driftprobe && cd driftprobe && npm install
# A demo API, and the proxy in front of it
node examples/demo-api.mjs &
node bin/driftprobe.mjs record --upstream http://127.0.0.1:4010 --port 8787 &
# Your agent calls the proxy instead of the API
curl -s http://127.0.0.1:8787/products > /dev/null
curl -s http://127.0.0.1:8787/products > /dev/null
# Learn a contract from those calls
node bin/driftprobe.mjs learn --observed .driftprobe/observed.jsonl --out-dir contracts
# The source degrades, and still answers 200 OK
curl -s -X POST http://127.0.0.1:4010/__break
curl -s -o /dev/null -w "HTTP %{http_code}\n" http://127.0.0.1:8787/products
Then the check:
$ node bin/driftprobe.mjs check contracts/*.yaml
GET /products
x price: only 16% of 50 items carry a value (contract: at least 90%)
x updated_at: newest value is 289 days old (contract: 3d)
1 contract(s): 0 ok, 1 broken, 0 could not be checked
Exit code 1, so it fails your CI. It also works on MCP tools, including over server-sent events.
How often does this really happen?
I do not know yet, and I could not find anyone who has measured it. So I am measuring it.
The official MCP registry listed 53,434 servers on 31 October 2026. I took a sample of 2,000 remote endpoints that do not declare a required credential, in a stable random order, at most 5 per domain:
- 655 asked for credentials anyway
- 1,115 answered with their tool list
- 770 declared at least one read only tool
- 415 returned data from a read only call with neutral arguments
From those, I picked 100 that return real lists, across 98 domains. I call one read only tool on each, twice a day, for 30 days, and record shapes only. I will publish the results whatever they show, including if almost nothing breaks.
To avoid crying wolf, the analysis only counts a break when the same rule fails on consecutive observations, and I tested the method on simulated servers where nothing changes: at most 2% get flagged.
What it does not do
It does not know if a value is true, only if it is there, fresh and varied. It learns what your agent receives: narrowing that to what your agent actually reads is manual for now (--fields name,price).
If you run LangGraph, quietfail watches behaviour from inside the agent, which is complementary. ToolBench grades how well MCP tools are defined, and FlareCanary watches tool schemas. This one sits outside the agent and looks at the data that comes back.
The code is MIT: github.com/driftprobe/driftprobe. If one of your sources has failed this way, I would like to hear how you found out.
Top comments (2)
The 42-of-50 null prices example is exactly the failure that slips past every status check. I like that the contract is learned from the consumer side, since the agent is the one that knows which fields it actually depends on. One addition that has helped me with web sources specifically: a minimum length check on fetched page text, because a login wall or cookie banner also comes back as a clean 200. Does the check run inline on each call through the proxy, or only in CI against the recorded traffic?
This matches what we see from the provider side. A lot of the scary failures are a clean 200 with the wrong body, and the consumer is the only one who knows which fields actually matter.
One MCP-specific case worth covering: per the spec, a tool that fails usually doesn't fail at the JSON-RPC level. It comes back as a normal successful result with
isError: trueand the error text incontent. So the transport is green, the response shape is valid, and an agent that doesn't check that flag happily treats an error message as data. Does driftprobe's MCP mode countisErrorresults as a coverage miss, or would that need its own rule?