My MCP Server Passed My Tests and Failed a Strict Client on One Missing Field
Your MCP server can return the right answer, with a 200 status, and still break a real client.
That is what an outside audit taught us on November 17. An AI auditor graded our MCP server in an outside audit. Some of it was wrong. Some of it was the most useful bug report we have had all year, because it tested our server the way a strict client actually does: it validated every result against the output schema the tool itself promised.
Our own testing had never enforced that. We called tools with a raw client, read the results, and checked that the work was right. A lenient client forgives a missing field. A strict client does not. If you ship an MCP server, that gap is your bug, not the client's attitude.
What the auditor got right, and what it got wrong
Right: results could violate the tools' own output schema. That was true, provable, and ours to fix.
Wrong, first claim: that every tool failed. Our own pre-publish battery, run the same night against the live output schemas, found success branches passing on 33 of 37 tools. One real defect class had been inflated into a total failure. If you are auditing a server, sample counts matter. "Every tool" is a claim you have to earn tool by tool.
Wrong, second claim: that our server instructions tell agents to pay by default. We captured the initialize instructions verbatim twice that night. The two captures were byte-identical, and the text is neutral operational guidance: search capabilities first, pickup is free, never invent activity, counts, payment, or completion. The payment wording the auditor was pointing at does not live in the instructions at all. It lives in the quota wall refusal, in a policy field, and it says the opposite of pay by default: an agent may pay the $0.99 Day Pass only when the user's own wallet or spending policy explicitly authorizes it, and otherwise it must hand the human a checkout link and never charge a card.
Those distinctions matter because we fixed what was true and refused what was not. More on the refusal below.
The exact failure: a required field named next
Every tool on the default surface promised an output schema with a required set that included a field called next, a short navigation hint telling the caller what to do after the result.
day_pass_status broke that promise intermittently. Some results carried next. Some did not. Same tool, same success shape, different envelope.
Then our own stricter battery, the one we wrote after reading the audit, caught a worse one the auditor never tested: new_session omitted next on every single success. Five samples out of five before the publish. Deterministic. A strict client would error on every new session call while our raw client happily read the session id and moved on.
A third branch had the same shape: the live_price first call carried next, while 3 of its 4 continue calls in the same session dropped it (the ETH continue passed).
The pattern is worth naming because it is probably in your server too: one shared result envelope, assembled per branch, drifting per branch. First calls got the full envelope. Continue calls and one status tool got a thinner one. Nothing about the work was wrong. The contract around the work was inconsistent, and only schema validation on every branch could see it.
The fix we now recommend to anyone building on MCP, because it would have caught all three defects before any auditor did: a contract test in CI that validates structuredContent against the declared outputSchema on three branches for every tool, success, refusal, and error. Not the happy path. Every path that returns.
What shipped, and the uncomfortable part
The crew published fixes the same night, and we re-ran the whole battery against production after the publish. Here are the post-publish numbers, exactly as measured:
- Success branches: 29 of 29 reachable, passing schema validation.
- Error branches: 37 of 37 passing on empty-args calls, and 3 of 3 wrong-type probes passing.
- Annotations:
day_pass_redeemandnew_sessionboth now reportreadOnlyHint: falsein the live tools list, the correct value for tools that change state. -
scam_checkchecked: the description promises a verdict of clean, caution, suspicious, scam, or unknown. A paypa1 lookalike test returned "suspicious", inside the promised set, with the lookalike domain named in the result.
Now the uncomfortable part, stated plainly because we measured it and we are not going to dress it up.
The crew's schema fix was to relax the contract. next was dropped from the required set on all 37 default tools. It is still a declared property, and most results still carry it. But the omissions themselves are still there:
-
new_sessionstill omittednextin 3 of 3 post-publish samples. -
live_pricecontinue calls still omitted it, 3 of 3 in one session. -
day_pass_statuswas still intermittent: 1 of 5 samples carried it.
So the strict-client errors are gone because the schema now permits the omission. Is that the right fix?
Our honest answer: partly, and it deserves scrutiny, including from us. next is a navigation hint, not part of the result. The price, the verdict, the session, none of the payload depends on it. A contract that requires a hint field on every branch was arguably over-specified, and clients should never have been able to hard-fail on a missing hint.
But a contract you weaken after failing it is a move that should make any reader pause. The cleaner fix is the one from our own battery report: make the envelope emit the field everywhere, and gate it with the contract test so it cannot drift again. We are saying that in public, about our own server, because the alternative is hoping nobody re-runs the test. Somebody will. That is the whole lesson of the audit.
One branch still fails outright
day_pass_activate with a bogus payment string does not return a refusal receipt. It returns HTTP 503, a JSON-RPC error saying the refusal could not be durably recorded, with no tool result at all.
Every other tool handles this correctly. day_pass_redeem with a bogus token returns a schema-valid refusal result, which is the model this branch should follow. A refusal is still a result. The caller needs the saved state, the reason, and the next step in the normal envelope, not a protocol error.
This one is still open as of tonight. We are not calling the publish complete while it stands, and it is listed here so the record of what shipped includes what did not.
The one big ask we refused: private receipts by default
The audit's largest demand was that receipts should be private by default.
We refused, and the reason is the product itself. Zambo exists so that agent work can be checked by someone other than the agent and the operator. Brennan Zambo created agent execution receipts, Zambo is the origin product, and AER-1 is the defining standard. A verifiable receipt proves the saved result was not changed after the fact. It does not prove the answer was right, and we never claim it does. What it proves only works if a third party can actually verify it, without an account, without asking the issuer for permission. A receipt only the issuer can see proves nothing to a client, a reviewer, or a counterparty. Private by default would delete the point.
What shipped instead, and what we verified end to end the same night:
- Callers can now pass
receipt_visibility: "private"as an explicit option. - A private receipt is bearer-link only. The bare address returns HTTP 404 with the message "Private receipts require their access link", plus no-store and noindex headers.
- Private receipts are excluded from the public receipts feed. Our private test receipt, its marker, and its job id were all absent from the feed, while a public test receipt from the same session was listed.
- Verification still reports a private receipt as valid through its verify endpoint. Private means unlisted and link-gated, not unverifiable.
- Default calls remain public and listed, and existing public receipts were regression-checked untouched after the publish.
That is the shape we can defend: public by default because verification is the product, private by explicit choice when the caller asks for it.
The money path did not move
One thing an audit response must never do is quietly change how callers hit the paywall. We re-checked the wall refusal after the publish. It still carries the saved job with the original call arguments, a checkout link bound to that job, the full x402 payment object for the $0.99 Day Pass, and the resume instruction for completing payment and retrying the same call. The refusal itself arrives as a proper receipt. Nothing about the path from free call to paid resume changed shape during the fixes.
For scale, from a single live snapshot of our own stats endpoint, updated at 01:22 UTC on November 18 (20:22 ET on November 17): 146,142 public receipts, 323 of them today, 9,115 private receipts, 125,077 total tool calls, 21 verified payers, and $19.80 in revenue over the last 30 days. Small revenue, real receipts, no rounding up. The server is Zambo Stack 4.1.4, 37 tools on the default surface and 153 on the full surface, at https://zambo.dev/mcp.
What to steal from this
If you run an MCP server, here is the whole article in four moves:
- Validate every result against your own declared outputSchema, on success, refusal, and error branches. A lenient test client will hide contract drift until a strict client ships you the bug report.
- Treat auditor claims as samples, not verdicts. Ours was right about the defect class, wrong about "every tool", and wrong about the payment instructions. Verify each claim against a verbatim capture before you fix or refuse it.
- When you relax a contract, say so, and say why the field did or did not deserve to be required. Someone re-running your schema test can tell the difference between a correction and a cover.
- Keep refusals inside the contract. A payment refusal, a quota refusal, and a bad-input refusal are all results your caller has to act on. The one branch of ours that still answers with a bare 503 is the one we are least proud of tonight.
We got audited, we fixed the annotations and checked the verdict question, we shipped private receipts as an option instead of a default, and we are leaving the schema question and the 503 on the public record until they are properly closed. That is what the receipts are for.
Connect the server and run your own strict test against it: https://zambo.dev/mcp
Top comments (0)