I asked a well-known API for my bounced emails. It answered 200 with five emails, and none of them had bounced. SilentProbe describes what an agent gets back in that case: "HTTP 200 with a parsable body, no exception to catch and no field to branch on."
The study tested 27 vendors' live APIs with 219 deliberately wrong requests, then gave twelve models ordinary tasks against them:
- Across 2,501 public OpenAPI documents, only 15.2% of parameters declare a constraint a machine can check.
- When the constraint was in the schema, a schema validator in front of the APIs returned an honest error 111 times out of 111. It caught 110 of those before the request reached the vendor.
- When it was only in the description, the API failed silently 44 times out of 61.
- Agents noticed the silent failure 12% of the time, and fixed it 0% of the time.
- In 41% of cases they told the user something didn't exist when it did (28% with no hedge). In 12% they made up a figure.
In the authors' words, "constraint form, not vendor identity, predicts honesty". The agents did what agents do with a 200: they believed it.
I've been running agents and fixed checks against two APIs, which I'll call A and B. These are the five ways their HTTP APIs misled my agents, ranked by damage, worst first.
1. It says 200 to a filter it ignored
That was API A, answering GET /emails?limit=5&status=bounced in 161 ms: 3 queued, 2 delivered, and nothing in the response saying the filter was ignored. An agent would hand that list to your user as their bounces.
I'm not the only one. The Colony is a forum and social network for AI agents. In August, one of its agents, ANP2 Network, probed the site's own API. Notification objects there carry a field named is_read, so it tried the obvious filter first: ?is_read=false returned 200 and all 50 rows. The parameter that works is ?unread=true, which returned 2. The same API answered ?sort=NOT_A_SORT with an honest 422. Its conclusion: "Strictness is a property of an individual parameter." In October, Nevermore followed up with GET /claims?status=banana: 200, the same row as without the filter. Nevermore's verdict: "the filter was never read".
2. It accepts a field it doesn't know, then drops it
I sent API A a POST /emails with "unknown_field": true. 200, an id, the email went out. Fine for a made-up field. Now picture an agent typing reply_too instead of reply_to: same 200, and your replies go nowhere.
Zod does this by default: z.object() strips unknown keys, which is how Shubham Shrivastav's API returned 201 "for a request whose main instruction it had thrown away".
3. Its responses break its own spec
API B lets you ask for only the fields you need, with a fields=… query parameter. Its spec also promises that some properties are always there: on a record, name, title and emails are marked required. Ask for a smaller set of fields and those "required" properties are missing from the response.
I ran an automated test of API B against its own spec. It found 9 responses that broke the spec, and every one of the 9 came from a request with fields. An agent that trusts the spec now holds objects that, according to that spec, can't exist.
4. It calls a bad key a bad request
HTTP has separate codes for two separate problems. A 400 Bad Request means "fix your request". A 401 Unauthorized means "fix your credentials". An agent needs to know which one it's facing, because the fixes have nothing in common.
API A gets it right when there's no key at all:
{"statusCode":401,"message":"Missing API Key","name":"missing_api_key"}
Send a wrong key instead, and listing emails or domains returns this:
{"statusCode":400,"message":"API key is invalid","name":"validation_error"}
Try to send an email with the same wrong key and you get a 401, still named validation_error. Same mistake, two status codes depending on the endpoint, and both times the name says the request failed validation.
In my run the model read "API key is invalid", reported it and stopped. The code around it is what breaks: the script the agent writes and the retry logic. That code branches on the status and the name, because that's what they're for. Handle auth errors with a check for 401 and you catch the dead key when sending, but miss it on every list call, where the same key comes back as a 400 and gets treated as a bad request.
5. It answers with someone else's error
I asked my agent to schedule a reminder email on API A, then cancel it. Its first cancel attempt said "Email is not scheduled" while the email sat in the queue, so it dropped to raw HTTP and wrote a short Python script with urllib. The cancel and a GET to check the email both came back with this, as plain text:
403 error code: 1010
The text comes from Cloudflare, which sits in front of API A and turns away requests that have no User-Agent header. urllib doesn't send one by default. The body isn't JSON, has no error name, and doesn't mention the header. API A does document the User-Agent requirement. The error just doesn't point to it.
My agent got past it on the next try. It added 'User-Agent': 'curl/8.7.1' to its headers, passing itself off as curl, and both calls came back 200. A 403 usually means "you're not allowed", and an agent that took this one at face value would have told you it had no permission to cancel its own email.
A 400 is a good day
Multigrid sorts parameters into four kinds: rejected, honoured, dropped and approximated. A dropped parameter is "indistinguishable from honoured without a test designed to tell them apart". Their line: "A 400 is a good day."
For agents, fail loudly. A 400 gives the agent something to fix. A 200 lets it tell your user something that isn't true.
Next week: how to fix all five, and which of those fixes can break your existing clients.
Subscribe to my newsletter so it lands in your inbox.
Thanks for reading :)
Top comments (0)