DEV Community

Cover image for Our API docs told AI agents to do the exact thing that fails
Kaven C
Kaven C

Posted on

Our API docs told AI agents to do the exact thing that fails

#webdev #ai #api #testing

We run a helpdesk that AI agents can operate over MCP: list tickets, read a thread, draft a reply for a human to approve. Last week a real agent paid for a call, chained it into a second call, and hit a wall. What we found underneath was embarrassing enough to write up, because I think half the "agent-ready" APIs out there have the same bug.

The bug

Our create_ticket tool returns this:

{ "ticketId": 47, "customerId": 18, "status": "active" }
Enter fullscreen mode Exit fullscreen mode

And our get_ticket_context tool accepts this:

{ "ticketId": { "type": "string", "minLength": 1 } }
Enter fullscreen mode Exit fullscreen mode

See it? The id comes OUT as a JSON number, because the database hands out integer ids. It goes IN as a string, because someone wrote z.string() in the input schema. So the most natural two-step an agent can perform, take the id from one response and pass it to the next tool, fails validation before the handler ever runs:

ticketId: Expected string, received number
Enter fullscreen mode Exit fullscreen mode

We audited every tool after the first report. All 24 fields that return an id emit numbers. All 14 fields that accept one demanded strings. Of 121 possible tool chains, 107 were broken.

The part that hurts: every input schema's own description said "the id, as returned by list_tickets". The documentation was actively instructing agents into the failure.

Why nobody noticed for months

Humans never chain raw ids; they click. Agents chain constantly, and they do it literally. They take your output and feed it to your input, exactly as documented.

Our test suite never caught it because every test wrapped ids defensively:

const res = await runTool(draftReply, { ticketId: String(ticket.id) })
Enter fullscreen mode Exit fullscreen mode

That String() is the whole story. The tests encoded what a careful human author would type, not what a literal-minded agent actually sends. The suite was green for months while the surface was broken for every real agent.

The fix, and two tempting fixes that are worse

We widened the acceptors. Changing the emitters (returning "47" instead of 47) would silently change the response shape for every existing client, so that was off the table.

But the obvious wideners both have traps:

z.union([z.string(), z.number()]) changes your published JSON Schema to an anyOf. If your tool list is advertised to clients (MCP's tools/list, an OpenAPI doc), that is a contract change every client can see, and some will handle it badly.

z.coerce.string() accepts everything. null becomes "null", undefined becomes "undefined", and a missing id turns from a clean validation error into a confusing "not found" three layers deeper.

What we shipped is a guarded preprocess:

const numericIdToString = (v: unknown) =>
  typeof v === 'number' && Number.isSafeInteger(v) && v > 0 ? String(v) : v

export const idSchema = () => z.preprocess(numericIdToString, z.string().min(1))
Enter fullscreen mode Exit fullscreen mode

Only a positive safe integer is rewritten. Everything else passes through untouched, so null, {}, floats, and negatives still fail with the same messages they always had. And the generated JSON Schema is byte-identical to the old z.string().min(1), so the published contract does not move at all. We verified that with a test that renders both schemas and compares the JSON.

The checklist we use now

  1. Round-trip your own outputs. For every id your API returns, write a test that feeds it back into every input that names the same entity, without any type massaging. No String(), no Number().
  2. Grep your tests for defensive casts around ids. Each one is a place your suite is politely covering for a bug.
  3. Widen acceptors, never emitters. Emitted shapes are contracts.
  4. Diff the generated schema before and after any validator change. "It still validates the same values" and "it advertises the same contract" are different claims.
  5. Read one real response with your own eyes. The paid call that exposed all this also showed us a grammar bug in the response text. Nobody had ever actually read what an agent receives.

Agents are the most literal API consumers you will ever have. They follow your docs exactly, which means your docs finally get tested.

If you want to poke at the surface that taught us this, the agent door is documented at deskcrew.io/agents. Free reads, and the paid actions quote you a price before you commit to anything.

What's the equivalent bug in your API? I'd genuinely like to know if the number-vs-string id split is as common as I suspect.

Top comments (0)