AI Writes Your API Tests Now. Here's the Bug It Introduces.

Ask an AI to write tests for your API endpoint and it will do something subtly dangerous: it will look at the response your API currently gives and write assertions that lock it in.

If that response is correct — great, you just saved twenty minutes. If that response is a bug — say, your API wrongly returns 200 OK for an invalid payload — the AI will happily generate pm.test("Status code is 200", ...) and your bug is now the expected behaviour, defended by a green test forever.

That's the whole tension of AI-assisted testing in one sentence: AI-generated tests validate what a response is, not what it should be. The specification lives in your head and your docs; the AI only sees the wire.

I use AI test generation daily and I'm not telling you to avoid it — I'm telling you how to use it without it quietly rotting your suite.

What the AI is genuinely great at

Postman's built-in assistant, Postbot, is the one most API testers will meet first (the pattern applies equally to Copilot, Claude, or whatever you use). Its strongest moves:

Boilerplate generation. Open a request, send it once so a response exists, open Scripts → Post-response, and prompt in plain English:

"Add tests for the status code, the content type, and that the response time is under 500 ms."

It writes idiomatic pm.test / pm.expect code straight into the editor. For an entire collection, Generate tests from the collection menu adds baseline tests to every request at once — hours of boilerplate become minutes of review.

Fixing broken scripts. When a test throws an error, Postbot's Fix script analyses it and proposes a correction. For syntax slips and stale JSON paths, this is close to magic.

Exploration. "What's wrong with this request?", "visualise this response as a table", "document this collection" — context-aware answers that used to mean twenty minutes of reading.

Test-case brainstorming. Outside Postman, feeding an OpenAPI spec to an LLM and asking for edge cases is genuinely productive: it will suggest boundary values, weird Unicode, and state transitions you hadn't listed. Treat the output as a checklist to evaluate, not a suite to paste.

Where it fails, specifically

Three failure modes recur:

1. The observed-behaviour trap (the big one, above). Every generated assertion needs review against the spec, not the response. The question is never "does this match what came back?" — the AI already guaranteed that. The question is "is what came back correct?"

2. Happy-path bias. Ask for tests and you'll get success cases. The negative suite — wrong passwords, missing fields, expired tokens, malformed bodies, the requests an attacker sends — still needs a human's adversarial instinct. My rule of thumb: a real endpoint needs five to ten negative/boundary cases per positive case, and the AI will volunteer roughly none of them unprompted.

3. Plausible nonsense. Occasionally the generated assertion is syntactically perfect and semantically empty — checking that an array "exists" rather than that it contains the right things. A suite full of these is worse than a small honest one, because it looks like coverage.

The working protocol

What this adds up to in practice:

1. Learn to write tests by hand first. Not for purity — because review is the whole job now, and you cannot review code you couldn't have written. A test you can't read is a test you can't trust.
2. Generate, then interrogate. For each AI assertion ask: does the spec say this? If the response were wrong in the most likely way, would this catch it?
3. Add the negatives yourself. Budget the time you saved on boilerplate for the adversarial cases the AI skipped.
4. Never paste secrets into prompts. Tokens, customer data, internal URLs — they don't belong in any AI conversation unless your plan contractually protects them. (Postbot on enterprise plans excludes inputs from training; check your tier's terms.)
5. Own the suite. If a generated test fails six months from now, someone has to understand it in ninety seconds. That someone is you.

The honest framing

Used this way, AI does for test authoring what Newman did for test execution: it removes the drudgery and leaves you the judgement. The testers who struggle with AI are the ones who hand over the judgement too — and then wonder why their green suite missed the bug that was green by design.

Machines for repetition, humans for meaning. That division of labour was true before LLMs and it will be true after whatever comes next.

---

Adapted from Chapter 14 of my free, open-source book *API Testing Using Postman: The Practical Guide to Modern API Testing** — 15 chapters from your first request to AI-assisted CI/CD pipelines. Read online, grab the PDF/EPUB, or contribute on GitHub.*

I'm a PhD researcher in Computer Science at Nottingham Trent University working on cybersecurity and AI-assisted security testing. More at imranalmunyeem.com.