I Automated My API Testing With AI — Here's What Actually Worked
I spent the last month trying to get AI to write and maintain my API tests. Not "generate test data" — I wanted the AI to understand my API, write real test suites, and keep them updated when endpoints changed.
Here's what I learned, with real code and real failure stories.
The Problem I Was Trying to Solve
I maintain an API that exposes endpoints for user management, billing, and webhook delivery. It's not massive — about 40 endpoints — but every time I add a feature, I'd spend 30-60 minutes updating tests. Multiply that by 4-5 features a month and it adds up.
I wanted an AI workflow that could:
- Read my OpenAPI spec
- Generate test suites that actually passed
- Update tests when the spec changed
- Catch edge cases I'd miss
Attempt 1: Dump the OpenAPI Spec Into a Chat Window
My first instinct: paste the entire spec YAML into the chat window and say "write tests."
Result: It generated beautiful-looking tests. pytest with fixtures, parameterized inputs, response validation — the works. Every single one failed.
The problem was subtle: it invented response fields that didn't exist. My /users/{id} returns {"id", "email", "created_at"}. The AI-generated test asserted response.json()["name"] — a field that endpoint never returns.
Lesson: LLMs hallucinate API shapes. Always validate generated tests against the actual schema before running them.
Attempt 2: Tool-Augmented Generation
Second try: I wrote a small Python script that feeds the AI structured context:
import json
# Extract schema into a flat structure the model can't misinterpret
with open("openapi.json") as f:
spec = json.load(f)
for path, methods in spec["paths"].items():
for method, details in methods.items():
# Pull out response status codes and schemas
if "responses" in details:
for status, resp in details["responses"].items():
schema = resp.get("content", {}).get(
"application/json", {}
).get("schema", {})
print(f"{method.upper()} {path} → {status}")
print(json.dumps(schema, indent=2))
print("---")
This worked much better. By giving the AI a computed summary rather than raw spec pages, the hallucination rate dropped dramatically. Tests started passing on first run about 70% of the time.
Attempt 3: The "Review Loop"
70% pass rate is better, but I still had to manually fix the other 30%. So I added a feedback loop:
# 1. Generate tests
ai-cli generate-tests --spec openapi.json > test_api_v2.py
# 2. Run them, capture failures
pytest test_api_v2.py -q --tb=short > failures.log 2>&1
# 3. Feed failures back to the AI
ai-cli fix-tests --source test_api_v2.py --failures failures.log > test_api_v3.py
This got me to ~95% pass rate on the second iteration. The remaining 5% were genuinely ambiguous cases — endpoints where the spec didn't fully describe behavior (like rate limit responses or conditional fields).
Key insight: Don't try to generate perfect tests in one shot. Two passes with real error feedback beats one pass with perfect prompting.
What Actually Worked (The Stack I Settled On)
After a month of iteration, here's where I landed:
┌─────────────────┐ ┌──────────────────┐
│ OpenAPI Spec │────▶│ Schema Extractor │
│ (openapi.json) │ │ (Python script) │
└─────────────────┘ └────────┬─────────┘
│ structured context
▼
┌──────────────────┐
│ AI Test Gen │
│ (Prompt + LLM) │
└────────┬─────────┘
│ test file
▼
┌──────────────────┐
│ pytest run │
└────────┬─────────┘
│ failures
▼
┌──────────────────┐
│ AI Fix Pass │◀─── loop until pass
└──────────────────┘
And the prompt template that made the biggest difference:
You are an API testing expert. Given the following endpoint schemas,
generate a pytest test suite.
RULES:
- Only assert fields that exist in the response schema
- For each endpoint, test: 200 OK, 404 Not Found, and at least one
edge case (invalid input, missing auth, etc.)
- Use pytest fixtures for shared setup
- Do NOT import libraries not in requirements.txt
ENDPOINT SCHEMAS:
{structured_context}
The last rule — about imports — saved me more time than anything else. Earlier versions would import random libraries I didn't have installed.
Numbers
| Metric | Manual | AI-Assisted |
|---|---|---|
| Time to write test suite (40 endpoints) | ~4 hours | ~45 minutes |
| % tests passing on first run | ~85% (human errors) | ~70% (hallucinations) |
| % after feedback loop | N/A | ~95% |
| Time to update tests after spec change | ~45 min | ~15 min |
| Edge cases caught | Depends on experience | Surprisingly good |
The time savings are real, but they come from speed of first draft, not elimination of human review. I still review every generated test. I just don't start from a blank file anymore.
What Didn't Work
- Fully autonomous mode. Every attempt to "set it and forget it" produced a test suite that looked good in CI but missed real bugs. AI doesn't know your business logic.
- Test data generation without context. AI-generated test data (names, emails, IDs) often violated database constraints. I ended up using factories with Faker for that.
- Complex assertion chains. Asked AI to write multi-step workflows (create user → update user → verify update → delete user → verify deletion). The models got confused about state across steps. Better to break these into independent tests.
Bottom Line
AI won't replace API testers, but it's an incredible accelerator for the boring parts. The workflow that worked for me:
- Extract structured context from your spec — don't dump raw YAML
- Generate + run + fix in a loop — not one-shot perfection
- Always review what the AI wrote
- Give it strict constraints — what libraries to use, what patterns to follow
I'm now spending less time on test boilerplate and more time thinking about what to test. That's the real win.
What's your experience with AI and testing? Have you found approaches that work better? Let me know in the comments.
Top comments (1)
The part that rings truest here is the failure of "set it and forget it." I'm not even a developer by trade — I build internal tools with AI — and I independently landed on your exact loop: generate, run, feed the real error back, regenerate. It's striking that the feedback pass is where all the value lives, whether you're writing pytest suites or a humble spreadsheet script. The deceptive-looking tests point deserves more attention than it usually gets: output that looks right is more dangerous than output that fails loudly. Curious — of the 45 minutes you ended up with, how much is you reviewing what it wrote? That review step feels like the one part that shouldn't ever be automated away.