My AI Test Data Nightmare: From Generic to Genuinely Gnarly JSON

Building out our new user profile API has been a blast. Mostly. But then came the part where we needed to test it. We're talking about comprehensive, real-world data, not just [e-mail] and password123. We needed variety, realism, and critically, a whole lot of edge cases.

My initial thought, like any developer in 2026, was, "AI can totally handle this!" I imagined myself throwing a simple prompt at ChatGPT or Copilot and getting a stream of perfectly structured, wildly diverse JSON ready to hammer our endpoints. Oh, how wrong I was.

The Problem: When AI Generates 'Too Perfect' or 'Perfectly Wrong'

We needed to test scenarios like:

• Users with incredibly long bios.
• Users with zero followers, or millions.
• Invalid email formats (e.g., missing '@', multiple '@'s).
• last_login_at dates from the future (oops!).
• Profiles missing optional fields entirely.
• Users with non-ASCII names or emoji in their descriptions.

I started with what felt like reasonable prompts: "Generate 20 JSON user profiles for a social media app. Include varied names, emails, bios, and follower counts." What I got back was… bland. Perfectly valid JSON, sure, but every bio was a generic sentence, follower counts clustered around hundreds, and emails were always [e-mail]. Utterly useless for finding real bugs.

So, I tried to guide it: "Now, include profiles with edge cases. Give me a user with a missing email, one with an invalid email, one with an empty bio, one with a really long bio (over 500 characters), and one with a last_login_at date in the future."

This is where it got frustrating. The AI would often try to comply, but it would either:

1. Be Repetitive: Every "missing email" was "email": null. Every "long bio" was the exact same paragraph copy-pasted.
2. Be Inconsistent: It would invent data types. Our is_verified field is a boolean, but I'd sometimes get "is_verified": "true" (a string!), which our API validation would immediately reject with: expected boolean for 'is_verified', got string.
3. Miss the Point: Asking for an "invalid email" might yield "email": "notanemail" instead of something like "email": "user@[e-mail]" that might bypass a basic regex but fail a more robust one.

I spent a solid 15 hours over three days, mostly re-prompting ChatGPT 4.0, tweaking instructions, trying to make it understand the nuance of an edge case. It felt like I was battling a very polite, very literal robot. It was well-formed, but not well-thought-out for my specific needs.

What Actually Fixed It: The AI-Assisted Faker Strategy

My breakthrough came when I realized I was asking the AI to do too much. AI is brilliant at understanding schemas and generating ideas, but for truly random, deliberately broken, or hyper-specific data, it's not a perfect direct generator. The fix involved leveraging AI for its strengths and then using a programmatic tool for the actual heavy lifting.

Here’s the two-step strategy that finally worked for me:

1. AI as an Edge Case Brainstormer: Instead of asking it to generate the data, I asked ChatGPT to describe edge cases and suggest properties for them. I gave it our API's basic schema and asked:

   "Given this user profile JSON structure, list 15 distinct edge cases for testing. For each edge case, describe the scenario and suggest which fields would be unusual or incorrect, and why. Be specific about data types or expected values."

   This was a game-changer! It gave me excellent descriptions: "User with international characters in name: name field contains non-ASCII characters like 'Björn', '你好'." or "User with empty array for friends list: friends field is [] instead of null or a populated array." This gave me a roadmap of what to test.

2. Faker.js (or Python's Faker) for Precision and Variety: With the AI's suggestions in hand, I wrote a Python script using the Faker library (version 18.10.1). This allowed me to generate a baseline of realistic data, and then programmatically override specific fields for each edge case the AI had described.

   For instance, to generate a user with an invalid email, Faker might give me a valid one. But my script could then explicitly take that valid email string and mangle it based on the AI's ideas:

python
from faker import Faker
import random

faker = Faker()

def generate_base_profile():
return {
"id": faker.uuid4(),
"name": faker.name(),
"email": faker.email(),
"bio": faker.paragraph(nb_sentences=3),
"followers_count": random.randint(0, 100000),
"is_verified": faker.boolean(),
"privacy_level": random.choice(["public", "private", "friends_only"]),
"last_login_at": faker.iso8601(),
"tags": faker.words(nb=random.randint(0, 5))
}

def generate_invalid_email_profile():
profile = generate_base_profile()
profile["email"] = faker.word() + "@" + faker.word() # e.g., 'foo@bar'
if random.random() < 0.5: # 50% chance for an even nastier one
profile["email"] = faker.word() + "@@" + faker.domain_name() # e.g., 'user@@domain.com'
return profile

def generate_long_bio_profile():
profile = generate_base_profile()
profile["bio"] = faker.paragraph(nb_sentences=15, variable_nb_sentences=True)
return profile

... and so on for other edge cases

profiles = []
for _ in range(50): # Generate 50 profiles
choice = random.choice([
generate_base_profile,
generate_invalid_email_profile,
generate_long_bio_profile,
# ... other edge case functions
])
profiles.append(choice())

Now, do something with 'profiles'

This ~150 line Python script became my workhorse. I could mix and match functions, ensuring a genuinely random distribution of both 'good' and 'bad' data. I got the variety I needed for edge cases, and the underlying structure was always correct, which the AI struggled with when trying to generate everything from scratch.

Takeaway: AI as a Consultant, Not a Coder

My big takeaway? AI is an incredible assistant for brainstorming, outlining, and even writing snippets, but for generating truly specific, nuanced, and programmatically validated test data, especially for diverse edge cases, it's better used as a consultant. Let it define the problems and suggest solutions. Then, use traditional coding tools like Faker to implement those solutions with precision and control. It saved me from manually crafting hundreds of JSON objects and gave me much more robust test coverage than I'd have achieved just prompting away.