The Armchair Expert Problem in Product Validation

This post was originally published on my Substack publication as The Armchair Expert Problem in Product Validation.

I spent three hours chatting with Claude to generate user personas for one of my products, Yahini.

After a few back and forths I ended up with five:

"SEO Manager Anna" who drowns in keyword spreadsheets but never knows which terms to prioritize.

"Content Lead Carlos" who publishes three posts a week but can't connect any of it to business goals.

"Startup Founder Fiona" who knows content matters but has no idea where to start.

"Agency Owner Andre" who runs strategy for eight clients and can't keep their briefs straight.

Each persona came with pain points, goals, objections, and even quotes they might say during a sales call. That made me feel like I understood my market.

Then I posted in a Slack community asking how people actually handle the problem I was solving. The first three responses described workflows completely different from anything Claude had generated. One person said the problem I was solving wasn't even a problem for them. Another said they solved it years ago with a Google Sheet and never thought about it again (the real world Anna).

My AI-generated personas were the product equivalent of an armchair travel agent who "visited" Kyoto through YouTube walking tours and "experienced" Lisbon through food bloggers.

Same as Claude.

AI personas aggregate what's already public and reflect your assumptions back disguised as validation. They tell you what "users" generally want.

But "generally" and "this specific person with budget authority" are different populations entirely.

AI-generated validation feels like research, but it's not. You asked questions and got answers complete with pain points and user quotes. Except those answers came from training data, not real customers who'll pay you.

You prompt Claude for ten personas, pick three that match your assumptions and start building. This means that weeks later, you've got a polished product that nobody uses.

The AI reflected general industry knowledge back at you while missing what actually matters for your specific market. Now you're stuck with features nobody wants, maintaining code for fake problems. Worse, your next decisions build on these false assumptions.

You missed your audience persona because you took an AI simulation instead of talking to humans about their pain.

You can prevent this waste by organizing your research into two distinct categories before you write a start working on your new project.

Experienced builders sort information into two buckets before making product decisions.

Bucket 1: Secondhand Data

This is information that's been aggregated or filtered through someone else's lens. AI personas, market research reports, competitor feature lists, blog posts about "what users want," survey responses about hypothetical future behavior.

Secondhand data tells you what's generally true. It's useful for generating hypotheses and identifying areas worth exploring. But it tells you nothing about what's specifically true for your users, in their context, with their constraints.

Bucket 2: Firsthand Data

This is information you gathered directly from someone who experienced the problem. Recorded interviews where users describe their last workaround, screenshots of the spreadsheet they're currently using to solve this problem.

Firsthand data is specific, behavioral and tied to real constraints. It's messier than secondhand data because real people have contradictory needs and unusual workflows. And the truth about your market is hidden in somewhere in that mess.

Use secondhand data to generate hypotheses and firsthand data to validate them before you build.

I use these five questions in order. Each one builds on the previous answer and forces the user to get specific about behavior instead of vague about preferences:

1. "How do you currently solve this problem?"

2. "Walk me through the last time you did this task."

3. "What's frustrating about how you do it now?"

4. "What have you tried to fix this?"

5. "If this was solved, what would you do differently?"

These questions come from the Five-User Interview Pattern in The 26 Mental Models To Build Better Article. The full framework covers more ground, but these five questions alone will save you from building for fictional users.

There are also red flags you need to pay attention to. Watch for these patterns:

• The user talks about what they "would" do instead of what they "did." Hypothetical future behavior is worthless. Past behavior predicts future behavior.

• The user agrees with your solution before you've described the problem. This means they're being polite, not honest.

• The user's workaround is "I just live with it." This sounds like validation but it's actually disqualification. If the problem doesn't hurt enough to motivate any action, it won't hurt enough to motivate a purchase.

• The user gives you the answer they think you want to hear. Friends and family are the worst validators because they love you. Strangers in online communities are better because they have no reason to protect your feelings.

If 4 out of 5 users describe the same workaround, that's your feature opportunity. Build the thing.

If 4 out of 5 users can't remember the last time they faced this problem, you're solving something that doesn't hurt enough. Go back to hypothesis generation.

If answers are scattered across five different problems, you haven't found your market yet.

You're just talking to people who look similar on paper but have different jobs to be done.

Talking to real humans feels slower than prompting AI to generate personas. Trust me, I get it. But you can compress the validation process into a single focused session.

Instead of scheduling five separate calls over two weeks, find one community where your target users gather and run rapid-fire research.

You can get firsthand data from multiple people in a single sitting.

Copy-paste this template for community research:

POST TITLE: Quick question for [target user type]

I'm researching how [target users] handle [specific problem].

If you've dealt with this recently, I'd love to hear:
1. What's your current workaround?
2. What's the most annoying part?

Not selling anything. Just trying to understand
the problem better before building something nobody needs.

Where to post this:

Identify the Reddit communities, Slack groups, and Discord servers where your industry talks. Check Facebook groups or LinkedIn if your specific demographic spends time there.

Success comes from meeting your target market in their existing habitat. You simply borrow their established audience for thirty minutes to gather the truth.

Why this works:

Because you're asking about current behavior. "What's your current workaround?" gets you truth. "Would you use a tool that does X?" gets you polite speculation.

You're not pitching a solution, so users have no reason to perform enthusiasm. They'll give you raw, unfiltered pain points in their own words. And those words often become the exact copy you use on your landing page later.

I've killed three feature ideas with this template. Each time, the responses made clear that the problem I thought was urgent was actually something people had already solved or didn't care about.

That's 30 minutes of research preventing three weeks of wasted building.

Just like the No List from the Scope Guillotine article, you need a pre-commitment system that prevents you from building before validating.

Create a Notion page, Google Doc, or sticky note with these rules. Keep it visible while you work.

## VALIDATION GATE
## Do not write prompts until you complete these steps.

Before building any feature, I must:

[ ] Talk to 5 real users (not AI personas)
[ ] Document 3+ users describing the same workaround
[ ] Record specific quotes about past behavior
[ ] Confirm the problem is painful enough to pay for

If I cannot check all boxes, I must:
- Post validation template in 2 communities
- Wait 48 hours for responses
- Revise hypothesis based on real feedback

Building without validation is procrastination.

This gate splits your personality into two modes:

1. The Planner sets the rules while rational.

2. The Builder follows them while coding.

When the temptation to skip validation arises, you don't have to make a difficult decision because your Notion page already says No.

AI makes it dangerously easy to feel productive while avoiding the uncomfortable work of talking to real people.

Generating personas feels like research. Analyzing competitor features feels like strategy. Building prototypes feels like progress. But none of this answers the only question that matters: will someone pay for this?

The armchair travel agent could have saved herself a furious client with one phone call. "Tell me about a trip that went wrong. What would you skip next time?"

Five minutes of firsthand data would have revealed that the temple closes for renovation every spring and the restaurant books out three months ahead.

You can save yourself weeks of wasted building with the same approach. Five conversations. Five questions about past behavior. Thirty minutes in a community where your users already gather.

The personas can wait. The real people can't.