Nobody writes a retrospective about the sprint that went smoothly. We move on, ship the next feature, and forget the details. But the project that hurt? That one stays with you line by line. It becomes the mental checklist you run every time someone says "let's just build it."
I recently came off a project like that. It wasn't a technical disaster the code shipped, the app worked, the demo looked fine. What broke it was process: no real user research, no design principles anyone could point to, and a growing habit of treating AI-generated "research" as a substitute for actually talking to users. Here's what I took away from it.
The pattern I kept seeing
In the AI era, it's tempting for product managers to skip the slow, expensive parts of building a product. Why run five user interviews when you can ask an LLM to "research user needs for a fintech onboarding flow" and get a confident, well-formatted answer in ten seconds?
The problem isn't that AI research is useless it's a decent starting point for hypotheses. The problem is when it quietly replaces the step where you validate those hypotheses against real humans. On this project, feature decisions increasingly traced back to "the research says users want X," where "the research" meant a chat transcript, not a single user conversation. Nobody could tell me who "users" actually referred to.
That's not research. That's a plausible-sounding guess wearing research's clothes.
What actually went wrong
A few concrete anti-patterns emerged, and I think they're worth naming plainly:
- AI output was treated as ground truth, not a hypothesis. Nobody followed up an AI-generated persona or use case with an actual interview, survey, or usability test. It went straight into the backlog as a requirement.
- UI decisions had no design principles behind them. There was no consistent spacing system, no defined interaction patterns, no accessibility baseline. Every screen was designed in isolation, so the app felt like five different products stitched together.
- Feedback loops were one-directional. Features shipped, nobody measured whether they solved the problem they claimed to solve, and the team moved on to the next AI-suggested feature.
- Speed was the only metric that mattered. "We shipped it fast" became the substitute for "we shipped the right thing." Velocity was celebrated; outcomes were rarely discussed.
- Engineers were kept out of the "why." We were handed tickets, not problems. When you don't understand the user problem you're solving, you can't push back when the solution doesn't fit it you just build what's written. None of this was one dramatic failure. It was a hundred small shortcuts that compounded until the product felt directionless, even though everyone individually was working hard.
Why bad projects teach you more than good ones
On a healthy project, good decisions look invisible. Nobody points at the moment someone insisted on a usability test before committing to a flow it just happens, and the project is better for it, quietly, forever.
On a bad project, every missing safeguard becomes visible because you feel its absence. You don't need someone to explain why user research matters when you watch a feature ship, confuse every real user who touches it, and get silently reworked three sprints later. The lesson isn't taught to you it's demonstrated, at cost, in real time.
That's the uncomfortable value of a bad project: it turns abstract principles ("talk to your users," "have a design system," "measure outcomes, not output") into concrete, felt memories. You stop treating them as things senior engineers say in blog posts and start treating them as things you personally never want to relearn.
What I'm doing differently now
- AI research is a first draft, not a final answer. If a PM shares an AI-generated set of user needs, my question is always: "Who did we validate this with?" If the answer is nobody, that's the next step, not a nice-to-have.
- I ask for the "why" before I estimate the "how." A ticket without a user problem attached doesn't get pointed. This isn't about gatekeeping it's about being able to catch a bad idea before I've built it.
- Design principles get written down, even informally. A shared doc of spacing, states, and interaction patterns saves so much rework it's not optional anymore.
- I treat shipped ≠ solved. After launch, I want to know if the feature actually moved the metric it was meant to move. If nobody's tracking it, I flag that as a gap, not a footnote.
- I push back on speed as the only success metric. Fast and wrong isn't a win it's the same failure with better PR. ### The takeaway
Good projects give you confidence. Bad projects give you conviction. Working on something that skipped user research and leaned on AI-generated assumptions instead didn't just teach me what not to do it taught me how expensive it is to skip the boring, human parts of product development, even when the tools make skipping them feel effortless.
AI can accelerate research. It can't replace the moment where a real user tells you your assumption was wrong. If your team is using AI to speed up thinking, make sure you're still checking that thinking against reality because the app I worked on didn't have a technology problem. It had a "we stopped listening to users" problem, and no model can fix that for you.
Have you worked on a project where AI-generated research quietly replaced real user research? I'd love to hear how your team caught it or didn't.
Top comments (0)