I've been working on a product where a surprisingly large part of the work comes down to answering questions.
At the beginning, I thought the difficult part would be getting the data. Connect the different sources, make sure the numbers are available, figure out how to pull everything together, and then the rest should be relatively straightforward.
That assumption didn't last very long.
The more I worked on it, the more I noticed something that felt slightly backwards. Getting an answer wasn't necessarily the end of the research. A lot of the time, getting the answer was what created the next problem.
And for a while, I thought that meant I wasn't doing the research properly.
The way I originally thought research worked
My mental model was pretty simple:
Question → Data → Answer → Done
It makes sense for a lot of basic questions.
How much revenue did we make?
How many users signed up?
Which page gets the most traffic?
You look something up, get the number, and move on.
But most of the questions that actually matter aren't really like that.
Take something simple like this:
Why did signups drop this week?
You look at the data and discover that traffic dropped.
Great. You have an answer.
Except now you have another questions.
Why did traffic drop?
You investigate that and find that most of the decline came from organic traffic.
Okay
Why did organic traffic drop?
Maybe a few pages lost rankings.
Then you start asking why those pages lost rankings.
Was there a technical change? Did the content change? Did search demand changes? Did competitors improve? Was there an indexing problem?
At this point, you've gone several questions away from where you started.
The strange part is that the original answer wasn't useless. It was actually what helped you get to the better question.
I used to think this meant the research was failing
This was probably the biggest mistake in how I thought about research.
If I asked a question and the answer immediately created another question, I felt like I hadn't really solved anything.
I was expecting a clean ending.
What I've started realizing is that real investigation usually doesn't work that way.
It's much closer to debugging.
When something is wrong in a codebase, you don't always know what the bug is before you start looking. You make an assumption, check something, find evidence that supports or disproves it, and then change your approach.
Sometimes the best result from debugging is discovering that the thing you thought was broken isn't actually the problem.
The same thing happens with business questions.
You might think you're investigating a conversion problem and discover that the conversion tracking changed.
You might think traffic is down because of SEO and discover that the bigger issue is a change in how the traffic is being measured.
You might think customers aren't using a feature because they don't want it, and then discover they couldn't figure out how to use it.
Those aren't failures.
They're corrections to the question.
More data didn't solve the problem either
This was another assumption I had.
If the problem is that you don't know what's happening, having access to more information should make things easier.
Sometimes it does.
Sometimes it makes things worse.
You can have analytics, payment data, product events, search data, customer feedback, spreadsheets, dashboards, and a dozen browser tabs open and still not know what you should actually do next.
I think I was confusing having information with making progress.
They're not the same thing.
A dashboard can tell you that something changed.
It doesn't necessarily tell you which explanation is worth investigating.
And having five different sources of data doesn't automatically help if you're not sure whether they're telling you the same story.
That was probably one of the more frustrating things to realize while working on this.
I didn't necessarily need more information.
I needed the information I already had to help narrow down what I should look at next.
The answer can be useful even when it isn't the answer you wanted
This has changed the way I look at research.
I used to think a successful investigation was one where I got the answer I was looking for.
Now I think there are several ways an answer can be useful.
It can confirm what you already suspected.
It can rule something out.
It can reveal something you weren't expecting.
Or it can completely change the question.
The last one might actually be the most valuable.
If I start with:
Why did revenue fall?
and discover that revenue didn't actually fall, but the reporting changed, that's a useful result.
I didn't solve the original problem.
I discovered that the original problem wasn't real.
That saves a lot of time.
I'm starting to think about questions differently
I've also noticed this changing the way I approach problems before I even start looking at the data.
Instead of only asking:
What's the answer?
I'm trying to think about:
What would I do differently depending on the answer?
That small change makes the investigation feel different.
If the answer could change the next thing I investigate, then it's probably worth digging into.
And sometimes the most useful question isn't the one that gives you the most information.
It's the one that eliminates the most possibilities.
I don't have this figured out completely. I'm still learning where that line is, and I'm sure I'll keep getting it wrong.
But I no longer see a new question appearing after an answer as a sign that the research failed.
Sometimes it means the research finally got somewhere.
What I'm taking away from this
The biggest change for me is that I don't think of research as:
Find the answer and finish.
I think of it more as:
Ask → investigate → learn something → adjust the question → investigate again.
That sounds obvious when I write it down, but it wasn't how I was thinking about it when we started building.
And maybe that's the part I find most interesting.
A good answer doesn't always close the investigation.
Sometimes it tells you what you should have been asking in the first place.
Top comments (1)
Hello Glad to see you, I am Kane Lim from Hong Kong. I have over 10 years of development experience. I am writing this because your post was interesting.
This is a very accurate description of research as iterative hypothesis reduction rather than simple information retrieval. I especially agree that the value of an answer can be measured by how many plausible hypotheses it eliminates.
I would formalize this into an evidence graph where each observation updates confidence across competing hypotheses. Bayesian inference or likelihood scoring can prioritize the next query by expected information gain rather than simply collecting more data.
For product analytics, I would also preserve provenance for every metric, including source, transformation, timestamp, and instrumentation version. That makes it much easier to detect when the apparent problem is actually a measurement artifact.
The strongest research systems I have worked with behave much like debugging systems: hypothesis, evidence, falsification, refinement, repeat. Your framing captures that extremely well. I would enjoy exchanging ideas on building research workflows around this model.