It's Monday's leadership review. The dashboard says average order value is up 12% month over month. Everyone nods, the marketing team gets a pat on the back, and the plan for next quarter is "do more of what we did last month."
Three weeks later, finance notices revenue is flat. What happened? One enterprise customer placed a single huge order that dragged the average up, while the number of orders from everyone else quietly fell. The number on the dashboard was correct. The decision built on it was wrong.
That's the uncomfortable truth about analytics: most bad decisions don't come from broken data. They come from correct data, read the wrong way. This guide walks through eight of the most common traps, with plain-English examples and a habit for each one that keeps your team honest. No statistics degree required.
1. Trusting the average when the story is in the spread
Averages are comforting because they turn a messy reality into one number. But an average can be moved by a handful of outliers, and it hides who is behind the number.
Example: "Average time to first response in support is 2 hours." Sounds fine. But if 80% of tickets get answered in 20 minutes and 20% wait two days, you have a serious problem with a specific group of customers that the average completely hides.
The habit: Whenever you see an average, ask for the median (the middle value) and a quick look at the extremes. If the average and the median are far apart, outliers are steering the number, and you should look at them directly.
2. Looking at the total and missing the segments
This one has a fancy name, Simpson's paradox, but the idea is simple: a trend can point one way in the total and the opposite way inside every group that makes up the total.
Example: Your overall trial-to-paid conversion rate drops from 10% to 8%. Panic? Then you split it by channel: conversion from paid ads went up, and conversion from organic went up. The overall rate fell only because a big campaign brought in lots of paid-ad trials, a channel that converts at a lower rate. Nothing is broken; the mix changed.
If you had "fixed" onboarding based on the headline number, you'd have been solving a problem that doesn't exist.
The habit: Before reacting to any big move in a top-line metric, break it down by the two or three dimensions that matter most for your business (channel, plan, region, customer size). Ask: "Did every group change, or did the mix of groups change?"
3. Only studying the winners
Survivorship bias is when you draw conclusions from the things that made it through a filter and ignore the ones that didn't.
Example: A product team surveys their most active users to find out which features drive retention. The answer: power users love the advanced reporting module. So the team invests heavily in it. But the users who churned in their first week (the real retention problem) never made it far enough to see that module. They were never in the survey.
The habit: When you analyze a group, ask "Who is not in this data?" Churned customers, abandoned carts, and trials that never activated often hold the most useful answers.
4. Treating correlation as cause
Two numbers moving together doesn't mean one is causing the other. Often a third factor drives both.
Example: "Customers who use our mobile app churn 40% less. Let's push everyone to download the app!" Maybe. Or maybe your most engaged customers, who were never going to churn, are simply the ones who bother to install the app. Forcing a disengaged customer to install it won't make them engaged.
The habit: When a correlation suggests an action, ask "What else could explain this?" If the decision is expensive, test it: roll the change out to a portion of users and compare against a group that didn't get it. A small experiment beats a confident assumption.
5. Calling a random wobble a trend
Numbers bounce around naturally. Small samples bounce around a lot. Teams that react to every wiggle end up changing strategy weekly based on noise.
Example: Signups were 140 last week and 118 this week, a 16% drop! An urgent Slack thread starts. But look at the last three months and weekly signups have ranged from 110 to 150 the whole time. This week is completely normal.
The habit: Always show a metric with enough history to see its normal range. A good rule of thumb: don't call something a trend until it has held for several periods in a row, or until it moves well outside its usual range. Context turns alarm into "this is fine."
6. Two people, two definitions, one metric name
Ask three people in your company what an "active user" is and you may get three answers: logged in this month, did a key action this week, or has a paid seat. Each is reasonable. Mixing them is a disaster.
Example: Sales reports 5,000 active customers in the board deck. Product reports 3,200 in the roadmap review. The board spends 20 minutes arguing about which team is wrong when both are right, just measuring different things.
The habit: Write down a one-line definition for each of your core metrics and put it right next to the number on the dashboard. "Active user = logged in and created at least one report in the last 30 days." It sounds bureaucratic; it saves hours of arguments.
7. Turning a measure into a target (and getting gamed)
Goodhart's law says that when a measure becomes a target, it stops being a good measure. People are smart. Reward them on a number and they will find the fastest way to move that number, not necessarily the outcome you wanted.
Example: Support is rewarded for closing tickets fast. Ticket close time drops beautifully. Customer satisfaction drops too, because agents are closing tickets before problems are really solved, and customers have to reopen them.
The habit: Pair every target metric with a "counter-metric" that would catch gaming:
| If you target... | Also watch... |
|---|---|
| Ticket close time | Reopen rate, customer satisfaction |
| New signups | Activation rate of those signups |
| Sales bookings | Early churn and discounting |
| Features shipped | Feature adoption, bug reports |
If the target improves and the counter-metric gets worse, you're probably being gamed, usually without anyone meaning harm.
8. Starting with the answer, then finding the data
This is the most human mistake of all. Someone already believes the new pricing page is working, so they go looking for the chart that proves it, and they'll find one. With enough slicing (by week, by region, by plan), almost any story can be "supported."
Example: A founder is convinced a new feature drove the spike in upgrades. They filter to "users who saw the feature" and see a high upgrade rate. They don't check that the feature was only shown to users already on the upgrade path.
The habit: Before you look at the data, write down what result would change your mind. "If upgrades among users who didn't see the feature also rose, the feature isn't the reason." Deciding the test in advance makes it much harder to fool yourself.
Bonus gotchas worth a quick check
A few smaller issues cause surprisingly big misreads:
- Stale data. A dashboard that silently stopped refreshing on Thursday will happily show you Thursday's world on Monday. Show a "last updated" time on every report.
- Incomplete periods. Comparing this month-to-date against all of last month will always look like a drop. Compare like with like.
- Time zones. "Yesterday" in your database may not be "yesterday" for your team, which shifts daily numbers in confusing ways.
- Test accounts and internal users. Your own team's activity can inflate early-stage metrics more than you think. Filter it out.
A simple pre-decision checklist
You don't need a data team to avoid these traps. Before acting on a number, run through five quick questions:
- What's the definition? Does everyone in the room mean the same thing?
- What's normal? Is this move outside the usual range, or just noise?
- What's underneath? Does the change hold across the main segments, or is it a mix shift?
- Who's missing? Are churned users, failed attempts, or other groups left out of this view?
- What else could explain it? And is the decision big enough to justify a small test first?
Teams that make this a habit (some even paste it into their weekly review agenda) catch most of these mistakes before they become expensive. Many BI tools, Draxlr among them, make it easy to save a metric with its definition and break it down by segment in a click, which takes a lot of the friction out of steps 1 and 3. But the checklist works just as well with a spreadsheet and a curious mind.
Key takeaways
- Most bad data-driven decisions come from correct numbers read the wrong way, not broken data.
- Check the median and the segments, not just the average and the total.
- Ask who's missing and what else could explain it before acting on a pattern.
- Give every metric a written definition and a normal range.
- Pair targets with counter-metrics so you notice when a number gets gamed.
- Decide what would change your mind before you go looking at the data.
Over to you
Which of these has bitten your team? I'd love to hear the war stories: the dashboard that looked great right up until it didn't, or the metric everyone defined differently. Share your best "the number was right but we were wrong" moment in the comments, along with any habits or tools your team uses to catch these early.
Top comments (0)