The data speaks for itself is usually offered as a defense. It means: I didn't put my thumb on the scale, I just showed you what's there. It sounds like humility. In practice it's the opposite — it's a claim that the analyst's choices aren't part of the result, when they always are.
Data never speaks. Someone chooses the time window, the bucket size, the baseline, which outliers stay in, and which comparison gets drawn on the same axis. Those choices are what turn raw numbers into a chart anyone can read in five seconds. The phrase the data speaks for itself is what lets those choices pass without anyone — including the analyst — examining them.
Here's what that looks like in practice, and why naming the choice out loud is the actual discipline.
1. Truncated axis
ax.set_ylim(80, 100) # "to make the trend visible"
A bar chart of conversion rate moving from 82% to 86% looks like a rounding error on a 0–100 axis. Truncate the y-axis to 80–100 and the same bars look like a dramatic surge. Neither chart is fabricated — every number on it is correct. The difference is entirely in a choice that's invisible unless someone asks what the axis range is and why.
Truncated axes aren't always wrong — sometimes the full 0–100 range genuinely hides a real, meaningful shift. The point isn't "never truncate." It's that the decision to truncate is doing real interpretive work, and "the data speaks for itself" is exactly the phrase that lets it go unstated.
2. Baseline nobody picked on purpose
df["growth"] = (df["revenue"] - df["revenue"].shift(12)) / df["revenue"].shift(12)
Year-over-year growth looks like a neutral, standard metric. It's a choice: this quarter compared to the same quarter last year, not last quarter, not a trailing average, not the pre-pandemic baseline. If last year had an unusual dip, this year's "growth" is partly an artifact of a low starting point — and the chart won't tell you that, because a percentage doesn't carry its own denominator's history along with it.
3. Outliers that got dropped "for clarity"
df = df[df["order_value"] < df["order_value"].quantile(0.99)] # cuts the long tail
Trimming the top 1% before plotting a distribution is often reasonable — a few extreme values can flatten a histogram into something unreadable. But it's a modeling decision about what counts as signal versus noise, made silently, before the reader ever sees the shape of the full data. If those extreme orders are disproportionately from one customer segment, "clarity" just deleted the most important 1% of the story.
4. Comparison that implies causation without claiming it
"Users who completed onboarding had 3x higher retention."
Looking at the chart on the left, every word of this is true and it will be read as onboarding causes retention by nearly everyone who sees it.
What is left unmentioned are possible confounders, like for instance as shown on the right, the people who complete onboarding are typically also the people motivated enough to complete onboarding — and hence we can never establish for sure that the training caused anything.
The chart didn't lie. The framing did the arguing that implies causation when there may be none.
5. p-value treated as a verdict
stats.ttest_ind(group_a, group_b) # p = 0.03 → "significant" → done
A p-value answers one narrow question — if there were actually no difference between these two groups, how likely would it be to see a difference this big just from random luck? A p-value of 0.03 means "about a 3% chance of seeing this by luck alone if nothing real is going on."
It says nothing about how big the difference is, whether it matters in practice, or whether the study was well-designed. It's a statement about coincidence, not about importance — and the 0.05 cutoff itself is just a convention, not a law of nature.
However, a p-value below 0.05 typically gets treated as an on/off switch: significant, ship it. When in reality, p-value is actually a statement about one specific null hypothesis, under one specific test, on one specific sample — not a verdict on whether the effect is real, large, or durable.
Reporting "p < 0.05" without reporting effect size, sample size, or how many other comparisons were run alongside this one lets a single threshold stand in for a judgment call about how much evidence is enough.
Take the example below of four studies (A, B, C, D) with roughly similar p-values (all < 0.05) but different effect sizes, measured using Cohen's d. Effect size is just "how big is the difference, really" — the gap between the two groups, scaled so it's comparable across studies. A small effect size means the two groups barely differ even if the math calls it "significant"; a large effect size means the difference is big enough to see with the naked eye.
The four studies in the chart above all clear the same p-value < 0.05 bar, but they're not telling the same story: Study D ran on 8,000 people and found a difference so tiny it's practically nothing — with a sample that large, even noise can become "significant." Study A ran on just 40 people and found a difference big enough to matter. Same headline, opposite substance.
That's why the p-value alone can't be the verdict — it only says a difference probably isn't zero, not that the difference is big enough to care about.
Why "just double-check the numbers" isn't the fix
Every example above involves numbers that are individually correct. Fact-checking the arithmetic won't catch any of them, because the distortion isn't in the math — it's in which math got chosen, and what got left off the chart. That's a different kind of review: not "is this number right," but "what would this look like if the axis, baseline, or trimming had been chosen differently, and why wasn't that shown?"
A defensible analysis — the kind that survives a skeptical stakeholder asking "wait, why does it look like that?" — means being able to answer yes to all of:
- The axis range and why it was chosen would survive being stated out loud
- The baseline or comparison point is named, not just implied by the metric's label
- Any trimming, filtering, or exclusion is disclosed next to the result it affects
- Causal language is reserved for claims that actually support causation
- A significance result is reported with effect size and sample size, not p-value alone
None of this is about being more honest in some abstract sense. It's about treating every visualization and every summary statistic as an argument with premises — because that's what it is — and being willing to state the premises instead of letting "the data speaks for itself" stand in for them.
This is the discipline behind SophiArch's Statistics for Data Science course, which teaches distributions, hypothesis testing, and inference specifically through the lens of directing and auditing AI-generated analysis — including catching exactly this kind of framing choice before it reaches a stakeholder.
Top comments (0)