Don't Choose another Statistical Test Until You Read This
Thank me later...
1. Introduction: The Beginner's Dilemma
Imagine you've collected some data. Maybe it's the heights of students in two different classes, or the scores from two versions of a quiz. You want to compare these two groups and answer a simple question: are they really different, or is it purely chance?
Yes, it is possible to work that out.
So you go online to find a statistical test that can help you answer this. And suddenly, you're staring at a huge list of names: t-test, Mann-Whitney U, ANOVA, Kruskal-Wallis, Wilcoxon... the list keeps going.
Which one do you pick?
Here's the good news: almost every test on that list belongs to one of just two families:
- Parametric tests
- Non-parametric tests
Once you understand what separates these two families, choosing the right test becomes much easier. Pick the wrong one, though, and you might draw the wrong conclusion from your data — which can lead to bad decisions down the line.
Bottom line, by the end of this article, you'll know exactly what makes these two approaches different...
And — more importantly — you'll know which one to reach for the next time you're staring at your own dataset.
2. Before Choosing a Test, Understand the Data
Here's a secret that makes statistics much less scary: it isn't really about memorizing a list of tests.
It's about learning to ask good questions about your data. Once you can answer those questions, the right test almost picks itself.
Think of it like a doctor before writing a prescription. A good doctor doesn't just grab the first medicine on the shelf.
They ask questions first — how old is the patient, what are the symptoms, are there any allergies? The answers guide the decision. Statistics works the same way. Before touching any test, you ask questions about your data.
Here are the four questions that matter most.
Is the data numerical?
Some data comes as numbers you can measure or count, like age, weight, temperature, or the price of a phone. Other data comes as categories or labels, like "red, blue, green" or "yes, no."
A number like height in centimeters behaves very differently from a label like favorite color, and that difference already shapes which tests make sense.
Example: If you're comparing the amount of time (in minutes) two groups of students spend on their phones each day, that's numerical data.
But if you're comparing which social media app each group prefers, that's categorical data.
Is the data normally distributed?
"Normally distributed" sounds technical, but the idea is simple: if you plotted the data on a graph, would it form a smooth, symmetric hump in the middle — like a bell? Think about the heights of adults in a city.
Most people cluster around an average height, with fewer very short or very tall people on each side. That's a bell curve, also called a normal distribution.
Not all data behaves this way. Income is a classic example: most people earn a moderate amount, but a small number of billionaires stretch the graph far to one side. That's not a bell curve — it's lopsided, or "skewed."
Example: Exam scores in a large class often form a bell curve — most students score near the average, with a few scoring very high or very low. But the number of likes on Instagram posts usually doesn't — a handful of posts go viral and get enormous likes, while most get very few. That second pattern is skewed, not normal.
Are there many outliers?
An outlier is a data point that's very different from the rest — the odd one out. A few outliers can distort your results if you're not careful.
Example: Imagine measuring the running speed of 20 athletes, and one of them is recovering from an injury and runs far slower than everyone else.
That one slow time can pull the average down and make the whole group look slower than they really are. Some tests are sensitive to this kind of distortion. Others are built to shrug it off.
How many observations do I have?
This one is about sample size — how much data you actually collected. Ten data points and ten thousand data points don't behave the same way statistically, even if they look similar on the surface.
Example: If you survey 8 friends about their favorite pizza topping, you have a very small sample.
If you survey 8,000 people across the country, you have a much larger one.
Small samples are notoriously shakier and harder to draw firm conclusions from, this affects which of the statistical tools are trustworthy.
Your answers to these four questions — numerical or not, normal or not, outliers or not, small or large sample — determine which family of tests fits your data: parametric or non-parametric. Let's look at exactly what separates the two.
3. What Are Parametric Tests?
In simple terms, a parametric test is a statistical test that makes assumptions about your data before it even starts working.
It assumes your data behaves in a certain, predictable way — and if that assumption is true, the test can give you very precise, powerful results.
Don't worry, we won't get into heavy math here. What matters is understanding the conditions these tests expect to be true. There are four big ones.
Normal distribution
Parametric tests usually expect your data to follow that smooth bell curve we talked about earlier. Think of the heights of students in a school — most cluster around an average, with fewer very short or very tall students. That's the shape parametric tests are built for.
Continuous numerical data
The data are measurable on a scale where numbers can take any value, not just fixed categories. Weight, temperature, and time are good examples — a person can weigh 62.4 kg, not just "light" or "heavy."
This is different from data like eye color, which only fits into fixed labels.
Equal variance
This one just means that when you're comparing two or more groups, they should be "spread out" by roughly the same amount.
Picture two classes taking the same test: if one class has scores tightly bunched around 70, and the other class has scores wildly ranging from 20 to 100, their spreads are very different.
Parametric tests work best when the spread — not just the average — looks similar across groups.
Independent observations
Each data point should stand on its own, without influencing or being influenced by another. For example, if you're measuring the exam scores of 30 different students, one student's score shouldn't affect another's.
But... if you accidentally measured the same student's score twice, that would break independence.
A simple analogy
Think of parametric tests like driving on a well-paved highway. When the road is smooth, straight, and free of potholes, your car can travel fast and get you there efficiently.
Parametric tests work the same way: when your data meets these assumptions — normal, numerical, equally spread, independent — the test runs smoothly and gives you sharp, reliable answers.
So, what happens when the road isn't smooth?
What if your data is bumpy, skewed, or full of surprises?
That's where the next family of tests comes in.
4. Examples of Parametric Tests
Now that you know what parametric tests expect from your data, let's meet the actual tests themselves.
These are the tools you'll reach for once your data checks the boxes we covered earlier — normal, numerical, similarly spread, and independent.
One-sample t-test
This test compares the average of a single group against a known or expected value. Imagine a phone company claims its batteries last 20 hours.
You test 15 phones and want to know if your results genuinely support or contradict that 20-hour claim. A one-sample t-test answers exactly that.
Independent t-test
This test compares the averages of two separate, unrelated groups. Suppose you want to know if students who study with music score differently on a test than students who study in silence.
Since these are two different groups of people, entirely independent of each other, an independent t-test is the right tool.
Paired t-test
This test compares two sets of measurements taken from the same group, usually before and after something happens.
Picture measuring the resting heart rate of 10 people before they start a fitness program, and then again after 8 weeks of training.
Since it's the same 10 people measured twice, a paired t-test checks whether the program made a real difference.
ANOVA (Analysis of Variance)
This test compares the averages of three or more groups at once.
Imagine a teacher wants to compare test scores across three different classes, each taught with a different teaching method.
Instead of running multiple t-tests, ANOVA compares all three groups in a single, cleaner test.
Pearson correlation
This test measures how strongly two numerical variables move together. Think about the relationship between hours spent studying and exam scores — does more study time tend to go with higher scores?
Pearson correlation gives you a number that tells you how strong (or weak) that relationship really is.
Each of these tests plays a slightly different role, but they all lean on the same foundation: the assumptions of normality, numerical data, similar spread, and independence.
When those assumptions hold, these tests are precise and powerful. When they don't, it's time to look at the non-parametric alternatives.
5. What Are Non-Parametric Tests?
A non-parametric test is a statistical test that makes far fewer assumptions about your data. It doesn't demand a smooth bell curve, it doesn't require perfectly numerical data, and it's much more forgiving of messiness.
Because of this flexibility, non-parametric tests can handle situations where parametric tests would struggle or give misleading results.
Let's look at exactly when these tests shine.
Skewed data
Remember the income example from earlier — most people earning a moderate amount, with a small number of billionaires stretching the graph to one side? That kind of lopsided data doesn't fit a bell curve, so a parametric test could easily be misled by it.
Non-parametric tests don't rely on that bell-curve shape, so they handle skewed data comfortably.
Ordinal data
Ordinal data is data that has a clear order, but the gaps between values aren't necessarily equal. Think of a customer satisfaction survey with options like "unhappy," "neutral," "happy," and "very happy."
You know "very happy" ranks above "happy," but you can't say the emotional distance between "unhappy" and "neutral" is exactly the same as between "happy" and "very happy."
Parametric tests are not built for this kind of ranked-but-not-measured data.
Non-parametric tests are exactly built for this — they compare the order and ranking of values instead of assuming precise numerical gaps between them.
Many outliers
Imagine measuring the running times of 20 athletes in a race, and one of them is recovering from an injury and finishes far slower than everyone else.
That one slow time can drag down the group's average and make the whole team look slower than they really are.
In situations like this, where a few extreme values could distort the results, non-parametric tests are far less sensitive to those outliers, because many of them work with the rank of the data (first, second, third...) rather than the exact numbers themselves.
Small sample size
When you only have a handful of data points — say, 6 or 7 observations — it becomes very hard to tell whether that data actually forms a bell curve or not.
There simply isn't enough information to be confident either way.
Say you are testing a new recipe on just 5 taste-testers and rating how much they liked it out of 10. With only 5 scores, there's no way to check whether ratings in general follow a bell-curve pattern — you simply don't have enough taste-testers to see that shape emerge..
In these cases, non-parametric tests are a safer choice, since they don't gamble on an assumption you can't properly verify.
A simple analogy
If parametric tests are like driving on a smooth, well-paved highway, non-parametric tests are like switching to a four-wheel-drive vehicle.
When the road gets rough
— full of bumps, uneven patches, and unpredictable turns
— a four-wheel-drive doesn't need a perfect road to get you where you're going. It's built to handle the mess.
That there is exactly what non-parametric tests do for messy, skewed, small, or outlier-heavy data: they still get you a reliable answer, even when conditions aren't ideal.
6. Examples of Non-Parametric Tests
Just like their parametric cousins, non-parametric tests have their own toolbox.
These are the go-to tests once you've decided your data is skewed, ordinal, full of outliers, or too small to trust a bell-curve assumption.
Mann-Whitney U
This test compares two separate, unrelated groups — the non-parametric version of the independent t-test.
Take comparing the tips earned by waiters at two different restaurants.
If the tip amounts are skewed (a few huge tips throwing off the pattern), Mann-Whitney U compares the two restaurants using rankings instead of raw averages, making it far less thrown off by those extreme tips.
Wilcoxon Signed-Rank
This test compares two sets of measurements from the same group, usually before and after something happens — the non-parametric version of the paired t-test.
Let's tend to rating your mood out of 10 before and after a two-week meditation habit.
Since mood ratings are ordinal (you can't say the emotional gap between a 6 and a 7 is exactly equal to the gap between a 9 and a 10), Wilcoxon Signed-Rank is designed to handle exactly this kind of data.
Kruskal-Wallis
This test compares three or more groups at once. It is the non-parametric version of ANOVA.
To compare customer satisfaction ratings you may rate as follows; ("poor," "average," "good," "excellent") across three different coffee shops.
Considering that satisfaction ratings are ordinal, not precisely numerical, Kruskal-Wallis lets you compare all three shops fairly.
Spearman correlation
This test measures how strongly two variables move together (relate to one another), without assuming their relationship is a perfectly straight-line or that the data is normally distributed.
The non-parametric version of Pearson correlation.
It's worth pointing out here that Pearson and Spearman are often mixed up, but they are genuinely different tests: Pearson measures a straight-line relationship between two normally distributed numerical variables, while Spearman measures whether two variables tend to rise and fall together in rank order, regardless of shape.
Think about the relationship between a movie's audience rating and its box office ranking. Spearman correlation checks whether higher-rated movies tend to rank higher at the box office, based on order of ranking rather than exact numbers.
Chi-square
Unlike the other tests on this list, Chi-square doesn't compare averages or rankings — it checks whether two categories tend to show up together more (or less) often than you'd expect if they had nothing to do with each other (commonly referred to as as: by chance).
Imagine surveying people on their favorite streaming service (Netflix, Disney+, or Amazon Prime) and their age groups (teen, adult, senior).
If teens overwhelmingly pick Netflix while seniors are spread evenly across all three, that's a sign the two categories are connected.
Chi-square puts a number on that connection and tells you whether streaming preference and age group are genuinely linked to one another, or whether the pattern you're seeing could easily have happened just by random luck in your sample — the way flipping a coin 10 times might give you 9 heads even though the coin is perfectly fair.
Chi-square is a bit of a special case in this family: because it works with categories rather than numbers, it doesn't have a direct parametric "partner" in the way the other tests do.
Categorical data simply doesn't fit the numerical, bell-curve assumptions parametric tests are built around, so Chi-square exists specifically to fill that gap.
Notice a pattern here: nearly every non-parametric test has a parametric "partner" that it stands in for when the data doesn't meet parametric assumptions. Here's a quick reference table so you can see those pairs at a glance:
| Parametric test | Non-parametric equivalent | Used for |
|---|---|---|
| One-sample t-test | Wilcoxon signed-rank (one-sample) | Comparing one group to a known value |
| Independent t-test | Mann-Whitney U | Comparing two separate groups |
| Paired t-test | Wilcoxon Signed-Rank (paired) | Comparing the same group, before vs. after |
| ANOVA | Kruskal-Wallis | Comparing three or more groups |
| Pearson correlation | Spearman correlation | Measuring how two variables move together |
| — (no numerical partner) | Chi-square | Checking a relationship between two categories |
Knowing these pairs is useful in theory, but in practice, it's exactly where beginners tend to slip up — picking the wrong test for their data, or skipping the checks that would have pointed them the other way.
That's what we'll untangle next.
7. Side-by-Side Comparison
At this point, you've seen both families up close. Here's a quick table that puts everything side by side, so you can compare them at a glance.
| Feature | Parametric | Non-parametric |
|---|---|---|
| Data assumptions | More assumptions required | Fewer assumptions required |
| Normal distribution | Usually required | Not required |
| Outliers | Sensitive | More robust |
| Ordinal data | Poor choice | Good choice |
| Statistical power | Usually higher | Usually lower |
| Flexibility | Less | More |
Let's ground each row in something real, so that the table is not just an abstract set of words.
Data assumptions: Parametric tests demand more boxes checked before you start testing: normal distribution, numerical data, similar spread, independence.
Non-parametric tests ask for much less, which is why they are the often safer default when you're unsure about your data.
Normal distribution: Think back to adult's heights (a natural bell curve) versus Instagram likes (a handful of viral posts skewing everything). Parametric tests are built for the first case, non-parametric tests handle the latter more comfortably.
Outliers: Recall the injured athlete whose slow time dragged the group average down? Well, a parametric test, which relies on averages, feels that distortion directly.
Instead, a non-parametric test, which often works with rankings barely notices it.
Ordinal data: Go back to the satisfaction survey — "unhappy," "neutral," "happy," "very happy." There's no way to say the gaps between these labels are mathematically equal, so parametric tests struggle here, while non-parametric tests handle ranked labels naturally.
Statistical power: "Statistical power" means how good a test is at detecting a real difference when one actually exists.
When your data genuinely meets the parametric assumptions, parametric tests squeeze more insight out of the same amount of data — like a well-tuned engine getting more mileage out of the same tank of fuel.
Non-parametric tests are a bit less efficient in the specific sense, that they trade off some of that power for being more forgiving of messy data.
Flexibility: Parametric tests are precise but rigid, they want the "well-paved road" conditions.
Non-parametric tests are the four-wheel-drive: less demanding about the terrain, and able to handle a wider range of real-world data straight out of the box.
The takeaway: parametric tests reward you with more precision if your data cooperates. Non-parametric tests trade in a little of that precision for the ability to handle messier, more unpredictable, real-world data.
8. Which Test Should You Choose?
By now you've got all the pieces — the assumptions, the tests, the trade-offs. Let's turn that into something you can actually use: a simple decision path you can walk through with any dataset.
Is your data numerical (measured, not categories)?
│
├── No → Use Non-parametric (e.g., Chi-square for categories)
│
└── Yes
│
Is it approximately normally distributed,
with no major outliers?
│
├── Yes → Use Parametric
│ (e.g., t-test, ANOVA, Pearson)
│
└── No → Use Non-parametric
(e.g., Mann-Whitney U, Kruskal-Wallis, Spearman)
Let's walk through this with a real example. Say you've collected the delivery times (in minutes), for two food delivery apps and you want to know if one app is genuinely faster than the other.
Step 1: Is your data numerical? Yes — delivery time in minutes is a measurable number, not a category.
So you move down the "Yes" branch.
Step 2: Is it approximately normally distributed, with no major outliers? Here's where you'd actually plot the data or check it in Python.
Delivery times are notorious for having outliers — a driver stuck in unexpected traffic, or a restaurant running late on one order. If your data has a few of these extreme delays, the answer is "No."
So you'd land on Non-parametric — in this case, the Mann-Whitney U test, since you're comparing two independent groups (two different apps).
If, instead, your delivery times were tightly clustered with no major outliers, you'd land on Parametric — reach for an independent t-test instead.
This is the real value of the decision tree: it turns a confusing wall of test names into a short series of yes-or-no questions you can answer just by looking at your data.
9. How Python Helps
Here's something important to understand: Python doesn't decide which test to use — you do. Python's job is much simpler.
Once you've walked through the decision tree and picked a test, Python runs the actual calculation for you, in a fraction of a second, without any risk of arithmetic mistakes.
Think of it like a calculator in an exam.
The calculator doesn't decide whether you need to add, subtract, or multiply — you figure that out by reading the question.
The calculator just executes the math accurately once you've made that call.
Python plays exactly that role for statistics.
A few libraries do most of the heavy lifting:
pandas — this is how you load, organize, and clean your data before testing anything. If your delivery time data is sitting in a spreadsheet, pandas reads it into a table you can work with in Python.
scipy.stats — this library contains the actual statistical tests themselves, both parametric and non-parametric. Once you've decided which test fits your data, this is where you call it.
statsmodels — this library goes a step further, offering more detailed statistical modeling and output, useful once you want deeper insight beyond a single test result.
Let's revisit the food delivery example from before. Say you've already walked through the decision tree and landed on Mann-Whitney U, because your delivery times had outliers. Here's roughly what that looks like in Python:
import pandas as pd
from scipy import stats
# Load delivery times for both apps
data = pd.read_csv("delivery_times.csv")
app_a_times = data["app_a"]
app_b_times = data["app_b"]
# Run the Mann-Whitney U test
result = stats.mannwhitneyu(app_a_times, app_b_times)
print(result)
Notice what happened here: you made every important decision — which columns to compare, and which test fits the data.
Python's only job was to crunch the numbers instantly and hand back a result. That division of labor never changes, no matter which test you're running: you bring the judgment, Python brings the speed.
10. A Real Data Science Example
Let's pull everything together with one more walkthrough, this time from a workplace setting you might run into in an actual data science job.
Imagine a company rolls out a new training program and wants to know: did this program actually improve employee productivity, or did nothing really change?
To find out, they measure productivity scores (say, units completed per day) for two groups — employees who went through the training, and employees who didn't.
Here's where the theory from earlier sections turns into an actual decision.
If the productivity scores are approximately normal, most employees clustered around a typical output, with a natural, even spread of a few doing slightly better or worse — then the data meets the conditions parametric tests expect.
In that case, the company would reach for an independent t-test, comparing the average productivity of the trained group against the untrained group.
If the scores are highly skewed instead — say, a handful of employees who were already top performers score dramatically higher than everyone else, stretching the data far to one side — then a t-test's reliance on the average would be misleading.
In this case, the company would use a Mann-Whitney U test instead, comparing the two groups by rank rather than raw averages, so that those few extreme scores don't distort the conclusion.
Notice that the business question never changed — "did the training work?" ... but, the right tool did change... hinging on what the data distribution actually looked like once collected.
This is exactly the discipline from Section 2 in action: check the data first, then let that answer point you to the test.
Skip that step, and you risk telling your company the training worked when it didn't, or vice versa — either way, a mistake with real consequences for budgets and decisions.
11. Common Beginner Mistakes
Even with the decision tree in hand, it's easy to trip up when you're just starting out. Here are five mistakes that show up again and again — and why each one matters.
Using a t-test on ordinal survey data
If you ran a customer survey with responses like "unhappy," "neutral," "happy," and "very happy," and you convert these to 1, 2, 3, 4 just to run a t-test.
The problem is that a t-test assumes equal distances between values — but there's no guarantee the emotional jump from "unhappy" to "neutral" is the same size as the jump from "happy" to "very happy."
Running a t-test here treats ranked opinions as if they were precise measurements, for which they are not.
Ignoring outliers
Picture analyzing employee commute times, where nearly everyone takes 20–40 minutes, except for one person who commutes 3 hours from another city.
If you run a parametric test without noticing that outlier, it can drag the average and distort your results, making the whole team look like they have unusually long commutes when really it's just one unusual case.
Never checking for normality
Suppose you're comparing salaries between two departments. Salaries are famous for being skewed — a few high earners can pull the whole picture out of shape.
If you jump straight to a t-test without first checking whether the data even resembles a bell curve, you might get a confident-looking result that's actually misleading.
Another way to phrase that is hallucination.
Assuming larger samples always solve everything
It's tempting to think "if I just collect more data, my problems go away."
Okay, imagine surveying 10,000 people about their favorite ice cream flavor, when what you actually recorded was flavor names — that's still categorical data, no matter how many people you ask.
A bigger sample doesn't turn categories into numbers, or fix data that was skewed to begin with. Sample size helps with some issues, but not all of them.
Choosing tests because someone on YouTube used them
It's common to watch a tutorial where someone runs an ANOVA on their dataset, then you quickly replicate them and apply the exact same test to your own data — without first checking if your data actually meets the same conditions theirs did.
A tutorial creator's dataset of clean, normally distributed lab measurements is not the same as your dataset of messy, real-world customer feedback.
There is no harm in copying other's tests. The handicap is doing so without applying the reasoning behind it; as that is how avoidable mistakes creep in.
The common thread in all five mistakes? Skipping the questions from Section 2.
Every one of these errors happens when someone reaches for a test before checking what their data actually looks like.
12. Key Takeaways
We've covered a lot of ground — assumptions, tests, decision trees, and the mistakes that trip up even careful beginners.
Here are five points worth carrying with you long after you close this article.
1. Statistics starts with understanding your data, not memorizing tests. Every example in this article — heights, incomes, delivery times, salaries — led with the same first move: look closely at what the data actually looks like before reaching for any formula.
2. Parametric tests consider multiple assumptions. They expect a bell curve, numerical data, similar spread, and independence — and in exchange for those demands, they hand back sharp, precise answers when your data actually meets them.
3. Non-parametric tests make fewer assumptions. They're the four-wheel-drive of the statistics world — built to handle skewed data, ordinal rankings, outliers, and small samples without needing ideal conditions first.
4. No test is "better" than another — best is relative. See it as the one test that best fits your data (not-rank). A Mann-Whitney U test isn't a "downgrade" from a t-test, any more than a 4x4 is a downgrade from a sports car. They are all built for different terrain.
5. Python is the tool — the statistician is still you.
Whether it's pandas, scipy.stats, or statsmodels, the code only runs the calculation you've already opted for. The judgment — which questions to ask, which test fits — is a decision no library can make for you.
Optional Bonus: Python Quick-Reference Cheat Sheet
Keep this table handy the next time you sit down with a dataset in Python.
| Goal | Parametric | Non-parametric |
|---|---|---|
| Compare two groups | t-test | Mann-Whitney U |
| Compare paired observations | Paired t-test | Wilcoxon |
| Compare three or more groups | ANOVA | Kruskal-Wallis |
| Correlation | Pearson | Spearman |
| Relationship between categories | — (no numerical partner) | Chi-square |
And remember: this table only helps if you've already answered the real question — what does your data actually look like?
That's where every good analysis starts, and it's the one habit worth taking away from this whole article.
Thanks for sticking with this one all the way through. Statistics can feel like a maze when you're starting out, and taking the time to actually understand why these tests work the way they do, puts you ahead of a lot of people who just duplicate code without checking in or asking why.
If this article helped things click for you, consider sharing it with a fellow data analyst Python learner who's stuck staring at the same maze of a myriad of test types and assumptions you once were.
Top comments (0)