Before we dive in, let's define some key terms that will help us understand, in layman's terms, what hypothesis testing is in prediction models.
1. Hypothesis testing: A statistical method of determining whether a specific theory/Argument/relationship in sample data is true or false by evaluating computed evidence. e.g., Does education level affect an employee's salary?
2. Test Statistic: A computed value from sample data that measures how different the sample result is from what is expected under a null hypothesis. e.g t-tests, Z-score, ANOVA.
3. Critical value: The threshold separating the rejection region from the non-rejection region based on alpha. (As shown below)
4. Type I error: This is a (false positive) where you reject the null hypothesis when it's true.
5. Type II error: This is a false Negative where we fail to reject the null hypothesis when it's actually false.
TYPES OF HYPOTHESIS TESTING:
A. Parametric testing:
(Numerical, continuous data types)
- This test assumes that your data has a normal distribution, where the mean, mode, and median are close to each other (classic bell curve). This test works best when this condition is met.
It is also suitable for large datasets with numerical, continuous values.
Core Assumptions
- Normality: Data in the population must be normally distributed.
- Homogeneity of Variance: Variances across groups should be approximately equal.
- Independence: Data points must be collected independently of each other.
Examples include: _One-way Anova_ and _T-test_
B. Non-Parametric testing:
(Categorical datatypes)
- This test is done on data without making any assumptions about the shape or distribution of data. -It works best with ranked or ordinal data and is suitable for a small dataset.
Examples include : _man-whitney_ and _kruskal wallis_
STEPS OF HYPOTHESIS TESTING:
Step 1:
- Define your business question. (what do you want to prove?) e.g "Does education level impact salary?"
Step 2:
- State your hypothesis.
- H₀ (Null hypothesis): "Nothing is happening. There is no difference." e.g "Education level has no impact on Salary amount"
- H₁ (Alternative hypothesis): "Something IS happening. There IS a difference." e.g "At least 1 educational level has an impact on Salary amount"
Step 3:
Choose your significance level denoted by (%/alpha%):
Usually the standard is 5%, i.e 0.05 (this indicates the percentage of data that you expect your model will get wrong when making a prediction.
Step 3:
Calculate the test statistic and evaluate your p_value.
The p_value denotes the percentage by which the results could still occur by pure luck or chance, which means the probability of rejecting the null hypothesis even if it is true.
If p-value ≤ 0.05 = (Reject H₀ ), meaning something real is happening.
If p-value > 0.05 = (Fail to reject H₀) — not enough evidence(chances of luck are high).
Directions in Hypothesis testing:
One - tailed test: It is a directional test; it looks for an effect in only one specific direction(expected to go higher or expected to go lower, i.e <= or >=), e.g (testing a suspected revenue increase of 50%)
Two - tailed test: This is a non-directional test. It looks for any difference regardless of direction. (Is the predicted value equal to or not equal to a known value) e.g (testing whether a class average is equal to the pass mark of 60%)
PARAMETRIC TESTS INCLUDE:
(For numerical, continuous data types)
The following are the most common and relevant parametric tests:
1. t-test:
- This test compares Means.
- This tests if the sample mean of a value is different from a known or predicted value. Use a t-test when your sample size is small (n<30) and when the population variance is known.
Types of T-Tests:
1. One-sample t-test:
- Compares a sample mean to a known population mean. It compares a value to a known benchmark. e.g The average mean score of boys in grade 7 is 60%
Imported from the scipy.stats library module name is ttest_1samp, t
2. Independent t-test:
— Compares the mean of two separate independent groups. That is, a Numerical vs two independent groups.
e.g If the mean score of girls is equal to the mean score of boys in a school.
Imported from scipy.stats library module name ttest_ind.
3. Paired t-test:
- Are the same people different before vs after A change/training, etc.?
- The data must consist of matched pairs. e.g., Netflix does a system upgrade; the users' behaviour is observed before and after the upgrade.
Business question: Did watch time of users increase after system upgrade?
Imported from scipy.stats library module ttest_rel
4. Z -test:
- Works exactly like a t-test but is a parametric test used when the sample size is n >= 30, and the variance is known.
5. Analysis of Variance (ANOVA):
This test compares the means of three or more independent groups simultaneously. e.g., evaluating the performance of three different marketing campaigns. (Influencer campaigns vs. Content-Based vs. Hybrid) on user sign-ups.
There are 2 types " One way Anova and Two-way ANOVA.
NON-PARAMETRIC TESTS INCLUDE:
(For Categorical data types)
1. Chi-Square test:
- Used when working with categorical data. It helps answer questions like: Is a disease caused by smoking? Does a specific gender prefer a specific product?
There are 2 common types of Chi-Square tests
- Chi-square test of independence
- Chi- square goodness of fit test.
a. Chi-Square test of independence:
- Used to determine whether 2 categorical variables are related. e.g does gender affect product preference?
Imported from scipy.stats library module chi2_contingency.
b. Chi-Square test of goodness of fit:
- Used to determine whether observed frequencies match the expected frequencies. e.g "Do customers choose products in the proportions we expected?"
Imported from scipy.stats import chisquare.
2. Tests for Comparing Distributions (Non-Parametric):
Use these when your data violates parametric assumptions like normality.
- Wilcoxon signed-rank test — equivalent of one-sample or paired t-test.
- Mann-Whitney U test — equivalent of an independent t-test.
- Kruskal-Wallis test — equivalent of ANOVA.
SUMMARY CHEATSHEET OF ALL TESTS AND THEIR USE-CASE
| Use Case | Data Type (Y) | Groups / X Variable | Parametric Test | Non-Parametric Test |
|---|---|---|---|---|
| Compare 1 Sample to Population Mean | Continuous | 1 Group vs Constant | One-Sample t-test | Wilcoxon Signed-Rank |
| Compare 2 Independent Groups | Continuous | 2 Independent Groups | Two-Sample t-test | Mann-Whitney U |
| Compare 2 Dependent Groups | Continuous | 2 Paired Groups | Paired t-test | Wilcoxon Signed-Rank |
| Compare 3+ Groups | Continuous | 3+ Groups | ANOVA | Kruskal-Wallis |
| Test Categorical Relationships | Categorical | Categorical Variables | N/A | Chi-Square Test |
| Test Linear Associations | Continuous | Continuous Variable | Pearson Correlation | Spearman Correlation |
AMAZING!!
You read my article to the end; tell me what you think about the contents of this article.
I would love to hear your thoughts :-)
Adios,
Jules
Top comments (0)