DEV Community

Samuel Mwai
Samuel Mwai

Posted on

# Statistics for Data Science

#datascience #learning #data

Introduction

Statistics is one of the foundations of data science. Data scientists work with large amounts of information, but simply having data is not enough. The real challenge is understanding what the data means, identifying patterns, measuring uncertainty, and using evidence to make reliable decisions.

Statistics provides the mathematical tools needed to do this. It helps a data scientist summarize a dataset, understand how variables behave, identify unusual observations, measure relationships, test assumptions, and make conclusions about populations using samples. Probability is also closely connected to statistics because real-world data contains uncertainty and randomness.

A useful way to think about the role of statistics in data science is:

Collect data → Explore data → Summarize data → Analyze relationships → Test assumptions → Make conclusions → Build models

1. What Is Statistics?

Statistics is the study of how to collect, organize, analyze, interpret, and communicate data.

Suppose a company has information about 10,000 customers. The company might want to know:

  • What is the average amount customers spend?
  • What percentage of customers are likely to leave?
  • Which customer group spends the most?
  • Is income related to spending?
  • Did a new marketing campaign increase sales?
  • How confident are we in our conclusions?

Statistics provides methods for answering these questions.

In data science, statistics is particularly important because data scientists often work with samples rather than complete populations. Inferential statistics helps use sample information to draw conclusions about the larger population.

2. Why Statistics Is Important in Data Science

Statistics is used throughout the data science workflow.

Exploratory Data Analysis

Statistics helps us understand a dataset before building a machine learning model.

For example:

df.describe()
Enter fullscreen mode Exit fullscreen mode

can provide:

  • Count
  • Mean
  • Standard deviation
  • Minimum
  • Quartiles
  • Maximum

Feature Selection

Statistical techniques can help determine whether variables have useful relationships with a target variable.

Machine Learning

Many machine learning techniques have statistical foundations. Understanding distributions, probability, sampling, variance, and estimation helps data scientists understand what models are doing rather than treating them as black boxes.

Experimentation

Statistics is essential for A/B testing and determining whether an observed difference is likely to represent a real effect rather than random variation.

Decision Making

Businesses can use statistical evidence to make decisions about customers, products, pricing, marketing, finance, and operations.

3. Types of Statistics

Statistics can broadly be divided into two major areas:

                    STATISTICS
                        │
            ┌───────────┴───────────┐
            │                       │
      DESCRIPTIVE              INFERENTIAL
      STATISTICS                STATISTICS
            │                       │
      Describe data          Draw conclusions
                              from samples
Enter fullscreen mode Exit fullscreen mode

Descriptive Statistics

Descriptive statistics summarizes the data that has been collected.

Examples include:

  • Mean
  • Median
  • Mode
  • Range
  • Variance
  • Standard deviation
  • Percentages
  • Frequencies
  • Quartiles

Descriptive statistics focuses on describing the observed data rather than making claims about data outside the dataset.

Inferential Statistics

Inferential statistics uses sample data to make conclusions about a larger population.

For example, suppose a company has 100,000 customers but surveys only 1,000 of them.

The sample can be analyzed to estimate characteristics of the larger customer population.

Inferential statistics includes:

  • Confidence intervals
  • Hypothesis testing
  • t-tests
  • Chi-square tests
  • ANOVA
  • Regression analysis

4. Population and Sample

Two important concepts in statistics are population and sample.

Population

A population is the complete group we are interested in studying.

For example:

All customers of a company.

Sample

A sample is a smaller subset selected from the population.

For example:

1,000 customers selected from the company's 100,000 customers.

POPULATION
100,000 customers
        │
        │ Sampling
        ↓
SAMPLE
1,000 customers
Enter fullscreen mode Exit fullscreen mode

Studying an entire population can be expensive, slow, or impossible. Therefore, data scientists frequently work with samples and use statistical methods to make inferences about the wider population.

5. Measures of Central Tendency

Central tendency describes the center or typical value of a dataset.

The three most common measures are:

  • Mean
  • Median
  • Mode

Mean

The mean is the arithmetic average.

For example:

10, 20, 30, 40, 50
Enter fullscreen mode Exit fullscreen mode

The mean is:

(10 + 20 + 30 + 40 + 50) / 5 = 30
Enter fullscreen mode Exit fullscreen mode

In Python:

df["salary"].mean()
Enter fullscreen mode Exit fullscreen mode

The mean is useful when the data is reasonably balanced, but it can be strongly affected by extreme values.

Median

The median is the middle value after the observations have been arranged in order.

For example:

10, 20, 30, 40, 50
Enter fullscreen mode Exit fullscreen mode

The median is:

30
Enter fullscreen mode Exit fullscreen mode

For an even number of observations:

10, 20, 30, 40
Enter fullscreen mode Exit fullscreen mode

the median is:

(20 + 30) / 2 = 25
Enter fullscreen mode Exit fullscreen mode

The median is often more appropriate than the mean when data contains extreme outliers.

For example, income data can be heavily skewed because a small number of people may earn extremely high incomes.

Mode

The mode is the most frequently occurring value.

Example:

10, 20, 20, 30, 40
Enter fullscreen mode Exit fullscreen mode

The mode is:

20
Enter fullscreen mode Exit fullscreen mode

Mode is particularly useful for categorical data.

For example, if the most common customer payment method is:

Mobile Money
Enter fullscreen mode Exit fullscreen mode

then Mobile Money is the mode.

6. Measures of Dispersion

Central tendency tells us where the data is centered, but it does not tell us how spread out the observations are.

Measures of dispersion include:

  • Range
  • Variance
  • Standard deviation
  • Interquartile range

Range

Range is:

Maximum - Minimum
Enter fullscreen mode Exit fullscreen mode

For:

10, 20, 30, 40, 50
Enter fullscreen mode Exit fullscreen mode

the range is:

50 - 10 = 40
Enter fullscreen mode Exit fullscreen mode

Variance

Variance measures how far observations tend to be from the mean.

A larger variance indicates greater spread.

The population variance can be represented as:

σ² = Σ(x - μ)² / N
Enter fullscreen mode Exit fullscreen mode

where:

  • σ² = population variance
  • x = individual observation
  • μ = population mean
  • N = population size

In practice, Python can calculate variance directly:

df["salary"].var()
Enter fullscreen mode Exit fullscreen mode

Standard Deviation

Standard deviation is the square root of variance.

It provides a measure of the typical spread of observations around the mean.

df["salary"].std()
Enter fullscreen mode Exit fullscreen mode

For example, a dataset with salaries tightly clustered around the average will have a smaller standard deviation than a dataset with salaries spread widely across different values.

The distinction between standard deviation and standard error is important: standard deviation describes variability among observations, while standard error describes uncertainty in an estimate such as a sample mean.

7. Percentiles and Quartiles

Percentiles divide data according to position.

For example:

  • 25th percentile
  • 50th percentile
  • 75th percentile

The 50th percentile is the median.

Quartiles divide data into four sections:

Q1 = 25th percentile
Q2 = 50th percentile
Q3 = 75th percentile
Enter fullscreen mode Exit fullscreen mode

The Interquartile Range (IQR) is:

IQR = Q3 - Q1
Enter fullscreen mode Exit fullscreen mode

The IQR is particularly useful for identifying outliers.

8. Understanding Distributions

A distribution describes how values are spread across a dataset.

One of the most important distributions in statistics is the normal distribution.

A normal distribution is:

  • Symmetrical
  • Bell-shaped
  • Centered around its mean
  • Characterized by its mean and standard deviation

In a perfectly normal distribution, the mean, median, and mode are equal.

Other important distributions include:

  • Binomial distribution
  • Poisson distribution
  • Uniform distribution
  • Exponential distribution
  • Bernoulli distribution

Understanding probability distributions helps data scientists model uncertainty and choose appropriate statistical methods.

9. Probability

Probability measures how likely an event is to occur.

It ranges from:

0 → Impossible
1 → Certain
Enter fullscreen mode Exit fullscreen mode

For example, if a fair coin is flipped:

P(Heads) = 0.5
Enter fullscreen mode Exit fullscreen mode

Probability becomes important in data science because real-world datasets contain uncertainty.

For example:

What is the probability that a customer will default on a loan?

or:

What is the probability that a transaction is fraudulent?

Probability forms part of the foundation for many statistical analyses and machine learning methods.

10. Correlation

Correlation measures the strength and direction of a relationship between two variables.

For example:

Hours studied ↑
        ↓
Exam score ↑
Enter fullscreen mode Exit fullscreen mode

If students who study more generally achieve higher scores, the two variables may have a positive correlation.

Correlation coefficients typically range from:

-1 to +1
Enter fullscreen mode Exit fullscreen mode

Positive correlation 

X ↑ → Y ↑
Enter fullscreen mode Exit fullscreen mode

Negative correlation 

X ↑ → Y ↓
Enter fullscreen mode Exit fullscreen mode

No linear correlation 

No clear linear relationship
Enter fullscreen mode Exit fullscreen mode

Correlation is useful, but an important principle is:

Correlation does not necessarily mean causation.

Two variables can be correlated without one directly causing the other.

11. Covariance

Covariance measures how two variables change together.

If two variables tend to increase together, covariance is positive.

If one tends to increase while the other decreases, covariance is negative.

Unlike correlation, covariance does not have a fixed range such as -1 to +1, which makes correlation easier to interpret when comparing relationships.

12. Hypothesis Testing

Hypothesis testing is used to evaluate claims about a population using sample data.

Suppose a company claims that a new training program increases employee productivity.

We could define:

Null hypothesis 

H₀: The training program does not increase productivity.
Enter fullscreen mode Exit fullscreen mode

Alternative hypothesis 

H₁: The training program increases productivity.
Enter fullscreen mode Exit fullscreen mode

We then collect data and perform an appropriate statistical test.

13. P-Values

The p-value is commonly used in hypothesis testing to quantify how compatible the observed data is with the null hypothesis under the assumptions of the test.

A small p-value provides stronger evidence against the null hypothesis.

For example, using a conventional significance level of 0.05:

p < 0.05
Enter fullscreen mode Exit fullscreen mode

may lead researchers to reject the null hypothesis.

However, a p-value should not be interpreted as the probability that the null hypothesis is true. It also does not tell you how large or practically important an effect is.

Statistical significance and practical significance are different concepts.

14. Type I and Type II Errors

Hypothesis testing can produce two important types of errors.

Type I Error

A Type I error occurs when we reject a true null hypothesis.

This is commonly described as a:

False positive

Example:

Concluding that a new medicine works when it actually does not.

Type II Error

A Type II error occurs when we fail to reject a false null hypothesis.

This is commonly described as a:

False negative

Example:

Concluding that a medicine does not work when it actually does.

15. Confidence Intervals

A confidence interval provides a range of plausible values for a population parameter based on sample data.

For example, suppose a survey estimates that average customer spending is:

KES 5,000
Enter fullscreen mode Exit fullscreen mode

with a 95% confidence interval of:

KES 4,700 – KES 5,300
Enter fullscreen mode Exit fullscreen mode

The interval communicates uncertainty around the estimate.

Confidence intervals are an important part of inferential statistics because sample statistics are estimates and therefore have uncertainty.

16. Statistical Tests

Different questions require different statistical tests.

Some common examples include:

Statistical Test Common Use
t-test Compare means
Chi-square test Analyze categorical variables
ANOVA Compare means across multiple groups
Pearson correlation Measure linear association
Regression Model relationships between variables
Mann-Whitney U Compare two groups without assuming normality
Kruskal-Wallis Compare multiple groups without assuming normality

The choice of test depends on factors such as:

  • Data type
  • Number of groups
  • Sample size
  • Distribution
  • Independence
  • Research question
  • Statistical assumptions

17. Regression and Statistics

Regression is another major statistical technique used in data science.

For example, linear regression can model the relationship between an independent variable and a dependent variable.

A simple linear regression equation is:

y = β₀ + β₁x + ε
Enter fullscreen mode Exit fullscreen mode

where:

  • y = dependent variable
  • x = independent variable
  • β₀ = intercept
  • β₁ = coefficient
  • ε = error term

For example, we could investigate whether advertising expenditure is associated with sales.

Advertising Spend → Sales
Enter fullscreen mode Exit fullscreen mode

Regression can help with:

  • Prediction
  • Understanding relationships
  • Estimating effects
  • Forecasting
  • Feature analysis

18. Statistics and Machine Learning

Statistics and machine learning are closely connected.

Consider a classification problem where we want to predict whether a customer will leave a company.

The dataset might contain:

Age
Income
Tenure
Monthly spending
Number of complaints
Customer status
Enter fullscreen mode Exit fullscreen mode

Statistics can help us:

  1. Understand the variables.
  2. Identify missing values.
  3. Detect outliers.
  4. Examine distributions.
  5. Analyze relationships.
  6. Select useful features.
  7. Evaluate model performance.
  8. Quantify uncertainty.

Machine learning then uses algorithms to learn patterns from the data and make predictions.

This is why a data scientist should not only know how to call:

model.fit(X_train, y_train)
Enter fullscreen mode Exit fullscreen mode

but should also understand what the data looks like and what assumptions may affect the model.

19. Statistics in Python

Python provides several libraries for statistical analysis.

Pandas

Pandas is commonly used for data manipulation and descriptive statistics.

import pandas as pd

df.describe()
Enter fullscreen mode Exit fullscreen mode

You can calculate individual statistics:

df["income"].mean()
df["income"].median()
df["income"].std()
df["income"].var()
df["income"].min()
df["income"].max()
Enter fullscreen mode Exit fullscreen mode

NumPy

NumPy provides numerical and statistical functions.

import numpy as np

np.mean(data)
np.median(data)
np.std(data)
Enter fullscreen mode Exit fullscreen mode

SciPy

SciPy provides many statistical tests.

from scipy import stats

stats.ttest_ind(group1, group2)
Enter fullscreen mode Exit fullscreen mode

Statsmodels

Statsmodels is useful for statistical modelling and detailed statistical inference.

import statsmodels.api as sm
Enter fullscreen mode Exit fullscreen mode

20. Example: Analyzing Business Income

Suppose a dataset contains the monthly income of several businesses:

30000
35000
40000
42000
50000
55000
60000
150000
Enter fullscreen mode Exit fullscreen mode

We could calculate:

df["personal_income"].mean()
df["personal_income"].median()
df["personal_income"].std()
Enter fullscreen mode Exit fullscreen mode

The extremely high value of 150000 may affect the mean substantially.

This is why it is important not to rely on a single statistic.

A data scientist should examine:

  • Mean
  • Median
  • Standard deviation
  • Distribution
  • Outliers
  • Quartiles

A boxplot can help identify potential outliers, while a histogram can show whether the distribution is symmetric or skewed.

21. The Central Limit Theorem

The Central Limit Theorem (CLT) is one of the most important ideas in statistics.

In simplified terms, when sufficiently large random samples are repeatedly drawn from a population, the distribution of their sample means tends toward a normal distribution under common conditions.

This helps explain why statistical inference can work even when the original population distribution is not normal.

The CLT is important for:

  • Confidence intervals
  • Hypothesis testing
  • Sampling distributions
  • Statistical estimation

Understanding sampling distributions also helps explain the difference between the variability of individual observations and the uncertainty of estimated population parameters.

22. Common Statistical Mistakes in Data Science

Knowing statistics is not only about knowing formulas. It is also about avoiding incorrect conclusions.

Mistake 1: Assuming correlation means causation

Two variables can move together without one causing the other.

Mistake 2: Ignoring outliers

Extreme observations can strongly influence statistics such as the mean and some models.

Mistake 3: Using the wrong statistical test

Different tests make different assumptions and answer different questions.

Mistake 4: Focusing only on p-values

A statistically significant result may have little practical importance.

Mistake 5: Ignoring sample bias

A large sample can still produce misleading conclusions if it is not representative of the population.

Mistake 6: Confusing standard deviation with standard error

Standard deviation describes variation among observations, while standard error describes the uncertainty of an estimate.

23. A Practical Statistics Workflow for Data Scientists

A useful workflow is:

1. Understand the dataset
          ↓
2. Identify variable types
          ↓
3. Check missing values
          ↓
4. Calculate descriptive statistics
          ↓
5. Visualize distributions
          ↓
6. Detect outliers
          ↓
7. Examine correlations
          ↓
8. Form statistical questions
          ↓
9. Choose appropriate tests
          ↓
10. Interpret the results
          ↓
11. Build statistical/ML models
          ↓
12. Communicate conclusions
Enter fullscreen mode Exit fullscreen mode

This approach prevents the common mistake of immediately building a machine learning model without first understanding the data.

24. Statistics vs Machine Learning

Statistics and machine learning overlap, but they often emphasize different goals.

Statistics Machine Learning
Understand relationships Make predictions
Estimate parameters Optimize predictive performance
Test hypotheses Learn patterns from data
Quantify uncertainty Evaluate generalization
Explain effects Predict outcomes

In practice, modern data scientists frequently use both.

For example, a data scientist might use statistical analysis to understand which variables are associated with customer churn and then use machine learning to predict which customers are most likely to churn.

Conclusion

Statistics is not an optional skill for data scientists. It provides the foundation for understanding data, measuring uncertainty, testing ideas, and making evidence-based decisions.

The most important areas to learn include:

  • Descriptive statistics
  • Inferential statistics
  • Mean, median, and mode
  • Variance and standard deviation
  • Percentiles and quartiles
  • Probability
  • Probability distributions
  • Sampling
  • Correlation and covariance
  • Hypothesis testing
  • P-values
  • Confidence intervals
  • Statistical tests
  • Regression
  • Sampling distributions
  • The Central Limit Theorem

The most important mindset is to understand the data before trusting the model.

A machine learning algorithm can produce predictions, but statistics helps you understand whether the data supports those predictions, how uncertain your conclusions are, and whether the patterns you see are meaningful.

In short:

Statistics helps data scientists turn data into evidence, and evidence into informed decisions.

Top comments (0)