Sami

Posted on Aug 8

Part 6 - STATISTICS

#statistics #mathematics

Bernoulli, Binomial Distribution & Central Limit Theorem Explained from First Principles

When I first started learning probability, I came across terms like:

Bernoulli Distribution
Binomial Distribution
Sampling Distribution
Central Limit Theorem

At first, they felt like completely different topics.

But after understanding the reason behind each concept, I realized that they are actually connected.

The entire story starts with a very simple idea:

What happens when I repeat a random experiment multiple times?

Let's understand it step by step.

Start With One Simple Experiment

Imagine I ask one person:

"Will you like this blog post?"

There are only two possible answers:

Yes
No

We can represent them as:

Yes → 1
No  → 0

This is the simplest possible random experiment.

And this is where the Bernoulli Distribution begins.

What is Bernoulli Distribution?

A Bernoulli Distribution models an experiment with exactly two possible outcomes.

We usually call them:

Success → 1
Failure → 0

For example:

Customer buys → 1
Customer doesn't buy → 0

Or:

Email is spam → 1
Email is not spam → 0

Or:

User clicks an advertisement → 1
User doesn't click → 0

The Bernoulli Distribution has only one parameter:

p = probability of success

The probability of failure is:

1 - p

So if the probability that a customer purchases something is 0.3:

P(success) = 0.3

P(failure) = 0.7

That's Bernoulli.

But What If We Repeat the Experiment?

Now let's make the problem slightly more interesting.

Suppose the probability that someone likes my blog is:

p = 0.5

I ask 3 people.

Each person can either:

Like → Success
Don't Like → Failure

Now I don't care about one person's answer.

Instead, I ask:

How many people out of 3 will like it?

Possible answers are:

0 people
1 person
2 people
3 people

Now we've moved from Bernoulli to Binomial Distribution.

What is Binomial Distribution?

The Binomial Distribution tells us:

How many successes will I get in a fixed number of independent trials?

Think of it as:

Bernoulli + repetition = Binomial

That's the easiest way I remember it.

Bernoulli vs Binomial

Let's make the difference very clear.

Bernoulli

One experiment:

Will this customer buy?

Yes / No

Binomial

Multiple experiments:

Out of 100 customers,

How many will buy?

So:

Bernoulli deals with one trial.

Binomial deals with multiple trials.

Conditions for Binomial Distribution

For something to follow a Binomial Distribution, we need four important conditions.

1. Fixed Number of Trials

We decide the number of experiments beforehand.

For example:

n = 100 customers

2. Only Two Outcomes

Each trial has two possible outcomes:

Success
Failure

3. Same Probability

The probability of success remains constant.

For example:

p = 0.5

for every trial.

4. Independent Trials

One trial should not affect another.

For example:

If one customer buys a product,

that shouldn't automatically change whether another randomly selected customer buys it.

A Simple Binomial Example

Suppose:

Probability of success = 0.5

Number of people = 3

We want to know:

What is the probability that exactly 2 out of 3 people like the blog?

Possible combinations include:

Like, Like, Don't Like

Like, Don't Like, Like

Don't Like, Like, Like

There are 3 ways to get exactly 2 successes.

The Binomial Distribution allows us to calculate the probability of this event.

The important thing isn't memorizing the formula.

The important idea is:

We are counting the number of successes across repeated Bernoulli trials.

Where is Binomial Distribution Used?

Binomial Distribution is extremely useful in Data Science.

For example:

Binary Classification

Suppose we build a spam detection model.

Every email is:

Spam
Not Spam

That's a binary outcome.

A/B Testing

Suppose we test two versions of a website.

Version A:

1000 visitors

Version B:

1000 visitors

For every visitor:

Clicked → Success
Didn't click → Failure

We can model the number of clicks using a Binomial Distribution.

Hypothesis Testing

We can use Binomial Distribution to calculate how likely a certain number of successes would be under a particular assumption.

This helps us decide whether the observed result provides evidence against our assumption.

Now Let's Make the Problem Much Bigger

So far, we've talked about individual observations.

But Data Science usually deals with populations and samples.

Suppose I want to know:

What is the average income of people in India?

I obviously cannot ask every single person.

There are far too many people.

So what do I do?

I take a sample.

For example:

Population

↓

Take a random sample of 100 people

↓

Calculate their average income

Maybe I get:

₹45,000

But what if I take another 100 people?

Maybe I get:

₹48,000

Another sample:

₹43,500

Another:

₹46,200

Something interesting has happened.

The sample means themselves are changing.

And this brings us to the next concept.

What is Sampling Distribution?

A Sampling Distribution is:

The probability distribution of a statistic calculated from many samples of the same size taken from a population.

That sounds complicated.

Let's break it down.

Imagine I have a population.

Population
100000 people

Instead of taking one sample, I take many samples.

Sample 1 → Mean = ₹45,000

Sample 2 → Mean = ₹47,000

Sample 3 → Mean = ₹44,500

Sample 4 → Mean = ₹46,200

Sample 5 → Mean = ₹45,800

...

Now I have a new dataset:

45,000
47,000
44,500
46,200
45,800
...

These aren't individual people's salaries.

They are sample means.

The distribution of these sample means is called the:

Sampling Distribution of the Mean

Why Is Sampling Distribution Important?

Because we usually don't have access to the entire population.

We only have a sample.

So we need to answer:

"How reliable is the statistic I calculated from my sample?"

Sampling distributions help us understand the variability of sample statistics.

This is the foundation for:

Confidence intervals
Hypothesis testing
Statistical inference
Population estimation

And Now We Reach the Central Limit Theorem

This is where everything starts coming together.

The Central Limit Theorem, or CLT, is one of the most important ideas in statistics.

And the basic idea is surprisingly simple.

Suppose the original population looks completely strange.

Maybe it is:

Skewed
Uniform
Log-normal
Non-Gaussian

Now repeatedly take sufficiently large random samples and calculate their means.

Something amazing happens.

The distribution of those sample means starts looking approximately Normal.

This is the Central Limit Theorem.

Let's Visualize the Idea

Imagine our original population looks like this:

       *
      **
    ****
***************

It is heavily skewed.

We take many samples.

For every sample, we calculate the mean.

Sample 1 → Mean
Sample 2 → Mean
Sample 3 → Mean
Sample 4 → Mean
...

Now plot all those means.

The resulting distribution starts looking more like:

           *
         *   *
       *       *
     *           *
___*_______________*___

A bell curve.

That's the magic of the Central Limit Theorem.

The Most Important Insight

The CLT does not say:

"Every dataset is normally distributed."

That's wrong.

It says:

The distribution of sample means approaches a Normal Distribution as the sample size becomes sufficiently large, under the required assumptions.

This distinction is extremely important.

Your original data can be non-normal.

The sampling distribution of the mean can still become approximately normal.

Conditions for the Central Limit Theorem

The source material gives the following important conditions.

1. Large Enough Sample

A commonly used rule of thumb is:

n ≥ 30

But this isn't a universal magic number.

The required sample size can depend on the underlying distribution and how strongly it deviates from normality.

2. Finite Variance

The population should have finite variance.

3. Independent and Identically Distributed Variables

The observations should be independent and come from the same underlying distribution.

Why Does CLT Matter So Much?

Because it allows us to make conclusions about a large population using samples.

Suppose I want to estimate the average income of Indians.

I don't need to collect everyone's income.

Instead:

Population

↓

Random Samples

↓

Sample Means

↓

Sampling Distribution

↓

Statistical Inference

The CLT gives us the theoretical foundation for many of the statistical methods we use.

Real-World Case Study: Average Income

Let's say we want to estimate the average income of a population.

We collect multiple random samples.

Each sample has more than 30 observations.

For every sample, we calculate:

Sample Mean
Sample Standard Deviation

Now we calculate the average of all the sample means.

This gives us an estimate of the population mean.

Then we calculate the standard error.

The standard error tells us how much the sample mean tends to vary from sample to sample.

A common formula for the standard error of the mean is:

SE = σ / √n

where:

σ = population standard deviation
n = sample size

When the population standard deviation is unknown, we commonly estimate it using the sample standard deviation.

Confidence Interval

Now suppose our estimated mean is:

₹50,000

Instead of saying:

"The true population average is exactly ₹50,000."

we can provide a range.

For a 95% confidence interval under the normal approximation:

Lower Limit = Mean - 1.96 × Standard Error

Upper Limit = Mean + 1.96 × Standard Error

So instead of giving one number,

we provide a range that represents uncertainty around our estimate.

This is much more useful in real-world statistics.

Why Does Sample Size Matter?

Imagine asking only 3 people about the average salary.

You might get a very unreliable estimate.

Now imagine asking 1,000 people.

The estimate is likely to be more stable.

This is one reason the CLT is so useful.

As sample size increases:

Sample means become less variable

and the sampling distribution becomes more concentrated around the population mean.

Connecting Everything Together

Let's put the entire story together.

Step 1 — Bernoulli

One experiment with two outcomes.

Success / Failure

Step 2 — Binomial

Repeat Bernoulli experiments.

How many successes?

Step 3 — Sampling

Take a subset from a population.

Population → Sample

Step 4 — Sampling Distribution

Take many samples and calculate a statistic.

Sample → Statistic

Many samples → Distribution of statistics

Step 5 — Central Limit Theorem

For sufficiently large samples, the distribution of sample means approaches a Normal Distribution.

Many Sample Means
        ↓
Approximately Normal Distribution

This is the connection that made these topics much easier for me to understand.

Where Does This Appear in Data Science?

These concepts aren't just theoretical.

They appear in many practical situations.

A/B Testing

Compare conversion rates between two groups.

Machine Learning

Understand the statistical behavior of samples and model estimates.

Hypothesis Testing

Determine whether observed results are consistent with a particular assumption.

Confidence Intervals

Estimate population parameters using sample data.

Business Analytics

Estimate things like:

Average customer spending
Conversion rates
Average order value
Customer satisfaction

Final Thoughts

When I first encountered Bernoulli, Binomial, Sampling Distribution, and CLT, they looked like separate chapters.

But after thinking from first principles, I realized they form a chain.

It starts with:

One random experiment.

Then:

Repeat the experiment.

Then:

Take samples from a population.

Then:

Look at the distribution of statistics from those samples.

And finally:

The Central Limit Theorem tells us that the distribution of sample means tends toward a Normal Distribution when the conditions are satisfied and the sample size is sufficiently large.

That's the bigger picture.

And once you see the connection, these concepts become much easier to remember.

Key Takeaways

Bernoulli Distribution models one binary experiment.
Binomial Distribution counts successes across a fixed number of Bernoulli trials.
Binomial trials require fixed n, two outcomes, constant probability, and independence.
Sampling Distribution describes how a statistic behaves across repeated samples.
Central Limit Theorem explains why sample means tend toward a Normal Distribution for sufficiently large samples under its assumptions.
CLT does not say the original population becomes normal.
Sampling distributions are fundamental for confidence intervals and hypothesis testing.
Larger samples generally make estimates of the population mean more stable.
These concepts connect probability theory with practical statistical inference.