Sami

Posted on Jul 15

PART 3 - STATISTICS

#datascience #maths

Probability Distributions Explained from First Principles (Beginner Friendly)

When I first heard the term Probability Distribution, I imagined something extremely mathematical.

Words like PMF, PDF, CDF, Random Variables, and Density Estimation sounded intimidating.

But after spending time understanding why these concepts exist, I realized something interesting:

Probability distributions are simply a smarter way of answering one question:

"What outcomes can happen, and how likely is each one?"

Let's understand everything from first principles.

Why Do We Need Probability?

Imagine you toss a coin.

Before tossing, can you tell me exactly what the result will be?

No.

You only know the possible outcomes.

Heads
Tails

This uncertainty is what probability tries to measure.

Instead of saying

"I know what will happen."

Probability says

"I know how likely each outcome is."

Random Variables — The First Building Block

Before learning probability distributions, we need to understand one important idea.

Variables in Algebra

In school, we learned

x + 5 = 10

Here,

x = 5

The variable has one unknown value.

Simple.

Random Variables in Statistics

Statistics looks at variables differently.

Instead of asking

"What is x?"

It asks

"What values can x possibly take?"

Suppose we roll a dice.

Possible outcomes are

The random variable isn't one unknown number.

It is the set of all possible outcomes of a random experiment.

Another Example

Suppose tomorrow I count the number of customers entering my shop.

Possible values could be

I don't know today's value yet.

But I know all the possible values it might take.

That's a random variable.

Now Comes the Real Question

Knowing the possible outcomes isn't enough.

I also want to know

Which outcomes are more likely?

For a coin,

Heads → 50%

Tails → 50%

For a dice,

1 → 1/6
2 → 1/6
...
6 → 1/6

This complete picture is called a Probability Distribution.

What is a Probability Distribution?

A probability distribution simply tells us

Every possible outcome

and

The probability of that outcome.

Think of it like a menu.

Instead of listing food prices,

it lists outcomes and their probabilities.

Outcome	Probability
1	1/6
2	1/6
3	1/6
4	1/6
5	1/6
6	1/6

That's it.

Nothing magical.

But There Is a Problem...

Imagine recording people's heights.

Possible values could be

170.1 cm

170.11 cm

170.112 cm

170.1125 cm

170.11253 cm

The list never ends.

How can we make a table with infinitely many values?

We can't.

So statisticians asked a brilliant question.

What if we use a mathematical function instead of a giant table?

That's exactly why Probability Distribution Functions were created.

Famous Probability Distributions

Some types of data appear again and again.

Instead of creating new distributions every time,

mathematicians created standard ones.

Examples include

Normal Distribution
Bernoulli Distribution
Binomial Distribution
Poisson Distribution
Exponential Distribution

Whenever real-world data behaves similarly,

we can use these existing distributions.

Why Are Probability Distributions Important?

Once we know which distribution our data follows,

we immediately know many things about it.

For example,

if exam marks approximately follow a Normal Distribution,

we can estimate

average score
spread
unusual scores
probabilities

without checking every student individually.

This is why probability distributions are one of the foundations of statistics and machine learning.

Parameters — The Settings of a Distribution

Think about making tea.

The recipe changes depending on

sugar
milk
tea powder

These are settings.

Probability distributions also have settings.

These settings are called parameters.

Changing parameters changes the shape of the distribution.

Just like changing sugar changes the taste of tea.

PMF — Probability Mass Function

Now we enter our first probability function.

PMF works only for Discrete Random Variables.

Discrete means

you can count every possible value.

Examples

Dice
Coin Toss
Number of Students
Number of Cars

PMF directly gives the probability of each value.

Example

Rolling a dice

P(1)=1/6

P(2)=1/6

...

P(6)=1/6

Two important rules

Every probability must be greater than or equal to 0.
All probabilities together must add up to 1.

CDF — Cumulative Distribution Function

Sometimes we don't want

P(X = 3)

Instead we ask

What is the probability that X is less than or equal to 3?

That's exactly what CDF does.

F(x)=P(X≤x)

Example

For a dice,

P(X ≤ 3)

=

P(1)+P(2)+P(3)

=

3/6

=

0.5

Notice something?

CDF keeps adding probabilities as we move forward.

That's why it's called Cumulative.

Continuous Data Changes Everything

Now imagine measuring

Height
Weight
Temperature
Salary

These values can have infinitely many decimals.

Can we assign probability to one exact value?

Not really.

The probability of observing exactly

170.111111111...

is practically zero.

So PMF no longer works.

We need something new.

PDF — Probability Density Function

Instead of probability,

continuous variables use Density.

At first this sounds strange.

Why density?

Imagine rain.

One single drop doesn't tell us much.

But if many drops fall in one area,

we say

Rainfall is dense here.

Similarly,

PDF tells us

where values are concentrated.

Higher curve

→ More observations likely to appear there.

Lower curve

→ Fewer observations.

Important Idea

For continuous distributions,

the height of the PDF curve is NOT the probability.

Instead,

the area under the curve represents probability.

This is one of the biggest beginner mistakes.

Always remember:

Area = Probability

Not

Height = Probability

Density Estimation — Finding the Shape of Data

Suppose someone gives you 50,000 customer ages.

You don't know which distribution they follow.

Normal?

Poisson?

Something else?

Instead of guessing,

we estimate the underlying distribution.

This process is called

Density Estimation.

Two Ways to Estimate Density

1. Parametric Density Estimation

Here we assume

"I think my data follows a known distribution."

For example

Normal Distribution.

Then we estimate its parameters.

This method is fast,

but only works if our assumption is correct.

2. Non-Parametric Density Estimation

Sometimes we don't know the distribution.

Instead of forcing one,

we let the data speak for itself.

This is called

Non-Parametric Density Estimation.

The most popular method is

Kernel Density Estimation (KDE).

Kernel Density Estimation (KDE)

Imagine placing a tiny smooth hill on every data point.

Now add all those hills together.

The final smooth curve becomes an estimate of your probability density.

Unlike histograms,

KDE produces a smooth continuous curve,

making it much easier to understand the distribution.

Where Do Data Analysts Use These Concepts?

These ideas appear everywhere.

Understanding customer ages
Salary analysis
Fraud detection
Anomaly detection
Machine Learning
Data Visualization
Risk Analysis
Forecasting

Even if you don't calculate these formulas manually,

libraries like NumPy, SciPy, Pandas, and Seaborn use these concepts behind the scenes.

Final Thoughts

When I first learned probability distributions, I thought they were just another chapter in statistics.

Now I see them differently.

They're simply a language for describing uncertainty.

Everything starts with one simple question:

What can happen?

Then another:

How likely is each outcome?

Every concept—Random Variables, PMF, PDF, CDF, and KDE—is just another tool to answer those two questions more effectively.

Once you understand the why, the formulas stop feeling difficult.

Key Takeaways

Random Variables represent possible outcomes of an experiment.
Probability Distributions assign probabilities to those outcomes.
PMF is used for discrete data.
PDF is used for continuous data.
CDF gives cumulative probability.
The area under a PDF curve represents probability.
Density Estimation helps estimate the underlying distribution of data.
KDE is a popular non-parametric density estimation technique.

If you're beginning your journey in Data Science or Machine Learning, mastering these concepts will make everything that comes later—hypothesis testing, regression, classification, and even deep learning—much easier to understand.
If you didn't read the previous part, read it here: PART 2

Happy Learning! 🚀

DEV Community