Probability - Introduction, axioms, Conditional and Bayes' Rule

#maths

Probability

Probability is everywhere. It helps determine the likelihood of certain events given the whole sample space. By understanding probability, we can make better decisions based on the chances of favorable outcomes and reduce the impact of unfavorable events. Probability is widely used in areas like:

Climate prediction
Machine learning algorithms
Competitive programming
Medical research
Decision-making in everyday life

Terminology

Sample Space (Ω): List of all the elements in a set / all possible outcomes.
Mutually Exclusive Events: Two or more events that cannot occur at the same time in a single trial of an experiment.
Collectively Exhaustive Events: At least one of the events must occur in a single trial of an experiment.
Event: A set of favorable outcomes.

Understanding Probability Visually

Visualization helps intuitively understand probability. It works well for discrete sample spaces but can get complex for continuous probability. For example, consider two rolls of a tetrahedral die.

Another way to visualize:

Probability Axioms

Non-negativity: P(A) ≥ 0
Normalization: P(Ω) = 1
Additivity Rule: If A ∩ B = ∅, then P(A ∪ B) = P(A) + P(B)

Conditional Probability

Conditional probability calculates the likelihood of an event occurring given that another event has already occurred:

P(A | B) = P(A ∩ B) / P(B), where P(B) ≠ 0

Equivalently:

P(A ∩ B) = P(A | B) × P(B)

Let's understand how this formula can be derived intuitively with an example.

Example: Student & Exam

Event B: Student studies for the exam
Event A: Student passes the exam

Suppose:

70% of students study: P(B) = 0.7
Given they studied, 90% pass: P(A | B) = 0.9

Then:

P(A ∩ B) = 0.7 × 0.9 = 0.63

Meaning there’s a 63% chance that a randomly chosen student both studies and passes. 🎯

Let’s visualize this with a probability tree diagram:

Bayes’ Theorem

From conditional probability, we get Bayes’ theorem:

P(A | B) = [P(A) × P(B | A)] / P(B)

Where:

P(A ∩ B) = P(A) × P(B | A)

Terminology in Bayes’ Theorem:

Posterior (P(H | E)): Probability of the hypothesis given the evidence.
Prior (P(H)): Probability of the hypothesis before observing the evidence.
Likelihood (P(E | H)): Probability of the evidence given the hypothesis is true.
Marginal (P(E)): Probability of the evidence under all possible hypotheses.