The first time I came across the term Confidence Level of 95%, I misunderstood that it means we are 95% sure that 95% of the data lies within a Confidence Interval. This is a common misconception and it took quite a few attempts to get the concept right. So here’s my attempt at explaining it in simple terms using a chai analogy.
Before I get into the details, let’s understand why we have a Confidence Interval and not a confidence value in statistics. I recently read in Practical Statistics for Data Scientists that humans are naturally averse to uncertainty, so we like single, confident numbers. But a single estimate from a sample hides how uncertain it is. A confidence interval gives a range of plausible values instead, which is a more honest answer.
In order to explain the concept of Confidence Intervals, I’ll use a simple example of a Chai/Tea dataset I created. There are two columns, personID and their milk-to-water preference ratio in tea. I have 60 records in the dataset – we call this the original sample set. In order to understand Confidence Intervals, I’ll first have to explain a crucial factor of sampling.
We take multiple resamples of the sample set with replacement. I’ll explain what replacement means here in a minute. Imagine I pick a chit from a bowl of samples and each one has one person’s ID and milk-to-water ratio, I make a note and put that chit back and pick again, this is how resampling with replacement works. Now in order to understand the Confidence Interval, we need many rounds of resamples to be able to confidently give a result.
So say we take the original chai samples (60 records), I make 1000 resamples using this set, each is of length 60 (the same length as the original sample set).
These 1000 resamples we made, they’re called bootstrap samples. Each of these bootstrap sample set has a mean of their own. The spread of the bootstrap means shows how much the average would vary from sample to sample.
Fact: If you’re familiar with how Random Forest models work, you must have come across this term Bootstrap aggregation. RF models use this method of drawing samples too.
Next, we choose a confidence level, usually 90% or 95%. So if we choose Confidence Level as 95%, we sort the 1,000 means and cut off the lowest 2.5% and highest 2.5% of the means.
So what is the 95% referring to?
After taking many (for example, 1,000) resamples of sample data, we want to be able to tell with some surety that 95% of means of all these 1000 resamples lie between what two points or a Confidence Interval. What are these two points? The lower limit and upper limit of the bootstrap means we found earlier. For my chai data, that range came out as 1.24 to 1.53. So we can say, with 95% confidence, that the average milk-to-water preference of the whole group lies between 1.24 and 1.53, even though individual people range from 1 to 3.
What are some real-world use cases where you’ve seen or used Confidence Intervals?
This was Day 2 of my 30 Days of Data Science writing challenge. Writing about this topic required a lot of effort and creativity: to understand the concept, come up with a chai related analogy and explain it as plainly as possible.
If you’d like to see the Python code for this post, you can read it at: https://github.com/Rupali-Gauravaram/dev-prompt-generator/blob/main/posts/post_02.ipynb
Top comments (0)