Descriptive Statistics: A Beginner's Guide

Introduction

In the previous article, we looked at how statistics helps us take a large amount of information and turn it into something we can understand.

When a hospital administrator receives data on 50,000 patient visits, they will see alot of information on it. There will probably be columns for dates, age, gender, patient vitals, diagnosis, admission status, length of stay, bill amount etc. The administrator can't reasonably look at all 50,000 rows and say, "Okay, I understand this dataset."

Naturally, they would have questions about who the patients are, what a typical patient looks like, patient flow by department, how much patients are paying, and many more. While the answers are contained within the dataset, they are not immediately obvious from looking at individual records.

These questions can be largely answered using descriptive statistics.

Four Questions to Guide Our Analysis

Very simply defined, descriptive statistics summarize and describe a complex set of raw data so it is easy to read and understand.

Descriptive statistics begins with 4 simple questions:

1. What type of data do I have?

2. Where is my data centered?

3. How spread out is my data?

4. What does the distribution of my data look like?

These four questions will form the structure of this article. As we work through each one, we will explore the descriptive statistics that help us summarize, understand and communicate what our data is telling us.

1. What type of data do I have?

Before we start calculating averages, percentages or looking at distributions, we first need to understand what kind of data we are working with.

In statistics, we have different types of data that are summarized in different ways. The first step is to understand what kind of variable we are working with. Data is broadly categorized into Qualitative (Categorical) and Quantitative (Numerical) data.

Qualitative Data

Qualitative data, also known as categorical data, describes characteristics, groups, labels, or names. It is further classified into two sub-types: Nominal and Ordinal data.

• Nominal Data: These are groups, labels or names that have no specific order or ranking. Examples include color (green, white), furniture type (desk, table), religion and gender.

• Ordinal Data: These are groups, labels or names that have a natural order or ranking. Examples include order of appearance (first, second, last), survey responses (disagree, neutral, agree) and education levels (primary, secondary, tertiary).

Quantitative Data

Quantitative data, also known as numerical data, uses numbers to measure or count things, indicating how much or how many. It is further classified into two types: Discrete and Continuous data.

• Discrete Data: Represents whole numbers that can be counted. They do not include decimals or fractions. Examples include the number of students in a class, the number of patients in a hospital or the number of wards in a hospital.

• Continuous Data: Represents measurements that can take a range of values, including decimals and fractions. Examples include weight, temperature and time.

Understanding data types is essential because they guide the methods we use to summarize our data.

For example, calculating the mean of the departments variable in a hospital dataset would not be meaningful because departments are categories. Instead, we might look at how many patients went to each department and the percentage they represent. For age, however, calculating the mean or median can be meaningful because age is a numerical variable.

In Python, we can use different methods to summarize these variables.

healthaccess_data["age"].describe()
# .describe() summarizes a numerical variable using statistics such as the mean, standard deviation, minimum, quartiles, and maximum.

Output:

count   50000.00
mean       36.81
std        19.02
min         0.00
25%        24.00
50%        36.00
75%        49.00
max        84.00
Name: age, dtype: float64

For a categorical variable such as department:

healthaccess_data["department"].value_counts()
# .value_counts() counts the number of observations in each category.

Output:

department
General Practice (Gp)    20783
Maternity                 8564
Tb/Hiv Clinic             4150
Oncology                  3362
Emergency & Casualty      3348
Paediatrics               2341
Renal/Dialysis            2206
Cardiology                2176
Surgery                   1127
Icu / Hdu                 1089
Ophthalmology              854
Name: count, dtype: int64

The important point is that different types of data require different ways of summarizing them.

Once we know what type of data we are working with, we can then explore where the data tends to be centered, leading us to the second question:

2. Where is my data centered?

When we talk about the center of a variable, we are looking for a value that gives us an idea of where the observations tend to cluster. These are known as measures of central tendency.
The three main measures are the mean, median and the mode.

• The mean

The mean is the sum of all values in the dataset, divided by the number of of values in the dataset.

Formula

$$\bar{x} = \frac{\sum x}{n}$$

Where:

$\bar{x}$ is the mean
$x$ is each value, therefore $\sum x$ is the sum of all values
$n$ is the number of values

If the ages are 20, 25, 30, 35 and 40:

$\bar{x} = \frac{20 + 25 + 30 + 35 + 40}{5}$

In Python, we calculate the mean using the .mean() method.

healthaccess_data["age"].mean() 

Output:

np.float64(36.80748)

Because it takes every value in the dataset into account, the mean is a strong measure of central tendency because no observation is ignored when calculating the average.

As each value contributes to the final result, an unusually large or unusually small value can pull the mean away from where most of the observations lie. These unusual observations are known as outliers.

In the presence of outliers, therefore, the mean cannot be relied on as an accurate representation of the center, and an alternative measure is chosen.

• The median

The median is the middle value of a set of data arranged in order of magnitude. It is less affected by outliers.

If you have ages as 35, 29, 28, 67, 27, 30, 32, 33 and 31

To get the median, you first you sort your ages in order of magnitude, i.e. from the smallest value to the largest value

ages = 27, 28, 29, 30, 31, 32, 33, 35, 67

The median is the value in the middle, i.e. 31

If the dataset has an even number of observations, then you simply add the two middle values, then divide by two to get the median

ages = 28, 29, 30, 31, 32, 33, 35, 67

The median is:

$\frac{31 + 32}{2} = 31.5$
In Python, we calculate the median using the .median() method.

We can compare the mean and median of the bill amount in our hospital dataset to see how the extremely high bill amounts affect the mean.

mean = healthaccess_data["bill_amount_ksh"].mean() 
median = healthaccess_data["bill_amount_ksh"].median() 
print(f"Mean: {mean}")
print(f"Median: {median}")

Output:

Mean: 30011.5429522
Median: 4420.695

The mean is significantly larger than the median because extremely high bill amounts are pulling the mean upwards. The median is less affected by extreme bill amounts, and is therefore a more representative measure of the typical bill amount.

• The mode

The mode is the most appearing observation in a dataset.
Unlike the mean and median, which can only be used with numerical data, the mode can be used with both numerical and categorical data. On a bar chart or histogram, the highest bar often represents the mode.

In Python, we get the mode using the .mode() method.

# Department with the most visits.
healthaccess_data["department"].mode() 

# The most observed patient age
healthaccess_data["age"].mode()

Output:

Department with most visits: 0    General Practice (Gp)
Name: department, dtype: object
Most common patient age: 0   36.00
Name: age, dtype: float64

Sometimes, the most frequently occurring value is not close to the center of the data and therefore the mode may not always be the best representation of the center of the data.

The mode can also become difficult to interpret when a dataset has two or more values with the same highest frequency. When two values tie in frequency, it is described as bimodal, while a tie with more than two is multimodal. It may be challenging to interpret and decide the center of the data in such cases.

There is no single measure of central tendency that is universally the best. The best choice depends on the nature of the data, its distribution and the objective of the analysis. Comparing the mean, median, and mode allows us to view our data from different perspectives and choose the measure that best represents it.

Now that we understand the type of data we are working with and where our data is centered, we need to understand how much the observations vary from that center. This brings us to our third question:

3. How spread out is my data?

2 datasets have the same mean of 10

Dataset A: 8, 9, 10, 11, 12

Dataset B: 2, 6, 10, 14, 18

While they have the same mean, we observe the values in Dataset A are closely grouped around the mean, while values in Dataset B are much more spread out.

To understand how spread out our data is, we use measures of dispersion. These are numerical values that describe the extent of variation among data points.

The main measures of dispersion that we will cover in this article are Range, Variance, Standard Deviation and Inter Quartile Range(IQR)

• Range

The range is the difference between the largest and smallest values in your observation.

$Range = Maximum - Minimum$

In Python, we can get the range using the .max() and .min() methods

age_range = healthaccess_data["age"].max() - healthaccess_data["age"].min()
bill_amount_range = healthaccess_data["bill_amount_ksh"].max() - healthaccess_data["bill_amount_ksh"].min()

print(f"Age Range : {age_range}")
print(f"Bill Amount Range : {bill_amount_range}")

Output:

Age Range : 84.0
Bill Amount Range : 634333.4900000001

The range is easy to calculate and gives a quick idea of the spread, but it only considers the two extreme values. This means it is very heavily influenced by outliers.

• Variance

Variance gives us a measure of how far the observations tend to be from the mean.

Variance is calculated by finding the difference between each observation and the mean, squaring those differences, adding all the squared differences, and then dividing their sum by the number of observations.
We square the differences so that negative and positive deviations do not cancel each other out.

Formula

$\text{Variance} = \frac{\sum(x-\bar{x})^2}{n}$

Where:

$x$ represents each observation (the values of your numeric column)

$\bar{x}$ represents the mean

$n$ is the number of observations

$\sum$ means adding all the squared differences

In Python, variance is calculated using the .var() method.

(healthaccess_data["age"].var()).round(2) 
# Calculates the variance and rounds the result to 2 decimal places.

Output:

np.float64(361.92)

Although variance tells us about the spread of our data, its units are squared. For example, as seen in the output, our age column in years ahs its variance expressed in squared years. This can make it difficult to interpret directly. This is why often standard deviation is preferred.

• Standard Deviation

Standard deviation is a widely used measure of dispersion. It is achieved by finding the square root of the variance, bringing the measure of spread back to the same units as the original data.

$\text{Standard Deviation} = \sqrt{\text{Variance}}$

A smaller standard deviation means that the observations tend to be closer to the mean, while a larger standard deviation indicates that the observations are more spread out.

In Python, standard deviation is calculated using the .std() method

healthaccess_data["age"].std().round(2)
# Calculates the standard deviation of patient ages and rounds the result to 2 decimal places.

Output:

np.float64(19.02)

The standard deviation of patient age is 19 years, meaning that patient ages have a substantial amount of variation around the mean, with observed ages ranging from 0 to 84 years.

• Inter Quartile Range (IQR)

Another useful measure of spread is the interquartile range (IQR), which looks at the middle 50% of the data and is less affected by extreme values. The IQR therefore captures the range containing the middle 50% of observations.

Formula

$$\text{IQR} = Q3 - Q1$$

Where:

Q1 = 25th percentile

Q3 = 75th percentile

In Python, Q1 and Q3 are obtained using the .quantile() method, which identifies the value at a specified position within the distribution. 0.25 gives us the 25th percentile (Q1), 0.5 gives us the 50th percentile (Q2), while 0.75 gives us the 75th percentile (Q3).

To find the IQR for our age variable therefore;

IQR = healthaccess_data["age"].quantile(0.75) - healthaccess_data["age"].quantile(0.25)

Output:

np.float64(25.0)

The IQR is 25 years, meaning the middle 50% of patient ages are spread across a range of 25 years.

Together, the measures of dispersion give us a better understanding of how much our observations vary. The choice of measure depends on the nature of the data and how affected it is by extreme values.

Having understood both the center and spread of data, the next step is to examine the shape and pattern of the distribution, which brings us to the final question:

4. What does the distribution of my data look like?

A distribution shows how the values in a dataset are spread across the possible range of values. Looking at a distribution helps us identify patterns such as where values are concentrated, how widely they are spread, and whether there are unusual observations.

In our hospital dataset, we can use histograms to visualize the distribution of our numerical variables.

A histogram groups numerical values into intervals, or bins, and shows how many observations fall within each interval.

sns.histplot(data=healthaccess_data, x="age")
plt.show()

Output:

We can use the same approach to examine the distribution of bill amounts:

sns.histplot(data=healthaccess_data, x="bill_amount_ksh", kde = True)
plt.show()

Output:

As we can observe from the histograms, the shape of a distribution can vary depending on how the observations are spread across the range of values. These patterns are commonly described as symmetrical, right-skewed, or left-skewed.

• Symmetrical Distributions

A symmetrical distribution is one where the observations are distributed in a similar way on either side of the center. A normal distribution is a specific type of symmetrical distribution with a characteristic bell-shaped curve.

In an approximately normal distribution, most observations are concentrated around the center, with fewer observations occurring toward either end. The mean, median, and mode are also typically equal or very close to one another because the data is balanced around the center.

The age distribution in our hospital dataset is approximately bell-shaped, with the mean, median, and mode all equal to 36 years. This suggests that the distribution is approximately symmetrical, with observations concentrated around the center and becoming less frequent toward either end.

• Right Skewed Distribution

In some cases, you may find most observations concentrated toward the lower values, while a smaller number of larger values extend the distribution toward the right. This is known as a right-skewed distribution, or positively skewed distribution.

In a right-skewed distribution, the tail extends toward the higher values, while most observations are concentrated on the lower end.

Our bill amount provides an example of this pattern. Most bill amounts are concentrated at the lower end, while a smaller number of extremely high bills create a long tail to the right.

Because the higher values pull the mean toward the right, the mean is typically larger than the median in a right-skewed distribution. making the median a more representative measure of the center.
This is why, when we examined the bill amounts earlier, the median was more useful than the mean for describing a typical bill.

Left Skewed Distribution
A left-skewed distribution, also known as a negatively skewed distribution, occurs when most observations are concentrated toward the higher values, while a smaller number of lower values create a longer tail toward the left.

Because the lower values pull the mean toward the left, the mean is typically smaller than the median in a left-skewed distribution.

Identifying Outliers

As seen above, the tail of a distribution can give us an initial clue about extreme values. To identify outliers more systematically, we need to look at the individual observations and use specific methods for detecting them. The interquartile range (IQR) can be used to identify potential outliers.

• The IQR rule

It defines potential outliers as observations that fall below the lower bound or above the upper bound:

$\text{Lower Bound} = Q1 - 1.5(IQR)$
$\text{Upper Bound} = Q3 + 1.5(IQR)$

Any observation below the lower bound or above the upper bound is considered a potential outlier.

In Python, we can calculate these boundaries using the first and third quartiles:

# First calculate the IQR
Q1 = healthaccess_data["bill_amount_ksh"].quantile(0.25)
Q3 = healthaccess_data["bill_amount_ksh"].quantile(0.75)

IQR = Q3 - Q1
# Then the upper and lower bounds
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

# Entries with potential outliers 
outliers = healthaccess_data[(healthaccess_data["bill_amount_ksh"] < lower_bound) |
    (healthaccess_data["bill_amount_ksh"] > upper_bound)]

For our hospital dataset, this is useful for the bill amount observations, where we have some extremely high values. These explain the right-skewed shape of the distribution and why the mean is substantially higher than the median.

Because it focuses on the middle 50% of observations, the IQR rule is less affected by extreme values.

• Visualizing Outliers with a Boxplot

A boxplot provides another way to visualize the distribution of a numerical variable. It displays the median, quartiles, upper and lower limits and potential outliers, making it useful for quickly assessing both the spread and outliers.

We can use a box plot to visualize outliers in the bill amounts

sns.boxplot(data=healthaccess_data, x="bill_amount_ksh")
plt.show()

Output:

The box represents the middle 50% of observations, with the line inside the box showing the median. Values that fall beyond the whiskers are typically shown as individual points and may represent potential outliers.

Conclusion

Descriptive statistics gives us a way to make sense of our data before moving on to more advanced analysis. Throughout this article, we have seen how asking four simple questions can help us build a clearer picture of what our data looks like.

The important lesson is that no single statistic tells us everything about our data. We get a much better understanding by looking at the type, center, spread, and distribution together. These descriptive summaries provide a foundation for making informed decisions about how to explore and analyze our data further.