Introduction
When we hear the term data science, the first things that may come to mind are Python, SQL, machine learning, or perhaps artificial intelligence.
And while all of these are important parts of data science, there is another field that sits underneath much of the work we do with data: statistics.
Statistics gives us ways to describe data, understand variation, identify patterns, quantify uncertainty, and draw conclusions from evidence. Data science brings these ideas together with programming, mathematics, computing, and domain knowledge to solve problems using data.
So before we get into distributions, probability, hypothesis testing, or statistical models, I think it is worth taking a step back and asking a more fundamental question:
Why does statistics matter in data science?
Data Alone Is Not Enough
Having a dataset does not automatically mean that we understand what is happening in it.
For example, imagine we have a dataset containing 50,000 hospital visits. We may have information such as patient age, diagnosis, department, insurance type, admission status, and length of stay. At first, we simply have a collection of records. But if we want to understand what is happening in the hospital, we might ask:
- Which departments have the longest patient stays?
- What is the typical length of stay?
- How much does length of stay vary between patients?
- Are some patient groups more likely to be admitted?
Answering these questions requires us to find patterns and relationships within the records.
We might find, for example, that patients in one department have a higher average length of stay than patients in another. But is that difference meaningful? How much variation exists within each department? Could the difference simply be due to the particular patients included in our data?
These are statistical questions.
Statistics gives us a framework for asking these kinds of questions and making sense of the answers. It provides the tools to describe what we are seeing, understand the variation in our data, and draw conclusions from the patterns we observe. We are then able to turn large collections of observations into answers
The University of Michigan describes statistics as a way of learning from data and emphasises the importance of understanding how data is collected and how it can be used to answer questions. This is an important distinction:
Collecting data is not the same as understanding data.
Where Does Statistics Fit Into Data Science?
Data science is an interdisciplinary field. It brings together statistics, mathematics, computer science, programming, analytics, and domain knowledge.
Each of these contributes something different.
- Programming helps us work with and manipulate data.
- Computer science provides the computational tools and systems we use to process data.
- Mathematics provides the foundations behind many of the methods we use.
- Statistics helps us understand data, variation, uncertainty, and evidence.
- Domain knowledge helps us understand what the data actually represents.
The American Statistical Association describes statistics as an important part of data science and highlights its role in areas such as statistical summaries, inference, uncertainty, and the extraction of knowledge from data.
UCLA similarly describes statistics as a foundation of data science, while recognising the increasingly close relationship between statistics, mathematics, and computer science.
So while data science is broader than statistics, statistics remains one of the disciplines that helps us reason about the data we work with.
Statistics Helps Us Understand Variation
One of the first things we encounter when working with real-world data is variation.
People do not all earn the same income.
Patients do not all spend the same amount of time in hospital.
Students do not all receive the same grades.
Households do not all have the same number of members.
Even when we are measuring the same thing, our observations can differ.
Statistics gives us tools for describing this variation. Measures such as the mean, median, range, variance, standard deviation, and interquartile range help us understand where our data is centred and how widely the observations are spread.
For example, two hospital departments could both have an average length of stay of 5 days. At first, it might seem that patients in the two departments have similar stays. But suppose the stays in Department A are mostly between 4 and 6 days, while Department B has patients staying anywhere from 1 to 12 days.
The average is the same, but the amount of variation is very different. Looking only at the average would therefore leave out an important part of the story.
A single summary value can sometimes fail to reveal all what is really happening in the data.
And this brings us to one of the concepts we will explore later: distributions.
Statistics Helps Us Understand Uncertainty
Real-world data is rarely perfect. When we analyse data, we are often working with some level of uncertainty.
We may not have information about everyone we are interested in, some observations may be missing or measured inaccurately, and even carefully collected data can contain differences that occur simply due to chance.
Consider a survey of 1,000 people. We might use their responses to learn about a much larger population, but those 1,000 people are only a sample of the population we are interested in. The results we observe in our sample may therefore not be exactly the same as what we would find if we could ask everyone.
Statistics gives us methods for understanding and quantifying this uncertainty, helping us determine how much confidence we can place in the conclusions we draw from our data.
This is another one of the reasons statistics is important in data science: it helps us understand how certain we can be about what the data says.
Statistics Helps Us Ask Better Questions
Statistical thinking is useful before we begin analysing our data. Good analysis starts with understanding the problem we are trying to solve and whether the data we have can actually help us answer it.
Before any analysis or creating models, we need to understand our data by asking:
- What are we trying to find out?
- What exactly are we measuring, and who or what does the data represent?
- How was the data collected?
- What limitations or potential sources of error exist in the data?
- Is the data appropriate for answering the question we are asking?
Only then can we determine whether the data is suitable for the analysis we want to perform.
The University of Michigan's introduction to data science places emphasis on starting with a question and using data systematically to investigate it.
This means statistical thinking can influence our analysis, even before we write our code.
Statistics and Data Science Tools
Data science involves many tools, from Python and SQL to visualisation and machine learning. Python can calculate a mean, but statistics helps us decide whether the mean is useful. A visualisation can reveal a pattern, but statistical thinking helps us determine what that pattern actually tells us. Statistics helps us move from working with data to reasoning about data.
So, Do Data Scientists Need Statistics?
The short answer is yes.
That does not mean every data scientist needs to become a theoretical statistician. Data science is broad, and different roles require different combinations of skills.
But understanding statistics provides a foundation for thinking critically about data.
What We Will Explore Next
This article is the starting point for a deeper exploration of the subject.
Before getting into statistical tests and models, we need to understand some of the building blocks that make those methods possible.
One of the first is distributions.
When we collect observations, those observations have a pattern of values and variation. Understanding that pattern helps us describe our data, identify unusual observations, choose appropriate summary measures, and eventually understand which statistical methods may be appropriate.
So in the next article, we will take a closer look at distributions and their impact on data science.
Final Thoughts
Learning statistics is more about learning how to reason with data.
As we explore the subject in future articles, we will build that understanding step by step, from distributions and descriptive statistics to probability, sampling, inference, and statistical testing.
There is a lot to cover, but we will take it one concept at a time.
I look forward to learning and sharing what I learn!
Top comments (0)