Handling missing values is one of the trickiest challenges in data analysis. For an analyst, incomplete datasets can easily lead to biased insights and faulty models. In practice, the wise approach is often not to discard incomplete observations but to impute the missing values in a systematic way.
Missing Data in Analysis
While working on real-world datasets, missing values are almost inevitable. Whether it’s customer surveys, medical records, financial transactions, or IoT sensor logs, data gaps creep in. If these are left unhandled, they can easily mislead statistical models or machine learning algorithms.
If the dataset is very large and the percentage of missing values is negligible (say less than 5%), it is sometimes acceptable to ignore them and move ahead. However, when the number of missing values is substantial, ignoring them may bias the results. In those cases, imputing the missing values provides a better alternative than discarding data.
What Are Missing Values?
Let’s think of a simple survey example. If a respondent indicates that they are “single,” it makes sense for the fields related to spouse’s name or number of children to be left blank. In this case, the missing values are perfectly logical and expected.
But missingness isn’t always so straightforward. A respondent may skip a question unintentionally, type in an incorrect entry (e.g., text where a date should go, or a negative age), or deliberately leave a field blank. These situations highlight why missing values occur and why analysts must classify them carefully before deciding how to handle them.
Types of Missing Values
Statisticians generally classify missing values into three categories: MCAR, MAR, and NMAR.
MCAR (Missing Completely at Random)
This is the rarest case where missingness has no systematic cause. In other words, the probability of data being missing is independent of both observed and unobserved values. For example, a sensor that randomly fails to log a value without any pattern.
MAR (Missing At Random)
Here, the missingness can be explained by the data we already have. For instance, men may be less likely than women to respond to a mental health survey, regardless of their actual mental health status. MAR assumes that the missingness is related to observed variables. This assumption cannot be proven but can often be reasonably defended.
NMAR (Not Missing At Random)
In this case, missingness depends on unobserved information. For example, if people with high cholesterol are less likely to report their cholesterol levels, then the missingness itself carries meaning. Ignoring NMAR data can lead to serious bias.
Understanding whether missing values are MCAR, MAR, or NMAR is crucial because it dictates how we should treat them. While MCAR or MAR can sometimes be ignored (if small in proportion), NMAR values must be carefully modeled or imputed.
Imputing Missing Values
One common approach is to fill missing entries with values that preserve certain statistical properties of the dataset.
For numerical data: A simple method is imputing with the mean or median. The mean preserves the central tendency, though it may underestimate variance, while the median is more robust to outliers.
For time series data: Moving averages or interpolation methods (linear, spline, polynomial) are often used to preserve temporal trends.
For categorical data: Mode imputation (filling with the most frequent value) is a straightforward choice.
For flagging missingness: Sometimes analysts use placeholder values outside the natural range, such as imputing age with -1. This makes missingness explicit and allows downstream models to detect the “missing” signal.
While quick fixes like zero imputation or extreme placeholders are sometimes used for exploratory analysis, business-critical models demand more thoughtful imputations that reflect plausible values.
R Packages for Missing Data Imputation
R offers a variety of packages for missing value imputation. Some of the most popular are:
Hmisc – provides functions for imputation as well as data summarization.
missForest – uses random forests to impute both categorical and numerical data.
Amelia – implements multiple imputation using bootstrapping and EM algorithms, useful for time-series cross-sectional data.
mice – stands for Multivariate Imputation by Chained Equations. Considered a gold standard, it generates multiple imputed datasets to account for uncertainty.
Among these, mice is especially popular because of its flexibility and statistical rigor. Let’s see how it works in R.
Using the mice Package – Dos and Don’ts
The mice package is best suited for MAR-type missingness. Its core idea is to use multivariate regression models to predict missing values based on other observed variables. By generating multiple imputations, it accounts for uncertainty rather than pretending that there’s only one “true” imputation.
The package supports several methods:
pmm (Predictive Mean Matching): Best for numerical variables.
logreg (Logistic Regression): For binary categorical variables.
polyreg (Bayesian Polytomous Regression): For categorical variables with 3+ levels.
Proportional odds model: For ordered categorical variables.
Example with NHANES Data
The NHANES dataset (National Health and Nutrition Examination Survey) is bundled with the mice package. Let’s walk through it.
Load required packages
library(mice)
library(VIM)
library(lattice)
Load NHANES data
data(nhanes)
First look
str(nhanes)
The dataset has 25 observations and 4 variables: age, bmi, hyp (hypertension), and chl (cholesterol). Some variables have missing values (NA).
Since age is coded as 1, 2, 3 (representing age bands), it is better treated as a factor:
nhanes$age <- as.factor(nhanes$age)
Exploring Missingness Patterns
The mice function md.pattern() gives an overview of missingness:
md.pattern(nhanes)
This output shows the different patterns of missingness across variables. Complementing this, the VIM package provides excellent visualization tools:
aggr(nhanes, col=mdc(1:2), numbers=TRUE, sortVars=TRUE,
labels=names(nhanes), cex.axis=.7, gap=3,
ylab=c("Proportion of missingness","Missingness Pattern"))
Margin plots allow us to compare distributions when values are present vs. missing, helping assess whether missingness is MCAR:
marginplot(nhanes[, c("chl", "bmi")], col = mdc(1:2),
cex.numbers = 1.2, pch = 19)
Imputing Missing Values
Now, let’s impute using mice:
mice_imputes <- mice(nhanes, m = 5, maxit = 40)
Here:
m = 5 → creates 5 imputed datasets.
maxit = 40 → maximum iterations per imputation.
Check the imputation methods:
mice_imputes$method
Since the variables are numeric, predictive mean matching (pmm) is used.
Extract an imputed dataset (say, the 5th):
Imputed_data <- complete(mice_imputes, 5)
Validating the Imputation
How do we know if the imputations are reasonable? The xyplot() and densityplot() functions help compare observed vs. imputed distributions.
xyplot(mice_imputes, bmi ~ chl | .imp, pch = 20, cex = 1.4)
densityplot(mice_imputes)
If the red (imputed) values resemble the blue (observed) values, imputations are consistent.
Going Beyond: Modelling with Multiple Imputations
One strength of mice is that it doesn’t stop at filling missing values—it enables robust modelling. Instead of using just one completed dataset, analysts can fit models across all imputed datasets and then pool the results.
Fit linear model across imputations
lm_5_model <- with(mice_imputes, lm(chl ~ age + bmi + hyp))
Pool results
combo_5_model <- pool(lm_5_model)
This ensures that the uncertainty from missing values is reflected in model estimates, yielding more reliable insights.
Summary – Why Imputation Matters
Imputing missing values is an essential preprocessing step that prevents loss of valuable data and reduces bias. R’s mice package stands out as a comprehensive tool, offering multiple imputations, flexibility across variable types, and seamless integration with modelling workflows.
Instead of relying on quick fixes like mean substitution, analysts can build more robust and trustworthy models by imputing intelligently. Missing data doesn’t have to be a nightmare—with the right tools, it becomes just another solvable problem in the data pipeline.
For more than 20 years, we’ve partnered with enterprises to solve complex analytics challenges. Our expertise spans Tableau Consulting Services, experienced Power BI consultants, and trusted Snowflake consultants— enabling businesses to transform data into strategic insights.
Top comments (0)