If you use the Python pandas library for data science and data analysis things, you'll eventually see
None in your DataFrame. These values all represent missing data. However, there are subtle and not-so-subtle differences in how they behave and when they appear..
Let's take a look at the three types of missing values and learn how to find them.
If a column is numeric and you have a missing value that value will be a
NaN stands for Not a Number.
NaNs are always floats. So if you have an integer column and it has a
NaN added to it, the column is upcasted to become a
float column. This behavior may seem strange, but it is based on NumPy's capabilities as of this writing. In general, floats take up very little space in memory, so pandas decided to treat them this way. The pandas dev team is hoping NumPy will provide a native NA solution soon.
If a column is a DateTime and you have a missing value, then that value will be a
NaT stands for Not a Time.
object dtype column - the dtype for strings as of this writing - can hold
NaT or all three at the same time!
NaN is a NumPy value.
NaT is a Pandas value.
None is a vanilla Python value.
However, they display in a DataFrame as
Behavior with missing values can get weird. Let's make a Series with each type of missing value.
pd.Series([np.NaN, pd.NaT, None])
Pandas created the Series as a DateTime dtype. Ok.
You can cast it to an object dtype if you like.
pd.Series([np.NaN, pd.NaT, None]).astype('object')
But you can't cast it to a numeric dtype.
pd.Series([np.NaN, pd.NaT, None]).astype('float')
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-255-66ec4de18835> in <module> ----> 1 pd.Series([np.NaN, pd.NaT, None]).astype('float') ... TypeError: cannot astype a datetimelike from [datetime64[ns]] to [float64]
Also note that you can change an object column with
Nones into a numeric column with
pd.to_numeric. No problem.
Another bizarre thing about missing values in Pandas is that some varieties are equal to themselves and others aren't.
NaN doesn't equal
np.NaN == np.NaN
NaT doesn't equal
pd.NaT == pd.NaT
None does equal
None == None
Now let's turn our attention finding missing values.
Finding Missing Values with df.isna()
df.isna() to find
None values. They all evaluate to
True with this method.
A boolean DataFrame is returned if
df.isna() is called on a DataFrame and a Series is returned if called on a Series.
df.isna() in action! Here's a DataFrame with all three types of missing values:
Here's the code to return a boolean DataFrame with
True for missing values.
A one-liner to return a DataFrame of all your missing values is pretty cool. Deciding what to do with those missing values is a whole nother question that I'll be exploring in my upcoming Memorable Pandas book.
Note that it's totally fine to have all three Pandas missing value types in your DataFrame at the same time, assuming you are okay with missing values.
I hope you found this intro to missing values in the Python pandas library to be useful. 😀
If you did, please do all the nice things on Dev and share it on your favorite social media so other people can find it, too. 👏
I write about Python, Docker, and data science things. Check out my other guides if you're into that stuff. 👍
You don't want to MISS them! (Missing values. Get it?) 🙄
Thanks to Kevin Markham of Data School for suggestions on an earlier version of this article!