DEV Community: Dunder Data

Why Matplotlib Figure Inches Don’t Match Your Screen Inches and How to Fix it

Ted Petrou — Thu, 09 Dec 2021 19:17:44 +0000

If you’ve worked with the matplotlib data visualization library before, then you’ll be familiar with the term figsize, which is measured…

Continue reading on Dunder Data »

Top 5 Reasons to use Seaborn for Data Visualizations

Ted Petrou — Tue, 16 Nov 2021 17:24:40 +0000

Continue reading on Dunder Data »

Daily Python and Pandas Challenges

Ted Petrou — Wed, 10 Nov 2021 20:51:00 +0000

I’m excited to announce Dunder Data Python and Pandas Challenges! One Python and one Pandas challenge will be released each weekday on python.dunderdata.com. They will vary in difficulty and cover a wide range of topics.

An image of the first challenge available at python.dunderdata.com

Video solutions for the Python Challenges will be available here with the Pandas Challenges available here. You’ll answer the challenges within your very own Jupyter Notebooks.

Can you solve them?

Think you have what it takes to solve the challenges? Get started at python.dunderdata.com now.

The Coronavirus Forecasting Dashboard — Modeling Deaths and Cases around the World

Ted Petrou — Thu, 21 Oct 2021 13:29:28 +0000

The Coronavirus Forecasting Dashboard — Modeling Deaths and Cases around the World

The coronavirus forecasting dashboard shows historical and predicted values for deaths and cases for all countries and US states from the ongoing coronavirus pandemic. In this post, I will present details of the model used and how the dashboard was created. If you are interested in a full tutorial of how this dashboard was built, check out my course Build an Interactive Data Analytics Dashboard with Python.

Dashboard Features

Before delving into the details of the model and technologies used to launch the dashboard, let’s cover some of the dashboard features.

Data Table

The data table on the left side of the dashboard contains totals for deaths and cases for each country of the world by default. You can switch the view to show totals for the last week and previous day via the tabs. Countries can also be aggregated by region of the world.

Totals for each US state have their own view.

Historical and predicted values

Each area in the table can be selected, which will have the effect of showing its historical and predicted values for deaths and cases appear in the graph to the right. Views for cumulative, daily, and weekly are available via the tabs above the graph. A slider below controls the date range.

Choropleth Maps

Choropleth maps (areas colored by specific values) are shown below the graphs as a visual representation of the data table. Hovering over a country reveals its data. The coloring can change by selecting one of the values above the map.

The map changes to show the USA when the US States tab is selected.

Model Performance

The model performance page shows how well the model performed over a chosen date range.

Constructing the Model

One goal of this dashboard was to build a model to predict the number of future cases and deaths. As with any data task, there are a wide variety of approaches for choosing and building a model. There are also many variables to consider. To keep things simple, I chose to only use historical cases as the single input into the model. Historical cases are known, easy to retrieve from a single source, and provide good predictive power for future cases. Let’s take a look at cumulative case count the UK from March 2020 to the end of October 2020.

The UK shows a typical pattern of cases — a first wave of cases beginning in March and ending in July with another wave starting in September. The majority of countries have now seen multiple waves that behave similarly. There appears to be exponential growth during the beginning of a wave, then steady growth, and then ending in exponential decline. The waves form shapes that are well-modeled by a class of functions called “S-Curves”, with the generalized logistic function proving to be a good choice with flexibility to model a single wave. Take a look at its form below:

The generalized logistic function

Where L represents the upper bound for cases, x0 controls the horizontal shift, s the vertical shift (useful when modeling new waves), k controls the growth rate (steepness of the curve), and v controls the amount of asymmetry difference between the exponential growth and decline phases (most coronavirus waves have long tails towards their upper bound like the UK does above).

Here, we use this model to make predictions for the UK after allowing the model to view data through April 10.

One formulation of the generalized logistic function is the Gompertz curve, also popular for modeling coronavirus waves. Each area of the world and US state has its model updated on a daily basis using this generalized logistic function. The results are displayed in the dashboard.

Dashboard Construction

The dashboard was constructed using the Dash Python library, an open-source project from the Plotly company. Dash provides the data tables, widgets, and interactivity necessary to create nearly any dashboard you wish. The data visualizations themselves are created by the Plotly library.

The Layout and Interactivity

Dash applications are composed of two parts, the layout and the interactivity. In the layout, you define each component of the dashboard, such as the data tables, tabs, links, graphs, maps, etc… You can use custom HTML and CSS to place more components, and style and arrange them.

After setting the layout, callback functions can be defined to add interactivity. A callback function is triggered by specific events in your dashboard such as clicking a radio button or selecting an item in a dropdown menu. In this dashboard, one callback is triggered by the clicking of a country or state name in the data table on the left. Doing so, changes the data displayed in the graphs on the right to that area. The radio buttons above the maps also trigger a callback to change the coloring of each are by the specific metric selected.

Dash Bootstrap Components

The third party library Dash Bootstrap Components has several more components styled using the popular Bootstrap JavaScript/CSS library and was used to make the radio buttons.

Visit the Dashboard

If you haven’t already done so, visit the dashboard, explore its features and let me know what you think of it.

Displaying Pandas DataFrames Horizontally in Jupyter Notebooks

Ted Petrou — Mon, 30 Aug 2021 15:47:28 +0000

Continue reading on Dunder Data »

Python Pandas Certification Courses

Ted Petrou — Mon, 22 Mar 2021 13:36:21 +0000

I am excited to announce Python Pandas Certification Courses — a series of courses to help you become an expert at using the pandas library to analyze data, each with a challenging certification exam given at the end.

The main goal of the Python Pandas Certification series of courses is to provide you with a path to gain mastery of the pandas library so that you can use it confidently in a professional environment.

First Course FREE — Begins March 29!

The first course in the certification series, Selecting Subsets of Data in Pandas, is free to take. It includes 8 hours of live instructions, over 100 exercises, and a challenging certification exam. Sign up to take this class for FREE here.

Courses

There are currently seven courses in the series that will cover nearly all of the pandas library. A description of each course is provided below. To get more info and to register, visit the Python Pandas Certification home page.

Course 1: Selecting Subsets of Data in Pandas

Selecting a subset of data is perhaps the most common task performed in pandas. Unfortunately, there are many different (and overlapping) ways to select various rows and columns of a pandas DataFrame. This course goes into great detail into all of the various ways to select data and gives guidance on choosing the most effective method for specific situations. This course is FREE to take and begins March 29.

Course 2: Essential Pandas Commands

In this course, you’ll learn the most common and fundamental commands used in just about every data analysis. You’ll learn the distinction between methods that aggregate (summarize the data with a single number) and those that do not. You’ll learn how to change the direction of the operation so that calculations are done horizontally as well as vertically (the default for DataFrames). You’ll work with missing values and perform other tasks such as sorting, ranking, and looking for unique values.

You will also learn how to operate on string and date columns, which require a different set of methods than numeric columns. You’ll learn about all of the many kinds of data types that pandas offers and how to operate with them.

Course 3: Grouping Data

In Essential Pandas Commands, you learn to operate on the entire dataset. In Grouping Data, you learn how to operate on independent groups with the dataset. You’ll learn how to aggregate and summarize each group by a single statistic and present the information as a pivot table.

Grouping data in pandas opens you up to the possibility of writing extremely inefficient code. It’s not unusual to see performance gains of 10–100x by rewriting code in a more efficient manner. You will learn specific approaches to writing efficient grouping commands.

Course 4: Time Series

Pandas provides a variety of tools to process time series data. You will learn how to sample and group time series by different periods of time. You’ll also learn how to perform moving aggregate operations. You will complete a project analyzing coronavirus cases and deaths from different countries.

Course 5: Cleaning Data

It’s often that you’ll want to transform the initial dataset given to you to a different form, before attempting any serious data analysis. Text data is particularly difficult to work with if it is not properly transformed. You’ll learn the fundamentals of regular expressions, which match patterns within text to help discover and extract particular pieces of information. You’ll also learn how to reshape your data and convert it to a form that makes data analysis simpler.

Course 6: Joining Data

Working with multiple datasets simultaneously often requires you to merge them into a single table. You’ll learn how to join together multiple pandas objects together. You’ll also learn how to join tables using SQL-like logic. You will learn about data normalization and complete exercises where you transform datasets so that they are normalized.

Course 7: Data Visualization

You will first learn the basics of matplotlib, a powerful and popular data visualization library in Python. You’ll then learn how to plot data with it as well as with the pandas and seaborn libraries. You will learn about the different approaches that different libraries take to plotting data.

Exercises, Projects, and Exams

To help you gain mastery of pandas so that you can use it effectively in a professional environment, each course will come with dozens of exercises and at least on project. All exercises have detailed solutions to help you reinforce the material.

Each course has a challenging comprehensive exam on the material covered. Passing it will earn you a certificate of completion.

Register for the entire series

If you desire a mastery of pandas with proof that you are indeed an expert, then you’ll want to sign up for the Python Pandas Certification Courses.

Instructor

My name is Ted Petrou and I have written two books on the pandas library:

I have completely mastered how to use the pandas library effectively and would love to be your guide for reaching the same level. I have taught more than 1,000 hours of live classes on pandas and know exactly where the pain points are and how to alleviate them. My classes are interactive and keep your hands on the keyboard so that you are coding along with me. Feel free to reach out to me with any questions — ted@dunderdata.

The Remarkable Similarity of Covid-19 Infection Waves throughout the World

Ted Petrou — Tue, 01 Dec 2020 19:26:56 +0000

I have been modeling covid-19 cases and deaths in all areas of the world for the past six months. I built the ![coronavirus forecasting dashboard][https://coronavirus.dunderdata.com] which makes predictions for covid cases/deaths for all countries and US states. In this post, I will explore the remarkable similarity of covid infection waves across many areas of the world.

Similarity of Covid Waves by Area

Understanding the shape and trajectory of covid waves can provide useful approximations for the time frame where many infections will occur. When I speak of “covid waves”, I refer to the rise and fall of cases (confirmed infections) in a particular area of the world.

One remarkable observation thus far, is that nearly all areas have covid waves that follow similar trajectories and last a similar amount of time. The most widely available data to track covid waves is the number of daily cases reported. While the severity of individual cases varies, the daily number provides a good indication of the overall covid activity in an area. To help improve the discussion, I have defined the following covid wave stages and approximate length each area spends within them.

Trough — same low number of daily cases
Start — slow increase in daily cases (1–3 weeks)
Outbreak — rapid (quadratic/exponential) increase in daily cases (1–3 weeks)
Growth — steady (linear) increase in daily cases (1–3 weeks)
Flattening — daily cases slowly stop increasing (1–3 weeks)
Peak — same number of daily cases (1–3 weeks)
Drop — steady (linear) decrease in daily cases (1–3 weeks)
Stop — slow decrease in daily cases (1–3 weeks)
Trough — same low number of daily cases (until the next wave)

Example Covid Waves

In this section, I’ll look at covid waves in specific areas. During the early parts of the pandemic (March-April 2020), covid struck areas that rank high in population density, industrialization, and connectedness to the world — Western Europe and the North Eastern United States. Although testing at this time was far lower than it is today, these countries and US states will be used as first examples of covid waves.

Let’s take a look at New York state which experienced its first major covid outbreak in mid-March. The data below is of smoothed daily cases. Using the raw data cases is much messier and more difficult to spot the wave stages. Although New York state is fairly large, most of the population (and initial covid activity) was from New York City, so it works well as a single “area”. As we will discover, its covid wave is typical, lasting about 15 weeks or 3–4 months.

Italy was the first western nation to have an outbreak, just a couple weeks before New York. It’s covid wave followed a very similar pattern.

Normalizing Covid Waves

Instead of plotting each covid wave individually, several of them can be plotted at the same time for easier comparison. Because, each has a different maximum value, they are normalized (scaled) such that the maximum value is 1 for each. Five North Eastern states are plotted below. You can see the similarity between each.

Shifting Covid Waves

Different states reach their peaks at different times. To make the comparison even easier, each wave is shifted so that their peaks align. Several Western European nations are plotted together with these states. Despite being in different areas of the world, the waves share a remarkably similar shape.

Examining Smaller Areas

Looking at aggregated state or country data hides covid outbreaks that happen at different dates within that particular state or country. Take a look at daily covid cases in the large state of Texas.

While there is clearly a wave from June to September and another one beginning in October, this aggregated total misses waves from different areas of the state.

Texas is the second largest US state, sitting around 270,000 square miles. It takes around 14 hours of driving to go from El Paso (Western border) to Orange (Eastern border). El Paso is very isolated from all the other major cities in Texas, separated by hundreds of miles of desert, and so far west it’s in a different time zone.

El Paso vs Houston

The wonderful John Hopkins Github repository has covid data for each county in the US. From this data, smoothed daily cases are made for both El Paso and Houston, the largest city in Texas, very far from El Paso. While both cities had summer waves, El Paso had a much larger wave beginning in mid-September and is just now close to completing. Houston appears to be in the growth stage of a second wave.

Even though El Paso is a fairly large city with nearly 700,000 residents, it accounts for less than 3% of the total population of Texas. Their latest major wave barely registers on the aggregated Texas total of daily cases from above.

El Paso’s latest wave appears to follow the typical pattern seen in the initial waves of the North Eastern US and Western Europe, and is likely to last the same amount of time (3–4 months).

New York City vs All Others

Getting back to New York state — splitting the counties by proximity to New York City shows a clear difference in their covid waves. The less dense areas outside of New York City are seeing their largest wave (and hopefully nearing the flattening stage). It remains to be seen whether this wave lasts the typical 3–4 months.

India

Turning to India, the country likely with the most overall total infections (currently second in confirmed infections), has daily case data that appears to be one long six or more month wave.

Like the US, India is a huge country with diverse geographic regions. Inspecting the nine largest Indian states reveals covid waves that are fairly similar to those in the US and Europe, lasting perhaps a bit longer (4–5 months).

All of these states are very large, with the smallest having 43 million residents. I imagine that dividing these states into even smaller regions could reveal more typical covid waves.
Isolated Areas

This pattern of covid waves seems to hold regardless of the geography or population density. The rural states of Montana, North Dakota, and South Dakota saw little covid activity until July/August with outbreaks happening in September. All three are following a typical covid wave trajectory and are into the drop phase and should complete their wave around the new year.

Eastern Europe

Eastern Europe (which I define as countries east of the border formed by Germany, Austria, and Italy) also had little covid activity until the late summer. These countries are far more dense than the central US states, but are less connected to the world than Western Europe, have less tourism, and less GDP per capita. The closest countries to Western Europe, Czechia and Slovakia, began their prototypical covid wave patterns first, followed by those countries further east.

Australia and New Zealand

The isolated islands of Australia and New Zealand with little overall covid activity due to harsh lockdown measures still exhibited similar covid waves.

Counterexamples

I have yet to locate an area of the world with a covid wave that differs significantly from the ones above. Nearly all appear to follow the same pattern from trough to peak back to trough in a span of 3–4 months. If an area appears to violate this pattern, it is likely because the are encompasses a geographic area that is too large with a large population.

Fractal Nature of Covid Waves

In this section, I explore the idea that covid waves have some properties of fractals all the way down to within the human body. From above, we saw how aggregated totals of large states or countries such as New York, Texas, and India can hide the underlying covid waves of smaller geographic areas. The best examples to reveal covid waves are small, rural, isolated cities, states, or countries, such as El Paso, North Dakota, or Czechia.

It could well be the case that examining smaller and smaller areas, for instance, neighborhoods in a city, similar covid waves would appear. Geometric shapes that have the same patterns for each increasingly smaller area that you magnify are called fractals.

The animated image on the left is a classic example of a fractal showing the same repeating pattern while magnification goes on indefinitely.

As a whole, the covid-19 pandemic is not quite fractal since the aggregated totals do have different trajectories than the small geographic areas. But, I suspect that as we examine smaller and smaller groups, the covid waves will converge to some limit.

City to Neighborhood to House to Self

The city level was the smallest area we looked at in this post with the example of the isolated city of El Paso having its own covid wave at a different time than the rest of Texas.

But what if we looked further down and performed an analysis by city neighborhood? Intuitively, people in the same neighborhood would have their own covid wave. Going further, isolated communities such as nursing homes have had some of the worst covid outbreaks and likely have their own wave.

How about our own bodies? Once someone has covid, the virus multiplies quickly before the immune system fights it and clears the infection. This process seems to last between 2–4 weeks. Could this be the limiting covid wave after peeling back all of the layers?

Black Friday Special 2020 — Get 50% Off — Limited Time Offer!

If you’ve enjoyed this article and are interested in mastering the python data science libraries so that you can produce trusted results in a professional environment, take a course with me. You can get 50% off all my courses today for a limited time!

Instantly Beautify Matplotlib Plots by Viewing all Available Styles

Ted Petrou — Fri, 26 Jun 2020 23:32:55 +0000

In this post, you’ll learn about the different available matplotlib styles that can instantly change the appearance of the plot. Let’s begin by making a simple line plot using the default style. This simple style is often the first (and sometimes only) style that many people encounter with matplotlib not realizing how easy it is to choose others.

import matplotlib.pyplot as plt
import numpy as np

x = np.arange(-2, 8, .1)
y = .1 \* x \*\* 3 - x \*\* 2 + 3 \* x + 2

fig, ax = plt.subplots(figsize=(4.5, 3), dpi=100)
ax.plot(x, y)
ax.set\_title('Default Matplotlib Style');

Viewing all of the available styles

There are nearly 30 builtin styles to matplotlib that can be activated with the plt.style.use function. The style names are available in the plt.style.available list. In the following code, we iterate through all of the available styles, then make the same line plot as above, setting the style temporarily for each Axes with plt.style.context.

fig = plt.figure(dpi=100, figsize=(10, 20), tight\_layout=True)
available = ['default'] + plt.style.available
for i, style in enumerate(available):
    with plt.style.context(style):
        ax = fig.add\_subplot(10, 3, i + 1)
        ax.plot(x, y)
    ax.set\_title(style)

Showing the style settings

Each style’s settings are stored in the plt.style.library dictionary. Here, we get all of the settings for the seaborn-darkgrid style.

plt.style.library['seaborn-darkgrid']

RcParams({'axes.axisbelow': True,
          'axes.edgecolor': 'white',
          'axes.facecolor': '#EAEAF2',
          'axes.grid': True,
          'axes.labelcolor': '.15',
          'axes.linewidth': 0.0,
          'figure.facecolor': 'white',
          'font.family': ['sans-serif'],
          'font.sans-serif': ['Arial',
                              'Liberation Sans',
                              'DejaVu Sans',
                              'Bitstream Vera Sans',
                              'sans-serif'],
          'grid.color': 'white',
          'grid.linestyle': '-',
          'image.cmap': 'Greys',
          'legend.frameon': False,
          'legend.numpoints': 1,
          'legend.scatterpoints': 1,
          'lines.solid\_capstyle': 'round',
          'text.color': '.15',
          'xtick.color': '.15',
          'xtick.direction': 'out',
          'xtick.major.size': 0.0,
          'xtick.minor.size': 0.0,
          'ytick.color': '.15',
          'ytick.direction': 'out',
          'ytick.major.size': 0.0,
          'ytick.minor.size': 0.0})

To set a style for the current session, do so with plt.style.use and reset to the default style with plt.style.use('default').

Completely Master Matplotlib

If you are interested in completely mastering matplotlib so that you can produce trusted results, take a look at my book Master Data Analysis with Python.

Matplotlib Builtin Styles

Ted Petrou — Fri, 26 Jun 2020 16:51:16 +0000

Instantly Beautify Matplotlib Plots by Viewing all Available Styles

import matplotlib.pyplot as plt
import numpy as np

x = np.arange(-2, 8, .1)
y = .1 \* x \*\* 3 - x \*\* 2 + 3 \* x + 2

fig, ax = plt.subplots(figsize=(4.5, 3), dpi=100)
ax.plot(x, y)
ax.set\_title('Default Matplotlib Style');

Viewing all of the available styles

fig = plt.figure(dpi=100, figsize=(10, 20), tight\_layout=True)
available = ['default'] + plt.style.available
for i, style in enumerate(available):
    with plt.style.context(style):
        ax = fig.add\_subplot(10, 3, i + 1)
        ax.plot(x, y)
    ax.set\_title(style)

Showing the style settings

Each style’s settings are stored in the plt.style.library dictionary. Here, we get all of the settings for the seaborn-darkgrid style.

plt.style.library['seaborn-darkgrid']

RcParams({'axes.axisbelow': True,
          'axes.edgecolor': 'white',
          'axes.facecolor': '#EAEAF2',
          'axes.grid': True,
          'axes.labelcolor': '.15',
          'axes.linewidth': 0.0,
          'figure.facecolor': 'white',
          'font.family': ['sans-serif'],
          'font.sans-serif': ['Arial',
                              'Liberation Sans',
                              'DejaVu Sans',
                              'Bitstream Vera Sans',
                              'sans-serif'],
          'grid.color': 'white',
          'grid.linestyle': '-',
          'image.cmap': 'Greys',
          'legend.frameon': False,
          'legend.numpoints': 1,
          'legend.scatterpoints': 1,
          'lines.solid\_capstyle': 'round',
          'text.color': '.15',
          'xtick.color': '.15',
          'xtick.direction': 'out',
          'xtick.major.size': 0.0,
          'xtick.minor.size': 0.0,
          'ytick.color': '.15',
          'ytick.direction': 'out',
          'ytick.major.size': 0.0,
          'ytick.minor.size': 0.0})

To set a style for the current session, do so with plt.style.use and reset to the default style with plt.style.use('default').

Completely Master Matplotlib

If you are interested in completely mastering matplotlib so that you can produce trusted results, take a look at my book Master Data Analysis with Python.