DEV Community: Nic Fox

How to Get Data from an API & Process XML in Python

Nic Fox — Sun, 05 Jul 2020 12:41:53 +0000

Thanks to the requests library it is quick and easy to get data from an API, even for beginners, and the ElementTree library makes it easy to process the XML that an API provides. This tutorial, aimed at beginners, will show you an end-to-end example using python to get information about property listings from the Zoopla API.

We start by going to the Zoopla API page and clicking 'Register':

Fill in the form:

You'll be given an API key (a string of text) that looks something like this:

abc79x5kxxx8xxx6def4abc1

You're then allowed to use their interactive tool to search for property listings. I searched for a postcode by typing in 'WR10', with 'area' left blank, and clicked the 'Try it!' button.

This produced a request URL which will be needed if we want to do this search programmatically in our python code:

(Note: my API key has been cropped from the image but your API key will be shown after 'api_key=')

Notice how the URL contains the postcode that was used in the search, as well as the API key.

The 'Try it' feature also produces the XML output containing the search results:

It's a good idea to look through this to get an idea of the structure of the XML to make it easier to use the information later. Tip: if you copy-paste the XML into a tool like Sublime Text, you can collapse/expand the tags to make it easier to see the hierarchy of attributes.

Now let's start coding. We first need to import the following libraries (install them first if you need to):



import requests  # for using API
import xml.etree.ElementTree as ET  # for parsing XML
import numpy as np  # for using pandas
import pandas as pd  # for using dataframes

Then save your api key as a variable. I've called mine api_key:



api_key = 'abc79x5kxxx8xxx6def4abc1'

Copy-paste the request URL that was created by the 'Try it!' feature on the Zoopla API website and save it as a variable called request_url:



request_url = 'http://api.zoopla.co.uk/api/v1/property_listings.xml?postcode=WR10&api_key=abc79x5kxxx8xxx6def4abc1'

(Note: the API key at the end of the request URL above is a dummy string. You should have your own API key shown there.)

We need to use this request_url to call the API from our python code, like this:



r = requests.get(request_url)  # call API

We can then use ElementTree to process the XML that the API sent back in its response:



root = ET.fromstring(r.content)  # parse XML

Let's say that we're only interested in the description for each property listing. We can print each description by looping through the description tags and printing the text, like this:



# loop through each "description" element in the XML 
for description in root.iter('description'):  
        print(description.text, # print the description
              '\n')  # leave space under the description

My output looks like this:

Let's modify the loop to put the descriptions into a dataframe instead of printing them:



# create a list to store the descriptions
desc_list = []

# loop through each "description" element in the XML
for description in root.iter('description'):
        # add the description text to the list of descriptions   
        desc_list.append(description.text)  

# convert the list of descriptions to a dataframe
properties_df = pd.DataFrame({"description":desc_list})

# view the properties dataframe
# (with a wider column "description" so we can see the full text)
properties_df\
.style.set_properties(subset=['description'], **{'width': '800px'}) \
.hide_index() # don't show the index column

My output:

Let's get the property type for each listing and add them to the dataframe:



# create a list to store the property type for each property
prop_types_list = []

# loop through the property listings
for prop_type in root.iter('property_type'):
        # add the property type to the list
        prop_types_list.append(prop_type.text)

# add the property types to the dataframe as a new column
properties_df["property_type"] = prop_types_list

# view the updated dataframe
properties_df\
.style.set_properties(subset=['description'], **{'width': '800px'}) \
.hide_index()

The output:

What if we wanted to get the property listings for multiple postcodes? This is one way of doing it using a list of postcodes to dynamically build a request URL for each postcode:



# create a list of the postcodes to get property listings for
postcodes = ['PE19','WR10', 'PO38'] 

# create a list to store the request URL for each postcode
request_urls = []

# loop through the postcodes
for postcode in postcodes:

    # create a request URL for the postcode
    request_url = "http://api.zoopla.co.uk/api/v1/property_listings.xml?postcode=" + postcode + "&api_key=" + api_key

    # add the request URL to the list
    request_urls.append(request_url)

# view the request URLs
request_urls

My output (cropped so you can't see my API key):

Now lets loop through these request URLs and get the description and property type for each listing in each postcode, and store them in a dataframe:



# create lists to store descriptions and property types
desc_list = []
prop_types_list = []

# loop through the request URLs to get
# the property descriptions and 
# property types
for request_url in request_urls:
    r = requests.get(request_url)
    root = ET.fromstring(r.content)

    for description in root.iter('description'):
        desc_list.append(description.text)

    for prop_type in root.iter('property_type'):
        prop_types_list.append(prop_type.text)


# create a dataframe of the descriptions and property types 
properties_df = pd.DataFrame({"description":desc_list, "property_type":prop_types_list})

# view the updated dataframe
properties_df\
.style.set_properties(subset=['description'], **{'width': '800px'}) \
.hide_index()

My output:

You could make the data frame more useful by adding the postcodes and other information from the XML such as the street name, post town, county, number of bedrooms and much more. Then you could extend the fun and do some data analysis on it and create some data visualisations. If you want some help getting started with text analysis then this tutorial might help, and if you'd like some tips on how to create good-looking charts in python you may like this tutorial.

How to Analyse & Clean Text Data in Python

Nic Fox — Mon, 29 Jun 2020 07:16:17 +0000

This is a beginner's tutorial (by example) on how to analyse text data in python, using a small and simple data set of dummy tweets and well-commented code. It will show you how to write code that will:

import a csv file of tweets
find tweets that contain certain things such as hashtags and URLs
create a wordcloud
clean the text data using regular expressions ("RegEx")
show you what tokenisation is and how to do it
explain what stopwords are and how to remove them
create a chart showing the most frequent words in the tweets, and their frequencies

The Jupyter Notebook is available on GitHub here.

The code uses pandas dataframes so if you're unfamiliar with those then this tutorial may help.

First, import the libraries and set the relevant configurations:

import numpy as np   # an essential python module
import pandas as pd   # for importing & transforming data
import re   # for regular expressions
import matplotlib.pyplot as plt   # for wordclouds & charts
import seaborn as sns   # for charts
sns.set_style("whitegrid");   # chart background style
plt.rcParams['figure.dpi'] = 360   # for high res chart output
from wordcloud import WordCloud   # for the wordcloud :)
import spacy   # for tokenising text
from spacy.lang.en import English  # for tokenising text
nlp = English()   # for tokenising text
from collections import Counter   # for getting freq of words

Then, let's import a csv file containing 2 columns (tweet_id and tweet) and 10 rows. I created this csv by just copy-pasting a few tweets from Twitter into an Excel file and saved it as a csv file.

# import the csv file into a Pandas dataframe
tweet_df = pd.read_csv("tweet_sample.csv")

# view the shape of the data (the number of rows and columns)
print(f"The shape of the data is: {tweet_df.shape}")

# view the data with the "tweet" column widened to 800px 
# so that the full tweet is displayed,
# and hide the index column
tweet_df\
.style.set_properties(subset=['tweet'], **{'width': '800px'})\
.hide_index()

My output looks like this:

(Note: If you're new to f strings, i.e. the print(f"...{}") bit in the code above, then this tutorial may help)

Now let's find all the tweets that contain a hashtag.

# let's find out how many tweets contain a hashtag
tweets_with_hashtags = tweet_df.loc[tweet_df["tweet"].str.contains("#")]

# view the number of tweets that contain a hashtag
print(f"Number of tweets containing hashtags: {len(tweets_with_hashtags)}")

# view the tweets that contain a hashtag
tweets_with_hashtags\
.style.set_properties(subset=['tweet'], **{'width': '800px'}).hide_index()

My output:

How many contain a URL?

# how many tweets contain a URL i.e. "http"?
tweets_with_URLs = tweet_df.loc[tweet_df["tweet"].str.contains("http")]

# view the number of tweets that contain a URL
print(f"Number of tweets containing URLs: {len(tweets_with_URLs)}")

# view the tweets that contain a URL
tweets_with_URLs\
.style.set_properties(subset=['tweet'], **{'width': '800px'}).hide_index()

My output:

Let's create a wordcloud. Before we can do that, we need to create one long string containing all the tweets.

# create a single string containing all the tweets, 
# as this will be needed to be able to create a wordcloud
tweet_string = " ".join(tweet for tweet in tweet_df["tweet"])

# view the first 200 elements of the string to check 
# this worked as expected
tweet_string[0:200]

My output:

Now we can create the wordcloud using this long string, and then view it. We'll only view the top 100 words so max_words will be set to 100.

# create the wordcloud
tweet_wordcloud = WordCloud(background_color="white", 
                              max_words=100, 
                             ).generate(tweet_string)

# view the wordcloud
plt.imshow(tweet_wordcloud, interpolation='bilinear')
plt.axis('off')
plt.show()

My output:

Let's clean up the data using the re python library to use regular expressions ("RegEx"). There is a useful guide on RegEx in Python here.

First, let's replace mentions (e.g. @Dilbert_Daily) with '@USER'.

# replace all the mentions (e.g. @Dilbert_Daily) 
# from the tweets with '@USER'
tweet_string = re.sub(r'@\w+','@USER ', tweet_string)

# view the first 200 elements of the string to check 
# this worked as expected
tweet_string[0:200]

My output:

...then replace all URLS with '_URL'

# replace all the URLs with '_URL_'
tweet_string = re.sub(r'http\S+','_URL_ ', tweet_string)

# view the first 200 elements of the string to check 
# this worked as expected
tweet_string[0:200]

My output:

...then convert the text to lower case so, for example, instead of having "Be" and "be" included as 2 separate words, we only have "be":

# convert the text to lower case so, for example, instead 
# of having "Be" and "be" included
# as 2 separate words, we'd only have "be"
tweet_string = tweet_string.lower()

# view the first 200 elements of the string to check 
# this worked as expected
tweet_string[0:200]

My output:

Let's remove the extra spaces so there's only one space between words:

# remove extra white spaces so there is only one 
# space between words
tweet_string = re.sub(r'\s+',' ', tweet_string)

# view the first 200 elements of the string to 
# check this worked as expected
tweet_string[0:200]

My output:

Let's view the wordcloud for this cleaned-up string:

# create the wordcloud
tweet_wordcloud = WordCloud(background_color="white", 
                              max_words=100, 
                             ).generate(tweet_string)

# view the wordcloud
plt.imshow(tweet_wordcloud, interpolation='bilinear')
plt.axis('off')
plt.show()

Let's change it so that the wordcloud only shows the top 25 words:

# create the wordcloud
tweet_wordcloud = WordCloud(background_color="white", 
                              max_words=25, 
                             ).generate(tweet_string)

# view the wordcloud
plt.imshow(tweet_wordcloud, interpolation='bilinear')
plt.axis('off')
plt.show()

Next, let's tokenise the string i.e. split the string into separate tokens (i.e. separate elements in a collection) using spacy. In this example, each token will be a word. This isn't the only option used in NLP. We could choose tokens to be individual characters, parts of words, 2 words (known as 2-grams/bi-grams), 3 words (3-grams/tri-grams), 4 words (4-grams), sentences... But for this example each token will be a word.

# create a spacy document by pointing spacy to the 
# tweet string
tweet_doc = nlp(tweet_string)

# get all tokens that aren't punctuation
tweet_words = [token.text for token in tweet_doc if token.is_punct != True]

# get the frequency of each word (token) in the tweet string
tweet_word_freq = Counter(tweet_words)

# get the 5 most frequent words
five_most_common_words = tweet_word_freq.most_common(5)

# view the 5 most common words
five_most_common_words

My output:

Now let's create a Pandas dataframe containing all the tokens (words) and the frequency of each word, so that we can use this to create a chart.

# create a Pandas dataframe containing the tokens 
# (words) and their frequencies
freq_df = pd.DataFrame.from_dict(tweet_word_freq, orient='index').reset_index()

# rename the columns to "word" and "freq"
freq_df.columns=["word", "freq"]

# sort the dataframe so that the most frequent word is 
# at the top and view the first 3 rows
freq_df.sort_values(by="freq", ascending=False).head(3)

My output:

Display a bar chart (using seaborn - which we aliased as sns) of the top 25 most frequent words, and their frequencies:

# display a bar chart showing the top 25 words 
# and their frequencies
fig, ax = plt.subplots(figsize=(12,6))
sns.barplot(data=freq_df.sort_values(by="freq", ascending=False).head(25), 
            y="word", 
            x="freq", 
            color='#7bbcd5')
plt.ylabel("Word")
plt.xlabel("Frequency")
plt.title("Top 25 Most Frequent Words")
sns.despine();

My output:

(Note: if you're new to creating charts in Python using seaborn then you may find this post useful)

Next, let's remove words that are really common but not very useful for understanding the topics being discussed, such as 'and'. In the NLP world these are called 'stopwords'. They can be removed easily using the spacy tokeniser's is_stop attribute:

# get all tokens that aren't punctuation 
# and aren't stopwords
tweet_words = [token.text for token in tweet_doc if token.is_punct != True and \
token.is_stop != True]

# get the frequency of each word (token) in the tweet string
tweet_word_freq = Counter(tweet_words)

# re-create the Pandas dataframe containing the 
# tokens (words) and their frequencies
freq_df = pd.DataFrame.from_dict(tweet_word_freq, orient='index').reset_index()

# rename the columns to "word" and "freq"
freq_df.columns=["word", "freq"]

# display a bar chart showing the top 25 words and their
# frequencies (which will exclude the stopwords this time)
fig, ax = plt.subplots(figsize=(12,6))
sns.barplot(data=freq_df.sort_values(by="freq", ascending=False).head(25), 
            y="word", 
            x="freq", 
            color='#7bbcd5')
plt.ylabel("Word")
plt.xlabel("Frequency")
plt.title("Top 25 Most Frequent Words (Excluding Stopwords)")
plt.xticks([0,1,2,3])
sns.despine();

My output:

The chart now gives us a much better indication of the topics being discussed in the tweet text.

There's lots more you can do of course, for example:

how many tokens (words) are in the longest tweet?
how many are in the shortest?
what's the mean average number of tokens? (Answers to these length questions are useful later on if you're going to use machine learning models)
are there any empty documents (tweets)? Our dataset is so small that we can see that there aren't any empty tweets but in real data sets that are larger you'd need to find out programmatically

...but that's probably enough for now. Data cleaning and analysis is a big part of working with text data, and deciding what to change, and how, will depend on the problem being solved and is part of the art of data science. For example, should mentions (e.g. @Dilbert_Daily) be removed/replaced or are they a useful predictor? Will removing punctuation improve or reduce a machine learning model's performance or make no difference at all? Should the text be converted to lower case? There's no right answer, so its useful to be able to easily play around with the text data and experiment.

I recommend playing around with your own dummy data, trying different regular expressions with the re module, and playing around with the wordcloud, spacy and seaborn modules. There's a great tutorial for spacy on their website.

Getting Started in NLP

Nic Fox — Sat, 27 Jun 2020 11:04:52 +0000

When I first started learning about NLP (Natural Language Processing - processing text data) I wanted to find a beginner's guide that gave me a framework for understanding the topics and terminology that I needed to search for to find learning resources. I struggled to find one. However, just this week, @omarsar0 ("elvis") on Twitter has posted some mind maps that look really useful, and I thought I should post them here in case other beginners find these useful too.

Text Mining Mind Map:

NLP Mind Map:

Also, here are some beginner's tutorials and code examples (in python) that I've found really helpful for getting started:

...and a useful book is "Applied Text Analysis with Python"

There are many more complicated "state of the art" (SOTA) methods not covered in the resources above (e.g. Word2Vec, GloVe, ELMo, BERT, and SOTA models since BERT) but I recommend staying away from those until you understand text mining with the traditional methods.

There are also many different tasks that can be performed using NLP techniques (e.g. translating between languages, summarising text, question answering, and more) and I recommend starting out with "text classification" or "sentiment analysis" (which is a type of text classification). There are lots of free tutorials and examples online for sentiment analysis e.g. trying to classify whether a Yelp review is a positive review or a negative review. Perhaps even before that I'd recommend importing text data and creating a wordcloud (this tutorial will help). If you don't know what a word cloud is, below is an example. It's a way to visualise the frequency of each word in some text.

I created the wordcloud above using this code:

# import matplotlib so that the wordcloud can be displayed
import matplotlib.pyplot as plt
%matplotlib inline

# import wordcloud so that the wordcloud can be created
from wordcloud import WordCloud

# create a string of text
text_string = "NLP, NLP, NLP, NLP, NLP, NLP, NLP, NLP, NLP, \
                text, text, text, spacy, spacy,\
                sentiment analysis, translation, stopwords,\
                tokenisation, tokenisation, tokenisation,\ 
                part-of-speech tagging, bag of words, TF-IDF,\
                embedding, summarisation, language modelling,\
                question answering, text classification,\
                text classification, RNN, LSTM"

# create a wordcloud from the string of text
my_wordcloud = WordCloud(background_color="white", 
                         max_words=50, 
                         ).generate(text_string)

# display the wordcloud
plt.imshow(my_wordcloud, interpolation='bilinear')
plt.axis('off')
plt.show()

Note: you may need to install wordcloud first (e.g. with !pip install wordcloud if you're writing python code in a Jupyter Notebook)

NLP is a massive field and it can be daunting and confusing to get started. It's a really interesting field though and well worth the effort. I'm planning on writing more about NLP in the future, as I'm learning a lot about it as part of my Data Science MSc project. In the meantime, I hope the resources I've mentioned here can help to make the journey a bit easier for total beginners.

How to Make Better Looking Charts in Python

Nic Fox — Fri, 19 Jun 2020 15:11:46 +0000

With the matplotlib and seaborn libraries it's easy to make charts in Python, but the default settings can result in an ugly looking chart. This might not be a problem if you only need the chart for your own purposes but if you’re going to share it you may wish to improve its appearance and make it easier to interpret.

Improving the chart’s appearance can be done with some simple changes. However, the documentation can be overwhelming for those new to coding/python and often its not obvious where to add the various code snippets to your chart code.

In this post, I’ll show you, step-by-step, a simple example of how I improved the default bar chart from this:

…to this:

First, let’s import the libraries and create some dummy data to create the charts with.

# import libraries
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# create and view data
severity = ["High", "Medium", "Low"]
freq = [10, 25, 50]
data = pd.DataFrame(list(zip(severity, freq)), 
               columns =['severity', 'freq'])

# view the data
data

So, we have a data-frame containing 2 columns and 3 rows, showing the number of accidents for each level of severity (High, Medium, Low):

The Default Plots

Using just the matplotlib library, we can create a bar chart with this one, simple line of code:

# default matplotlib bar plot
data.plot.bar(x="severity", y="freq")

…but it doesn’t look great:

It outputs text above the chart that we don’t want, the x axis tick labels are vertical, and it just doesn’t look very professional.

The seaborn default looks a little better:

# default seaborn bar plot
sns.barplot(data=data, x="severity", y="freq")

This also displays unwanted text above the plot but at least the x axis tick labels are horizontal and so easier to read. Still, there’s a lot that can be improved.

Remove the Unwanted Text

To remove the unwanted text above the plot, add a semicolon to the end of the code. This applies to both the matplotlib and seaborn plots. From this point, we’ll work on improving the seaborn plot.

# without the matplotlib text
sns.barplot(data=data, x="severity", y="freq");

Change Axis Labels

Next, let’s change the axis labels to “Accident Severity” and “Number of Accidents” using plt.xlabel() and plt.ylabel().

# change the axis labels
sns.barplot(data=data, x="severity", y="freq")
plt.xlabel('Accident Severity')
plt.ylabel('Number of Accidents');

Add a Title

At the moment it’s not clear what the chart is about so let’s add a title using plt.title().

# add a title
sns.barplot(data=data, x="severity", y="freq")
plt.xlabel('Accident Severity')
plt.ylabel('Number of Accidents')
plt.title('Number of Accidents By Severity');

Remove X Axis Label

Now that we have a title, the x axis label is not really needed. Charts look better when there is less clutter. So let’s remove the x axis label by removing the text in the quotes.

If we just deleted the whole plt.xlabel() line then matplotlib would display the name of the dataframe column mapped to the x axis (“severity”) which is not what we want. So, instead, we use plt.xlabel(‘’).

# remove x axis label
sns.barplot(data=data, x="severity", y="freq")
plt.xlabel('')
plt.ylabel('Number of Accidents')
plt.title('Number of Accidents By Severity');

Remove Top and Right Border

Removing the top and right part of the border also helps to reduce clutter and makes the chart look “cleaner” and more modern. This can be done with sns.despine() like this:

# remove top and right border
sns.barplot(data=data, x="severity", y="freq")
plt.xlabel('')
plt.ylabel('Number of Accidents')
plt.title('Number of Accidents By Severity')
sns.despine();

Remove Left Border

We could go further and also remove the border on the left (the y axis). This approach can make the chart look even better and is used by many publications these days, like the Financial Times for example. The y axis border can be removed by adding left=True in sns.despine() like this:

# remove left border
sns.barplot(data=data, x="severity", y="freq")
plt.xlabel('')
plt.ylabel('Number of Accidents')
plt.title('Number of Accidents By Severity')
sns.despine(left=True);

Add Horizontal Grid Lines

This doesn’t look quite right now that we’ve removed the y axis line because the chart doesn’t have any gridlines. Let’s add horizontal grid lines so its easier to see the value of each of the bars. This can be done by adding sns.set(style=”whitegrid”) like this:

# add horizontal grid lines to the background
sns.set(style="whitegrid")
sns.barplot(data=data, x="severity", y="freq")
plt.xlabel('')
plt.ylabel('Number of Accidents')
plt.title('Number of Accidents By Severity')
sns.despine(left=True);

Make the Title and Axis Labels Bigger

To make the title and axis labels easier to read, let’s make them bigger by adding , size=18 to the list of parameters in plt.ylabel() and plt.title():

# Make the title and axis labels bigger
sns.set(style="whitegrid")
sns.barplot(data=data, x="severity", y="freq")
plt.xlabel('')
plt.ylabel('Number of Accidents', size=18)
plt.title('Number of Accidents By Severity', size=18)
sns.despine(left=True);

Make the Axis Tick Labels Bigger

Let’s also increase the size of the axis tick labels too. This can be done with plt.xticks and plt.yticks() like this:

# Make the axis tick labels bigger
sns.set(style="whitegrid")
sns.barplot(data=data, x="severity", y="freq")
plt.xlabel('')
plt.ylabel('Number of Accidents', size=18)
plt.title('Number of Accidents By Severity', size=18)
plt.xticks(size=14)
plt.yticks(size=14)
sns.despine(left=True);

Change the Colour of the Bars

It would be better if the colours of the bars were meaningful, and represented the severity. Let’s change the colours to red, orange and yellow using the palette parameter in sns.barplot().

# Change the colours of the bars
sns.set(style="whitegrid")
sns.barplot(data=data, x="severity", y="freq", 
            palette=['red', 'orange', 'yellow'])
plt.xlabel('')
plt.ylabel('Number of Accidents', size=18)
plt.title('Number of Accidents By Severity', size=18)
plt.xticks(size=14)
plt.yticks(size=14)
sns.despine(left=True);

This is better but those shades of the colours hurt my eyes!
I tend to define colours using the hex values after choosing a particular colour using an online colour picker tool (just google “colour picker” and a tool is shown as the first result). You can slide the cursor to the exact colour and shade that you want and copy-paste the hex value to your code.

Here’s my updated code after replacing the colour names with hex values of the shades I wanted:

# Change the colours of the bars
sns.set(style="whitegrid")
sns.barplot(data=data, x="severity", y="freq", 
            palette=['#eb3434', '#eb7a34', '#ebae34'])
plt.xlabel('')
plt.ylabel('Number of Accidents', size=18)
plt.title('Number of Accidents By Severity', size=18)
plt.xticks(size=14)
plt.yticks(size=14)
sns.despine(left=True);

Ahhh…that’s easier on the eyes!

Increase the Sharpness of the Chart

The resolution of the chart looks a bit low at the moment. To improve how sharp the chart looks, we can define the “dots per inch” of the chart display with plt.rcParams[‘figure.dpi’] = 360

# Increase the sharpness of the display
plt.rcParams['figure.dpi'] = 360
sns.set(style="whitegrid")
sns.barplot(data=data, x="severity", y="freq", 
            palette=['#eb3434', '#eb7a34', '#ebae34'])
plt.xlabel('')
plt.ylabel('Number of Accidents', size=18)
plt.title('Number of Accidents By Severity', size=18)
plt.xticks(size=14)
plt.yticks(size=14)
sns.despine(left=True);

This looks much better! It’s a bit too big now though.

Change the Size of the Chart

To change the dimensions of the chart, define the width and height in fig, ax = plt.subplots(figsize=()). I’ve set the dimensions to a width of 12 and height of 4 like this:

# change the size of the plot
plt.rcParams['figure.dpi'] = 360
sns.set(style="whitegrid")
fig, ax = plt.subplots(figsize=(12,4))
sns.barplot(data=data, x="severity", y="freq", 
            palette=['#eb3434', '#eb7a34', '#ebae34'])
plt.xlabel('')
plt.ylabel('Number of Accidents', size=18)
plt.title('Number of Accidents By Severity', size=18)
plt.xticks(size=14)
plt.yticks(size=14)
sns.despine(left=True);

Change the Colour of the Title, Axis Labels and Tick Mark Labels

I prefer to have the text in a dark grey rather than pure black, so I chose a dark grey that I liked using Google’s colour picker and changed the colour to #4f4e4e using the color parameters in plt.ylabel(), plt.title(), plt.xticks() and plt.yticks() like this:

# change the colour of the title, axis labels and tick labels
plt.rcParams['figure.dpi'] = 360
sns.set(style="whitegrid")
fig, ax = plt.subplots(figsize=(12,4))
sns.barplot(data=data, x="severity", y="freq", 
            palette=['#eb3434', '#eb7a34', '#ebae34'])
plt.xlabel('')
plt.ylabel('Number of Accidents', size=18, color='#4f4e4e')
plt.title('Number of Accidents By Severity', size=18, color='#4f4e4e')
plt.xticks(size=14, color='#4f4e4e')
plt.yticks(size=14, color='#4f4e4e')
sns.despine(left=True);

Add Text Annotations

It’s often helpful to add text annotations to charts to help the reader to understand what the key insight is that the chart is providing. Or perhaps there is something important about the chart that you need to communicate when the chart is being looked at when you’re not there to explain it. It’s not really needed in this chart but let’s add one as an example so you know how to do it. It can be done using plt.text() like this:

# Add a text annotation
plt.rcParams['figure.dpi'] = 360
sns.set(style="whitegrid")
fig, ax = plt.subplots(figsize=(12,4))
sns.barplot(data=data, x="severity", y="freq", 
            palette=['#eb3434', '#eb7a34', '#ebae34'])
plt.xlabel('')
plt.ylabel('Number of Accidents', size=18, color='#4f4e4e')
plt.title('Number of Accidents By Severity', size=18, color='#4f4e4e')
plt.xticks(size=14, color='#4f4e4e')
plt.yticks(size=14, color='#4f4e4e')
plt.text(x=1, y=48, s='Most accidents were low severity', 
                 color='#4f4e4e', fontsize=12, horizontalalignment='center')
sns.despine(left=True);

It’s often tricky to know what x and y co-ordinates to set for your text. I just put some numbers in and then change them through trial and error until the text is positioned where I want it.

Add Data Labels

At the moment, if the person viewing the chart wants to know how many accidents were medium severity, they have to visualise a line between the bar and the y axis and guess what the value is. This is relatively easy on this chart but on other charts this could be more difficult.

A good chart doesn’t make the reader work hard to get the information from it. One way we can make it easier for the reader is to add data labels, so it’s obvious what the values are.

Note: If there were lots of bars on the chart then this may not be a good idea as it could end up looking too cluttered. In that situation, it may be better to only add data labels for the “headline” bars e.g. the biggest and smallest, or the bar that you’re focussing on in your analysis (e.g. if its a bar chart showing sales for your team compared to other teams then you may wish to add a data label for the bar representing your team).

Data labels can be added using plt.text() like this:

# Add data labels
plt.rcParams['figure.dpi'] = 360
sns.set(style="whitegrid")
fig, ax = plt.subplots(figsize=(12,4))
sns.barplot(data=data, x="severity", y="freq", 
            palette=['#eb3434', '#eb7a34', '#ebae34'])
plt.xlabel('')
plt.ylabel('Number of Accidents', size=18, color='#4f4e4e')
plt.title('Number of Accidents By Severity', size=18, color='#4f4e4e')
plt.xticks(size=14, color='#4f4e4e')
plt.yticks(size=14, color='#4f4e4e')
plt.text(x=1, y=48, s='Most accidents were low severity', 
                 color='#4f4e4e', fontsize=12, horizontalalignment='center')
plt.text(x=0, y=2, s="10", 
                 color='white', fontsize=18, horizontalalignment='center')
plt.text(x=1, y=2, s="25", 
                 color='white', fontsize=18, horizontalalignment='center')
plt.text(x=2, y=2, s="50", 
                 color='white', fontsize=18, horizontalalignment='center')
sns.despine(left=True);

Remove Y Axis Tick Labels

Now that we have data labels for all 3 bars, the y axis is somewhat redundant so let’s remove it. We’ll need to remove the y axis label, the y axis tick labels, and also the background grid lines.

The y axis label can be removed by setting plt.ylabel() to an empty string
The y axis tick labels can be removed by setting plt.yticks() to 2 empty lists like this: plt.yticks([],[])
We won’t need the background grid lines if we no longer have a y axis displayed. The changes described above will automatically remove the grid lines but you may wish to remove the sns.set(style=”whitegrid”) line of code to keep your code tidy. I’ve just commented it out here.

# Remove y axis tick labels and y axis label
plt.rcParams['figure.dpi'] = 360
#sns.set(style="whitegrid")
fig, ax = plt.subplots(figsize=(12,4))
sns.barplot(data=data, x="severity", y="freq", 
            palette=['#eb3434', '#eb7a34', '#ebae34'])
plt.xlabel('')
plt.ylabel('')
plt.title('Number of Accidents By Severity', size=18, color='#4f4e4e')
plt.xticks(size=14, color='#4f4e4e')
plt.yticks([], [])
plt.text(x=1, y=48, s='Most accidents were low severity', 
                 color='#4f4e4e', fontsize=12, horizontalalignment='center')
plt.text(x=0, y=2, s="10", 
                 color='white', fontsize=18, horizontalalignment='center')
plt.text(x=1, y=2, s="25", 
                 color='white', fontsize=18, horizontalalignment='center')
plt.text(x=2, y=2, s="50", 
                 color='white', fontsize=18, horizontalalignment='center')
sns.despine(left=True);

Save the Chart as an Image

That’s it! We now have a much more professional, clutter-free and easier-to-read chart that’s shareable with others.

If you’d like to use the chart in another application (e.g. Powerpoint) then you may want to save it as an image using plt.savefig() like this:

# save your chart as an image
plt.rcParams['figure.dpi'] = 360
#sns.set(style="whitegrid")
fig, ax = plt.subplots(figsize=(12,4))
sns.barplot(data=data, x="severity", y="freq", 
            palette=['#eb3434', '#eb7a34', '#ebae34'])
plt.xlabel('')
plt.ylabel('')
plt.title('Number of Accidents By Severity', size=18, color='#4f4e4e')
plt.xticks(size=14, color='#4f4e4e')
plt.yticks([], [])
plt.text(x=1, y=48, s='Most accidents were low severity', 
                 color='#4f4e4e', fontsize=12, horizontalalignment='center')
plt.text(x=0, y=2, s="10", 
                 color='white', fontsize=18, horizontalalignment='center')
plt.text(x=1, y=2, s="25", 
                 color='white', fontsize=18, horizontalalignment='center')
plt.text(x=2, y=2, s="50", 
                 color='white', fontsize=18, horizontalalignment='center')
sns.despine(left=True)
plt.savefig('accidents.png');

There are many more aspects of how the chart looks that you can tweak, and many different types of chart of course, but hopefully that gives you enough to get you started. I recommend having a play around with chart settings and also looking at some popular professional publications that regularly contain charts to get some inspiration.

Below are links to some places that inspire me.

Financial Times — e.g. their corona virus charts
BBC — e.g. the 2019 UK general election results
FiveThirtyEight — e.g. How (Un)Popular is Donald Trump?

How to Create an Interactive Dashboard in Python Using HoloViz Panel

Nic Fox — Thu, 18 Jun 2020 07:47:04 +0000

Thanks to the Panel library from HoloViz (previously PyViz), it’s now relatively simple to create an interactive dashboard of plots in Python, similar to an R Shiny app. This is a short tutorial-by-example that walks you through a very basic dashboard, created in a Jupyter Notebook.

TL;DR: See the dashboard which includes a button to view all the code

Using a small dummy data set of animal ratings data, the interactive dashboard will allow the user to choose an animal and view a box plot & data table for the ratings for that animal.

I’m not going to spend much time explaining individual lines of code or underlining mechanisms, as my intention is to provide a very quick and simple end-to-end deployable example that can be copy-pasted to get you started, rather then getting bogged down and confused in abstract details that you may not be interested in yet. You can then build your knowledge from there by playing around with this sample code and improving on it using the HoloViz documentation and other tutorials.

Libraries and Set-Up

import param
import pandas as pd 
import matplotlib.pyplot as plt
import seaborn as sns
import panel as pn
pn.extension()

# make plots static
%matplotlib inline

Many python users will be used to using pandas, matplotlib and seaborn, but the new elements for the interactive dashboard are:

import param,
import panel,
pn.extension()

You may have to install these if you’ve not used them before. (In Jupyter Notebook, I used !pip install param etc)

Data

# create a simple dataframe of animal ratings
df = pd.DataFrame({'Animal':['Pig', 'Goat','Sheep', 'Frog',
                             'Goat', 'Goat', 'Pig', 'Sheep', 
                             'Octopus', 'Goat', 'Pig',
                             'Sheep', 'Octopus', 'Frog', 
                             'Goat', 'Goat', 'Pig', 'Pig',
                             'Sheep', 'Frog', 'Frog', 
                             'Octopus', 'Octopus'], 
                 'Rating':[3, 10, 3, 2, 9, 10, 4, 1, 
                           1, 8, 5, 6, 2, 4, 10, 9,
                           5, 5, 3, 2, 3, 1, 1]})

The code above creates a simple 2-column (Animal and Rating) pandas data frame with 23 ratings for 5 animals (Pig, Goat, Sheep, Frog, Octopus).

Class of Dashboard Elements

The first step for creating a dashboard is to create a class containing the data elements that will appear in the dashboard. In this case, the elements are:

an ‘Animal’ drop down to allow the user to choose the animal
a subset of the data containing the ratings for the chosen animal
a seaborn box plot of the ratings for the chosen animal (using the subset of data)
a data table output of the chosen animal (using the subset of data)

This class can be called anything you like. I called it RatingsDashboard. A more meaningful name such as DashboardDataElements may have been more representative but thats quite a long name.

The dashboard will contain more than the 4 things defined in this class, but those other things are not data-specific (e.g. dashboard title and description) and so will be defined outside of the class, later.

It’s important to note that any plots should be closed before they’re returned. E.g: plt.close()

If you forget this step then the plot may not update when the user chooses a different animal in the drop down.

Also, the plot’s figure (not the axis) should be returned.

return ax.figure is correct

return ax will not display the plot when the dashboard is displayed.

(I learnt this the hard way and had to post on StackOverflow to resolve the issue.)

# create a class containing an animal selector drop-down,
# various plots, and a data table output
class RatingsDashboard(param.Parameterized):

    # drop down selector widget containing the list of
    # animals, with the default being 'Goat'
    Animal = param.ObjectSelector(default='Goat',
                             objects=list(df.Animal.unique()))

    # create data set containing only the data applicable 
    # to the animal in the drop down selector
    def get_data(self):
        class_df = df[(df.Animal==self.Animal)].copy()
        return class_df

    # seaborn box plot for the chosen animal
    def box_view(self):
        data = self.get_data() 
        ax = sns.boxplot(data['Rating'])
        plt.close()
        return ax.figure

    # table of data for the chosen animal
    def table_view(self):
        data = self.get_data()
        return data

Create Class Instance

The next step is to create an instance of the class. I called it rd. This will be used in the next step to access the data-related elements within the class when the dashboard layout is defined.

# create an instance of the class
rd = RatingsDashboard(name='')

Define Dashboard Layout & Content

# create a title for the dashboard
dashboard_title = '# Animal Ratings Dashboard'

# create some text describing the dashboard
dashboard_desc = 'An example of a simple interactive HoloViz
Panel dashboard using a dummy data set of animal ratings.'

# create a dashboard, defining the layout as one column containing the
# dashboard title, dashboard description, 'Animal' drop down selector,
# box plot, and data table
dashboard = pn.Column(dashboard_title, 
                      dashboard_desc,   
                      rd.param,       # 'Animal' drop down selector
                      rd.box_view,    # box plot
                      rd.table_view   # data table
                     )

pn.Column is just one way of doing this. If you’d like to explore more layouts, there’s also pn.Row, pn.Tabs and more. I wanted to keep it simple for this beginner’s tutorial so chose a single column, which also has the added advantage that the dashboard will be more readable on a mobile device.

Display Dashboard in Jupyter Notebook

The easiest way to display this dashboard is to just type it’s name, like you would for a data frame or another variable:
dashboard

If you’re only creating the dashboard for you and/or other Jupyter Notebook users then you could stop here. This will let you view and interact with the dashboard to check that it’s working. However, if you want to share the dashboard with others, perhaps via an html page, then embedding the data with the dashboard is one way to achieve that…

Display Dashboard With Data Embedded

Use dashboard.embed() to embed the data with the dashboard. I used this as follows:

# show the dashboard with the data embedded 
# (for using in an html download of the notebook so that 
# no 'callback' is required from the browser to get the data)
dashboard.embed(max_opts = 5)

This will display the dashboard in the notebook in exactly the same way as just calling the dashboard variable. However, the advantage is that when you download the notebook as html and view the dashboard in the html file, choosing another animal from the drop down will update the plot, and there is no need for any ‘back-end’ to be called to get the data for the plot.

max_opts = 5 is used so that all 5 animals are available in the drop down widget. If this parameter wasn’t defined, and the brackets were empty, the default of 3 would be used and only 3 of the animals would be available to choose from in the drop down. The default is probably low for performance reasons, since the data is being embedded. More info on this function and its parameters are in the ‘Deploy and Export’ section of the user guide.

Download as HTML

To download the dashboard to an html file, go to File > Download as > html

Save the html file and view the dashboard by opening the html file in your browser. I made the html version of the dashboard publicly available by adding it to the ‘Projects’ page on my website which is hosted for free on GitHub but that’s not the only option of course.

Add Show Code/Hide Code Button

When you share the html version of your Notebook, it will display the code as well as the dashboard. If you want to hide the code by default but make it possible for the user to view the code if they wish, then you could add a Show Code / Hide Code button with some JavaScript.

To add this button and hide the code by default, create 2 new cells in your Notebook, one at the top (the very 1st cell) and one at the bottom (the very last cell) and make both cells of type Raw NBConvert:

In the top cell, copy-paste the following code:

<!–– Hide code cells and show a button to allow user to show/hide code cells ––!>
<script>
  function code_toggle() {
    if (code_shown){
      $('div.input').hide('500');
      $('#toggleButton').val('Show Code')
    } else {
      $('div.input').show('500');
      $('#toggleButton').val('Hide Code')
    }
    code_shown = !code_shown
  } 

  $( document ).ready(function(){
    code_shown=false; 
    $('div.input').hide()
  });
</script>
<form action="javascript:code_toggle()"><input type="submit" id="toggleButton" value="Show Code"></form>

In the bottom cell, copy-paste this:

<!–– Hide JupyterNotebook cell numbers etc ––!>
<script>
  $(document).ready(function(){
    $('div.prompt').hide();
    $('div.back-to-top').hide();
    $('nav#menubar').hide();
    $('.breadcrumb').hide();
    $('.hidden-print').hide();
  });
</script>
<footer id="attribution" style="float:right; color:#999; background:#fff;">
Attribution: Created with Jupyter. Code toggle possible thanks to github.com/csaid.
</footer>

This includes an attribution to the GitHub account of the person (Chris Said) who created this code.

That’s it! That’s all you need to create the interactive dashboard and deploy it as html!

If you wanted to add another plot you just need to:

add it to the class in the same way as the existing plot
add it in the dashboard = pn.Columns(…) code in the position you want it displayed

…and as long as the plot uses the subset of code based on the selected animal, the new plot will also update along with the original plot, whenever you choose a new animal.

Complete Code and Dashboard

The complete code and dashboard can be viewed at https://foxnic.github.io/simple_dashboard.html

Other python (and R) code examples created by me are on my Projects page.

More Examples & Tutorials

The HoloViz Panel website contains a gallery of examples with code, as well as a Getting Started guide and User Guide which includes a section on how to customise the style of the dashboard.

I discovered this functionality thanks to the following YouTube videos:

Turn Any Notebook Into a Deployable Dashboard (~30 mins SciPy 2019 Talk by James Bednar)
Visualize any Data Easily, from Notebooks to Dashboards (~3 hour ScipPy 2019 Tutorial by James Bednar)