DEV Community: Max Humber

How to make an animated gif fit for /r/dataisbeautiful

Max Humber — Tue, 13 Oct 2020 19:19:00 +0000

A good visualization should capture the interest of the audience and make an impression. Few things capture interest more than bright colors and movement.

In this post I'm going to show you exactly how to make an animated gif, so that you can go farm some internet points on /r/dataisbeautiful, maybe~

Here's what we're going to make:

Step 0 - Data

Before you make a graph you've gotta get your hands on some data. I grabbed some business data from StatsCanada available here. The data isn't in the best shape, so here's a pinch of pandas to make it suck less:

import pandas as pd

df = pd.read_csv("3310027001-noSymbol.csv", skiprows=7).iloc[1:5]
df = df.rename(columns={"Business dynamics measure": "status"})
df['status'] = df['status'].apply(lambda x: x[:-13])
df = pd.melt(df, id_vars="status", var_name="date", value_name="count")
df["date"] = pd.to_datetime(df["date"], format="%B %Y")
df['count'] = df['count'].apply(lambda x: int(x.replace(",", "")))

print(df.head())
#        status       date   count
# 0      Active 2015-01-01  775497
# 1     Opening 2015-01-01   40213
# 2  Continuing 2015-01-01  731116
# 3     Closing 2015-01-01   30979
# 4      Active 2015-02-01  778554

Step 1 - Graph

If you want to make a gif, you first have to make a single frame. Which, coincidentally, is just a graph:

from matplotlib import pyplot as plt

da = df[df["status"] == "Active"]
plt.plot(da["date"], da["count"])

Step 2 - Size

The graph could be bigger, and the y-axis limits could be adjusted. No problem, that's just two extra lines of code: some code to size and limit:

plt.figure(figsize=(8, 5), dpi=300)
plt.plot(da["date"], da["count"])
plt.ylim([0, da['count'].max() * 1.1])

Step 3 - Tick

I like to manually set the ticks on my graphs, you don't have to, but if you want to:

ymax = int(da['count'].max() * 1.1 // 1)

plt.figure(figsize=(8, 5), dpi=300)
plt.plot(da["date"], da["count"])
plt.ylim([0, ymax])
plt.yticks(range(0, ymax, 200_000))

Step 4 - Label

If someone saw our graph right now, without any context, they'd have know idea what's going on. Let's fix that by adding some labels:

plt.figure(figsize=(8, 5), dpi=300)
plt.plot(da["date"], da["count"])
plt.ylim([0, ymax])
plt.yticks(range(0, ymax, 200_000))
plt.title("Active Businesses in Canada (Seasonally Adjusted)")
plt.xlabel("Year")
plt.ylabel("Count")

Step 4.5 - Detour

Our graph is exclusively about "Active" businesses in Canada. Here's what the "Opening" and "Closing" numbers look like:

dc = df[df["status"] == "Closing"]
do = df[df["status"] == "Opening"]

plt.plot(dc["date"], dc["count"], color='red', label="closing")
plt.plot(do["date"], do["count"], color='green', label="opening")
plt.legend()

Step 5 - Combine

The "Opening and Closing" graph adds some interesting colour to the "Active" data. Let's combine both with some fancy-pant matplotlib:

rows = 7
figure = plt.figure(figsize=(8, 4), constrained_layout=False, dpi=300)
grid = plt.GridSpec(
    nrows=rows,
    ncols=1,
    wspace=0,
    hspace=0.5,
    figure=figure
)

main = plt.subplot(grid[:5, 0])
sub = plt.subplot(grid[5:, 0])

main.plot(da["date"], da["count"])
sub.plot(do["date"], do["count"])
sub.plot(dc["date"], dc["count"])

Step 6 - Colour

I'm not keen on the colours or spacing of what we have right now. To fix, along with some axis adjustments, here's what you'll need:

figure = plt.figure(figsize=(8, 4), constrained_layout=False, dpi=300)
grid = plt.GridSpec(
    nrows=rows,
    ncols=1,
    wspace=0,
    hspace=0.75,
    figure=figure
)

main = plt.subplot(grid[:5, 0])
sub = plt.subplot(grid[5:, 0])

main.plot(da["date"], da["count"], color="purple")
sub.plot(do["date"], do["count"], color="blue")
sub.plot(dc["date"], dc["count"], color="red")

main.set_xticks([])
main.set_ylim([0, ymax])
main.set_yticks(range(0, ymax, 200_000))
main.set_yticklabels([0, "200K", "400K", "600K", "800K\nbusinesses"])

sub.set_ylim([0, 110_000])
sub.set_yticks([0, 100_000])
sub.set_yticklabels([0, "100K"])

Step 7 - Refactor

Our graph code is nearly ready to go. We just need to refactor it so that we can take an individual date and build an individual frame for that date. I've also added some vlines and fixed the xlims to improve legibility and ensure that the plotting space is consistent across plots:

date = pd.Timestamp("2019-08-01")

xmin = df['date'].min()
xmax = df['date'].max()

dd = df[df["date"] <= date]
dc = dd[dd["status"] == "Closing"]
do = dd[dd["status"] == "Opening"]
da = dd[dd["status"] == "Active"]

figure = plt.figure(figsize=(8, 4), constrained_layout=False, dpi=300)
grid = plt.GridSpec(
    nrows=rows,
    ncols=1,
    wspace=0,
    hspace=1.25,
    figure=figure
)

main = plt.subplot(grid[:5, 0])
sub = plt.subplot(grid[5:, 0])

main.plot(da["date"], da["count"], color="#457b9d")
main.vlines(date, ymin=0, ymax=1e20, color="#000000")
sub.plot(do["date"], do["count"], color="#a8dadc")
sub.plot(dc["date"], dc["count"], color="#e63946")
sub.vlines(date, ymin=0, ymax=1e20, color="#000000")

main.set_xlim([xmin, xmax])
main.set_xticks([])
main.set_ylim([0, ymax])
main.set_yticks(range(0, ymax, 200_000))
main.set_yticklabels([0, "200K", "400K", "600K", "800K"])
main.set_title("Active Businesses in Canada")

sub.set_xlim([xmin, xmax])
sub.set_xticks([date])
sub.set_xticklabels([date.strftime("%B '%y")])
sub.set_ylim([0, 110_000])
sub.set_yticks([0, 100_000])
sub.set_yticklabels([0, "100K"])
sub.set_title("Businesses Opening and Closing")

Step 7.5 - Functionize

In order build a bunch of frames on a bunch of dates, we should wrap our code in a function:

def plot(date):
    dd = df[df["date"] <= date]
    dc = dd[dd["status"] == "Closing"]
    do = dd[dd["status"] == "Opening"]
    da = dd[dd["status"] == "Active"]

    figure = plt.figure(figsize=(8, 4), constrained_layout=False, dpi=300)
    grid = plt.GridSpec(
        nrows=rows,
        ncols=1,
        wspace=0,
        hspace=1.25,
        figure=figure
    )

    main = plt.subplot(grid[:5, 0])
    sub = plt.subplot(grid[5:, 0])

    main.plot(da["date"], da["count"], color="#457b9d")
    main.vlines(date, ymin=0, ymax=1e20, color="#000000")
    sub.plot(do["date"], do["count"], color="#a8dadc")
    sub.plot(dc["date"], dc["count"], color="#e63946")
    sub.vlines(date, ymin=0, ymax=1e20, color="#000000")

    main.set_xlim([xmin, xmax])
    main.set_xticks([])
    main.set_ylim([0, ymax])
    main.set_yticks(range(0, ymax, 200_000))
    main.set_yticklabels([0, "200K", "400K", "600K", "800K"])
    main.set_title("Active Businesses in Canada")

    sub.set_xlim([xmin, xmax])
    sub.set_xticks([date])
    sub.set_xticklabels([date.strftime("%b '%y")])
    sub.set_ylim([0, 110_000])
    sub.set_yticks([0, 100_000])
    sub.set_yticklabels([0, "100K"])
    sub.set_title("Businesses Opening and Closing");

So that we can build a frame with just one call:

plot(pd.Timestamp("2017-06-01"))

Step 8 - `import gif`

To turn static frames into an animated gif, all we have to do now is to install and import the gif package:

import gif

Decorate the plot function with gif.frame:

@gif.frame
def plot(date):
    dd = df[df["date"] <= date]
    dc = dd[dd["status"] == "Closing"]
    do = dd[dd["status"] == "Opening"]
    da = dd[dd["status"] == "Active"]

    figure = plt.figure(figsize=(8, 4), constrained_layout=False, dpi=300)
    grid = plt.GridSpec(
        nrows=7,
        ncols=1,
        wspace=0,
        hspace=1.25,
        figure=figure
    )

    main = plt.subplot(grid[:5, 0])
    sub = plt.subplot(grid[5:, 0])

    main.plot(da["date"], da["count"], color="#457b9d")
    main.vlines(date, ymin=0, ymax=1e20, color="#000000")
    sub.plot(do["date"], do["count"], color="#a8dadc")
    sub.plot(dc["date"], dc["count"], color="#e63946")
    sub.vlines(date, ymin=0, ymax=1e20, color="#000000")

    main.set_xlim([xmin, xmax])
    main.set_xticks([])
    main.set_ylim([0, ymax])
    main.set_yticks(range(0, ymax, 200_000))
    main.set_yticklabels([0, "200K", "400K", "600K", "800K"])
    main.set_title("Active Businesses in Canada")

    sub.set_xlim([xmin, xmax])
    sub.set_xticks([date])
    sub.set_xticklabels([date.strftime("%b '%y")])
    sub.set_ylim([0, 110_000])
    sub.set_yticks([0, 100_000])
    sub.set_yticklabels([0, "100K"])
    sub.set_title("Businesses Opening and Closing");

Build all the frames:

dates = pd.date_range(df['date'].min(), df['date'].max(), freq="1MS")

frames = [plot(date) for date in dates]

And save the animation to disk:

gif.save(frames, "businesses.gif", duration=5, unit="s", between="startend")

Now it's your turn to find some interesting data and turn it into a gif.

And if you want to learn more, I'm running a workshop on gifs with ODSC/AI+ on October 22. Hope to see you in class!

BeautifulSoup is so 2000-and-late: Web Scraping in 2020

Max Humber — Fri, 09 Oct 2020 16:43:00 +0000

BeautifulSoup (bs4) was created over a decade-and-a-half ago. And it's been the standard for web scraping ever since. But it's time for something new, because bs4 is so 2000-and-late.

In this post we'll explore 10 reasons why gazpacho is the future of web scraping, by scraping parts of this post!

1. No Dependencies

gazpacho is installed at command line:

pip install gazpacho

With no extra dependencies:

pip freeze
# gazpacho==1.1

In contrast, bs4 is packaged with soupsieve and lxml. I won't tell you how to write software, but minimizing dependencies is usually a good idea...

2. Batteries Included

The html for this blog post can be fetched and made parse-able with Soup.get:

from gazpacho import Soup

url = "https://dev.to/maxhumber/beautifulsoup-is-so-2000-and-late-web-scraping-in-2020-2528"
soup = Soup.get(url)

Unfortunately, you'll need requests on top of bs4 to do the same thing:

import requests
from bs4 import BeautifulSoup

url = "https://dev.to/maxhumber/beautifulsoup-is-so-2000-and-late-web-scraping-in-2020-2528"
html = requests.get(url).text
bsoup = BeautifulSoup(html)

3. Simple `find`ing

bs4 is a monster. There are 184 methods and attributes attached to every BeautifulSoup object. Making it hard to know what to use and when to use it:

len(dir(BeautifulSoup()))
# 184

In contrast, Soup objects in gazpacho are simple; there are just seven methods and attributes to keep track of:

[method for method in dir(Soup())]
# ['attrs', 'find', 'get', 'html', 'strip', 'tag', 'text']

Looking at that list it's clear that to find the title of this post (nested inside of an h1 tag), for example, we'll need to use .find:

soup.find('h1')

4. Prototyping to Production

gazpacho is awesome for prototyping and even better for production. By default, .find will return one Soup object if it finds just one element, or a list of Soup objects if it finds more than one.

To guarantee and enforce return types in production the mode= argument in .find can be set manually:

title = (soup
    .find("header", {'id': 'main-title'}, mode="first")
    .find("h1", mode="all")[0]
    .text
)

In contrast, bs4 has 27 find methods and they all return something different:

[method for method in dir(BeautifulSoup()) if 'find' in method]

5. PEP 561 Compliant

As of version 1.1, gazpacho is PEP 561 compliant. Meaning that the entire library is typed and will work with your typed (or standard duck/un-typed!) code-base:

help(soup.find)
# Signature:
# soup.find(
#     tag: str,
#     attrs: Union[Dict[str, Any], NoneType] = None,
#     *,
#     partial: bool = True,
#     mode: str = 'automatic',
#     strict: Union[bool, NoneType] = None,
# ) -> Union[List[ForwardRef('Soup')], ForwardRef('Soup'), NoneType]

6. Automatic Formatting

The html on dev.to and this post is well formatted. But if it weren't:

header = soup.find("div", {'class': 'crayons-article__header__meta'})
html = str(header.find("div", {'class': 'mb-4 spec__tags'}))
bad_html = html.replace("\n", "") # remove new line characters
print(bad_html)
# <div class="mb-4 spec__tags">  <a class="crayons-tag mr-1" href="/t/python" style="background-color:#1E38BB;color:#FFDF5B">    <span class="crayons-tag__prefix">#</span>    python  </a>  <a class="crayons-tag mr-1" href="/t/webscraping" style="background-color:;color:">    <span class="crayons-tag__prefix">#</span>    webscraping  </a>  <a class="crayons-tag mr-1" href="/t/gazpacho" style="background-color:;color:">    <span class="crayons-tag__prefix">#</span>    gazpacho  </a>  <a class="crayons-tag mr-1" href="/t/hacktoberfest" style="background-color:#29161f;color:#ffa368">    <span class="crayons-tag__prefix">#</span>    hacktoberfest  </a></div>

gazpacho would be able to automatically format and indent the bad/malformed html:

tags = Soup(bad_html)

Making things easier to read:

print(tags)
# <div class="mb-4 spec__tags">
#   <a class="crayons-tag mr-1" href="/t/python" style="background-color:#1E38BB;color:#FFDF5B">
#     <span class="crayons-tag__prefix">#</span>
#         python
#   </a>
#   <a class="crayons-tag mr-1" href="/t/webscraping" style="background-color:;color:">
#     <span class="crayons-tag__prefix">#</span>
#         webscraping
#   </a>
#   <a class="crayons-tag mr-1" href="/t/gazpacho" style="background-color:;color:">
#     <span class="crayons-tag__prefix">#</span>
#         gazpacho
#   </a>
#   <a class="crayons-tag mr-1" href="/t/hacktoberfest" style="background-color:#29161f;color:#ffa368">
#     <span class="crayons-tag__prefix">#</span>
#         hacktoberfest
#   </a>
# </div>

7. Speed

gazpacho is fast. It takes just 258 µs to scrape the tag links for this post:

%%timeit
tags = Soup(bad_html)
tags = tags.find("a")
tag_links = ["https://dev.to" + tag.attrs['href'] for tag in tags]
# 258 µs ± 10.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

While bs4 takes nearly twice as long to do the same thing:

%%timeit
tags = BeautifulSoup(bad_html)
tags = tags.find_all("a")
tag_links = ["https://dev.to" + tag.attrs['href'] for tag in tags]
# 465 µs ± 2.61 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

8. Partial Matching

gazpacho can partially match html element attributes. For instance, the sidebar for this page is displayed with the following html:

<aside class="crayons-layout__sidebar-right" aria-label="Right sidebar navigation">

And can be matched exactly with:

soup.find("aside", {"class": "crayons-layout__sidebar-right"}, partial=False)

Or partially (the default behaviour) with:

sidebar = soup.find("aside", {'aria-label': 'Right sidebar'}, partial=True)

# finding my name
sidebar.find("span", {'class': 'crayons-subtitle-2'}, partial=True).text

9. Debt-free

gazpacho is Python 3 first, Black, typed with mypy, and about ~400 sloc. It's easy to read through the source:

import inspect

source = inspect.getsource(Soup.find)
print(source)

And like bs4 isn't riddled with Python 2 technical debt.

10. Open (and Friendly)!

Most importantly, gazpacho is open-source, hosted on GitHub (instead of some clunky custom platform) and looking for contributors.

If you're participating in #hacktoberfest, we'd love to have you. There's a couple of open issues that could use some help!

How to Send and Schedule Emails with Python

Max Humber — Wed, 09 Sep 2020 15:37:31 +0000

Sending an email with Python isn't all that difficult. But most tutorials leave out the most important bits! Namely:

How to get Gmail to work with Python
How to properly manage secrets (passwords)
How to schedule message delivery

In this tutorial you're going to learn how to do it all and how do it properly!

Configuring Gmail

First things first, you have to configure Gmail to work with Python...

Sign into your Google Account and go to: https://myaccount.google.com/

Find and click on Security in the sidebar on the left:

Scroll down and click on "Turn on access" in the "Less secure app access" panel:

Flip the toggle:

Google will probably send you a menacing email about a "Critical security alert"... don't worry, you're about to deal with this...

Managing Secrets

Google's freaking out right now because they know you're about to give Python your email and password. And if you copy-and-pasted your credentials as plaintext into a Python script they would have cause for concern! But you're not going to do that... you're going to manage your secrets the right way.

Create a new directory and cd into it:

mkdir email-tut
cd email-tut

Open text editor and save a .env file to the directory you just created with your email and password:

GMAIL_USER=replace_me@gmail.com
GMAIL_PASSWORD=Sup3r$ecretP@assW0rd!

Because you saved your credentials outside of Python you can .gitignore this .env file so that if/when you upload the repo to Github you won't expose your password!

Now you can create a Python script called emailbot.py inside the email-tut directory (right beside your .env) and fill it with:

import os
from dotenv import load_dotenv

load_dotenv(".env")

SENDER = os.environ.get("GMAIL_USER")
PASSWORD = os.environ.get("GMAIL_PASSWORD")

The load_dotenv function from the python-dotenv package will allow you to safely import your secrets.

Sending an email with the Standard Library requires the following imports and convenience function:

from email.message import EmailMessage
import smtplib

def send_email(recipient, subject, body):
    msg = EmailMessage()
    msg.set_content(body)
    msg["Subject"] = subject
    msg["From"] = SENDER
    msg["To"] = recipient
    server = smtplib.SMTP_SSL("smtp.gmail.com", 465)
    server.login(SENDER, PASSWORD)
    server.send_message(msg)
    server.quit()

Calling the function should fire off an email:

send_email("replace_me@hey.com", subject="test", body="test")

Save the entire python script:

from email.message import EmailMessage
import smtplib
import os
from dotenv import load_dotenv

load_dotenv(".env")

SENDER = os.environ.get("GMAIL_USER")
PASSWORD = os.environ.get("GMAIL_PASSWORD")

def send_email(recipient, subject, body):
    msg = EmailMessage()
    msg.set_content(body)
    msg["Subject"] = subject
    msg["From"] = SENDER
    msg["To"] = recipient
    server = smtplib.SMTP_SSL("smtp.gmail.com", 465)
    server.login(SENDER, PASSWORD)
    server.send_message(msg)
    server.quit()

send_email("replace_me@hey.com", subject="test", body="test")

And test it at the command line to make sure that it works:

python emailbot.py

Sending just "test" is a little pedestrian, spice up your script with some inspirational quotes:

from quote import quote
import random

quotes = quote("William Shakespeare", limit=50)
body = random.sample(quotes, k=1)[0]['quote']

send_email("replace_me@hey.com", subject="Quote of the Day", body=body)

Save the script and call it from the command line again to make sure you didn't screw anything up 😉

Scheduling Delivery

Hopefully you've got a cute little Shakespearean Email Bot working and now you're itching to schedule this bot to send you a quote every day at 9:00am. To accomplish this install hickory:

pip install hickory

Call hickory schedule at the command line with emailbot.py as the script argument:

hickory schedule emailbot.py --every=day@9:00am

And that's literally it!

If you want to check the status of the schedule you can with:

hickory status
# ID      FILE         STATE    RUNS
# ea6b74  emailbot.py  waiting  0

And to kill the schedule:

hickory kill emailbot.py

Happy emailing!

TL;DR: The entire script:

from email.message import EmailMessage
import smtplib
import os
import random
from dotenv import load_dotenv
from quote import quote

load_dotenv(".env")

SENDER = os.environ.get("GMAIL_USER")
PASSWORD = os.environ.get("GMAIL_PASSWORD")

def send_email(recipient, subject, body):
    msg = EmailMessage()
    msg.set_content(body)
    msg["Subject"] = subject
    msg["From"] = SENDER
    msg["To"] = recipient
    server = smtplib.SMTP_SSL("smtp.gmail.com", 465)
    server.login(SENDER, PASSWORD)
    server.send_message(msg)
    server.quit()

quotes = quote("William Shakespeare", limit=50)
body = random.sample(quotes, k=1)[0]['quote']

send_email("replace_me@hey.com", subject="Quote of the Day", body=body)

How to Build a Price Monitoring Slack Bot 🤑

Max Humber — Thu, 03 Sep 2020 15:36:16 +0000

I just got a new Switch (because what else is there to do these days?!) And I really wanna get the new Paper Mario game. Except, I'm not keen on paying full pop for the title... So. I built a price monitoring Slack Bot to tell me if it's on sale! Here's how you can build one too~

First, install gazpacho, hickory, and slackclient:

pip install gazpacho hickory slackclient

Identify the page you want to scrape:

Grab the URL, open a Text Editor (I like Atom), create a python file called mario.py and import gazpacho:

from gazpacho import get, Soup

Download the html with the get function and stuff it in a Soup wrapper:

url = "https://www.bestbuy.ca/en-ca/product/paper-mario-the-origami-king-switch/14643866"
html = get(url)
soup = Soup(html)

Figure out how the price is being rendered on the page by right clicking on it and opening "Inspect Element:

Note the specific tag and element attributes:

In this case the relevant block of html on the page is:

<div class="price_FHDfG large_3aP7Z " aria-hidden="true">
  $79
  <span class="rightEndPrice_6y_hS">99</span>
</div>

Use the find method with partial matching (strict=False) enabled to extract the price and format it as a float:

price = soup.find("div", {"class": "price"}, strict=False)
lhs = price.text
rhs = price.find("span", {'class': 'rightEndPrice'}).text
price = float(f"{lhs}.{rhs}".replace('$', ""))

Might as well grab the title too:

title = soup.find('h1', {'class': 'productName'}).text

Print out both to confirm that everything is working:

print(title, price)
# Paper Mario: The Origami King (Switch) 79.99

Your entire script should look like this:

# mario.py
from gazpacho import get, Soup

url = "https://www.bestbuy.ca/en-ca/product/paper-mario-the-origami-king-switch/14643866"
html = get(url)
soup = Soup(html)

title = soup.find('h1', {'class': 'productName'}).text
price = soup.find("div", {"class": "price"})
lhs = price.text
rhs = price.find("span", {'class': 'rightEndPrice'}).text
price = float(f"{lhs}.{rhs}".replace('$', ""))

print(title, price)

To schedule this scraper and turn it into a Slack Bot go to https://api.slack.com/apps and click Create New App:

Find and click on OAuth & Permissions:

Search for the chat:public.write scope and add it:

Scroll back up to the top of the page and install the app into your Slack Workspace:

Copy the generated token to the mario.py file as a string:

token = "xoxb-00000000-0000000000-00000000000"

Import slack and instantiate an instance of WebClient:

from slack import WebClient
client = WebClient(token=token)

Send a message to a Slack channel with the chat_postMessage method:


client.chat_postMessage(channel="mario", text=f"{price} - {title}")

Re-factor everything to make it look like:

# mario.py

from gazpacho import get, Soup
from slack import WebClient

token = "xoxb-00000000-00000000-000000000000000"
client = WebClient(token=token)

url = "https://www.bestbuy.ca/en-ca/product/paper-mario-the-origami-king-switch/14643866"
html = get(url)
soup = Soup(html)

title = soup.find('h1', {'class': 'productName'}).text
price = soup.find("div", {"class": "price"})
lhs = price.text
rhs = price.find("span", {'class': 'rightEndPrice'}).text
price = float(f"{lhs}.{rhs}".replace('$', ""))

client.chat_postMessage(channel="mario", text=f"${price} - {title}")

You really shouldn't expose your token as plaintext but, whatever...

Run the script at the command line to make sure it works:

python mario.py

Scheduling this script to every 30 seconds with hickory is dead simple. Just use hickory schedule at the command line:

hickory schedule mario.py --every=30s

To check the status of the schedule run:

hickory status
# ID      FILE         STATE    RUNS  INTERVAL
# ea6b74  mario.py     waiting  0     30