DEV Community: Greg

Best Books for Learning Python in 2021

Greg — Wed, 18 Nov 2020 23:12:15 +0000

its almost that time of year when people want to learn python

As we get closer to the end of one of the oddest years on record and everyone changes their Google search queries to 'best python books 2021', I thought it might be a good time to tackle the incredibly popular question of "If I want to learn Python, what books should I read?". There are endless articles on the topic - and basically any article will lead you to Automate the Boring Stuff - but I thought it was time to throw my hat in the ring, so to speak. If you're not a regular reader, I've done this a few times in the past with newsletters, twitter accounts, and podcasts.

Just as a forewarning, in a few past articles I documented all the code I used to get to the results of the post. This time I'm not doing so, and I'm sorry.

I mean I'm not really sorry - it takes a lot more effort to format my code from a sketchy jupyter notebook to a presentable-ish jupyter notebook. So we're just looking at results and random commentary from me this time!

Anyways, on to the stuff you care about - the top python books to read this year.

title	author	average_rating	ratings_count	publish_date
Fluent Python: Clear, Concise, and Effective Programming	Luciano Ramalho	4.66	777	2015-01-25
Grokking Algorithms An Illustrated Guide For Programmers and Other Curious People	Aditya Y. Bhargava	4.41	1,576	2015-01-01
Deep Learning with Python	Francois Chollet	4.64	680	2018-12-04
Hands-On Machine Learning with Scikit-Learn and TensorFlow	Aurélien Géron	4.55	651	2017-04-09
Automate the Boring Stuff with Python: Practical Programming for Total Beginners	Al Sweigart	4.28	1,432	2014-11-25
Python Crash Course: A Hands-On, Project-Based Introduction to Programming	Eric Matthes	4.35	976	2015-02-25
Python for Data Analysis	Wes McKinney	4.13	1,226	2011-12-30
Learning Python	Mark Lutz	3.96	1,976	2013-07-24
Effective Python: 59 Specific Ways to Write Better Python	Brett Slatkin	4.29	630	2015-02-01
Python Machine Learning	Sebastian Raschka	4.27	411	2015-09-23
Programming Python	Mark Lutz	3.98	848	1996-08-01
Python Data Science Handbook: Tools and Techniques for Developers	Jake Vanderplas	4.32	285	2016-03-25
Introduction to Machine Learning with Python: A Guide for Data Scientists	Andreas C. Müller	4.34	257	2015-06-25
Python for Everybody: Exploring Data in Python 3	Charles Severance	4.31	256	2016-07-10
Python Cookbook	David Beazley	4.16	374	2002-07-15
Learn Python The Hard Way	Zed A. Shaw	3.87	881	2010-01-01
Flask Web Development: Developing Web Applications with Python	Miguel Grinberg	4.19	302	2014-04-28
Data Science from Scratch: First Principles with Python	Joel Grus	3.93	552	2015-04-14
Test-Driven Web Development with Python	Harry Percival	4.21	237	2010-01-01
Black Hat Python: Python Programming for Hackers and Pentesters	Justin Seitz	4.04	319	2014-11-25
Python Pocket Reference	Mark Lutz	4.00	307	1998-03-15
Introducing Python: Modern Computing in Simple Packages	Bill Lubanovic	4.21	170	2013-11-22
A Common-Sense Guide to Data Structures and Algorithms: Level Up Your Core Programming Skills	Jay Wengrow	4.22	160	2017-08-22
Head First Python	Paul Barry	3.83	320	2010-01-01
Python Testing with Pytest: Simple, Rapid, Effective, and Scalable	Brian Okken	4.06	119	2017-10-25
Think Stats	Allen B. Downey	3.63	315	2011-01-01
The Self-Taught Programmer: The Definitive Guide to Programming Professionally	Cory Althoff	4.00	71	2017-01-24
Serious Python: Black-Belt Advice on Deployment, Scalability, Testing, and More	Julien Danjou	4.03	64	2018-12-01
Feature Engineering for Machine Learning	Alice Zheng	3.81	80	2018-04-10
Impractical Python Projects: Playful Programming Activities to Make You Smarter	Lee Vaughan	4.24	31	2018-11-27
Python Flash Cards: Syntax, Concepts, and Examples	Eric Matthes	4.08	12	2019-01-15
High Performance Python: Practical Performant Programming for Humans	Micha Gorelick	4.17	7	2013-10-22

So which one should I read if I'm a beginner?

So if we filter down the list above which books are primarily for beginners, you end up with a solid starting point:

Automate the Boring Stuff with Python: Practical Programming for Total Beginners by Al Sweigart (4.28 avg, 1,432 ratings)
Python Crash Course: A Hands-On, Project-Based Introduction to Programming by Eric Matthes (4.35 avg, 976 ratings)
Learning Python by Mark Lutz (3.96 avg, 1,976 ratings)
Grokking Algorithms An Illustrated Guide For Programmers and Other Curious People by Aditya Y. Bhargava (4.41 avg, 1,576 ratings)
Python for Data Analysis by Wes McKinney (4.13 avg, 1,226 ratings)

And those are only the top five books on there. Grokking Algorithms is a bit more focused on the computer science aspects of coding rather than the python language, but it does use Python for its examples. And Python for Data Analysis is basically focused on a python package called pandas, which is what any data scientist/analyst spends most of their day in. So if you're truly a beginner, focus on Automate the Boring Stuff, Python Crash Course, and Learning Python.

My other recommendation for beginners is to keep an eye out for bundle deals, like humble bundle. You can normally get ~10 ebooks for $15. Which is a pretty good deal, given a lot of these books go for $30+ on their own...

The Most Popular Data Science Newsletters

Greg — Thu, 17 Sep 2020 20:52:06 +0000

Or at least, the most often cited when looking through articles about data science newsletters.

title	mention_count
Data Elixir	13
Data Science Weekly	12
O’Reilly Data Newsletter	12
The Data Science Roundup	11
Import AI	8
KDnuggets News	8
The Analytics Dispatch	8
Data Machina	7
Machine Learnings	6
Data is Plural	5
Deep Learning Weekly	5
The Algorithm**	5
The Wild Week in AI	5
Topbots Applied AI	5
AI Weekly	4
Center for Data Innovation	4
Inside AI	4
insideBIGDATA	4
NYU Data Science Community News	4
Data Science Central	3
Deep Hunt	3
Hacker News Digest	3
OpenDS	3
The Week in Data	3
Analytics Vidhya	2
Dataconomy	2
Kaggle Newsletter	2
Towards Data Science	2

** Paid Newsletter

Note: This was collected by hand because my web scraping gameplan didn't account for newsletters that go to multiple different links... Could I have built some super cool algorithm to collect this? ...Maybe? But I didn't. Instead, I just collected links for about an hour or two and here we are. Although the process was rather tedious, it probably provides a lot of value to you guys, so I'd say it was worth it.

honorable mention

These newsletters were mentioned one time each in the lists I was referencing. FYI to the two people who are combing through the source links wondering why newsletter x or newsletter y wasn't included - I took out newsletters where the links were no longer functional.

AI Times, AI Trends, Big Data News Weekly, Bootstrap Labs, CB Insights, ChinAI, Creative AI, dair.ai NLP Newsletter, Data Coalition, Data Community DC, Data Eng Weekly, DataQuest Newsletter, Exponential View, Eye on AI, Flowing Data, Gary’s Guide, Hacker Noon, Hilary Mason, Humane AI, Lionbridge AI, Machine Learning Blueprint, ML in Production, NLP News, Open AI, R-Bloggers, Skynet Today, Stratechery, Talking Machines, The Art of Data Science, The Batch, The Belamy, The Gradient, The Pudding, TLDR, TWIML

sources for the list

newsletter time!

You've made it to the body of the article! Congratulations!

In today's post, I'm continuing my completely absurd crusade to find publications to subscribe to - next up on my list is newsletters. My previous articles were about twitter accounts and podcasts, if you're inclined to check them out.

But similar to my views on podcasts, I love newsletters. I find them to be most useful for keeping up with all the cool, new developments in the fields that I'm interested in. Unlike other platforms - cough Twitter cough - where you get bogged down with a deluge of information, newsletters typically boil everything down into 5-10 bullet points per week (or month). And brevity is nice.

And I recently found a cool app* that lets me subscribe to them outside of my normal email, so I can have a separate 'Data Science/Tech Newsletter' app, effectively. Which is fantastic. Like truly, honestly, fantastic. Highly recommend.

*For reference, the newsletter app is called slick inbox and it's still in beta. There's also apparently one called stoop inbox, which seems pretty similar. I don't endorse either one of these, nor do I have any affiliation - I just like having newsletters out of my inbox so that I can actually read them when I have free time.

Anyways, you probably don't care about that, lets get on to the process.

gameplanning the process

So, off the back of some silly articles where I found popular twitter accounts and podcasts, I thought... what next?

The answer? Sleep. Also, work. But then, newsletters. Newsletters are definitely next.

But how to go about doing this?

My first thought was to find a site that aggregates newsletter listings and estimates subscribers, but I struck out hard on that front. So I moved to plan B - google-ing what the best data science newsletters were. That search brought me to a lot of blog posts in list format, which had links to - you guessed it - newsletters. My bright data-science-y mind thought, "Hey, you can scrape these links and count the frequency they occur in the articles - that'll be an easy way to get your answer". So that's exactly what I did.

Well, that's what I scoped out and built... unfortunately, I had to scrap the code because links to newsletters are inconsistent (in that some people link to the main site, some to a subscribe-specific link, a newsletter-hosting service, etc). But since I did already build the code, I think its worth sharing... even if it looks like its held together with duct tape.

code to find top data science newsletters

I'm going to throw the code below with a bit less than the standard amount of commentary. Since it doesn't actually work, no one's really going to end up using it... probably?

# import default python packages
import urllib
import requests
import time

# import non-standard python packages
# if you dont have these installed, install them with pip or conda
from bs4 import BeautifulSoup
from readability import Document
import pandas as pd

# define the desktop user-agent
USER_AGENT = "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0.1"
# mobile user-agent
MOBILE_USER_AGENT = "Mozilla/5.0 (iPhone; CPU iPhone OS 12_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1"

# function to create google query string based on a query text
def string_to_google_query(string, pg=None):
    query = urllib.parse.quote_plus(f'{string}')
    if pg != None:
        query = query + "&start=" + str(10*(pg-1))
    return f"https://google.com/search?hl=en&lr=en&q={query}", string

# Queries to feed scraper
list_kwargs = [
    {"string": 'best data science newsletters'},
    {"string": 'best data science newsletters', "pg": 2},
    {"string": 'best data science newsletters', "pg": 3},
    {"string": 'best data engineering newsletters'},
    {"string": 'best data visualization newsletters'},
    {"string": 'best artificial intelligence newsletters'},
    {"string": 'best machine learning newsletters'},

    {"string": 'data science newsletters'},
    {"string": 'data science newsletters', "pg": 2},
    {"string": 'data science newsletters', "pg": 3},
    {"string": 'data engineering newsletters'},
    {"string": 'data visualization newsletters'},
    {"string": 'artificial intelligence newsletters'},
    {"string": 'machine learning newsletters'},
]

def google_scraper_mobile(link):
    results = []
    headers = {"user-agent" : MOBILE_USER_AGENT}
    resp = requests.get(url, headers=headers)
    if resp.status_code == 200:
        soup = BeautifulSoup(resp.content, "html.parser")

    for g in soup.find_all('div', class_='mnr-c'):
        anchors = g.find_all('a')
        if len(anchors) > 0:
            try:
                # this code will fail on featured snippets
                link = anchors[0]['href']
            except:
                next

            try:
                title = anchors[0].find_all('div')[1].get_text().strip()
            except:
                title = anchors[0].get_text().strip()

            item = {
                "title": title,
                "link": link,
                "search_term": search_term
            }
            results.append(item)
    return results


# Crawling Google as a mobile user
headers = {"user-agent" : MOBILE_USER_AGENT}

results = []
for x in list_kwargs:
    url, search_term = string_to_google_query(**x)
    scrape_res = google_scraper_mobile(url)
    results = results + scrape_res

    time.sleep(2.5)


# put results into a dataframe
newsletter_df = pd.DataFrame(results)

# Check there is a number in the title (such as 'top 10 newsletters')
# or that the title contains newsletter
newsletter_df = newsletter_df.loc[(newsletter_df['title'].str.contains('[0-9]') | newsletter_df['title'].str.lower().str.contains('newsletter'))]
newsletter_df.drop_duplicates(subset='link',inplace=True)

# switch the user agent to desktop - articles shouldn't differ on desktop vs mobile and will likely have fewer issues on desktop
headers = {"user-agent" : USER_AGENT}

#define the crawler for each article
def article_link_crawl(link):
    """
    Returns links and either a 1 or 0 if it was a success / failure.
    Only Crawls articles, so there should be a few failures
    """
    try:
        domain = link.split('://')[1].split('/')[0] # defines the site domain
        article_links = []
        resp = requests.get(link, headers=headers) # get request for the link

        # pass the article through readibility to get the article content rather than the full webpage
        rd_doc = Document(resp.text)
        if resp.status_code == 200:
            soup = BeautifulSoup(rd_doc.content(), "html.parser")
        link_soup = soup.find_all('a') # find all links
        for link in link_soup:
            # if the link has a href, create an item to add to the aggregate article_links list
            if link.has_attr('href'):
                item = {
                    "text": link.get_text().strip(),
                    "link": link['href']
                    }
                # dont add any blank links, internal links, links starting with '/' or '#' (also internal links)
                if item['text'] != '' and item['link'].find(domain) == -1 and item['link'][0] != '/' and item['link'][0] != '#':
                    article_links.append(item)
        return article_links, 1
    except:
        return None, 0


# loop through results
agg_links = []
total_success = 0
total_fail = 0
fail_urls = []
for link in newsletter_df['link']:
    res_links, is_success = article_link_crawl(link)
    if is_success == 1:
        total_success = total_success+1
    else:
        total_fail = total_fail+1
        fail_urls.append(link)

    if res_links != None:
        for lnk in res_links:
            agg_links.append(lnk)
    time.sleep(2.5)

# function to get frequency of a link
def list_freq(tgt_list): 
    freq = {} 
    for item in tgt_list: 
        if (item in freq): 
            freq[item] += 1
        else: 
            freq[item] = 1

    result = []
    for key, value in freq.items(): 
        result.append({
            "link": key,
            "count": value
        })
    return result

# count occurrences of each link
clean_link_list = [x['link'].replace('http://','https://') for x in agg_links]
link_freq = list_freq(clean_link_list)

# count occurrences of anchor text
clean_text_list = [x['text'].replace('http://','https://') for x in agg_links]
text_freq = list_freq(clean_text_list)

# move variables to data frames
link_freq_df = pd.DataFrame(link_freq)
text_freq_df = pd.DataFrame(text_freq)

link_freq_df = link_freq_df.sort_values('count',ascending=False)
link_freq_df = link_freq_df.loc[link_freq_df['count'] >= 3]

This was the result of the code - not great, but not horrible. In the end, I just decided the 'old fashioned way' was the proper way to find newsletters...

link	count
https://roundup.fishtownanalytics.com/	7
https://www.oreilly.com/data/newsletter.html	7
https://dataelixir.com/	7
https://www.datascienceweekly.org/	7
https://us13.campaign-archive.com/home/?u=67bd06787e84d73db24fb0aa5&id=6c9d98ff2c	6
https://github.com/EthicalML/awesome-machine-learning-operations	5
https://www.instagram.com/data4sci/	4
https://www.linkedin.com/company/dataforscience/	4
https://github.com/DataForScience	4
https://medium.com/data-for-science	4
https://twitter.com/data4sci	4
https://tinyletter.com/data-is-plural	4
https://theodi.org/knowledge-opinion/the-week-in-data/	4
https://www.facebook.com/data4sci/	4
https://towardsdatascience.com/doing-machine-learning-the-uber-way-five-lessons-from-the-first-three-years-of-michelangelo-da584a857cc2	3
https://www.datasciencecentral.com/profiles/blogs/check-out-our-dsc-newsletter	3
https://www.podcastinit.com	3
https://www.getrevue.co/profile/datamachina	3
https://www.datascienceweekly.org/newsletters	3
https://medium.com/towards-data-science/newsletters/the-daily-pick?source=newsletter_v3_promo--------------------------newsletter_v3_promo-	3

Finding popular data science podcasts via web scraping

Greg — Fri, 11 Sep 2020 13:21:58 +0000

The article will go over the process I used to create the list of podcasts you see below. If you're just here for the podcasts, then have at it...

the most popular data science podcasts

title	author	avg_rtg	rtg_ct	episodes
Lex Fridman Podcast	Lex Fridman	4.9	2400	126
Machine Learning Guide	OCDevel	4.9	626	30
Data Skeptic	Kyle Polich	4.4	431	300
Data Stories	Enrico Bertini and Moritz Stefaner	4.5	405	162
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)	Sam Charrington	4.7	300	300
DataFramed	DataCamp	4.9	188	59
The AI Podcast	NVIDIA	4.5	162	125
SuperDataScience	Kirill Eremenko	4.6	161	300
Partially Derivative	Partially Derivative	4.8	141	101
Machine Learning	Stanford	3.9	138	20
Talking Machines	Tote Bag Productions	4.6	133	106
AI in Business	Daniel Faggella	4.4	102	100
Learning Machines 101	Richard M. Golden, Ph.D., M.S.E.E., B.S.E.E.	4.4	87	82
storytelling with data podcast	Cole Nussbaumer Kna	4.9	80	33
Data Crunch	Data Crunch Corporation	4.9	70	64
Data Viz Today	Alli Torban	5.0	64	62
Artificial Intelligence	MIT	4.1	61	31
O'Reilly Data Show Podcast	O'Reilly Media	4.2	59	60
Machine Learning – Software Engineering Daily	Machine Learning – Software Engineering Daily	4.5	59	115
Data Science at Home	Francesco Gadaleta	4.2	58	100
Data Engineering Podcast	Tobias Macey	4.7	58	150
Big Data	Ryan Estrada	4.6	58	13
Follow the Data Podcast	Bloomberg Philanthropies	4.3	57	82
Making Data Simple	IBM	4.3	56	104
Analytics on Fire	Mico Yuk	4.4	51	48
Learn to Code in One Month	Learn to Code	4.9	50	26
Becoming A Data Scientist Podcast	Renee Teate	4.5	49	21
Practical AI: Machine Learning & Data Science	Changelog Media	4.5	48	105
The Present Beyond Measure Show: Data Visualization, Storytelling & Presentation for Digital Marketers	Lea Pica	4.9	44	58
The Data Chief	Mission	4.9	43	16
AI Today Podcast: Artificial Intelligence Insights, Experts, and Opinion	Cognilytica	4.2	42	161
Data Driven	Data Driven	4.9	41	257
HumAIn Podcast - Artificial Intelligence, Data Science, and Developer Education	David Yakobovitch	4.8	39	78
Data Gurus	Sima Vasa	5.0	39	106
Masters of Data Podcast	Sumo Logic hosted by Ben Newton	5.0	38	74
The PolicyViz Podcast	The PolicyViz Podcast	4.7	36	180
The Radical AI Podcast	Radical AI	4.9	34	35
Women in Data Science	Professor Margot Gerritsen	4.9	28	24
Towards Data Science	The TDS team	4.6	26	50
Data in Depth	Mountain Point	5.0	22	24
Data Science Imposters Podcast	Antonio Borges and Jordy Estevez	4.4	22	88
The Artists of Data Science	Harpreet Sahota	4.9	19	41
#DataFemme	Dikayo Data	5.0	17	30
The Banana Data Podcast	Dataiku	4.9	15	33
Experiencing Data with Brian T. O'Neill	Brian T. O'Neill from Designing for Analytics	4.9	14	13
Secrets of Data Analytics Leaders	Eckerson Group	4.8	13	82
Data Journeys	AJ Goldstein	5.0	13	26
Data Driven Discussions	Outlier.ai	5.0	12	8
Data Futurology - Leadership And Strategy in Artificial Intelligence, Machine Learning, Data Science	Felipe Flores	4.4	11	135
Artificially Intelligent	Christian Hubbs and Stephen Donnelly	4.9	11	100

why i want to find data science podcasts

This would normally be at the top of an article on finding data science podcasts. Well it would be at the top of any article. But realistically, most people are finding this from google, and they're just looking for the answer that's at the top of the page. If you type in 'the most popular data science podcasts', you really don't want to have to scroll down endlessly to find the answer you're looking for. So to make their experience better, we're just leaving the answer up there. And giving them sass. Lots of sass.

Anyways, I really like listening to things. While newsletters are great for keeping up with current events and blogs are great for learning specific things, podcasts have a special place in my heart for allowing me to aimlessly learn something new every day. The format really lends itself to delivering information efficiently, but in a way where you can multitask. Pre-COVID, my morning commute was typically full of podcasts. While COVID has rendered my commute a nonexistent affair, I still try to listen to at least a podcast a day if I can manage it. My view is that 30 minutes of learning a day will really add up in the long run, and I feel that podcasts are a great way to get there.

Now that we've been through my love affair with podcasts, you can imagine my surprise when I started looking for a few data science ones to subscribe to and I didn't find a tutorial on how to use web scraping to find the most popular data science podcasts to listen to. I know, crazy. There's a web scraping tutorial on everything under the sun except for - seemingly - podcasts. I mean there's probably not one on newsletters either, but we'll leave that alone for now...

So if no one else is crazy enough to write about finding data science podcasts with web scraping, then...

gameplanning the process

By now we're almost certainly rid of those savages who are only here for the answer (gasp, how could they), so we'll go into the little process I went through to gather the data. It's not particularly long, and took me probably an hour to put it together, so it should be a good length for an article.

I'm using python here with an installation of Anaconda (which is a common package management / deployment system for python). I'll be running this in a Jupyter notebook, since its a one-off task that I don't need to use ever again... hopefully.

In terms of what I'm going to do, I'll run a few google keyword searches which are limited to the 'https://podcasts.apple.com/us/podcast/' domain and scrape the results for the first few pages. From there I'll just be scraping the apple podcast page to get the total number of ratings and the average rating. Yea, the data will be biased, but its a quick and dirty way to get the answer I'm looking for.

code to find top data science podcasts - version 1

# import default python packages
import urllib
import requests
import time

The above packages are included in python, the below ones aren't always included. If you don't have them installed, you'll have to download them. You can find out how to use pip to do it or conda.

# import non-standard python packages
# if you dont have these installed, install them with pip or conda
from bs4 import BeautifulSoup
import pandas as pd

Now that the packages have been imported, you should define your user agent. First off, because its polite if you're scraping anything. Secondly, google gives different results for mobile and desktop searches. This isn't actually my user-agent, I took it from another tutorial since I'm a bit lazy. I actually use linux...

# define your desktop user-agent
USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0"

Alright now we're going to define the queries we want to run. And then create a function that spits out the URL we want to scrape on google. I'm putting the queries in a kwargs format, since I want to put them through a function. That means I can just loop through the list of kwargs and get the results that the function returns.

# Queries
list_kwargs = [
    {"string": 'data podcast'},
    {"string": 'data podcast', "pg": 2},
    {"string": 'data podcast', "pg": 3},
    {"string": 'data science podcast'},
    {"string": 'data engineering podcast'},
    {"string": 'data visualization podcast'},
]

def string_to_podcast_query(string, pg=None):
    query = urllib.parse.quote_plus(f'site:https://podcasts.apple.com/us/podcast/ {string}')
    if pg != None:
        query = query + "&start=" + str(10*(pg-1))
    return f"https://google.com/search?hl=en&lr=en&q={query}", string

# define the headers we will add to all of our requests
headers = {"user-agent" : USER_AGENT}

# set up an empty list to push results to
results = []

# cycle through the list of queries 
for x in list_kwargs:
    # return the query url and the search term that was used to create it (for classification later)
    url, search_term = string_to_podcast_query(**x)

    # make a get request to the url, include the headers with our user-agent
    resp = requests.get(url, headers=headers)

    # only proceed if you get a 200 code that the request was processed correctly
    if resp.status_code == 200:
        # feed the request into beautiful soup
        soup = BeautifulSoup(resp.content, "html.parser")

    # find all divs (a css element that wraps page areas) within google results
    for g in soup.find_all('div', class_='r'):
        # within the results, find all the links 
        anchors = g.find_all('a')
        if anchors:
            # get the link and title, add them to an object, and append that to the results array
            link = anchors[0]['href']
            title = g.find('h3').text
            item = {
                "title": title,
                "link": link,
                "search_term": search_term
            }
            results.append(item)

    # sleep for 2.5s between requests.  we don't want to annoy google and deal with recaptchas
    time.sleep(2.5)

Alright, now we have the google results back - nice. From here, lets put that in a pandas dataframe and filter it a bit.

google_results_df = pd.DataFrame(results)

# create a filter for anything that is an episode.  They should contain a ' | '.
# drop any duplicate results as well.
google_results_df['is_episode'] = google_results_df['title'].str.contains(' | ',regex=False)
google_results_df = google_results_df.drop_duplicates(subset='title')

google_results_podasts = google_results_df.copy().loc[google_results_df['is_episode']==False]

Ok cool, we have a list of podcasts. Lets define our apple podcasts scraper.

def podcast_scrape(link):
    # get the link, use the same headers as had previously been defined.
    resp = requests.get(link, headers=headers)
    if resp.status_code == 200:
        soup = BeautifulSoup(resp.content, "html.parser")

    # find the figcaption element on the page
    rtg_soup = soup.find("figcaption", {"class": "we-rating-count star-rating__count"})
    # the text will return an avg rating and a number of reviews, split by a •
    # we'll spit that out, so '4.3 • 57 Ratings' becomes '4.3', '57 Ratings'
    avg_rtg, rtg_ct = rtg_soup.get_text().split(' • ')
    # then we'll take numbers from the rtg_ct variable by splitting it on the space
    rtg_ct = rtg_ct.split(' ')[0]

    # find the title in the document, get the text and strip out whitespace
    title_soup = soup.find('span', {"class":"product-header__title"})
    title = title_soup.get_text().strip()
    # find the author in the document, get the text and strip out whitespace
    author_soup = soup.find('span', {"class":"product-header__identity podcast-header__identity"})
    author = author_soup.get_text().strip()

    # find the episode count div, then the paragraph under that, then just extract the # of episodes
    episode_soup = soup.find('div', {"class":"product-artwork__caption small-hide medium-show"})
    episode_soup_p = episode_soup.find('p')
    episode_ct = episode_soup_p.get_text().strip().split(' ')[0]

    # format the response as a dict, return that response as the result of the function
    response = {
        "title": title,
        "author": author,
        "link": link,
        "avg_rtg": avg_rtg,
        "rtg_ct": rtg_ct,
        "episodes": episode_ct
    }
    return response

Cool, we now have a podcast scraper. You can try it with the below code.

podcast_scrape('https://podcasts.apple.com/us/podcast/follow-the-data-podcast/id1104371750')


{'title': 'Follow the Data Podcast',
'author': 'Bloomberg Philanthropies',
'link': 'https://podcasts.apple.com/us/podcast/follow-the-data-podcast/id1104371750',
'avg_rtg': '4.3',
'rtg_ct': '57'}

Back to the code. Lets now loop through all the podcast links we have.

# define the result array we'll fill during the loop
podcast_summ = []
for link in google_results_podcasts['link']:
    # use a try/except, since there are a few episodes still in the list that will cause errors if we don't do this.  This way, if there is an error we just wont add anything to the array.
    try:
        # get the response from our scraper and append it to our results
        pod_resp = podcast_scrape(link)
        podcast_summ.append(pod_resp)
    except:
        pass
    # wait for 5 seconds to be nice to apple
    time.sleep(5)

Now to put everything into a dataframe and do a little bit of sorting and filtering.

pod_df = pd.DataFrame(podcast_summ)

# Remove non-english podcasts, sorry guys...
pod_df = pod_df.loc[~pod_df['link'].str.contains('l=')]
pod_df.drop_duplicates(subset='link', inplace=True)

# merge with the original dataframe (in case you want to see which queries were responsible for which podcasts)
merge_df = google_results_podcasts.merge(pod_df,on='link',suffixes=('_g',''))
merge_df.drop_duplicates(subset='title', inplace=True)

# change the average rating and rating count columns from strings to numbers
merge_df['avg_rtg'] = merge_df['avg_rtg'].astype('float64')
merge_df['rtg_ct'] = merge_df['rtg_ct'].astype('int64')

# sort by total ratings and then send them to a csv
merge_df.sort_values('rtg_ct',ascending=False).to_csv('podcasts.csv')

From here I exported the file to csv and did a bit of cheating where I combined the title and link to create a <a hrer="link">title</a>, but that's mainly because I got a bit lazy...

Anyways, that was the full process in creating the above list of data science podcasts. You now have the top podcasts, sorted by total reviews. I considered also using castbox as a source of scraping (since they have an approximation of subscribers / downloads), but I couldn't find any good way to search for generally popular podcasts. Or podcasts that contained a certain word.

The first version of this article stopped here and showed results from this code

code to find top data science podcasts - version 2

Well, that was fine, but I think its actually lacking a bit. There seem to be a few podcasts that I've stumbled across that are missing which I was hoping this would capture. So we're going to switch some stuff up. First, I'm going to use a mobile user agent to tell Google I'm searching from my phone.

Why? Well Google shows different results for desktop searches vs mobile searches, so if we're looking to find the best podcasts, we want to be where most of the searches are actually happening. And since you basically always listen to podcasts on your phone, it probably makes sense to search from your phone... The code for that is below, the main changes are in

# Mobile Search Version
headers = {"user-agent" : MOBILE_USER_AGENT}

results = []
for x in list_kwargs:
    url, search_term = string_to_podcast_query(**x)
    resp = requests.get(url, headers=headers)
    if resp.status_code == 200:
        soup = BeautifulSoup(resp.content, "html.parser")

    for g in soup.find_all('div', class_='mnr-c'): # updated target class
        anchors = g.find_all('a')
        if anchors:
            link = anchors[0]['href']
            title = anchors[0].find_all('div')[1].get_text().strip() # updated title crawler
            item = {
                "title": title,
                "link": link,
                "search_term": search_term
            }
            results.append(item)

    time.sleep(2.5)

What else did I switch up? I switched the Google queries up a bit and added a few more. I figure if I'm actually trying to find the best podcasts, it makes sense to search for them. That way, you get the ones that typically show up on these types of blog lists.

# Queries
list_kwargs = [
    {"string": 'best data podcast'},
    {"string": 'best data podcast', "pg": 2},
    {"string": 'best data podcast', "pg": 3},
    {"string": 'best data podcast', "pg": 4},
    {"string": 'best data science podcast'},
    {"string": 'best data science podcast', "pg": 2},
    {"string": 'best data science podcast', "pg": 3},
    {"string": 'best artificial intelligence podcast'},
    {"string": 'best machine learning podcast'},
    {"string": 'best data engineering podcast'},
    {"string": 'best data visualization podcast'},
]

And that's it - all of the changes I made for the second version. The results are updated up top, but it gets a more complete

code to find top data science podcasts - version 3

And I'm an idiot. 'Fixing' my queries to only find the 'best data science podcasts' ended up making me miss a few of the good ones I found earlier. So I'm going to do as any good data scientist does and just combine the results of both sets of queries...

# Queries

list_kwargs = [

    {"string": 'best data podcast'},

    {"string": 'best data podcast', "pg": 2},

    {"string": 'best data podcast', "pg": 3},

    {"string": 'best data podcast', "pg": 4},

    {"string": 'best data science podcast'},

    {"string": 'best data science podcast', "pg": 2},

    {"string": 'best data science podcast', "pg": 3},

    {"string": 'best artificial intelligence podcast'},

    {"string": 'best machine learning podcast'},

    {"string": 'best data engineering podcast'},

    {"string": 'best data visualization podcast'},

]

closing note

This is a cross-post from my blog. My current readership is a solid 0 views per month, so I thought it might be worth actually sharing it here...

DEV Community: Greg

Best Books for Learning Python in 2021

its almost that time of year when people want to learn python

top python books to read in 2021

So which one should I read if I'm a beginner?

The Most Popular Data Science Newsletters

honorable mention

sources for the list

newsletter time!

gameplanning the process

code to find top data science newsletters

Finding popular data science podcasts via web scraping

the most popular data science podcasts

why i want to find data science podcasts

gameplanning the process

code to find top data science podcasts - version 1

code to find top data science podcasts - version 2

code to find top data science podcasts - version 3

closing note