Scrape Google Top Stories using Python

Contents: intro, imports, what will be scraped, process, code, links, outro.

Intro

This blog post is a continuation of Google's web scraping series. Here you'll see how to scrape Google Top Stories from Organic Search Results using Python with beautifulsoup, requests libraries. An alternative API solution will be shown.

Imports

from bs4 import BeautifulSoup
import requests, lxml
from serpapi import GoogleSearch

What will be scraped

Process

Selecting container, published date, title CSS selectors.

Selecting link CSS selector.

Code

from bs4 import BeautifulSoup
import requests, lxml

headers = {
    'User-agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

params = {
  'q': 'john cena',
  'hl': 'en',
  'gl': 'us',
}

html = requests.get('https://www.google.com/search?q=', headers=headers, params=params).text
soup = BeautifulSoup(html, 'lxml')

for result in soup.select('.yLWA8b'):
    title = result.select_one('.nDgy9d').text.replace('\n', '')
    link = result.select_one('.WlydOe')['href']
    published_date = result.select_one('.ecEXdc span').text
    print(f'{title}\n{link}\n{published_date}\n')

----------------
'''
Nikki Bella's Reported Response to John Cena's New Attitude on Having Kids Is All Class
https://www.yahoo.com/entertainment/nikki-bellas-reported-response-john-194941780.html
2 days ago

When will John Cena win his seventeenth World Championship?
https://www.sportskeeda.com/wwe/when-will-john-cena-win-seventeenth-world-championship
16 hours ago

Is Vin Diesel Really The Same Height As John Cena? | Screen Rant
https://screenrant.com/f9-john-cena-vin-diesel-height-comparison-same-different/
20 hours ago
'''

Using Google Top Stories API

SerpApi is a paid API with a free trial of 5,000 searches.

In this example, Google Top Stories API provides quite a few more results in relation to the code above. Also, if you do not interested in maintaining the parse, figuring out why things don't behave as they should, or bypassing blocks from Google, an API solution is a way to go.

from serpapi import GoogleSearch
import os, json

params = {
    "api_key": os.environ["API_KEY"], # pycharm environment
    "engine": "google",
    "q": "john cena",
    "gl": "us",
    "hl": "en"
}

search = GoogleSearch(params)
results = search.get_dict()

for result in results['top_stories']:
    print(json.dumps(result, indent=2, ensure_ascii=False))

-------------
'''
{
  "title": "Nikki Bella's Reported Response to John Cena's New Attitude on Having Kids Is All Class",
  "link": "https://www.yahoo.com/entertainment/nikki-bellas-reported-response-john-194941780.html",
  "source": "Yahoo",
  "date": "2 days ago",
  "thumbnail": "https://serpapi.com/searches/60e30aaf1071f423feda11f5/images/b544986d11640bd39c43fea8c2bbc111dec815b3d8089f08.jpeg"
}
{
  "title": "When will John Cena win his seventeenth World Championship?",
  "link": "https://www.sportskeeda.com/wwe/when-will-john-cena-win-seventeenth-world-championship",
  "source": "Sportskeeda",
  "date": "17 hours ago",
  "thumbnail": "https://serpapi.com/searches/60e30aaf1071f423feda11f5/images/b544986d11640bd38557cb4783461be1ca3dfe7ff6bcb2a0.jpeg"
}
...
'''

Links

Code in the online IDE • Google Top Stories API

Outro

If you have any questions or something isn't working correctly or you want to write something else, feel free to drop a comment in the comment section or via Twitter at @serp_api.

Yours,
Dimitry, and the rest of SerpApi Team.