Scrape Google Autocomplete Suggestion with Python

Contents: intro, imports, what will be scraped, process, code, links, outro.

Intro

This blog post is a continuation of Google's web scraping series.
Here you'll see examples of how you can scrape Autocomplete Suggestion using Python. An alternative API solution will be shown.

Imports

import os, requests, json

What will be scraped

Process

All that needs to be done is to add /complete/ and client parameter (...&client=chrome...) the search URL.

This URL: https://www.google.com/search?q=minecraft is better than  # no '/complete/' word in the url
Becomes this URL: http://google.com/complete/search?client=chrome&q=minecraft is better than # added '/complete/' as well as 'client' param

Code

import requests, json

headers = {
    "User-Agent":
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}
# client param could be replaced with firefox or other browser
response = requests.get('http://google.com/complete/search?client=chrome&q=minecraft is better than', headers=headers)
for result in json.loads(response.text)[1]:
    print(result)

-----------
'''
minecraft is better than roblox
minecraft is better than fortnite
minecraft is better than terraria
minecraft is better than fortnite memes
minecraft is better than among us
minecraft is better than roblox and fortnite
minecraft is better than fortnite song
minecraft is better than fortnite reddit
'''

Using Google Autocomplete API

SerpApi is a paid API with a free trial of 5,000 searches and essentially it does the same as a code above except you can use some of the advanced autocomplete parameters such as:

• Cursor pointer which can be used to refine completion.

Example with below query q: minecraft is better than and cp:8 will give completely different results as you see in the code below.

import os 
from serpapi import GoogleSearch

params = {
  "engine": "google_autocomplete",
  "q": "minecraft",
  "api_key": os.getenv("API_KEY"), # environment variable 
}

search = GoogleSearch(params)
results = search.get_dict()

for result in results["suggestions"]:
  print(result['value'])

--------------
'''
default (0) "cp" param
minecraft java
minecraft 1.17
minecraft classic
minecraft mods
....

"cp" param set to 2
minecraft classic
minecraft 1.17
minecraft education edition
minecraft free
'''

Links

Code in the online IDE • Google Autocomplete API