DEV Community: Artur Chukhrai

Python SEO Keyword Research Tool: Google Autocomplete, People Also Ask and Related Searches

Artur Chukhrai — Mon, 15 May 2023 14:00:35 +0000

Intro
Demonstration
Installation
CLI Usage
In-code Usage
Links

Intro

Here's a simple user guide to help you get started with using seo-keyword-research-tool.

SEO Keyword Research Tool is a Python package designed to generate keywords for SEO purposes. The package is available on PyPi and can be installed and used in your Python project or as CLI. It is used to generate keyword ideas using Google Autocomplete, Related Searches and People also ask results and save them to TXT, CSV, JSON file.

This tool uses SerpApi as a tool to parse data from Google search results. Other search engines coming soon.

Demonstration

Installation

$ pip install seo-keyword-research-tool

📌Note: If you want to use this tool for personal or commercial purposes, you must register on SerpApi and get your own API key (free plan available). The default API key used is for informational purposes and only works in the CLI.

Make sure you have Python installed on your computer.

Before use, you need to define your API key from SerpApi. There are several ways to do this:

Add an environment variable API_KEY with your key. $ export API_KEY=<API_KEY> (for Linux based terminal)
Add your key directly in code: SeoKeywordResearch.api_key = "your_api_key" (for in-code usage only)

CLI Usage

You can see available usage options by typing in the terminal:

$ seo -h

Available arguments

SerpApi SEO Keyword Research Tool [-h] -q  [-e  [...]] [-dl] [-st] [-ak] [-gd] [-gl] [-hl]

Extract keywrods from: Google Autocomplete, People Also Ask, and People Also Search and saves data to CSV/JSON/TXT.

optional arguments:
  -h, --help            show this help message and exit
  -q , --query          Search query (required).
  -e  [ ...], --engines  [ ...]
                        Choices of engines to extract: Autocomplete (ac), Related Searches (rs), People Also Ask (rq). You can select multiple engines. All engines are selected by default.
  -dl , --depth-limit   Depth limit for People Also Ask. Default is 0, first 2-4 results.
  -st , --save-to       Saves the results in the current directory in the selected format (CSV, JSON, TXT). Default CSV.
  -ak , --api-key       Your SerpApi API key: https://serpapi.com/manage-api-key. Default is a test API key to test CLI.
  -gd , --domain        Google domain: https://serpapi.com/google-domains. Default google.com.
  -gl , --country       Country of the search: https://serpapi.com/google-countries. Default us.
  -hl , --lang          Language of the search: https://serpapi.com/google-languages. Default en.

Found a bug? Open issue: https://github.com/chukhraiartur/seo-keyword-research-tool/issues

📌Note: You can use any of the available arguments in any combination. You can see the full description of each of them below.

To get results for a specific query, you need to define the --query argument (or short: -q):

$ seo -q "python how to"

📌Note: This is a required argument.

Output

{
  "auto_complete": [
    "python how to add to a list",
    "python how to round to 2 decimals",
    "python how to install pip",
    "python how to comment out a block",
    "python how to sort a list",
    "python how to call a function",
    "python how to reverse a string",
    "python how to write to a file",
    "python how to add to a dictionary",
    "python how to convert string to int",
    "python how to remove item from list",
    "python how to split a string",
    "python how to open a file",
    "python how to reverse a list",
    "python how to concatenate strings"
  ],
  "related_searches": [
    "python for beginners",
    "python course",
    "python code examples",
    "python for beginners pdf",
    "learn python",
    "python w3schools",
    "python download",
    "python programming"
  ],
  "related_questions": [
    "How do I start programming in Python?",
    "How do I teach myself Python?",
    "Is Python easy to learn?",
    "Can I self teach myself Python?"
  ]
}
Saving data in CSV format...
Data successfully saved to python_how_to.csv file

To get results from all supported engines, you don't need to pass an engine argument (as used in the example above). If you want to get results from one or two selected engines you need to define the --engine argument (or short: -e) divided by a space, e.g. ac rs (Autocomplete and Related Searches):

$ seo -q "lofi" -e ac

Value	Engine
`ac`	Autocomplete
`rs`	Related Searches (People Also Search)
`rq`	Related Questions (People Also Ask)

Output

{
  "auto_complete": [
    "lofi girl",
    "lofi music",
    "lofi hip hop",
    "lofi boy",
    "lofi beats",
    "lofi jersey city",
    "lofi study",
    "lofi meaning",
    "lofi girl 34",
    "lofi youtube",
    "lofibra",
    "lofi study music",
    "lofi girl youtube",
    "lofi girl lore",
    "lofi girl synthwave boy"
  ]
}
Saving data in CSV format...
Data successfully saved to lofi.csv file

You can change the default google domain google.com to any of supported domains list by defining the --domain (or short: -gd) argument:

$ seo -q "big ben" -e rs rq -gd "google.co.uk"

Output

{
  "related_searches": [
    "big ben london",
    "big ben country",
    "big ben clock",
    "big ben facts",
    "how old is big ben",
    "why is big ben called big ben",
    "why is big ben famous",
    "big ben tickets"
  ],
  "related_questions": [
    "Is Big Ben still under construction 2023?",
    "Why is Elizabeth Tower called Big Ben?",
    "How much does it cost to go inside Big Ben?",
    "Can you go inside Big Ben?"
  ]
}
Saving data in CSV format...
Data successfully saved to big_ben.csv file

Also, you can change the default country us (United States) of the search using the --country (or short: -gl) argument and set it to any of supported country codes list:

$ seo -q "hotel in" -e rs -gl de

Output

{
  "related_searches": [
    "cheap hotel in germany",
    "cheap hotel in cologne, germany",
    "hotel in baesweiler restaurant",
    "hotel in baesweiler telefonnummer",
    "germany hotel booking website",
    "hotel baesweiler speisekarte",
    "germany hotel address",
    "hotel baesweiler öffnungszeiten"
  ]
}
Saving data in CSV format...
Data successfully saved to hotel_in.csv file

If you want to change the default language en (English) of the search, you need to use the --language (or short: -hl) argument and set it to any of supported language codes list:

$ seo -q "anime" -e ac -hl "ja"

Output

{
  "auto_complete": [
    "アニメ",
    "アニメイト",
    "アニメーション",
    "アニメ無料",
    "アニメ映画",
    "アニメ 英語",
    "アニメダンス",
    "アニメイト 池袋",
    "アニメ 2023",
    "アニメ人気",
    "アニメ9",
    "アニメ無料サイト",
    "anime expo",
    "anime girl",
    "anime store japan"
  ]
}
Saving data in CSV format...
Data successfully saved to anime.csv file

The default file type to save search results is CSV. You can change it to JSON or TXT using the --save-to (or short: -st) argument:

$ seo -q "coffee" -st json

Output

{
  "auto_complete": [
    "coffee near me",
    "coffee belvidere il",
    "coffee rockford",
    "coffee crush",
    "coffee nearby",
    "coffee table",
    "coffee crush menu",
    "coffee maker",
    "coffee shop",
    "coffee cake recipe",
    "coffee shops rockford il",
    "coffee grinder",
    "coffee shops rockford",
    "coffee meets bagel",
    "coffee creamer"
  ],
  "related_searches": [
    "coffee near me",
    "coffee types",
    "coffee beans",
    "coffee benefits",
    "coffee starbucks",
    "coffee origin",
    "coffee plant",
    "coffee tree"
  ],
  "related_questions": [
    "Is coffee good for health?",
    "How coffee affects the body?",
    "What are the 4 types of coffee?",
    "What are 10 benefits of coffee?"
  ]
}
Saving data in JSON format...
Data successfully saved to coffee.json file

When you open People Also Ask result, it loads more results.

Default:

One of the results is open:

So if you open all initial results that was mean you see next-depth results.

The --depth-limit (or short: -dl) argument for People Also Ask can be set from 0 to 4. For each depth limit value, the number of results returned grows exponentially. Below is a table showing how the depth limit argument is affected:

Depth limit	Number of results	Explanation
0	4	Standard results
1	12	4*2 = 8 + 4 = 12
2	36	8*3 = 24 + 12 = 36
3	108	24*3 = 72 + 36 = 108
4	324	72*3 = 216 + 108 = 324

📌Note: This is how the logic works for the google.com domain, on other domains the results may differ.

The example for depth 2 means that we open all standard results, then after the new results are loaded, we open them and wait until the new results are loaded:

$ seo -q "minecraft" -e rq -st txt -dl 2

Output

{
  "related_questions": [
    "Is Minecraft OK for a 7 year old?",
    "Is Minecraft appropriate for a 5 year old?",
    "Can I play Minecraft for free?",
    "Is Roblox better than Minecraft?",
    "Is Minecraft good for kids?",
    "Is Minecraft appropriate for 4 year olds?",
    "Is Minecraft OK for 18 year olds?",
    "Is 15 too old to play Minecraft?",
    "Why is my son obsessed with Minecraft?",
    "Is Minecraft OK for 2 year olds?",
    "What is Roblox age limit?",
    "Do 30 year olds play Minecraft?",
    "What is Roblox age limit?",
    "Is Minecraft OK for 2 year olds?",
    "What is Roblox age limit?",
    "Is 15 too old to play Minecraft?",
    "Is Roblox ok for a 40 year old?",
    "Are there inappropriate games on Roblox?",
    "Why is Fortnite not for kids?",
    "Do 30 year olds play Minecraft?",
    "Is Minecraft OK for 2 year olds?",
    "Is Minecraft OK for 18 year olds?",
    "How long is Minecraft free?",
    "How to get Minecraft for free legally?",
    "How long is 100 Minecraft day?",
    "Does Minecraft cost per month?",
    "How long does 1 days in Minecraft last?",
    "What is the free version of Minecraft called?",
    "How to download real Minecraft without money?",
    "How much does Minecraft cost?",
    "Which is bigger Minecraft or Roblox?",
    "Who is the best Minecraft player alive?",
    "How do I get Robux for free?",
    "What parents need to know about Roblox?",
    "What is better fortnite or Roblox?",
    "Who is PvP God in Minecraft?",
    "Who is the clutch god in Minecraft?",
    "Who is the Minecraft King?",
    "Is Minecraft good for ADHD?",
    "What is the appropriate age to play Minecraft?",
    "Why do autistic kids like Minecraft so much?",
    "Why is my son obsessed with Minecraft?",
    "Why is my kid addicted to Minecraft?",
    "Is Minecraft good for the brain?",
    "Is 16 too old to play Minecraft?",
    "Is Minecraft addictive game?",
    "What is Roblox age limit?",
    "What age is appropriate for Roblox?",
    "Is Roblox ok for a 40 year old?",
    "Are there inappropriate games on Roblox?",
    "Why is Fortnite not for kids?",
    "How do I make Roblox safe for my child?",
    "Should I let my child chat on Roblox?",
    "Is Roblox ok for a 40 year old?"
  ]
}
Saving data in TXT format...
Data successfully saved to minecraft.txt file

In-code Usage

You can also use this tool in code. In fact, the arguments that you enter in the CLI are passed to the SeoKeywordResearch class.

Import the SeoKeywordResearch class to your file:

from SeoKeywordResearch import SeoKeywordResearch

After that, you need to create an object of this class and pass all the necessary parameters:

keyword_research = SeoKeywordResearch(
    query='starbucks coffee',
    api_key='<your_serpapi_api_key>',
    lang='en',
    country='us',
    domain='google.com'
)

To extract data from a particular engine, you need to use the appropriate methods. In the example below, I show how you can create a dictionary and add the required data by keys:

data = {
    'auto_complete': keyword_research.get_auto_complete(),
    'related_searches': keyword_research.get_related_searches(),
    'related_questions': keyword_research.get_related_questions(depth_limit=1)
}

📌Note: The get_related_questions() method has the depth_limit parameter that allows you to load more results from People Also Ask.

You can use a convenient format to save data using the following methods:

keyword_research.save_to_csv(data)
keyword_research.save_to_json(data)
keyword_research.save_to_txt(data)

If you need to print data in JSON format, then use the following method:

keyword_research.print_data(data)

Output

{
  "auto_complete": [
    "starbucks coffee menu",
    "starbucks coffee cups",
    "starbucks coffee traveler",
    "starbucks coffee gear",
    "starbucks coffee drinks",
    "starbucks coffee pods",
    "starbucks coffee sizes",
    "starbucks coffee creamer",
    "starbucks coffee cake",
    "starbucks coffee traveler price",
    "starbucks coffee bags",
    "starbucks coffee beans",
    "starbucks coffee near me",
    "starbucks coffee mugs",
    "starbucks coffee press"
  ],
  "related_searches": [
    "starbucks coffee menu",
    "starbucks coffee price",
    "starbucks near me",
    "starbucks coffee beans",
    "starbucks coffee cups",
    "starbucks coffee bag size",
    "starbucks coffee on sale this week",
    "amazon starbucks coffee instant"
  ],
  "related_questions": [
    "Which one is the best coffee in Starbucks?",
    "Which coffee is Starbucks famous for?",
    "What coffee is sold most at Starbucks?",
    "What is the Starbucks drink for St Patrick's Day?",
    "Which Starbucks coffee is best for beginners?",
    "How do I choose a Starbucks coffee?",
    "Why is Starbucks coffee so popular?",
    "Who is known for the best coffee?",
    "What is the most ordered drink in the world?",
    "What is the most expensive drink at Starbucks?",
    "What is the name of the green coffee at Starbucks?",
    "Does Starbucks have a healthy green drink?"
  ]
}

Links

Feel free to open bug issue, something isn't working, what feature to add, or anything else related to Google autocomplete, related searches or people also ask.

If you want other functionality added to this demo project or if you want to see some other projects made with SerpApi, write me a message.

Join us on Twitter | YouTube

Add a Feature Request💫 or a Bug🐞

Get Product Data from Bing Shopping with Python and SerpApi

Artur Chukhrai — Tue, 25 Apr 2023 10:25:35 +0000

What will be scraped
Full Code
Preparation
Code Explanation
- Top-level code environment
- Get inline shopping results
- Get shopping results
Output
Links

What will be scraped

📌Note: Inline shopping results may not always be present.

Full Code

If you don't need an explanation, have a look at the full code example in the online IDE.

from selectolax.lexbor import LexborHTMLParser
import requests, json, re


def get_inline_shopping_results(parser: LexborHTMLParser, product_counter: int) -> list:
    data = []

    for position, product in enumerate(parser.root.css('.br-gOffCard'), start=product_counter):
        raw_title = product.css_first('.br-offTtl span')
        raw_shipping = product.css_first('.br-offDec')
        raw_rating = product.css_first('.tags > span')

        title = raw_title.attributes.get('title', '') if raw_title else product.css_first('.br-offTtl').text().strip()
        product_link = product.css_first('.br-offLink').attributes.get('href', '')
        brand = product.css_first('.br-offSecLbl').text().strip() if product.css_first('.br-offSecLbl') else None
        seller = product.css_first('.br-offSlr').text().strip()
        price = product.css_first('.br-price').text().strip()
        # https://regex101.com/r/lap8lr/1
        extracted_price = float(re.search(r'[\d|,|.]+', price).group().replace(',', ''))
        old_price = product.css_first('.br-standardPriceDemoted').text().strip() if product.css_first('.br-standardPriceDemoted') else None
        shipping_text = raw_shipping.text().strip() if raw_shipping else ''
        shipping = shipping_text if 'shipping' in shipping_text else None
        rating_text = raw_rating.attributes.get('aria-label', '') if raw_rating else None
        # https://regex101.com/r/D2YTRI/1
        rating = float(re.search(r'[\d.\d|\d]+', rating_text).group(0)) if rating_text else None
        reviews_text = raw_rating.text().strip() if raw_rating else None
        reviews = reviews_text.replace('(', '').replace(')', '') if reviews_text else None
        thumbnail = product.css_first('.cico img').attributes.get('src', None) if product.css_first('.cico img') else None

        data.append({
            'position': position,
            'title': title,
            'product_link': product_link,
            'brand': brand,
            'seller': seller,
            'price': price,
            'extracted_price': extracted_price,
            'old_price': old_price,
            'shipping': shipping,
            'rating': rating,
            'reviews': reviews,
            'thumbnail': thumbnail
        })

    return data


def get_shopping_results(parser: LexborHTMLParser, product_counter: int) -> list:
    data = []

    for position, product in enumerate(parser.root.css('.br-fullCard'), start=product_counter):
        raw_title = product.css_first('.br-title span')

        title = raw_title.attributes.get('title', '') if raw_title else product.css_first('.br-title').text().strip()
        product_link = f"https://www.bing.com{product.css_first('.br-titlelink').attributes.get('href', '')}"
        seller = product.css_first('.br-seller').text().strip()
        price = product.css_first('.pd-price').text().strip()
        # https://regex101.com/r/lap8lr/1
        extracted_price = float(re.search(r'[\d|,|.]+', price).group().replace(',', ''))
        sale = True if product.css_first('.br-saletag') else False
        old_price = product.css_first('.br-pdOfferPrice').text().strip() if product.css_first('.br-pdOfferPrice') else None
        shipping = product.css_first('.br-decoSlot').text().strip() if product.css_first('.br-decoSlot') else None
        thumbnail = product.css_first('.br-pdMainImg img').attributes.get('src', None) if product.css_first('.br-pdMainImg img') else None

        data.append({
            'position': position,
            'title': title,
            'product_link': product_link,
            'seller': seller,
            'price': price,
            'extracted_price': extracted_price,
            'sale': sale,
            'old_price': old_price,
            'shipping': shipping,
            'thumbnail': thumbnail
        })

    return data


def main():
    # https://docs.python-requests.org/en/master/user/quickstart/#custom-headers
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36'
    }

    url = 'https://www.bing.com/shop?q=ps4 controller'

    inline_shopping_results_counter: int = 1
    shopping_results_counter: int = 1

    bing_shopping_results = {
        'inline_shopping_results': [],
        'shopping_results': []
    }

    while True:
        html = requests.get(url, headers=headers)
        parser = LexborHTMLParser(html.text)

        inline_shopping_page_results = get_inline_shopping_results(parser, inline_shopping_results_counter)
        bing_shopping_results['inline_shopping_results'].extend(inline_shopping_page_results)

        shopping_page_results = get_shopping_results(parser, shopping_results_counter)
        bing_shopping_results['shopping_results'].extend(shopping_page_results)

        next_page_button = parser.css_first('.sb_pagN_bp')

        if next_page_button:
            url = f"https://www.bing.com{next_page_button.attributes.get('href', '')}"
            inline_shopping_results_counter += len(inline_shopping_page_results)
            shopping_results_counter += len(shopping_page_results)
        else:
            break

    print(json.dumps(bing_shopping_results, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()

Preparation

Install libraries

pip install requests selectolax

Reduce the chance of being blocked

Make sure you're using request headers user-agent to act as a "real" user visit. Because default requests user-agent is python-requests and websites understand that it's most likely a script that sends a request. Check what's your user-agent.

There's a how to reduce the chance of being blocked while web scraping blog post that can get you familiar with basic and more advanced approaches.

Code Explanation

Import libraries:

from selectolax.lexbor import LexborHTMLParser
import requests, json, re

Library	Purpose
`LexborHTMLParser`	a fast HTML5 parser with CSS selectors using Lexbor engine.
`requests`	to make a request to the website.
`json`	to convert extracted data to a JSON object.
`re`	to extract parts of the data via regular expression.

The next part of the code is divided into functions. Each function is described in the corresponding heading below.

Top-level code environment

The request headers are generated:

# https://docs.python-requests.org/en/master/user/quickstart/#custom-headers
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36'
}

The url variable contains a link to the Bing Shopping page:

url = 'https://www.bing.com/shop?q=ps4 controller'

Variables are created to determine the position of the current product, taking into account pagination. For example, if on the first page the position of the last product is 36, then on the second page the position of the first product will be 37 (not 1):

inline_shopping_results_counter: int = 1
shopping_results_counter: int = 1

The bing_shopping_results dictionary is formed, where data on the corresponding keys will subsequently be added:

bing_shopping_results = {
    'inline_shopping_results': [],
    'shopping_results': []
}

Requests are sent in a loop for each page:

while True:
    # pagination

Make a request, pass url and headers. The data extraction itself is done with selectolax because it has Lexbor parser which is incredibly fast, like 186% faster compared to bs4 with lxml backend when parsing data with 3000 iterations 5 times:

html = requests.get(url, headers=headers)
parser = LexborHTMLParser(html.text)

By calling the appropriate functions, all the necessary data is retrieved and written to the bing_shopping_results dictionary. These functions are detailed in the sections below.

inline_shopping_page_results = get_inline_shopping_results(parser, inline_shopping_results_counter)
bing_shopping_results['inline_shopping_results'].extend(inline_shopping_page_results)

shopping_page_results = get_shopping_results(parser, shopping_results_counter)
bing_shopping_results['shopping_results'].extend(shopping_page_results)

At the end of the iteration, it checks whether the "Next" button is present. If such a button is present, then the url and counters are updated. Otherwise, the loop is stopped.

next_page_button = parser.css_first('.sb_pagN_bp')

if next_page_button:
    url = f"https://www.bing.com{next_page_button.attributes.get('href', '')}"
    inline_shopping_results_counter += len(inline_shopping_page_results)
    shopping_results_counter += len(shopping_page_results)
else:
    break

After receiving the data from all pages, they are output in JSON format:

print(json.dumps(bing_shopping_results, indent=2, ensure_ascii=False))

This code uses boilerplate __name__ == "__main__" construct that protects users from accidentally invoking the script when they didn't intend to. This indicates that the code is a runnable script:

def main():
    # https://docs.python-requests.org/en/master/user/quickstart/#custom-headers
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36'
    }

    url = 'https://www.bing.com/shop?q=ps4 controller'

    inline_shopping_results_counter: int = 1
    shopping_results_counter: int = 1

    bing_shopping_results = {
        'inline_shopping_results': [],
        'shopping_results': []
    }

    while True:
        html = requests.get(url, headers=headers)
        parser = LexborHTMLParser(html.text)

        inline_shopping_page_results = get_inline_shopping_results(parser, inline_shopping_results_counter)
        bing_shopping_results['inline_shopping_results'].extend(inline_shopping_page_results)

        shopping_page_results = get_shopping_results(parser, shopping_results_counter)
        bing_shopping_results['shopping_results'].extend(shopping_page_results)

        next_page_button = parser.css_first('.sb_pagN_bp')

        if next_page_button:
            url = f"https://www.bing.com{next_page_button.attributes.get('href', '')}"
            inline_shopping_results_counter += len(inline_shopping_page_results)
            shopping_results_counter += len(shopping_page_results)
        else:
            break

    print(json.dumps(bing_shopping_results, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()

This check will only be performed if the user has run this file. If the user imports this file into another, then the check will not work.

You can watch the video Python Tutorial: if name == 'main' for more details.

Get inline shopping results

The function takes a parser and a product_counter. Returns a list with the retrieved data.

The data list is declared to which the extracted data will be added:

data = []

To retrieve all products from a page, you need to use the css() method and pass the .br-gOffCard selector there.

After that, you need to iterate over the resulting list of products using the for loop with the built-in enumerate() function. This function adds a counter to the iterable and returns it. The counter is needed to assign a position to each product:

for position, product in enumerate(parser.root.css('.br-gOffCard'), start=product_counter):
    # data extraction

Retrieving the required data is done as follows: we find the required selector for current product using the css_first() method, after this we extract the text value or attribute value:

product_link = product.css_first('.br-offLink').attributes.get('href', '')
seller = product.css_first('.br-offSlr').text().strip()
price = product.css_first('.br-price').text().strip()

There are data that may not be present in some products. In this case, checks are added:

raw_title = product.css_first('.br-offTtl span')
raw_shipping = product.css_first('.br-offDec')
raw_rating = product.css_first('.tags > span')

title = raw_title.attributes.get('title', '') if raw_title else product.css_first('.br-offTtl').text().strip()
brand = product.css_first('.br-offSecLbl').text().strip() if product.css_first('.br-offSecLbl') else None
old_price = product.css_first('.br-standardPriceDemoted').text().strip() if product.css_first('.br-standardPriceDemoted') else None
shipping_text = raw_shipping.text().strip() if raw_shipping else ''
shipping = shipping_text if 'shipping' in shipping_text else None
rating_text = raw_rating.attributes.get('aria-label', '') if raw_rating else None
reviews_text = raw_rating.text().strip() if raw_rating else None
reviews = reviews_text.replace('(', '').replace(')', '') if reviews_text else None
thumbnail = product.css_first('.cico img').attributes.get('src', None) if product.css_first('.cico img') else None

If you need to get a specific value, then you can use regular expression to extract them:

# https://regex101.com/r/lap8lr/1
extracted_price = float(re.search(r'[\d|,|.]+', price).group().replace(',', ''))
# https://regex101.com/r/D2YTRI/1
rating = float(re.search(r'[\d.\d|\d]+', rating_text).group(0)) if rating_text else None

At the end of each iteration, product data is added to the data list:

data.append({
    'position': position,
    'title': title,
    'product_link': product_link,
    'brand': brand,
    'seller': seller,
    'price': price,
    'extracted_price': extracted_price,
    'old_price': old_price,
    'shipping': shipping,
    'rating': rating,
    'reviews': reviews,
    'thumbnail': thumbnail
})

After all the operations are done, return the data list:

return data

The function looks like this:

def get_inline_shopping_results(parser: LexborHTMLParser, product_counter: int) -> list:
    data = []

    for position, product in enumerate(parser.root.css('.br-gOffCard'), start=product_counter):
        raw_title = product.css_first('.br-offTtl span')
        raw_shipping = product.css_first('.br-offDec')
        raw_rating = product.css_first('.tags > span')

        title = raw_title.attributes.get('title', '') if raw_title else product.css_first('.br-offTtl').text().strip()
        product_link = product.css_first('.br-offLink').attributes.get('href', '')
        brand = product.css_first('.br-offSecLbl').text().strip() if product.css_first('.br-offSecLbl') else None
        seller = product.css_first('.br-offSlr').text().strip()
        price = product.css_first('.br-price').text().strip()
        # https://regex101.com/r/lap8lr/1
        extracted_price = float(re.search(r'[\d|,|.]+', price).group().replace(',', ''))
        old_price = product.css_first('.br-standardPriceDemoted').text().strip() if product.css_first('.br-standardPriceDemoted') else None
        shipping_text = raw_shipping.text().strip() if raw_shipping else ''
        shipping = shipping_text if 'shipping' in shipping_text else None
        rating_text = raw_rating.attributes.get('aria-label', '') if raw_rating else None
        # https://regex101.com/r/D2YTRI/1
        rating = float(re.search(r'[\d.\d|\d]+', rating_text).group(0)) if rating_text else None
        reviews_text = raw_rating.text().strip() if raw_rating else None
        reviews = reviews_text.replace('(', '').replace(')', '') if reviews_text else None
        thumbnail = product.css_first('.cico img').attributes.get('src', None) if product.css_first('.cico img') else None

        data.append({
            'position': position,
            'title': title,
            'product_link': product_link,
            'brand': brand,
            'seller': seller,
            'price': price,
            'extracted_price': extracted_price,
            'old_price': old_price,
            'shipping': shipping,
            'rating': rating,
            'reviews': reviews,
            'thumbnail': thumbnail
        })

    return data

Get shopping results

The function takes a parser and a product_counter. Returns a list with the retrieved data.

The data list is declared to which the extracted data will be added:

data = []

In the previous function, the data extraction process was described in detail. In this function, the process is very similar except for different data and correspondingly different selectors:

for position, product in enumerate(parser.root.css('.br-fullCard'), start=product_counter):
    raw_title = product.css_first('.br-title span')

    title = raw_title.attributes.get('title', '') if raw_title else product.css_first('.br-title').text().strip()
    product_link = f"https://www.bing.com{product.css_first('.br-titlelink').attributes.get('href', '')}"
    seller = product.css_first('.br-seller').text().strip()
    price = product.css_first('.pd-price').text().strip()
    # https://regex101.com/r/lap8lr/1
    extracted_price = float(re.search(r'[\d|,|.]+', price).group().replace(',', ''))
    sale = True if product.css_first('.br-saletag') else False
    old_price = product.css_first('.br-pdOfferPrice').text().strip() if product.css_first('.br-pdOfferPrice') else None
    shipping = product.css_first('.br-decoSlot').text().strip() if product.css_first('.br-decoSlot') else None
    thumbnail = product.css_first('.br-pdMainImg img').attributes.get('src', None) if product.css_first('.br-pdMainImg img') else None

At the end of each iteration, product data is added to the data list:

data.append({
    'position': position,
    'title': title,
    'product_link': product_link,
    'seller': seller,
    'price': price,
    'extracted_price': extracted_price,
    'sale': sale,
    'old_price': old_price,
    'shipping': shipping,
    'thumbnail': thumbnail
})

After all the operations are done, return the data list:

return data

The function looks like this:

def get_shopping_results(parser: LexborHTMLParser, product_counter: int) -> list:
    data = []

    for position, product in enumerate(parser.root.css('.br-fullCard'), start=product_counter):
        raw_title = product.css_first('.br-title span')

        title = raw_title.attributes.get('title', '') if raw_title else product.css_first('.br-title').text().strip()
        product_link = f"https://www.bing.com{product.css_first('.br-titlelink').attributes.get('href', '')}"
        seller = product.css_first('.br-seller').text().strip()
        price = product.css_first('.pd-price').text().strip()
        # https://regex101.com/r/lap8lr/1
        extracted_price = float(re.search(r'[\d|,|.]+', price).group().replace(',', ''))
        sale = True if product.css_first('.br-saletag') else False
        old_price = product.css_first('.br-pdOfferPrice').text().strip() if product.css_first('.br-pdOfferPrice') else None
        shipping = product.css_first('.br-decoSlot').text().strip() if product.css_first('.br-decoSlot') else None
        thumbnail = product.css_first('.br-pdMainImg img').attributes.get('src', None) if product.css_first('.br-pdMainImg img') else None

        data.append({
            'position': position,
            'title': title,
            'product_link': product_link,
            'seller': seller,
            'price': price,
            'extracted_price': extracted_price,
            'sale': sale,
            'old_price': old_price,
            'shipping': shipping,
            'thumbnail': thumbnail
        })

    return data

Output

{
  "inline_shopping_results": [
    {
      "position": 1,
      "title": "Sony Playstation 4 Dual Shock 4 Controller",
      "product_link": "https://www.bing.com/aclick?ld=e8y8UnkNEA5yLnmFwFXVJsIDVUCUxGDjCRfBB7hezInXrQF_5sOcYbnlwQ7FroV_Zn5FefQcj7dQqTSlvA3lj2I21y0MviXMAVnyW-3FkoUoi16_LPCXsLblfhhQ2D_DBXPc7yCF56HaNeXUDrxLymGBGLDEWL241igNH5h1ZNrEBK3Hy1&u=aHR0cHMlM2ElMmYlMmZ3d3cuYW1hem9uLmNvbSUyZlNvbnktUGxheVN0YXRpb24tRHVhbHNob2NrLVdpcmVsZXNzLUNvbnRyb2xsZXItQmxhY2slMmZkcCUyZkIwMEQ4Mlo0WU8lMmZyZWYlM2Rhc2NfZGZfQjAwRDgyWjRZTyUzZnRhZyUzZGJpbmdzaG9wcGluZ2EtMjAlMjZsaW5rQ29kZSUzZGRmMCUyNmh2YWRpZCUzZDc5OTg5NTk3MTQ0NTE1JTI2aHZuZXR3JTNkbyUyNmh2cW10JTNkZSUyNmh2Ym10JTNkYmUlMjZodmRldiUzZGMlMjZodmxvY2ludCUzZCUyNmh2bG9jcGh5JTNkJTI2aHZ0YXJnaWQlM2RwbGEtNDU4MzU4OTExODI4NTM1MCUyNnBzYyUzZDE&rlid=044e0ea8d1d91e49a33a6a95b021d2b3",
      "brand": "580+ viewed",
      "seller": "Amazon.com",
      "price": "$56.00",
      "extracted_price": 56.0,
      "old_price": null,
      "shipping": null,
      "rating": 4.5,
      "reviews": null,
      "thumbnail": "https://th.bing.com/th?id=OP.javWvhvskVjF3A474C474&w=140&h=140&pid=21.1"
    },
    {
      "position": 2,
      "title": "Wireless Controller For PS4/Slim/Pro, Berry Blue",
      "product_link": "https://www.bing.com/aclick?ld=e8K5Qjv1rx7b91q1TGu2kIHDVUCUzljX6PIJ1_Ag71Iy5QZiGWnbe49xj4OnlDYYwXIZtyHeZHN66fGgfhttXqkZjtRPPAG3TZF7rGH0z6XerddgQBQx_dfYoJlYMoPPnC1ujLoNsFoUL6VTMyu4Ln14u3EoPqCJV8wSvMZ7FHfLWcEayH&u=aHR0cHMlM2ElMmYlMmZjbGlja3NlcnZlLmRhcnRzZWFyY2gubmV0JTJmbGluayUyZmNsaWNrJTNmJTI2JTI2ZHNfZV9hZGlkJTNkNzUzMTY2OTg4MTk5MTglMjZkc19lX3RhcmdldF9pZCUzZHBsYS00NTc4OTE2MjM5MDc0Njc5JTI2ZHNfZV9wcm9kdWN0X2dyb3VwX2lkJTNkNDU3ODkxNjIzOTA3NDY3OSUyNmRzX2VfcHJvZHVjdF9pZCUzZDEzOTY2MDA3OV8xMDAwMTEzNTk4OCUyNmRzX2VfcHJvZHVjdF9jb3VudHJ5JTNkVVMlMjZkc19lX3Byb2R1Y3RfbGFuZ3VhZ2UlM2RFTiUyNmRzX2VfcHJvZHVjdF9jaGFubmVsJTNkT25saW5lJTI2ZHNfdXJsX3YlM2QyJTI2ZHNfZGVzdF91cmwlM2RodHRwcyUzYSUyZiUyZnd3dy53YWxtYXJ0LmNvbSUyZmlwJTJmV2lyZWxlc3MtQ29udHJvbGxlci1mb3ItUFM0LVNsaW0tUHJvLUJlcnJ5LUJsdWUlMmYxMzk2NjAwNzklM2Z3bWxzcGFydG5lciUzZHdscGElMjZzZWxlY3RlZFNlbGxlcklkJTNkMTAxMTE2NjI2JTI2YWRpZCUzZDIyMjIyMjIyMjIyNTc2ODA1MzgwJTI2d21sc3BhcnRuZXIlM2R3bXRsYWJzJTI2d2wwJTNkZSUyNndsMSUzZG8lMjZ3bDIlM2RjJTI2d2wzJTNkNzUzMTY2OTg4MTk5MTglMjZ3bDQlM2RwbGEtNDU3ODkxNjIzOTA3NDY3OSUyNndsNSUzZCUyNndsNiUzZCUyNndsNyUzZCUyNndsMTAlM2RXYWxtYXJ0JTI2d2wxMSUzZE9ubGluZSUyNndsMTIlM2QxMzk2NjAwNzlfMTAwMDExMzU5ODglMjZ3bDE0JTNkcHM0JTI1MjBjb250cm9sbGVyJTI2dmVoJTNkc2VtJTI2Z2NsaWQlM2Q1NjkxZjc3M2I3ODQxMTQ2ZTVhYjkzNGVkNjdjNTQ0OSUyNmdjbHNyYyUzZDNwLmRzJTI2bXNjbGtpZCUzZDU2OTFmNzczYjc4NDExNDZlNWFiOTM0ZWQ2N2M1NDQ5&rlid=5691f773b7841146e5ab934ed67c5449",
      "brand": "Brand: SPBPQY",
      "seller": "Walmart",
      "price": "$19.99",
      "extracted_price": 19.99,
      "old_price": null,
      "shipping": "Free shipping",
      "rating": null,
      "reviews": null,
      "thumbnail": "https://th.bing.com/th?id=OP.AjeAqUx49FCBaQ474C474&w=140&h=140&pid=21.1"
    },
    {
      "position": 3,
      "title": "YCCTEAM Wireless Ps4 Controller ,Wireless Game Controller Compatible With Playstation 4/Slim/Pro, Built-In 1000Mah Battery With Turbo/Dual Vibration",
      "product_link": "https://www.bing.com/aclick?ld=e8XOuCV1B-F1vb3cgxnxYFWjVUCUxeNZ_G0jSncEe8IzWF8UzLDZihTUvXFtS_U7DFDfJNV_L4aBCyLLwDxIW3jnYGUaW0Q0C4MhhYByxkPdaXmHEexgmFgmfKA1EITFmoVR4_Yla_FXAV-Nmx6yJiQmPLtRwNLSr6eWXTZE07O-ZjlAPX&u=aHR0cHMlM2ElMmYlMmZ3d3cuYW1hem9uLmNvbSUyZkNvbnRyb2xsZXItQ2hhcmdpbmctWUNDVEVBTS1JbmRpY2F0b3ItSm95c3RpY2tzJTJmZHAlMmZCMDdXN0gyMVZWJTJmcmVmJTNkYXNjX2RmX0IwN1c3SDIxVlYlM2Z0YWclM2RiaW5nc2hvcHBpbmdhLTIwJTI2bGlua0NvZGUlM2RkZjAlMjZodmFkaWQlM2Q3OTg1MjEyNDE1MTUzNSUyNmh2bmV0dyUzZG8lMjZodnFtdCUzZGUlMjZodmJtdCUzZGJlJTI2aHZkZXYlM2RjJTI2aHZsb2NpbnQlM2QlMjZodmxvY3BoeSUzZCUyNmh2dGFyZ2lkJTNkcGxhLTQ1ODM0NTE2NjkzOTg1NzklMjZwc2MlM2Qx&rlid=15ad8be3c1af10f584ffefdea057bbd8",
      "brand": null,
      "seller": "Amazon.com",
      "price": "$19.99",
      "extracted_price": 19.99,
      "old_price": null,
      "shipping": null,
      "rating": null,
      "reviews": null,
      "thumbnail": "https://th.bing.com/th?id=OP.NsyWoG%2bGBv4G3A474C474&w=140&h=140&pid=21.1"
    },
    ... other inline shopping results
  ],
  "shopping_results": [
    {
      "position": 1,
      "title": "Sony PS4 Dualshock 4 Wireless Controller - Midnight Blue",
      "product_link": "https://www.bing.com/shop/productpage?q=ps4+controller&filters=scenario%3a%2217%22+gType%3a%223%22+gId%3a%22IKQSlWYtjMJCH45YKbCW1INZT5%22+gIdHash%3a%221034604334%22+gGlobalOfferIds%3a%2297596786905%22+AucContextGuid%3a%220%22+GroupEntityId%3a%22IKQSlWYtjMJCH45YKbCW1INZT5%22+NonSponsoredOffer%3a%22True%22&productpage=true&FORM=SHPPDP&browse=true",
      "seller": "Walmart",
      "price": "$64.00",
      "extracted_price": 64.0,
      "sale": false,
      "old_price": null,
      "shipping": "Free shipping",
      "thumbnail": "https://th.bing.com/th?id=OP.5KY3rYxghNq1ng474C474&w=272&h=272&o=5&pid=21.1"
    },
    {
      "position": 2,
      "title": "Sony Dual Shock 4 Bluetooth Controller For PS4 - Black",
      "product_link": "https://www.bing.com/shop/productpage?q=ps4+controller&filters=scenario%3a%2217%22+gType%3a%223%22+gId%3a%22IjzPcGXIjh5kEfHzF4DdKkdm7O%22+gIdHash%3a%22503285226%22+gGlobalOfferIds%3a%2213378540660%22+AucContextGuid%3a%220%22+GroupEntityId%3a%22IjzPcGXIjh5kEfHzF4DdKkdm7O%22+NonSponsoredOffer%3a%22True%22&productpage=true&FORM=SHPPDP&browse=true",
      "seller": "Tech Import World",
      "price": "$51.92",
      "extracted_price": 51.92,
      "sale": false,
      "old_price": null,
      "shipping": null,
      "thumbnail": "https://th.bing.com/th?id=OP.2zrQISY7yRggYA474C474&w=272&h=272&o=5&pid=21.1"
    },
    {
      "position": 3,
      "title": "Sony PS4 Dualshock 4 Wireless Controller - Green Camouflage",
      "product_link": "https://www.bing.com/shop/productpage?q=ps4+controller&filters=scenario%3a%2217%22+gType%3a%223%22+gId%3a%22IsUwpe7StYMy9zWBEooypnccb6%22+gIdHash%3a%221451490157%22+gGlobalOfferIds%3a%2285554586355%22+AucContextGuid%3a%220%22+GroupEntityId%3a%22IsUwpe7StYMy9zWBEooypnccb6%22+NonSponsoredOffer%3a%22True%22&productpage=true&FORM=SHPPDP&browse=true",
      "seller": "Walmart",
      "price": "$64.00",
      "extracted_price": 64.0,
      "sale": false,
      "old_price": null,
      "shipping": "Free shipping",
      "thumbnail": "https://th.bing.com/th?id=OP.bmIHvaU977kYvw474C474&w=272&h=272&o=5&pid=21.1"
    },
    ... other shopping results
  ]
}

Links

Join us on Twitter | YouTube

Add a Feature Request💫 or a Bug🐞

Scrape Google Flights with Python

Artur Chukhrai — Fri, 21 Apr 2023 10:40:23 +0000

What will be scraped
Full Code
Preparation
Code Explanation
- Working with a Playwright
- Get page
- Scrape Google Flights Listings
Output
Links

What will be scraped

Full Code

from playwright.sync_api import sync_playwright
from selectolax.lexbor import LexborHTMLParser
import json, time


def get_page(playwright, from_place, to_place, departure_date, return_date):
    page = playwright.chromium.launch(headless=False).new_page()
    page.goto('https://www.google.com/travel/flights?hl=en-US&curr=USD')

    # type "From"
    from_place_field = page.query_selector_all('.e5F5td')[0]
    from_place_field.click()
    time.sleep(1)
    from_place_field.type(from_place)
    time.sleep(1)
    page.keyboard.press('Enter')

    # type "To"
    to_place_field = page.query_selector_all('.e5F5td')[1]
    to_place_field.click()
    time.sleep(1)
    to_place_field.type(to_place)
    time.sleep(1)
    page.keyboard.press('Enter')

    # type "Departure date"
    departure_date_field = page.query_selector_all('[aria-label="Departure"]')[0]
    departure_date_field.click()
    time.sleep(1)
    departure_date_field.type(departure_date)
    time.sleep(1)
    page.query_selector('.WXaAwc .VfPpkd-LgbsSe').click()
    time.sleep(1)

    # type "Return date"
    return_date_field = page.query_selector_all('[aria-label="Return"]')[0]
    return_date_field.click()
    time.sleep(1)
    return_date_field.type(return_date)
    time.sleep(1)
    page.query_selector('.WXaAwc .VfPpkd-LgbsSe').click()
    time.sleep(1)

    # press "Explore"
    page.query_selector('.MXvFbd .VfPpkd-LgbsSe').click()
    time.sleep(2)

    # press "More flights"
    page.query_selector('.zISZ5c button').click()
    time.sleep(2)

    parser = LexborHTMLParser(page.content())
    page.close()

    return parser


def scrape_google_flights(parser):
    data = {}

    categories = parser.root.css('.zBTtmb')
    category_results = parser.root.css('.Rk10dc')

    for category, category_result in zip(categories, category_results):
        category_data = []

        for result in category_result.css('.yR1fYc'):
            date = result.css('[jscontroller="cNtv4b"] span')
            departure_date = date[0].text()
            arrival_date = date[1].text()
            company = result.css_first('.Ir0Voe .sSHqwe').text()
            duration = result.css_first('.AdWm1c.gvkrdb').text()
            stops = result.css_first('.EfT7Ae .ogfYpf').text()
            emissions = result.css_first('.V1iAHe .AdWm1c').text()
            emission_comparison = result.css_first('.N6PNV').text()
            price = result.css_first('.U3gSDe .FpEdX span').text()
            price_type = result.css_first('.U3gSDe .N872Rd').text() if result.css_first('.U3gSDe .N872Rd') else None

            flight_data = {
                'departure_date': departure_date,
                'arrival_date': arrival_date,
                'company': company,
                'duration': duration,
                'stops': stops,
                'emissions': emissions,
                'emission_comparison': emission_comparison,
                'price': price,
                'price_type': price_type
            }

            airports = result.css_first('.Ak5kof .sSHqwe')
            service = result.css_first('.hRBhge')

            if service:
                flight_data['service'] = service.text()
            else:
                flight_data['departure_airport'] = airports.css_first('span:nth-child(1) .eoY5cb').text()
                flight_data['arrival_airport'] = airports.css_first('span:nth-child(2) .eoY5cb').text()

            category_data.append(flight_data)

        data[category.text().lower().replace(' ', '_')] = category_data

    return data


def run(playwright):
    from_place = 'Seattle'
    to_place = 'Las Vegas'
    departure_date = '14-6-2024'
    return_date = '20-6-2024'

    parser = get_page(playwright, from_place, to_place, departure_date, return_date)
    google_flights_results = scrape_google_flights(parser)

    print(json.dumps(google_flights_results, indent=2, ensure_ascii=False))


with sync_playwright() as playwright:
    run(playwright)

Preparation

Install libraries:

pip install playwright selectolax

Install the required browser:

playwright install chromium

Code Explanation

Import libraries:

from playwright.sync_api import sync_playwright
from selectolax.lexbor import LexborHTMLParser
import json, time

Library	Purpose
`sync_playwright`	for synchronous API. `playwright` have asynchronous API as well using `asyncio` module.
`LexborHTMLParser`	a fast HTML5 parser with CSS selectors using Lexbor engine.
`json`	to convert extracted data to a JSON object.
`time`	to work with time in Python.

The next part of the code is divided into functions. Each function is described in the corresponding heading below.

Working with a Playwright

Declare a function:

def run(playwright):
    # further code ...

Passing user data to form a request:

from_place = 'Seattle'
to_place = 'Las Vegas'
departure_date = '14-6-2024'
return_date = '20-6-2024'

The playwright object and the previous data are then passed to the get_page(playwright, from_place, to_place, departure_date, return_date) function. The parser returned by this function is passed to the scrape_google_flights(parser) function to extract all the data. The explanation of these functions will be in the corresponding headings below.

parser = get_page(playwright, from_place, to_place, departure_date, return_date)
google_flights_results = scrape_google_flights(parser)

After the all data is retrieved, it is output in JSON format:

print(json.dumps(google_flights_results, indent=2, ensure_ascii=False))

Run your code using context manager:

with sync_playwright() as playwright:
    run(playwright)

The whole part of the code looks like this:

def run(playwright):
    from_place = 'Seattle'
    to_place = 'Las Vegas'
    departure_date = '14-6-2024'
    return_date = '20-6-2024'

    parser = get_page(playwright, from_place, to_place, departure_date, return_date)
    google_flights_results = scrape_google_flights(parser)

    print(json.dumps(google_flights_results, indent=2, ensure_ascii=False))


with sync_playwright() as playwright:
    run(playwright)

Get page

The function takes a playwright object and parameters from_place, to_place, departure_date, return_date. Returns a parser.

Initialize playwright, connect to chromium, launch() a browser new_page() and goto() to the URL:

page = playwright.chromium.launch(headless=False).new_page()
page.goto('https://www.google.com/travel/flights?hl=en-US&curr=USD')

Parameters	Explanation
`playwright.chromium`	is a connection to the Chromium browser instance.
`launch()`	will launch the browser, and headless argument will run it in headless mode. Default is True.
`new_page()`	creates a new page in a new browser context.
`page.goto()`	will make a request to provided website.

The next section of code is easier to show on the GIF:

In short, with the help of browser automation, we pass user parameters and generate search results:

# type "From"
from_place_field = page.query_selector_all('.e5F5td')[0]
from_place_field.click()
time.sleep(1)
from_place_field.type(from_place)
time.sleep(1)
page.keyboard.press('Enter')

# type "To"
to_place_field = page.query_selector_all('.e5F5td')[1]
to_place_field.click()
time.sleep(1)
to_place_field.type(to_place)
time.sleep(1)
page.keyboard.press('Enter')

# type "Departure date"
departure_date_field = page.query_selector_all('[aria-label="Departure"]')[0]
departure_date_field.click()
time.sleep(1)
departure_date_field.type(departure_date)
time.sleep(1)
page.query_selector('.WXaAwc .VfPpkd-LgbsSe').click()
time.sleep(1)

# type "Return date"
return_date_field = page.query_selector_all('[aria-label="Return"]')[0]
return_date_field.click()
time.sleep(1)
return_date_field.type(return_date)
time.sleep(1)
page.query_selector('.WXaAwc .VfPpkd-LgbsSe').click()
time.sleep(1)

# press "Explore"
page.query_selector('.MXvFbd .VfPpkd-LgbsSe').click()
time.sleep(2)

# press "More flights"
page.query_selector('.zISZ5c button').click()
time.sleep(2)

After all the data has been loaded, you need to process HTML using from selectolax because it has Lexbor parser which is incredibly fast, like 186% faster compared to bs4 with lxml backend when parsing data with 3000 iterations 5 times. Please note that selectolax does not currently support XPath:

parser = LexborHTMLParser(page.content())

After all the operations are done, close the browser and the parser is returned:

page.close()

return parser

The function looks like this:

def get_page(playwright, from_place, to_place, departure_date, return_date):
    page = playwright.chromium.launch(headless=False).new_page()
    page.goto('https://www.google.com/travel/flights?hl=en-US&curr=USD')

    # type "From"
    from_place_field = page.query_selector_all('.e5F5td')[0]
    from_place_field.click()
    time.sleep(1)
    from_place_field.type(from_place)
    time.sleep(1)
    page.keyboard.press('Enter')

    # type "To"
    to_place_field = page.query_selector_all('.e5F5td')[1]
    to_place_field.click()
    time.sleep(1)
    to_place_field.type(to_place)
    time.sleep(1)
    page.keyboard.press('Enter')

    # type "Departure date"
    departure_date_field = page.query_selector_all('[aria-label="Departure"]')[0]
    departure_date_field.click()
    time.sleep(1)
    departure_date_field.type(departure_date)
    time.sleep(1)
    page.query_selector('.WXaAwc .VfPpkd-LgbsSe').click()
    time.sleep(1)

    # type "Return date"
    return_date_field = page.query_selector_all('[aria-label="Return"]')[0]
    return_date_field.click()
    time.sleep(1)
    return_date_field.type(return_date)
    time.sleep(1)
    page.query_selector('.WXaAwc .VfPpkd-LgbsSe').click()
    time.sleep(1)

    # press "Explore"
    page.query_selector('.MXvFbd .VfPpkd-LgbsSe').click()
    time.sleep(2)

    # press "More flights"
    page.query_selector('.zISZ5c button').click()
    time.sleep(2)

    parser = LexborHTMLParser(page.content())
    page.close()

    return parser

Scrape Google Flights Listings

The function takes a Lexbor parser and returns the extracted data.

The data dictionary is declared to which the extracted data will be added:

data = {}

In order to extract all the data, you need to extract the category names and the elements of these categories separately. You need to use the css() method and pass the appropriate selectors there:

categories = parser.root.css('.zBTtmb')
category_results = parser.root.css('.Rk10dc')

You then need to iterate over the resulting item lists using the zip() method to successfully categorize the flights. For each category, a category_data dictionary is created, in which flights associated with a corresponding category will be added:

for category, category_result in zip(categories, category_results):
    category_data = []

We iterate each flight to extract all the necessary data:

for result in category_result.css('.yR1fYc'):
    # data extraction will be here

Most of the data is easily retrieved:

company = result.css_first('.Ir0Voe .sSHqwe').text()
duration = result.css_first('.AdWm1c.gvkrdb').text()
stops = result.css_first('.EfT7Ae .ogfYpf').text()
emissions = result.css_first('.V1iAHe .AdWm1c').text()
emission_comparison = result.css_first('.N6PNV').text()
price = result.css_first('.U3gSDe .FpEdX span').text()
price_type = result.css_first('.U3gSDe .N872Rd').text()

Code	Explanation
`css_first()`	to find the desired element.
`text()`	to retrieve the text content.

The selector responsible for the date contains 2 elements: departure_date and arrival_date.

date = result.css('[jscontroller="cNtv4b"] span')
departure_date = date[0].text()
arrival_date = date[1].text()

After extracting the main data, we form the flight_data dictionary:

flight_data = {
    'departure_date': departure_date,
    'arrival_date': arrival_date,
    'company': company,
    'duration': duration,
    'stops': stops,
    'emissions': emissions,
    'emission_comparison': emission_comparison,
    'price': price,
    'price_type': price_type
}

After adding the main data, you need to check the data, which may differ. These are the departure_airport and the arrival_airport or other service.

Depending on what data is present in this flight, we add them to the flight_data dictionary. Then add this dictionary to the category_data list:

airports = result.css_first('.Ak5kof .sSHqwe')
service = result.css_first('.hRBhge')

if service:
    flight_data['service'] = service.text()
else:
    flight_data['departure_airport'] = airports.css_first('span:nth-child(1) .eoY5cb').text()
    flight_data['arrival_airport'] = airports.css_first('span:nth-child(2) .eoY5cb').text()

category_data.append(flight_data)

When all flights have been retrieved for a specific category, you need to add them to the data dictionary by key. The key is the name of the category:

data[category.text().lower().replace(' ', '_')] = category_data

At the end of the function, the data dictionary is returned:

return data

The function looks like this:

def scrape_google_flights(parser):
    data = {}

    categories = parser.root.css('.zBTtmb')
    category_results = parser.root.css('.Rk10dc')

    for category, category_result in zip(categories, category_results):
        category_data = []

        for result in category_result.css('.yR1fYc'):
            date = result.css('[jscontroller="cNtv4b"] span')
            departure_date = date[0].text()
            arrival_date = date[1].text()
            company = result.css_first('.Ir0Voe .sSHqwe').text()
            duration = result.css_first('.AdWm1c.gvkrdb').text()
            stops = result.css_first('.EfT7Ae .ogfYpf').text()
            emissions = result.css_first('.V1iAHe .AdWm1c').text()
            emission_comparison = result.css_first('.N6PNV').text()
            price = result.css_first('.U3gSDe .FpEdX span').text()
            price_type = result.css_first('.U3gSDe .N872Rd').text() if result.css_first('.U3gSDe .N872Rd') else None

            flight_data = {
                'departure_date': departure_date,
                'arrival_date': arrival_date,
                'company': company,
                'duration': duration,
                'stops': stops,
                'emissions': emissions,
                'emission_comparison': emission_comparison,
                'price': price,
                'price_type': price_type
            }

            airports = result.css_first('.Ak5kof .sSHqwe')
            service = result.css_first('.hRBhge')

            if service:
                flight_data['service'] = service.text()
            else:
                flight_data['departure_airport'] = airports.css_first('span:nth-child(1) .eoY5cb').text()
                flight_data['arrival_airport'] = airports.css_first('span:nth-child(2) .eoY5cb').text()

            category_data.append(flight_data)

        data[category.text().lower().replace(' ', '_')] = category_data

    return data

Output

{
  "best_departing_flights": [
    {
      "departure_date": "10:03 PM",
      "arrival_date": "12:42 AM+1",
      "company": "Frontier",
      "duration": "2 hr 39 min",
      "stops": "Nonstop",
      "emissions": "99 kg CO2e",
      "emission_comparison": "-27% emissions",
      "price": "$242",
      "price_type": "round trip",
      "departure_airport": "Seattle–Tacoma International Airport",
      "arrival_airport": "Seattle–Tacoma International Airport"
    },
    {
      "departure_date": "8:05 AM",
      "arrival_date": "10:41 AM",
      "company": "Alaska",
      "duration": "2 hr 36 min",
      "stops": "Nonstop",
      "emissions": "130 kg CO2e",
      "emission_comparison": "Avg emissions",
      "price": "$412",
      "price_type": "round trip",
      "departure_airport": "Seattle–Tacoma International Airport",
      "arrival_airport": "Seattle–Tacoma International Airport"
    },
    {
      "departure_date": "9:10 AM",
      "arrival_date": "11:38 AM",
      "company": "Delta",
      "duration": "2 hr 28 min",
      "stops": "Nonstop",
      "emissions": "139 kg CO2e",
      "emission_comparison": "Avg emissions",
      "price": "$447",
      "price_type": "round trip",
      "departure_airport": "Seattle–Tacoma International Airport",
      "arrival_airport": "Seattle–Tacoma International Airport"
    }
  ],
  "other_departing_flights": [
    {
      "departure_date": "8:04 AM",
      "arrival_date": "6:35 AM+1",
      "company": "Frontier",
      "duration": "22 hr 31 min",
      "stops": "1 stop",
      "emissions": "186 kg CO2e",
      "emission_comparison": "+37% emissions",
      "price": "$329",
      "price_type": "round trip",
      "departure_airport": "Seattle–Tacoma International Airport",
      "arrival_airport": "Seattle–Tacoma International Airport"
    },
    {
      "departure_date": "3:34 PM",
      "arrival_date": "6:35 AM+1",
      "company": "Frontier",
      "duration": "15 hr 1 min",
      "stops": "1 stop",
      "emissions": "185 kg CO2e",
      "emission_comparison": "+36% emissions",
      "price": "$329",
      "price_type": "round trip",
      "departure_airport": "Seattle–Tacoma International Airport",
      "arrival_airport": "Seattle–Tacoma International Airport"
    },
    {
      "departure_date": "11:59 PM",
      "arrival_date": "9:43 AM+1",
      "company": "Separate tickets booked togetherThis trip includes tickets from multiple airlines. Missed connections may be protected by Gotogate. Learn moreSun Country Airlines",
      "duration": "9 hr 44 min",
      "stops": "1 stop",
      "emissions": "353 kg CO2e",
      "emission_comparison": "+160% emissions",
      "price": "$379",
      "price_type": "round trip",
      "departure_airport": "Seattle–Tacoma International Airport",
      "arrival_airport": "Seattle–Tacoma International Airport"
    },
    ... other results
  ]
}

📌Note: Other departing trains are not displayed due to the fact that they are not available for this route.

How to Extract Bing Images Data with SerpApi and Python

Artur Chukhrai — Wed, 12 Apr 2023 19:12:30 +0000

Intro
What will be scraped
Why using API?
Full Code
Preparation
Code Explanation
Output
Links

Intro

In this blog post, we'll go through the process of extracting Bing Images using the Bing Images API and the Python programming language. You can look at the complete code in the online IDE (Replit).

What will be scraped

Why using API?

There're a couple of reasons that may use API, ours in particular:

No need to create a parser from scratch and maintain it.
Bypass blocks from Google: solve CAPTCHA or solve IP blocks.
Pay for proxies, and CAPTCHA solvers.
Figure out the legal part of scraping data.

SerpApi handles everything on the backend with fast response times under ~1.7 seconds per request and without browser automation, which becomes much faster. Response times and status rates are shown under SerpApi Status page.

Full Code

This code retrieves all the data with pagination:

from serpapi import BingSearch
import json

params = {
    'api_key': '...',               # https://serpapi.com/manage-api-key
    'q': 'Coffee',                  # search query
    'engine': 'bing_images',        # search engine
    'cc': 'US',                     # country of the search
    'first': 1,                     # pagination
    'count': 50,                    # number of results per page
    # 'imagesize': 'wallpaper',     # filtering by size
    # 'color2': 'color',            # filtering by color
    # 'photo': 'photo',             # filtering by image type
    # 'aspect': 'wide',             # filtering by layout
    # 'face': 'portrait',           # filtering by people
    # 'age': 'lt525600',            # filtering by date
    # 'license': 'Type-Any'         # filtering by license
}

search = BingSearch(params)         # data extraction on the SerpApi backend
results = search.get_dict()         # JSON -> Python dict

bing_images_results = {
    'images_results': [],
    'suggested_searches': [],
    'refined_searches': results.get('refined_searches', []),
    'related_searches': results.get('related_searches', []),
    'shopping_results': results.get('shopping_results', [])
}

page_count = 0
page_limit = 10

while 'error' not in results and page_count < page_limit:
    bing_images_results['images_results'].extend(results.get('images_results', []))
    bing_images_results['suggested_searches'].extend(results.get('suggested_searches', []))

    params['first'] += params['count']
    page_count += 1
    results = search.get_dict()

print(json.dumps(bing_images_results, indent=2, ensure_ascii=False))

Preparation

Install library:

pip install google-search-results

google-search-results is a SerpApi API package.

Code Explanation

Import libraries:

from serpapi import BingSearch
import json

Library	Purpose
`BingSearch`	to scrape and parse Bing results using SerpApi web scraping library.
`json`	to convert extracted data to a JSON object.

The parameters are defined for generating the URL. If you want to pass other parameters to the URL, you can do so using the params dictionary:

params = {
    'api_key': '...',               # https://serpapi.com/manage-api-key
    'q': 'Coffee',                  # search query
    'engine': 'bing_images',        # search engine
    'cc': 'US',                     # country of the search
    'first': 1,                     # pagination
    'count': 50,                    # number of results per page
    # 'imagesize': 'wallpaper',     # filtering by size
    # 'color2': 'color',            # filtering by color
    # 'photo': 'photo',             # filtering by image type
    # 'aspect': 'wide',             # filtering by layout
    # 'face': 'portrait',           # filtering by people
    # 'age': 'lt525600',            # filtering by date
    # 'license': 'Type-Any'         # filtering by license
}

Parameters	Explanation
`api_key`	Parameter defines the SerpApi private key to use.
`q`	Parameter defines the search query. You can use anything that you would use in a regular Bing Images search.
`engine`	Set parameter to `bing_images` to use the Bing Images API engine.
`cc`	Parameter defines the country to search from. It follows the 2-character ISO_3166-1 format. (e.g., `us` for United States, `de` for Germany, `gb` for United Kingdom, etc.).
`first`	Parameter controls the offset of the organic results. This parameter defaults to `1`. (e.g., `first=10` will move the 10th organic result to the first position).
`count`	Parameter controls the number of results per page. This parameter is only a suggestion and might not reflect the returned results.
`imagesize`	Parameter is used for filtering images by size. It can be set to: `small`, `medium`, `large`, `wallpaper`.
`color2`	Parameter is used for filtering images by color. It can be set to: `color` - Color Only, `bw` - Black & white, `FGcls_GREEN` - Green, etc.
`photo`	Parameter is used for filtering images by image type. It can be set to: `photo`, `clipart`, `linedrawing`, `animatedgif`, `transparent`.
`aspect`	Parameter is used for filtering images by layout. It can be set to: `square`, `wide`, `tall`.
`face`	Parameter is used for filtering images by people. It can be set to: `face` - Faces Only, `portrait` - Head & Shoulders.
`age`	Parameter is used for filtering images by date. It can be set to: `lt1440` - Past 24 hours, `lt10080` - Past week, `lt43200` - Past month, `lt525600` - Past year.
`license`	Parameter is used for filtering images by license. It can be set to: `Type-Any` - All Creative Commons, `L1` - Public Domain, `L2_L3_L4_L5_L6_L7` - Free to share and use, `L2_L3_L4` - Free to share and use commercially, `L2_L3_L5_L6` - Free to modify, share and use, `L2_L3` - Free to modify, share, and use commercially.

📌Note: You can also add other API Parameters.

Then, we create a search object where the data is retrieved from the SerpApi backend. In the results dictionary we get data from JSON:

search = BingSearch(params)         # data extraction on the SerpApi backend
results = search.get_dict()         # JSON -> Python dict

At the moment, the results dictionary only stores data from 1 page. Before extracting data, the bing_images_results dictionary is created where this data will be added later. Since some of the data is only displayed on the first page, you can extract them immediately:

bing_images_results = {
    'images_results': [],
    'suggested_searches': [],
    'refined_searches': results.get('refined_searches', []),
    'related_searches': results.get('related_searches', []),
    'shopping_results': results.get('shopping_results', [])
}

The page_limit variable defines the page limit. If you want to extract data from a different number of pages, then simply write the required number into this variable.

page_limit = 10

To get all results, you need to apply pagination. This is achieved by the following check: while there is no error in the results and the current page_count value is less than the specified page_limit value, we extract the data, increase the first parameter by the value of the count parameter to get the results from next page and update the results object with the new page data:

page_count = 0

while 'error' not in results and page_count < page_limit:
    # data extraction from current page will be here

    params['first'] += params['count']
    page_count += 1
    results = search.get_dict()

Lists by corresponding keys are extended with new data from each page:

bing_images_results['images_results'].extend(results.get('images_results', []))
bing_images_results['suggested_searches'].extend(results.get('suggested_searches', []))
# thumbnail= results['images_results'][0]['thumbnail']
# link = results['images_results'][0]['link']
# title = results['images_results'][0]['title']

📌Note: In the comments above, I showed how to extract specific fields. You may have noticed the results['images_results'][0]. This is the index of a organic result, which means that we are extracting data from the first organic result. The results['images_results'][1] is from the second organic result and so on.

After the all data is retrieved, it is output in JSON format:

print(json.dumps(bing_images_results, indent=2, ensure_ascii=False))

Output

{
  "images_results": [
    {
      "thumbnail": "https://th.bing.com/th/id/OIP.-ACVKbQLg-aRb2bVxy1jfAHaE8?w=237&h=180&c=7&r=0&o=5&pid=1.7",
      "link": "https://www.bing.com/images/search?view=detailV2&ccid=%2bACVKbQL&id=16A3CD2F2C1C0D71AE4EA5B08475E79A9A2D543C&thid=OIP.-ACVKbQLg-aRb2bVxy1jfAHaE8&mediaurl=https%3a%2f%2fimages.freeimages.com%2fimages%2flarge-previews%2fbc0%2fcoffee-1317648.jpg&cdnurl=https%3a%2f%2fth.bing.com%2fth%2fid%2fR.f8009529b40b83e6916f66d5c72d637c%3frik%3dPFQtmprndYSwpQ%26pid%3dImgRaw%26r%3d0&exph=1066&expw=1599&q=Coffee&simid=608042879963195855&FORM=IRPRST&ck=733C3F62E8891EC1D9ECD99852D9CB5E&selectedIndex=0",
      "title": "coffee Free Photo Download | FreeImages",
      "size": "1599 x 1066 · jpeg",
      "source": "https://www.freeimages.com/photo/coffee-1317648",
      "domain": "freeimages.com",
      "original": "https://images.freeimages.com/images/large-previews/bc0/coffee-1317648.jpg",
      "description": "coffee freeimages",
      "position": 1
    },
    {
      "thumbnail": "https://th.bing.com/th/id/OIP.ceGbTJIrj6ajM2XlsyFXAwAAAA?w=238&h=180&c=7&r=0&o=5&pid=1.7",
      "link": "https://www.bing.com/images/search?view=detailV2&ccid=ceGbTJIr&id=A7A5C7AA478FAAAE30E29D2927035138A7ED4F86&thid=OIP.ceGbTJIrj6ajM2XlsyFXAwAAAA&mediaurl=https%3a%2f%2fwww.indigofinance.com.au%2fwp-content%2fuploads%2f2018%2f07%2fistock-157528129.jpg&cdnurl=https%3a%2f%2fth.bing.com%2fth%2fid%2fR.71e19b4c922b8fa6a33365e5b3215703%3frik%3dhk%252ftpzhRAycpnQ%26pid%3dImgRaw%26r%3d0&exph=315&expw=474&q=Coffee&simid=608037970811556740&FORM=IRPRST&ck=03F56E55E9A229798D3F9E0E47710CF6&selectedIndex=1",
      "title": "Coffee Addiction! Can it be tamed? - Indigo Finance",
      "size": "474 x 315 · jpeg",
      "source": "https://www.indigofinance.com.au/coffee-addiction-can-tamed/",
      "domain": "indigofinance.com.au",
      "original": "https://www.indigofinance.com.au/wp-content/uploads/2018/07/istock-157528129.jpg",
      "description": "coffee addiction",
      "position": 2
    },
    {
      "thumbnail": "https://serpapi.com/searches/641c636f47ccf1fac0af6454/images/d8b77d8a83f199ee4f68a702661ae07622e8f07644945d882e253318c4e63e4d.gif",
      "link": "https://www.bing.com/images/search?view=detailV2&ccid=%2fO2T2GEH&id=CDAE3AA25A9EAAC5AAFCFEAE226C4E400B130445&thid=OIP._O2T2GEHLIPfkjeW55c4fAHaEo&mediaurl=https%3a%2f%2fth.bing.com%2fth%2fid%2fR.fced93d861072c83df923796e797387c%3frik%3dRQQTC0BObCKu%252fg%26riu%3dhttp%253a%252f%252fi.slimg.com%252fsc%252fsl%252fphoto%252ff%252ffo%252ffood-coffeeandcoffeebeans-dd.jpg%26ehk%3dd3WNdg6Q8xrjnwbPLfOtvTsmTGRas1ZKp81pa7PKQqw%253d%26risl%3d%26pid%3dImgRaw%26r%3d0&exph=382&expw=610&q=Coffee&simid=607997220165786271&FORM=IRPRST&ck=03A478D663783F2B09856F5B2AAAE68B&selectedIndex=2",
      "title": "America's Best Coffee Shops - SmarterTravel.com",
      "size": "610 x 382 · jpeg",
      "source": "http://www.smartertravel.com/blogs/today-in-travel/america-best-coffee-shops.html?id=10909975",
      "domain": "smartertravel.com",
      "original": "http://i.slimg.com/sc/sl/photo/f/fo/food-coffeeandcoffeebeans-dd.jpg",
      "description": "coffee shops america cafe coffe food cup foods wine facts espresso coffees tweet caffeine un good healthy give con flavors",
      "position": 3
    },
    ... other images results
  ],
  "suggested_searches": [
    {
      "thumbnail": "https://th.bing.com/th?q=Coffee+Maker&w=42&h=42&c=7&rs=1&p=0&o=5&pid=1.7&mkt=en-US&cc=US&setlang=en&adlt=moderate&t=1",
      "link": "https://www.bing.com/images/search?q=Coffee+Maker&FORM=RESTAB",
      "serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=bing_images&q=Coffee+Maker",
      "name": "Coffee Maker"
    },
    {
      "thumbnail": "https://th.bing.com/th?q=Best+Coffee+Beans&w=42&h=42&c=7&rs=1&p=0&o=5&pid=1.7&mkt=en-US&cc=US&setlang=en&adlt=moderate&t=1",
      "link": "https://www.bing.com/images/search?q=Best+Coffee+Beans&FORM=RESTAB",
      "serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=bing_images&q=Best+Coffee+Beans",
      "name": "Best Coffee Beans"
    },
    {
      "thumbnail": "https://th.bing.com/th?q=Coffee+Flavors&w=42&h=42&c=7&rs=1&p=0&o=5&pid=1.7&mkt=en-US&cc=US&setlang=en&adlt=moderate&t=1",
      "link": "https://www.bing.com/images/search?q=Coffee+Flavors&FORM=RESTAB",
      "serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=bing_images&q=Coffee+Flavors",
      "name": "Coffee Flavors"
    },
    ... other suggested searches
  ],
  "refined_searches": [
    {
      "thumbnail": "https://th.bing.com/th?q=Coffee+Cart&w=120&h=120&c=1&rs=1&qlt=90&cb=1&pid=InlineBlock&mkt=en-US&cc=US&setlang=en&adlt=moderate&t=1&mw=247",
      "link": "https://www.bing.com/images/search?q=Coffee+Cart&qft=&FORM=IRTRRL",
      "serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=bing_images&q=Coffee+Cart",
      "name": "Cart"
    },
    {
      "thumbnail": "https://th.bing.com/th?q=Healthy+Coffee&w=120&h=120&c=1&rs=1&qlt=90&cb=1&pid=InlineBlock&mkt=en-US&cc=US&setlang=en&adlt=moderate&t=1&mw=247",
      "link": "https://www.bing.com/images/search?q=Healthy+Coffee&qft=&FORM=IRTRRL",
      "serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=bing_images&q=Healthy+Coffee",
      "name": "Healthy"
    },
    {
      "thumbnail": "https://th.bing.com/th?q=Coffee+Packaging&w=120&h=120&c=1&rs=1&qlt=90&cb=1&pid=InlineBlock&mkt=en-US&cc=US&setlang=en&adlt=moderate&t=1&mw=247",
      "link": "https://www.bing.com/images/search?q=Coffee+Packaging&qft=&FORM=IRTRRL",
      "serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=bing_images&q=Coffee+Packaging",
      "name": "Packaging"
    },
    ... other refined searches
  ],
  "related_searches": [
    {
      "link": "https://www.bing.com/images/search?q=Teeccino+Coffee&qft=&fsm=1&FORM=SHOPSO",
      "serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=bing_images&q=Teeccino+Coffee",
      "name": "Teeccino Coffee"
    },
    {
      "link": "https://www.bing.com/images/search?q=Espresso+Coffees&qft=&fsm=1&FORM=SHOPSO",
      "serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=bing_images&q=Espresso+Coffees",
      "name": "Espresso Coffees"
    },
    {
      "link": "https://www.bing.com/images/search?q=Folgers+Coffee&qft=&fsm=1&FORM=SHOPSO",
      "serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=bing_images&q=Folgers+Coffee",
      "name": "Folgers Coffee"
    }
  ],
  "shopping_results": []
}

📌Note: Head to the playground for a live and interactive demo.

Links

Join us on Twitter | YouTube

Add a Feature Request💫 or a Bug🐞

How to Extract Bing News Data with SerpApi and Python

Artur Chukhrai — Wed, 29 Mar 2023 04:00:00 +0000

Intro
What will be scraped
Why using API?
Full Code
Preparation
Code Explanation
Output
Links

Intro

In this blog post, we'll go through the process of extracting Bing News using the Bing News Engine Results API and the Python programming language. You can look at the complete code in the online IDE (Replit).

What will be scraped

Why using API?

There're a couple of reasons that may use API, ours in particular:

No need to create a parser from scratch and maintain it.
Bypass blocks from Google: solve CAPTCHA or solve IP blocks.
Pay for proxies, and CAPTCHA solvers.
Figure out the legal part of scraping data.

SerpApi handles everything on the backend with fast response times under ~1.5 seconds (~1.0 seconds with Ludicrous speed) per request and without browser automation, which becomes much faster. Response times and status rates are shown under SerpApi Status page.

Full Code

This code retrieves all news with pagination:

from serpapi import BingSearch
import json

params = {
    'api_key': '...',                   # https://serpapi.com/manage-api-key
    'q': 'Coffee',                      # search query
    'engine': 'bing_news',              # search engine
    'cc': 'US',                         # country of the search
    'first': 1,                         # pagination
    'count': 10,                        # number of results per page
    'qft': 'interval="7"'               # news for past 24 hours
}

search = BingSearch(params)             # data extraction on the SerpApi backend
results = search.get_dict()             # JSON -> Python dict

bing_news_results = []

page_count = 0
page_limit = 5

while 'error' not in results and page_count < page_limit:
    bing_news_results.extend(results.get('organic_results', []))

    params['first'] += params['count']
    page_count += 1
    results = search.get_dict()

print(json.dumps(bing_news_results, indent=2, ensure_ascii=False))

Preparation

Install library:

pip install google-search-results

google-search-results is a SerpApi API package.

Code Explanation

Import libraries:

from serpapi import BingSearch
import json

Library	Purpose
`BingSearch`	to scrape and parse Bing results using SerpApi web scraping library.
`json`	to convert extracted data to a JSON object.

The parameters are defined for generating the URL. If you want to pass other parameters to the URL, you can do so using the params dictionary:

params = {
    'api_key': '...',                   # https://serpapi.com/manage-api-key
    'q': 'Coffee',                      # search query
    'engine': 'bing_news',              # search engine
    'cc': 'US',                         # country of the search
    'first': 1,                         # pagination
    'count': 10,                        # number of results per page
    'qft': 'interval="7"'               # news for past 24 hours
}

Parameters	Explanation
`api_key`	Parameter defines the SerpApi private key to use.
`q`	Parameter defines the search query. You can use anything that you would use in a regular Bing search. (e.g., `'query'`, `NOT`, `OR`, `site:`, `filetype:`, `near:`, `ip:`, `loc:`, `feed:` etc.).
`engine`	Set parameter to `bing_news` to use the Bing News API engine.
`cc`	Parameter defines the country to search from. It follows the 2-character ISO_3166-1 format. (e.g., `us` for United States, `de` for Germany, `gb` for United Kingdom, etc.).
`first`	Parameter controls the offset of the organic results. This parameter defaults to `1`. (e.g., `first=10` will move the 10th organic result to the first position).
`count`	Parameter controls the number of results per page. This parameter is only a suggestion and might not reflect actual results returned.
`qft`	Parameter defines results sorted by date. If the parameter is not set, it will default to the "Best match" sorting. It can be set to: `interval="4"` - Past hour, `interval="7"` - Past 24 hours, `interval="8"` - Past 7 days, `interval="9"` - Past 30 days, `sortbydate="1"` - Most Recent.

📌Note: You can also add other API Parameters.

Then, we create a search object where the data is retrieved from the SerpApi backend. In the results dictionary we get data from JSON:

search = BingSearch(params)         # data extraction on the SerpApi backend
results = search.get_dict()         # JSON -> Python dict

Before extracting data, the bing_news_results list is created where this data will be added later:

bing_news_results = []

The page_limit variable defines the page limit. If you want to extract data from a different number of pages, then simply write the required number into this variable.

page_limit = 5

page_count = 0

while 'error' not in results and page_count < page_limit:
    # data extraction from current page will be here

    params['first'] += params['count']
    page_count += 1
    results = search.get_dict()

Extending the bing_news_results list with new data from each page:

bing_news_results.extend(results.get('organic_results', []))
# title = results['organic_results'][0]['title']
# link = results['organic_results'][0]['link']
# snippet = results['organic_results'][0]['snippet']
# source = results['organic_results'][0]['source']
# date = results['organic_results'][0]['date']
# thumbnail= results['organic_results'][0]['thumbnail']

📌Note: In the comments above, I showed how to extract specific fields. You may have noticed the results['organic_results'][0]. This is the index of a organic result, which means that we are extracting data from the first organic result. The results['organic_results'][1] is from the second organic result and so on.

After the all data is retrieved, it is output in JSON format:

print(json.dumps(bing_news_results, indent=2, ensure_ascii=False))

Output

[
  {
    "title": "Coffee drinkers get more steps but also less sleep, study finds",
    "link": "https://www.cbsnews.com/sacramento/news/coffee-drinkers-get-more-steps-but-also-less-sleep-study-finds/",
    "snippet": "Coffee is one of the most consumed beverages worldwide, but the pendulum has swung back and forth about its benefits and ...",
    "source": "CBS News",
    "date": "5h",
    "thumbnail": "https://serpapi.com/searches/6421c51af26ac67a4930a857/images/85e52d2243238795454091b5f6b3f41e7f9969cd089f02e76ad21147c47e4f69.jpeg"
  },
  {
    "title": "Cedarburg School Board candidate accused of sexually harassing young workers at his former coffee shop",
    "link": "https://www.jsonline.com/story/news/education/2023/03/27/cedarburg-school-board-candidate-accused-of-sexual-harassment-coffee-shop/70039331007/",
    "snippet": "Former employees accused Scott Sidney, a Cedarburg School Board candidate, of inappropriate words and actions at his former ...",
    "source": "Milwaukee Journal Sentinel",
    "date": "5h",
    "thumbnail": "https://serpapi.com/searches/6421c51af26ac67a4930a857/images/85e52d2243238795454091b5f6b3f41e8897032186a0a3603444deba3cdd76a7.jpeg"
  },
  {
    "title": "New development near WoodTrust in Grand Rapids; new merch from local coffee shop | Streetwise",
    "link": "https://news.yahoo.com/development-near-woodtrust-grand-rapids-100428122.html",
    "snippet": "If you know of a new business, a development or a place that’s closing, send me a note at cshuda@gannett.com. If you have ...",
    "source": "YAHOO!News",
    "date": "6h",
    "thumbnail": "https://serpapi.com/searches/6421c51af26ac67a4930a857/images/85e52d2243238795454091b5f6b3f41e5f1483416a23fbf49adc15101a042c82.jpeg"
  },
  ... other news
]

📌Note: Head to the playground for a live and interactive demo.

Links

Join us on Twitter | YouTube

Add a Feature Request💫 or a Bug🐞

Scraping Bing Organic Results using Python and SerpApi

Artur Chukhrai — Wed, 29 Mar 2023 02:05:54 +0000

Intro
What will be scraped
Why using API?
Full Code
Preparation
Code Explanation
Output

Intro

In this blog post, we'll go through the process of extracting all the data that appears in Bing organic search results using the Bing Search Engine Results API and other APIs contained in the dropdown below.

Used APIs:

You can look at the complete code in the online IDE (Replit).

What will be scraped

📌Note: Search results may vary depending on the parameters passed.

Why using API?

There're a couple of reasons that may use API, ours in particular:

No need to create a parser from scratch and maintain it.
Bypass blocks from Google: solve CAPTCHA or solve IP blocks.
Pay for proxies, and CAPTCHA solvers.
Figure out the legal part of scraping data.

SerpApi handles everything on the backend with fast response times under ~2.1 seconds (~1.4 seconds with Ludicrous speed) per request and without browser automation, which becomes much faster. Response times and status rates are shown under SerpApi Status page.

Full Code

This code retrieves all the data with pagination:

from serpapi import BingSearch
import json

params = {
    'api_key': '...',                        # https://serpapi.com/manage-api-key
    'q': 'Coffee',                           # search query
    'engine': 'bing',                        # search engine
    'cc': 'US',                              # country of the search
    'location': 'New York,United States',    # location of the search
    'first': 1,                              # pagination
    'count': 10                              # number of results per page
}

search = BingSearch(params)                  # data extraction on the SerpApi backend
results = search.get_dict()                  # JSON -> Python dict

bing_results = {
    'ads': [],
    'organic_results': [],
    'local_map': results.get('local_map', {}),
    'local_results': results.get('local_results', {}),
    'inline_images': results.get('inline_images', {}),
    'inline_videos': results.get('inline_videos', {}),
    'knowledge_graph': results.get('knowledge_graph', {}),
    'answer_box': results.get('answer_box', {}),
    'recipes_results': results.get('recipes_results', {}),
    'related_questions': results.get('related_questions', []),
    'related_searches': []
}

page_count = 0
page_limit = 10

while 'error' not in results and page_count < page_limit:
    bing_results['ads'].extend(results.get('ads', []))
    bing_results['organic_results'].extend(results.get('organic_results', []))
    bing_results['related_searches'].extend(results.get('related_searches', []))

    params['first'] += params['count']
    page_count += 1
    results = search.get_dict()

print(json.dumps(bing_results, indent=2, ensure_ascii=False))

Preparation

Install library:

pip install google-search-results

google-search-results is a SerpApi API package.

Code Explanation

Import libraries:

from serpapi import BingSearch
import json

Library	Purpose
`BingSearch`	to scrape and parse Bing results using SerpApi web scraping library.
`json`	to convert extracted data to a JSON object.

The parameters are defined for generating the URL. If you want to pass other parameters to the URL, you can do so using the params dictionary:

params = {
    'api_key': '...',                        # https://serpapi.com/manage-api-key
    'q': 'Coffee',                           # search query
    'engine': 'bing',                        # search engine
    'cc': 'US',                              # country of the search
    'location': 'New York,United States',    # location of the search
    'first': 1,                              # pagination
    'count': 10                              # number of results per page
}

Parameters	Explanation
`api_key`	Parameter defines the SerpApi private key to use.
`q`	Parameter defines the search query. You can use anything that you would use in a regular Bing search. (e.g., `'query'`, `NOT`, `OR`, `site:`, `filetype:`, `near:`, `ip:`, `loc:`, `feed:` etc.).
`engine`	Set parameter to `bing` to use the Bing API engine.
`cc`	Parameter defines the country to search from. It follows the 2-character ISO_3166-1 format. (e.g., `us` for United States, `de` for Germany, `gb` for United Kingdom, etc.).
`location`	Parameter defines from where you want the search to originate. If several locations match the location requested, we'll pick the most popular one. Head to the /locations.json API if you need more precise control.
`first`	Parameter controls the offset of the organic results. This parameter defaults to `1`. (e.g., `first=10` will move the 10th organic result to the first position).
`count`	Parameter controls the number of results per page. Minimum: `1`, Maximum: `50`. This parameter is only a suggestion and might not reflect actual results returned.

📌Note: You can also add other API Parameters.

Then, we create a search object where the data is retrieved from the SerpApi backend. In the results dictionary we get data from JSON:

search = BingSearch(params)     # data extraction on the SerpApi backend
results = search.get_dict()     # JSON -> Python dict

At the moment, the results dictionary only stores data from 1 page. Before extracting data, the bing_results dictionary is created where this data will be added later. Since some of the data is only displayed on the first page, you can extract them immediately:

bing_results = {
    'ads': [],
    'organic_results': [],
    'local_map': results.get('local_map', {}),
    'local_results': results.get('local_results', {}),
    'inline_images': results.get('inline_images', {}),
    'inline_videos': results.get('inline_videos', {}),
    'knowledge_graph': results.get('knowledge_graph', {}),
    'answer_box': results.get('answer_box', {}),
    'recipes_results': results.get('recipes_results', {}),
    'related_questions': results.get('related_questions', []),
    'related_searches': []
}

The page_limit variable defines the page limit. If you want to extract data from a different number of pages, then simply write the required number into this variable.

page_limit = 10

page_count = 0

while 'error' not in results and page_count < page_limit:
    # data extraction from current page will be here

    params['first'] += params['count']
    page_count += 1
    results = search.get_dict()

Lists by corresponding keys are extended with new data from each page:

bing_results['ads'].extend(results.get('ads', []))
bing_results['organic_results'].extend(results.get('organic_results', []))
bing_results['related_searches'].extend(results.get('related_searches', []))
# price= results['ads'][0]['price']
# title = results['organic_results'][0]['title']
# link = results['organic_results'][0]['link']
# snippet = results['organic_results'][0]['snippet']
# query = results['related_searches'][0]['query']

After the all data is retrieved, it is output in JSON format:

print(json.dumps(bing_results, indent=2, ensure_ascii=False))

Output

{
  "ads": [
    {
      "position": 1,
      "block_position": "inline-top",
      "title": {
        "visible": "Flavored Coffee Of",
        "hidden": "The Month Club - 6 Months"
      },
      "price": "$124.49",
      "store": "Boca Java",
      "link": "https://www.bing.com/aclk?ld=e8Hfu6ijJpE-SJ-E2odZFsDTVUCUxXbdrWFM3S-7j7JLeWi1v3bGHZ1Aq1nJuIVz7RC0o_fJsYZoqFE5atgN0ii6xA7uv4-LyWrc0C835nImKbzlaakp2M2WSjk05bxpnSTJV6iYyYtnq9k0tlEkln_uSMu7wmP_YZy8Q8HPwa6Z9yIZNbR5YIQshvvzEykwkqwXTdew&u=aHR0cHMlM2ElMmYlMmZ3d3cuYm9jYWphdmEuY29tJTJmcHJvZHVjdCUyZjk5NTQlM2Ztc2Nsa2lkJTNkZTRmYmE4MzAxNGQ1MWMxNjVmYjJjYzJiZTY1OWIzOTQlMjZ1dG1fc291cmNlJTNkYmluZyUyNnV0bV9tZWRpdW0lM2RjcGMlMjZ1dG1fY2FtcGFpZ24lM2RTaG9wcGluZyUyNTIwLSUyNTIwTmV3JTI2dXRtX3Rlcm0lM2Q0NTg1NTEzMjQzOTA4ODEyJTI2dXRtX2NvbnRlbnQlM2RUb3Vycw&rlid=e4fba83014d51c165fb2cc2be659b394",
      "thumbnail": "https://www.bing.com/th?id=OP.D7DcGzD8CA5qcQ474C474&w=94&h=94&c=17&o=5&pid=21.1"
    },
    ... other ads results
  ],
  "organic_results": [
    {
      "position": 1,
      "title": "27 Outstanding Coffee Shops Open Right Now in NYC",
      "link": "https://ny.eater.com/maps/best-cafe-coffee-shop-new-york-city-brooklyn-queens",
      "displayed_link": "https://ny.eater.com/maps/best-cafe-coffee-shop...",
      "rich_snippet": {
        "extensions": [
          "Estimated Reading Time: 6 mins"
        ]
      },
      "thumbnail": "https://serpapi.com/searches/641bf28a37014c25c6bae18a/images/00dad7eafb7da3761aec6080896f69f20ece50ab3511be0312fb414fc343556c.gif",
      "snippet": "Variety first started roasting coffee in 2014, and is known for its nutty, chocolatey roasts. Open in Google Maps. 1269 Lexington Ave, …",
      "cached_page_link": "https://cc.bingj.com/cache.aspx?q=Coffee&d=4691689351434387&w=zDQ1K8HqZ2VViOImKdoLD7jCQPkb1BtF"
    },
    ... other organic results
  ],
  "local_map": {
    "link": "https://www.bing.com/maps/geoplat/REST/v1/Imagery/Map/RoadVibrant/40.687304,-73.976673/15?ms=646,200&pp=40.689369,-73.980797;S518;Starbucks&pp=40.689491,-73.972549;S518;Bittersweet&pp=40.685116,-73.976021;S518;Starbucks&ppsc=dpss,cfp&ml=Basemap,Landmarks,OsmBuildings&key=AnTcaqBi2ypp0xI-OZNi4W_ik2KhjgpqioTAtXLC8GzkMBQRMlyxvxyTnd5b73im&c=en-US&fmt=jpeg&od=1&logo=n&da=ro",
    "image": "https://www.bing.com/maps/geoplat/REST/v1/Imagery/Map/RoadVibrant/40.687304,-73.976673/15?ms=646,200&pp=40.689369,-73.980797;S518;Starbucks&pp=40.689491,-73.972549;S518;Bittersweet&pp=40.685116,-73.976021;S518;Starbucks&ppsc=dpss,cfp&ml=Basemap,Landmarks,OsmBuildings&key=AnTcaqBi2ypp0xI-OZNi4W_ik2KhjgpqioTAtXLC8GzkMBQRMlyxvxyTnd5b73im&c=en-US&fmt=jpeg&od=1&logo=n&da=ro",
    "gps_coordinates": {
      "latitude": "40.689491271972656",
      "longitude": "-73.98079681396484"
    }
  },
  "local_results": {
    "places": [
      {
        "position": 1,
        "place_id": "YN873x133137763",
        "title": "Starbucks",
        "rating": 4.0,
        "reviews": 60,
        "hours": "Closed · Opens 5:30 AM",
        "address": "395 Flatbush Avenue Ext, Brooklyn",
        "mention": "Delivery • Takeout",
        "links": {
          "directions": "https://www.bing.com/maps/directions?rtp=adr.~pos.40.689369201660156_-73.98079681396484_395+Flatbush+Avenue+Ext%2c+Brooklyn%2c+NY+11201_Starbucks_(718)+858-8070",
          "website": "https://www.bing.com/alink/link?url=https%3a%2f%2fwww.starbucks.com%2fstore-locator%2fstore%2f1005807%2f&source=serp-local&h=F2sOrXCeAlgbx8McGSutCp9gvyBDYDU6w4WmuoOg9YM%3d&p=localwebsitelistingentitycard&ig=DC3B1BE012E2422F8DEA1C8F86AAB99B&ypid=YN873x133137763"
        },
        "gps_coordinates": {
          "latitude": "40.68937",
          "longitude": "-73.9808"
        }
      },
      ... other local places
    ]
  },
  "inline_images": {},
  "inline_videos": {
    "title": "Videos of Coffee",
    "displayed_link": "bing.com/videos",
    "see_more_link": "https://www.bing.com/videos/search?q=Coffee&FORM=VDRESM",
    "items": [
      {
        "position": 1,
        "link": "https://www.bing.com/videos/search?q=coffee&docid=608034461997991710&mid=0A195DA22867D85A814A0A195DA22867D85A814A&view=detail&FORM=VIRE",
        "thumbnail": "https://www.bing.com/th?id=OVP.r2hQaGFwkWYBsPgw_CbqWwEsDh&w=236&h=132&c=7&rs=1&qlt=90&o=6&pid=1.7",
        "duration": "7:52",
        "title": "How to Make the 3 Most Popular Milk Coffees #barista #coffee",
        "views": "239.9K views",
        "date": "6 months ago",
        "platform": "YouTube",
        "channel": "Artisti Coffee Roasters."
      },
      ... other inline videos
    ]
  },
  "knowledge_graph": {
    "title": "Coffee",
    "type": "Drink",
    "description": "A brewed drink prepared from roasted coffee beans, the seeds of berries from certain 'Coffea' species.",
    "thumbnails": [
      {
        "image": "https://www.bing.com/th?id=ALSTU1C05162DF22F2C2E8EF4206C8DBBEA9FA785433DDD73D0BBEB59978C47020D96&w=296&h=176&rs=2&o=6&oif=webp&pid=SANGAM",
        "source": "https://www.bing.com/images/search?q=coffee&cbn=KnowledgeCard&stid=e7f69601-fd37-6560-319a-6bd7037a3cf1&FORM=KCHIMM"
      }
    ],
    "nutrition_facts": {
      "serving_size": "Coffee (100g)",
      "serving_facts": [
        {
          "avg_cals": {
            "ratio": "1"
          }
        },
        {
          "protein": {
            "ratio": "0.1g"
          }
        },
        {
          "fat": {
            "ratio": "0g",
            "percentage": "0%"
          }
        },
        {
          "carbs": {
            "ratio": "0g",
            "percentage": "0%"
          }
        }
      ]
    },
    "facts": [
      {
        "title": "The most expensive coffee in world is Black Ivory Coffee.",
        "link": "https://en.wikipedia.org/wiki/Black_Ivory_Coffee"
      },
      {
        "title": "Place of origin Yemen",
        "link": "http://en.wikipedia.org/wiki/Coffee"
      },
      {
        "title": "Coffee was supposedly discovered by a goat herder in Ethiopia.",
        "link": "https://www.insider.com/coffee-facts-you-didnt-know-2018-12#coffee-was-supposedly-discovered-by-a-goat-herder-in-ethiopia-1"
      },
      {
        "title": "Brazil is the largest coffee producer in the world, followed by Vietnam and Colombia.",
        "link": "https://www.insider.com/coffee-facts-you-didnt-know-2018-12#brazil-is-the-largest-coffee-producer-in-the-world-followed-by-vietnam-and-colombia-5",
        "thumbnail": "https://www.bing.com/th?id=ALSTU18E12B038F083FF3EBF5C20DDBC4667D3C2D9C0C33ED0C72F89309B8AA54A77A&w=124&h=73&o=6&oif=webp&pid=SANGAM"
      }
    ],
    "profiles": [
      {
        "title": "Wikipedia",
        "link": "https://en.wikipedia.org/wiki/Coffee"
      }
    ],
    "common_ingredients": [
      {
        "title": "Water",
        "link": "https://www.bing.com/search?q=Water",
        "thumbnail": "https://www.bing.com/th?id=ALSTU17F805D3AED894E8E9AE96144DD06EBD1C6AC45E9451BB5A0C6C7B14AF960BE2&w=40&h=40&o=6&pid=SANGAM"
      },
      {
        "title": "Roasted coffee beans",
        "link": "https://www.bing.com/search?q=Roasted+coffee+beans",
        "thumbnail": "https://www.bing.com/th?id=ALSTUB0B37AFC3D8E307EACBFE8F6487BF02B27069B6E49309A8117A98653F56D0CA9&w=40&h=40&o=6&pid=SANGAM"
      }
    ],
    "life_cycle": [
      {
        "link": "https://www.bing.com/search?FORM=SNAPST&q=coffee+harvesting+methods&filters=fcid:\"0-9fc66bf7-b836-d612-71d6-44c649d20f6f\"+itemid:\"0\"",
        "thumbnail": "https://www.bing.com/th?id=OSK.TMLNVBRWzFmoZAQSp2N0WZzCNqvBKsl8VOaADH9cw3cfd4k&w=150&h=220&c=7&o=6&pid=SANGAM"
      },
      {
        "link": "https://www.bing.com/search?FORM=SNAPST&q=coffee+processing+methods&filters=fcid:\"0-9fc66bf7-b836-d612-71d6-44c649d20f6f\"+itemid:\"1\"",
        "thumbnail": "https://www.bing.com/th?id=OSK.TMLNaqLBMSDvjyLkQy5TBEc7r3otnt1Qe6KSft9h7e56yTw&w=150&h=220&c=7&o=6&pid=SANGAM"
      },
      {
        "link": "https://www.bing.com/search?FORM=SNAPST&q=coffee+roasting+process&filters=fcid:\"0-9fc66bf7-b836-d612-71d6-44c649d20f6f\"+itemid:\"2\"",
        "thumbnail": "https://www.bing.com/th?id=OSK.TMLNhtdA7jtVcEjyF_T3ekW4hShQWZVSbdVUV1NyHF5QLT4&w=150&h=220&c=7&o=6&pid=SANGAM"
      },
      {
        "link": "https://www.bing.com/search?FORM=SNAPST&q=coffee+grinding+levels&filters=fcid:\"0-9fc66bf7-b836-d612-71d6-44c649d20f6f\"+itemid:\"3\"",
        "thumbnail": "https://www.bing.com/th?id=OSK.TMLNW_2cWIzfYJ9gLZlvpYxWx37CKCsPJh5HXdpEnSVGQTE&w=150&h=220&c=7&o=6&pid=SANGAM"
      }
    ]
  },
  "answer_box": {},
  "recipes_results": {},
  "related_questions": [],
  "related_searches": [
    {
      "query": "best tasting coffee in america",
      "link": "https://www.bing.com/search?q=best+tasting+coffee+in+america&FORM=QSRE1"
    },
    {
      "query": "best tasting coffee",
      "link": "https://www.bing.com/search?q=best+tasting+coffee&FORM=QSRE2"
    },
    {
      "query": "best tasting store bought coffee",
      "link": "https://www.bing.com/search?q=best+tasting+store+bought+coffee&FORM=QSRE3"
    },
    {
      "query": "how to order drip coffee at starbucks",
      "link": "https://www.bing.com/search?q=how+to+order+drip+coffee+at+starbucks&FORM=QSRE4"
    },
    {
      "query": "starbucks coffee box",
      "link": "https://www.bing.com/search?q=starbucks+coffee+box&FORM=QSRE5"
    },
    {
      "query": "how much caffeine in coffee",
      "link": "https://www.bing.com/search?q=how+much+caffeine+in+coffee&FORM=QSRE6"
    },
    {
      "query": "starbucks coffee catering",
      "link": "https://www.bing.com/search?q=starbucks+coffee+catering&FORM=QSRE7"
    },
    {
      "query": "top 10 best tasting coffee",
      "link": "https://www.bing.com/search?q=top+10+best+tasting+coffee&FORM=QSRE8"
    }
  ]
}

📌Note: Head to the playground for a live and interactive demo.

Join us on Twitter | YouTube

Add a Feature Request💫 or a Bug🐞

Scrape Google Hotels with Python

Artur Chukhrai — Thu, 02 Mar 2023 20:34:43 +0000

What will be scraped
Full Code
Preparation
Code Explanation
- Top-level code environment
- Scroll page
- Scrape Google Hotels Listings
Output
Links

What will be scraped

Full Code



from selenium import webdriver
from selenium_stealth import stealth
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selectolax.lexbor import LexborHTMLParser
import time, json


def scroll_page(url):
    service = Service(ChromeDriverManager().install())

    options = webdriver.ChromeOptions()
    options.add_argument('start-maximized')
    options.add_argument('--headless')
    options.add_argument('--no-sandbox')
    options.add_experimental_option('excludeSwitches', ['enable-automation'])
    options.add_experimental_option('useAutomationExtension', False)

    driver = webdriver.Chrome(service=service, options=options)

    stealth(
        driver,
        user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
        languages=['en-US', 'en'],
        vendor='Google Inc.',
        platform='Win32',
        webgl_vendor='Intel Inc.',
        renderer='Intel Iris OpenGL Engine',
        fix_hairline=True,
    )

    driver.get(url)
    time.sleep(2)

    old_height = driver.execute_script("""
        function getHeight() {
            return document.querySelector('.t8aeve').scrollHeight;
        }
        return getHeight();
    """)

    body = driver.find_element(By.TAG_NAME, 'body')

    while True:
        driver.execute_script("window.scrollTo(0, document.querySelector('.t8aeve').scrollHeight)")
        time.sleep(1)

        body.send_keys(Keys.PAGE_UP)
        time.sleep(2)

        new_height = driver.execute_script("""
            function getHeight() {
                return document.querySelector('.t8aeve').scrollHeight;
            }
            return getHeight();
        """)

        if new_height == old_height:
            break

        old_height = new_height

    parser = LexborHTMLParser(driver.page_source)
    driver.quit()

    return parser


def scrape_google_hotels(parser):
    data = []

    for result in parser.root.css('.uaTTDe'):
        result_dict = {}

        if result.css_first('.hVE5 .ogfYpf'):
            result_dict['ad'] = result.css_first('.hVE5 .ogfYpf').text().replace('  ', ' ')

        result_dict['title'] = result.css_first('.QT7m7 h2').text()
        result_dict['link'] = 'https://www.google.com' + result.css_first('.PVOOXe').attributes.get('href')
        price = result.css_first('.OxGZuc .kixHKb span')
        result_dict['price'] = price.text() if price else None
        result_dict['extracted_price'] = float(price.text().split()[0]) if price else None
        rating = result.css_first('.FW82K .KFi5wf')
        result_dict['rating'] = float(rating.text()) if rating else None
        reviews = result.css_first('.FW82K .jdzyld')
        result_dict['reviews'] = int(reviews.text()[2:-1].replace(',', '')) if reviews else None
        result_dict['extensions'] = [extension.css_first('.sSHqwe').text() for extension in result.css('.RJM8Kc .HlxIlc div, li')]
        result_dict['thumbnails'] = [
            thumbnail.attributes.get('src') if thumbnail.attributes.get('src') else thumbnail.attributes.get('data-src')
            for thumbnail in result.css('.NBZP0e .q5P4L')
        ]

        data.append(result_dict)

    return data


def main():
    URL = 'https://www.google.com/travel/hotels?q=hotels%20in%20tulsa%20oklahoma&gsas=1&rp=CgpYAGAAcgQIAhgAOAGoAgA&ved=0CAAQ5JsGahcKEwioxL-n2oT9AhUAAAAAHQAAAAAQBA&utm_campaign=sharing&utm_medium=link&utm_source=htls&ts=CAESCgoCCAMKAggDEAAaIAoCGgASGhIUCgcI5w8QAhgJEgcI5w8QAhgKGAEyAhAAKgkKBToDVVNEGgA'

    parser = scroll_page(URL)
    google_hotels = scrape_google_hotels(parser)

    print(json.dumps(google_hotels, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()

Preparation

Install libraries:



pip install selenium selenium-stealth webdriver webdriver_manager selectolax

Code Explanation

Import libraries:



from selenium import webdriver
from selenium_stealth import stealth
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selectolax.lexbor import LexborHTMLParser
import time, json

Library	Purpose
`webdriver`	to drive a browser natively, as a user would, either locally or on a remote machine using the Selenium server.
`selenium-stealth`	to prevent almost all website detections (CAPTCHA and so on).
`Service`	to manage the starting and stopping of the ChromeDriver.
`By`	to set of supported locator strategies (`By.ID`, `By.TAG_NAME`, `By.XPATH` etc).
`Keys`	to simulate button presses (`Keys.PAGE_UP`, `Keys.PAGE_DOWN`, `Keys.END` etc).
`LexborHTMLParser`	a fast HTML5 parser with CSS selectors using Lexbor engine.
`time`	to work with time in Python.
`json`	to convert extracted data to a JSON object.

The next part of the code is divided into functions. Each function is described in the corresponding heading below.

Top-level code environment

The URL variable contains a link to the Google Hotels:



URL = 'https://www.google.com/travel/hotels?q=hotels%20in%20tulsa%20oklahoma&gsas=1&rp=CgpYAGAAcgQIAhgAOAGoAgA&ved=0CAAQ5JsGahcKEwioxL-n2oT9AhUAAAAAHQAAAAAQBA&utm_campaign=sharing&utm_medium=link&utm_source=htls&ts=CAESCgoCCAMKAggDEAAaIAoCGgASGhIUCgcI5w8QAhgJEgcI5w8QAhgKGAEyAhAAKgkKBToDVVNEGgA'

Next, the URL is passed to the scroll_page(URL) function to start selenium in stealth mode and scroll page. The parser returned by this function is passed to the scrape_google_hotels(parser) function to extract all the data. The explanation of these functions will be in the corresponding headings below.



parser = scroll_page(URL)
google_hotels = scrape_google_hotels(parser)

After the all data is retrieved, it is output in JSON format:



print(json.dumps(google_hotels, indent=2, ensure_ascii=False))

This code uses boilerplate __name__ == "__main__" construct that protects users from accidentally invoking the script when they didn't intend to. This indicates that the code is a runnable script:



def main():
    URL = 'https://www.google.com/travel/hotels?q=hotels%20in%20tulsa%20oklahoma&gsas=1&rp=CgpYAGAAcgQIAhgAOAGoAgA&ved=0CAAQ5JsGahcKEwioxL-n2oT9AhUAAAAAHQAAAAAQBA&utm_campaign=sharing&utm_medium=link&utm_source=htls&ts=CAESCgoCCAMKAggDEAAaIAoCGgASGhIUCgcI5w8QAhgJEgcI5w8QAhgKGAEyAhAAKgkKBToDVVNEGgA'

    parser = scroll_page(URL)
    google_hotels = scrape_google_hotels(parser)

    print(json.dumps(google_hotels, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()

This check will only be performed if the user has run this file. If the user imports this file into another, then the check will not work.

You can watch the video Python Tutorial: if name == 'main' for more details.

Scroll page

The function takes the URL and returns a parser.

First, let's understand how pagination works on the Google Hotels page. Data does not load immediately. If the user needs more data, they will simply scroll the page and site download a small package of data.

In this case, selenium library is used, which allows you to simulate user actions in the browser. For selenium to work, you need to use ChromeDriver, which can be downloaded manually or using code. In our case, the second method is used. To control the start and stop of ChromeDriver, you need to use Service which will install browser binaries under the hood:



service = Service(ChromeDriverManager().install())

You should also add options to work correctly:



options = webdriver.ChromeOptions()
options.add_argument('start-maximized')
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_experimental_option('excludeSwitches', ['enable-automation'])
options.add_experimental_option('useAutomationExtension', False)

Now we can start webdriver:



driver = webdriver.Chrome(service=service, options=options)

After starting the webdriver, you need to pass it and other attributes to the stealth() function. This will make selenium more stealthy to be able to bypass CAPTCHA (Cloudflare one also):



stealth(
    driver,
    user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
    languages=['en-US', 'en'],
    vendor='Google Inc.',
    platform='Win32',
    webgl_vendor='Intel Inc.',
    renderer='Intel Iris OpenGL Engine',
    fix_hairline=True,
)

Pass the URL to the get() method and wait a few seconds for the site to load:



driver.get(url)
time.sleep(2)

In order to load all the data, you need to implement page scrolling. By scrolling down the page, you can notice that new data is not being loaded. In order for the new data to load, you need to scroll up the page a little and wait. Yes, I'm also surprised that it works that way 🙂

The GIF below shows what it looks like:

The page scrolling algorithm looks like this:

Find out the initial page height and write the result to the old_height variable.
Scroll to the bottom of the page using the script.
Simulating a "Page Up" button press, scroll up a little on the page to load new data.
Find out the new page height and write the result to the new_height variable.
If the variables new_height and old_height are equal, then we complete the algorithm, otherwise we write the value of the variable new_height to the variable old_height and return to step 2.

Getting the page height and scroll is done by pasting the JavaScript code into the execute_script() method:



# 1 step
old_height = driver.execute_script("""
    function getHeight() {
        return document.querySelector('.t8aeve').scrollHeight;
    }
    return getHeight();
""")

body = driver.find_element(By.TAG_NAME, 'body')

while True:
    # 2 step
    driver.execute_script("window.scrollTo(0, document.querySelector('.t8aeve').scrollHeight)")
    time.sleep(1)

    # 3 step
    body.send_keys(Keys.PAGE_UP)
    time.sleep(2)

    # 4 step
    new_height = driver.execute_script("""
        function getHeight() {
            return document.querySelector('.t8aeve').scrollHeight;
        }
        return getHeight();
    """)

    # 5 step
    if new_height == old_height:
        break

    old_height = new_height



parser = LexborHTMLParser(driver.page_source)

After all the operations are done, stop the driver and the parser is returned:



driver.quit()

return parser

The function looks like this:



def scroll_page(url):
    service = Service(ChromeDriverManager().install())

    options = webdriver.ChromeOptions()
    options.add_argument('start-maximized')
    options.add_argument('--headless')
    options.add_argument('--no-sandbox')
    options.add_experimental_option('excludeSwitches', ['enable-automation'])
    options.add_experimental_option('useAutomationExtension', False)

    driver = webdriver.Chrome(service=service, options=options)

    stealth(
        driver,
        user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
        languages=['en-US', 'en'],
        vendor='Google Inc.',
        platform='Win32',
        webgl_vendor='Intel Inc.',
        renderer='Intel Iris OpenGL Engine',
        fix_hairline=True,
    )

    driver.get(url)
    time.sleep(2)

    old_height = driver.execute_script("""
        function getHeight() {
            return document.querySelector('.t8aeve').scrollHeight;
        }
        return getHeight();
    """)

    body = driver.find_element(By.TAG_NAME, 'body')

    while True:
        driver.execute_script("window.scrollTo(0, document.querySelector('.t8aeve').scrollHeight)")
        time.sleep(1)

        body.send_keys(Keys.PAGE_UP)
        time.sleep(2)

        new_height = driver.execute_script("""
            function getHeight() {
                return document.querySelector('.t8aeve').scrollHeight;
            }
            return getHeight();
        """)

        if new_height == old_height:
            break

        old_height = new_height

    parser = LexborHTMLParser(driver.page_source)
    driver.quit()

    return parser

Scrape Google Hotels Listings

The function takes a Lexbor parser and returns the extracted data.

The data list is declared to which the extracted data will be added:



data = []

In order to find all the hotels, you need to use the css() method and pass the .uaTTDe selector there. For each hotel, the result_dict dictionary is created, which will be supplemented with all the data about this hotel:



for result in parser.root.css('.uaTTDe'):
    result_dict = {}

First of all, you need to check if this is an advertisement. If so, then the key with the corresponding value is added to the result_dict dictionary:



if result.css_first('.hVE5 .ogfYpf'):
    result_dict['ad'] = result.css_first('.hVE5 .ogfYpf').text().replace('  ', ' ')

Data such as title and link are easily retrieved:



result_dict['title'] = result.css_first('.QT7m7 h2').text()
result_dict['link'] = 'https://www.google.com' + result.css_first('.PVOOXe').attributes.get('href')

Code	Explanation
`css_first()`	to find the desired element.
`text()`	to retrieve the text content.
`attributes`	to get all attributes of an element.

Sometimes it happens that in some hotels there is no such data as price and rating. So a ternary expression is used for such cases:



price = result.css_first('.OxGZuc .kixHKb span')
result_dict['price'] = price.text() if price else None
rating = result.css_first('.FW82K .KFi5wf')
result_dict['rating'] = float(rating.text()) if rating else None

When retrieving reviews the user gets an element like this:



<span class="jdzyld XLC8M" aria-hidden="true"> (1,602)</span>

In order to extract a numeric value, you need to do the following operations: slice a list by removing space and brackets; remove the comma.



reviews = result.css_first('.FW82K .jdzyld')
result_dict['reviews'] = int(reviews.text()[2:-1].replace(',', '')) if reviews else None

Hotels may have a different number of extensions. Sometimes they may not exist at all. Therefore, when extracting them, list comprehension is used:



result_dict['extensions'] = [extension.css_first('.sSHqwe').text() for extension in result.css('.RJM8Kc .HlxIlc div, li')]

When extracting thumbnails, I noticed that the required value is stored in different attributes. Therefore, when extracting a thumbnails, we first check whether there is an attribute src. If this attribute exists, then retrieve its value. Otherwise, we extract the value from the data-src attribute:



result_dict['thumbnails'] = [
    thumbnail.attributes.get('src') if thumbnail.attributes.get('src') else thumbnail.attributes.get('data-src')
    for thumbnail in result.css('.NBZP0e .q5P4L')
]

Add result_dict dictionary with all extracted data to the data list:



data.append(result_dict)

At the end of the function, the data list is returned:



return data

The function looks like this:



def scrape_google_hotels(parser):
    data = []

    for result in parser.root.css('.uaTTDe'):
        result_dict = {}

        if result.css_first('.hVE5 .ogfYpf'):
            result_dict['ad'] = result.css_first('.hVE5 .ogfYpf').text().replace('  ', ' ')

        result_dict['title'] = result.css_first('.QT7m7 h2').text()
        result_dict['link'] = 'https://www.google.com' + result.css_first('.PVOOXe').attributes.get('href')
        price = result.css_first('.OxGZuc .kixHKb span')
        result_dict['price'] = price.text() if price else None
        rating = result.css_first('.FW82K .KFi5wf')
        result_dict['rating'] = float(rating.text()) if rating else None
        reviews = result.css_first('.FW82K .jdzyld')
        result_dict['reviews'] = int(reviews.text()[2:-1].replace(',', '')) if reviews else None
        result_dict['extensions'] = [extension.css_first('.sSHqwe').text() for extension in result.css('.RJM8Kc .HlxIlc div, li')]
        result_dict['thumbnails'] = [
            thumbnail.attributes.get('src') if thumbnail.attributes.get('src') else thumbnail.attributes.get('data-src')
            for thumbnail in result.css('.NBZP0e .q5P4L')
        ]

        data.append(result_dict)

    return data

Output



[
  {
    "ad": "From SpringHill Suites by Marriott Tulsa",
    "title": "SpringHill Suites by Marriott Tulsa",
    "link": "https://www.google.com/travel/hotels/entity/CgoIsq_EvdysoNAlEAEagAFBQUJuQjNtSE1LSm45bGJ0OGFEZ1gtRnhVbnhsVkFtN3Z4cHk3cG5QWmhYX1NCdU9lbHd1V3poM0JFNVpHbGdpeGRXU2xnbUZsSkxUZDJ6eno3MllScFJmOUlodVFPNEFwcDA5TDJMOVFYN0RiVi1CZDZwRkhENmptYURQX0tRbA?q=hotels%20in%20tulsa%20oklahoma&gsas=1&rp=CgpYAGAAcgQIAhgAOAGoAgA&ved=2ahUKEwiPtOao34T9AhUZ1xEIHRoyAsoQyvcEegQIAxAw&utm_campaign=sharing&utm_medium=link&utm_source=htls&ts=CAESCgoCCAMKAggDEAAaIAoCGgASGhIUCgcI5w8QAhgJEgcI5w8QAhgKGAEyAhAAKgkKBToDVVNEGgA&ap=EgRDS2dCMANapwMKBQjwLhAAIgNVQUgqFgoHCOcPEAIYCRIHCOcPEAIYChgBKACwAQBYAWAAcgQIAhgAogESCgkvbS8wMTNrY3YSBVR1bHNhqgEXCgIIEhIDCJsBEgIIaBICCGsSAghuGAGqASwKAggUEgIIPxICCB0SAggbEgMImAESAggwEgIIUhICCE4SAggXEgMIpQIYAaoBBwoDCJwBGACqASMKAggcEgMIlwESAghREgIIWBICCHMSAghHEgIIJBICCE0YAaoBDgoCCCUSAgh5EgIIehgBqgEeCgIIERICCH0SAgg0EgIIQBICCDgSAghXEgIIKxgBqgErCgIILhICCDwSAgg7EgIIVhICCDoSAgg9EgMIgwESAghLEgIIUxICCCgYAaoBBgoCCCwYAKoBCwoDCOECEgIIYxgBqgEKCgIIUBICCBkYAaoBBgoCCAQYAKoBBgoCCAoYAKoBIgoCCDUSAggeEgIIExICCAsSAggxEgIIMhICCF0SAghEGAGSAgIIEJICAggSkgICCBOSAgIIDZICAggRkgICCA-SAgIIDJICAggUkgECIAE",
    "price": "107 USD",
    "rating": 4.0,
    "reviews": 428,
    "extensions": [
      "3-star hotel",
      "Free breakfast",
      "Free Wi-Fi",
      "Free parking",
      "Indoor pool",
      "Air conditioning",
      "Fitness center",
      "Accessible",
      "Business center"
    ],
    "thumbnails": [
      "https://lh6.googleusercontent.com/proxy/npSnFI0zPjhU7MB5eo7Bl4y7FufLgyU8A_fsErIYsi1f0Va_aYbqw5jMKZxc0_ShXxvOVxy-1Dz8DnY4JEyL6ixaUe8j7YfdJdoUGQl4DA8XcB1YB_zmxZxiUaR7RcUlNldKHzKfFfy6p35kX2_ftWvHE2zvREQ=w592-h404-n-k-no-v1",
      "https://lh4.googleusercontent.com/proxy/zOawslbUt33W_gbP_4SeVbDmtYKUkISF1i2By_w7PJSys9TgI7zlGSrYyA8Cc1K-h4iyqyF-s7sxs8GHzWY0_uBqYG_sFqhpfHl5Hss_br0cNF93VlauapN_gCE92QMhzscWvnXF0HDO13K3hRUYQHeXTO7SHlo=w592-h404-n-k-no-v1",
      "https://lh5.googleusercontent.com/proxy/P79WRpA9In1qfrl5tnjRTxY0_UyJhdI2Wp4tIQHSjT47uo0tuN-KKizhTUUMmXnseA3SeootIW1nV7zXbZpnQjuYmvRALMiYtFpoMJDj3VL8xtAY3w4gcsjRVFsbUcxY5WTGYMK0NdF7FC5a4CQvjoFZTpoTIVE=w592-h404-n-k-no-v1",
      "https://lh5.googleusercontent.com/p/AF1QipO99f-u7Cjgv_GSweNP4GQ6jLVAz__pe9KQV-7w=w592-h404-n-k-no-v1",
      "https://lh4.googleusercontent.com/proxy/3UrQRyhdntDx_ak23X4I9L-nSBDckWMXZIjuDWOpxLgWE0fY16sZujVoPhImY5nL-cvCoTlpQf5xMc1JFLgSBZfMARHdAdGxBTXVqCWmw26GcChdBgzC3TYat_zhNtEnexvos_40pdzp1yvPi_Rg3Hy8qfBz2w=w592-h404-n-k-no-v1",
      "https://lh6.googleusercontent.com/proxy/iGLJat0Rz_HP8N-EX0AOG2f28yERdh2hxjJaZCWEgH1Rr0yh7ZIHDBhFBtd9KLyx-n4kw7M-ewIs3JOhfHtyXl1D02Kc8l_WCiTHp0JIkG83PuMwgqb5zmSCkxZ6FkE02HFYSw4d16z6w-Jinwxgn1c1qLcRHMg=w592-h404-n-k-no-v1"
    ]
  },
  {
    "ad": "From Candlewood Suites Tulsa Hills - Jenks, an IHG Hotel",
    "title": "Candlewood Suites Tulsa Hills - Jenks, an IHG Hotel",
    "link": "https://www.google.com/travel/hotels/entity/CgoIqLmTiPOD-rp1EAEae0FBQm5CM25SczRoM2drV1JJYnJ0U0dXcmduZk5mTEl3Sm1fNkpqZEJWODBmeUFhVlQ3T0ZLWEhXMTVOQTRFYnhaSzc4dE1iWlZPYU1sd2l6b3VZZnhjcjVXblZGU1JLc2RkXzFyUkhUZ3A3c1NRWWRGTUxKMGNHN21XWQ?q=hotels%20in%20tulsa%20oklahoma&gsas=1&rp=CgpYAGAAcgQIAhgAOAGoAgA&ved=2ahUKEwiPtOao34T9AhUZ1xEIHRoyAsoQyvcEegQIAxBG&utm_campaign=sharing&utm_medium=link&utm_source=htls&ts=CAESCgoCCAMKAggDEAAaIAoCGgASGhIUCgcI5w8QAhgJEgcI5w8QAhgKGAEyAhAAKgkKBToDVVNEGgA&ap=EgRDS2dCMANapwMKBQjwLhAAIgNVQUgqFgoHCOcPEAIYCRIHCOcPEAIYChgBKACwAQBYAWAAcgQIAhgAogESCgkvbS8wMTNrY3YSBVR1bHNhqgEXCgIIEhIDCJsBEgIIaBICCGsSAghuGAGqASwKAggUEgIIPxICCB0SAggbEgMImAESAggwEgIIUhICCE4SAggXEgMIpQIYAaoBBwoDCJwBGACqASMKAggcEgMIlwESAghREgIIWBICCHMSAghHEgIIJBICCE0YAaoBDgoCCCUSAgh5EgIIehgBqgEeCgIIERICCH0SAgg0EgIIQBICCDgSAghXEgIIKxgBqgErCgIILhICCDwSAgg7EgIIVhICCDoSAgg9EgMIgwESAghLEgIIUxICCCgYAaoBBgoCCCwYAKoBCwoDCOECEgIIYxgBqgEKCgIIUBICCBkYAaoBBgoCCAQYAKoBBgoCCAoYAKoBIgoCCDUSAggeEgIIExICCAsSAggxEgIIMhICCF0SAghEGAGSAgIIEJICAggSkgICCBOSAgIIDZICAggRkgICCA-SAgIIDJICAggUkgECIAE",
    "price": "107 USD",
    "rating": 4.5,
    "reviews": 32,
    "extensions": [
      "2-star hotel",
      "Free Wi-Fi",
      "Free parking",
      "Outdoor pool",
      "Air conditioning",
      "Pet-friendly",
      "Fitness center",
      "Bar",
      "Restaurant"
    ],
    "thumbnails": [
      "https://lh5.googleusercontent.com/p/AF1QipNfsuPQsi_AU4VMrL-1GV6Mg2Xv3UU_GGsX-rJy=w592-h404-n-k-no-v1",
      "https://lh5.googleusercontent.com/p/AF1QipPaGhgG0rfakDzyPsjRwrKVRTb2FJjE97yZxMdH=w592-h404-n-k-no-v1",
      "https://lh4.googleusercontent.com/proxy/Yz8RTylxuiUnuDEQgzUkh9I0EkFdUGBYn7cyHbrG-mD65GGB0YtPAVoC9QWp-9Fa6bRR29Bc3e-INVDBax9h6SVVGwUSF2KHh3nU56NbdFB4LQ5DDho8B5Up-JDZJ4BW1ESoehQQ-cxzX4PSj4JIBDZpu-bjSfU=w592-h404-n-k-no-v1",
      "https://lh5.googleusercontent.com/p/AF1QipO7qih8wFIYOmU7ovsMhi8Hj03XyvDfWQb93OpQ=w592-h404-n-k-no-v1",
      "https://lh5.googleusercontent.com/p/AF1QipP-qzQMCQcxIwdLybDudgA7mOdc2yD-Eut4WnKR=w592-h404-n-k-no-v1",
      "https://lh5.googleusercontent.com/p/AF1QipNdAZiBXI-ZCRAFAx95gL_dZoIiINeiSFj6FQUC=w592-h404-n-k-no-v1"
    ]
  },
  {
    "title": "Hyatt Regency Tulsa Downtown",
    "link": "https://www.google.com/travel/hotels/entity/ChYIgefFrp_A-5tJGgovbS8wcGRkbGtxEAE?q=hotels%20in%20tulsa%20oklahoma&gsas=1&rp=CgpYAGAAcgQIAhgAOAGoAgA&ved=2ahUKEwiPtOao34T9AhUZ1xEIHRoyAsoQyvcEegQIAxBc&utm_campaign=sharing&utm_medium=link&utm_source=htls&ts=CAESCgoCCAMKAggDEAAaIAoCGgASGhIUCgcI5w8QAhgJEgcI5w8QAhgKGAEyAhAAKgkKBToDVVNEGgA&ap=EgRDS2dCMANapwMKBQjwLhAAIgNVQUgqFgoHCOcPEAIYCRIHCOcPEAIYChgBKACwAQBYAWAAcgQIAhgAogESCgkvbS8wMTNrY3YSBVR1bHNhqgEXCgIIEhIDCJsBEgIIaBICCGsSAghuGAGqASwKAggUEgIIPxICCB0SAggbEgMImAESAggwEgIIUhICCE4SAggXEgMIpQIYAaoBBwoDCJwBGACqASMKAggcEgMIlwESAghREgIIWBICCHMSAghHEgIIJBICCE0YAaoBDgoCCCUSAgh5EgIIehgBqgEeCgIIERICCH0SAgg0EgIIQBICCDgSAghXEgIIKxgBqgErCgIILhICCDwSAgg7EgIIVhICCDoSAgg9EgMIgwESAghLEgIIUxICCCgYAaoBBgoCCCwYAKoBCwoDCOECEgIIYxgBqgEKCgIIUBICCBkYAaoBBgoCCAQYAKoBBgoCCAoYAKoBIgoCCDUSAggeEgIIExICCAsSAggxEgIIMhICCF0SAghEGAGSAgIIEJICAggSkgICCBOSAgIIDZICAggRkgICCA-SAgIIDJICAggUkgECIAE",
    "price": "$174",
    "rating": 4.3,
    "reviews": 2755,
    "extensions": [
      "4-star hotel",
      "Breakfast ($)",
      "Free Wi-Fi",
      "Parking ($)",
      "Pools",
      "Air conditioning",
      "Pet-friendly",
      "Fitness center",
      "Spa"
    ],
    "thumbnails": [
      "https://lh5.googleusercontent.com/p/AF1QipPrPKhl9HabLa_NGsOtbimCikCZWsJcDB29icgS=w296-h202-n-k-no-v1",
      "https://lh5.googleusercontent.com/p/AF1QipPBoWJnCSqgupKMQ5hZxeLuMjUm6JvcHBecSVT2=w296-h202-n-k-no-v1",
      "https://lh5.googleusercontent.com/p/AF1QipOsNY1x2kyhAWzE5dxjo4ZrZAg6-pXTscTP675-=w296-h202-n-k-no-v1",
      "https://lh5.googleusercontent.com/p/AF1QipOOPTxY6Mxtyr5A3jS3mJ2kzClrTVtkirjr7vXS=w296-h202-n-k-no-v1",
      "https://lh5.googleusercontent.com/p/AF1QipMCzxsTCjtsWJLu8w4BB-Nh9fh1dedKJ9G2pZXk=w296-h202-n-k-no-v1",
      "https://lh5.googleusercontent.com/p/AF1QipNrQC1BObMJw6w_QHnnmF3M-WOYsh1JmEwW4mX4=w296-h202-n-k-no-v1"
    ]
  },
  ... other results
]

Scrape Google "Things to do" page with Python

Artur Chukhrai — Tue, 14 Feb 2023 19:00:09 +0000

What will be scraped
Full Code
Preparation
Code Explanation
- Top-level code environment
- Start selenium stealth
- Scrape Google "Things to do" page
- Get category data
Output
Links

What will be scraped

Full Code

If you don't need an explanation, have a look at the full code example in the online IDE.

from selenium import webdriver
from selenium_stealth import stealth
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selectolax.lexbor import LexborHTMLParser
import time, json


def start_selenium_stealth(url):    
    service = Service(ChromeDriverManager().install())

    options = webdriver.ChromeOptions()
    options.add_argument('start-maximized')
    options.add_argument('--headless')
    options.add_argument('--no-sandbox')
    options.add_experimental_option('excludeSwitches', ['enable-automation'])
    options.add_experimental_option('useAutomationExtension', False)

    driver = webdriver.Chrome(service=service, options=options)

    stealth(
        driver,
        user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
        languages=['en-US', 'en'],
        vendor='Google Inc.',
        platform='Win32',
        webgl_vendor='Intel Inc.',
        renderer='Intel Iris OpenGL Engine',
        fix_hairline=True,
    )

    driver.get(url)
    time.sleep(2)

    return driver


def scrape_google_things_to_do(driver):
    data = []

    parser = LexborHTMLParser(driver.page_source)

    data.append({
        'category': 'Top sights',
        'results': get_category_data(parser)
    })

    buttons = driver.find_elements(By.CSS_SELECTOR, '.iydyUc button')
    categories = parser.root.css('.m1GHmf')

    for button, category in zip(buttons, categories):
        driver.execute_script('arguments[0].click();', button)
        time.sleep(2)
        parser = LexborHTMLParser(driver.page_source)

        data.append({
            'category': category.text(),
            'results': get_category_data(parser)
        })

    driver.quit()

    return data


def get_category_data(parser):
    category_data = []

    for result in parser.root.css('.T1Yjbc'):
        title = result.css_first('.YmWhbc').text()
        description = result.css_first('.nFoFM').text()
        rating = result.css_first('.lA0BZ')
        rating = float(rating.text()) if rating else None
        reviews = result.css_first('.XLC8M')
        reviews = int(reviews.text()[2:-1].replace(',', '')) if reviews else None
        thumbnail = result.css_first('.R1Ybne').attributes
        thumbnail = thumbnail.get('src') if thumbnail.get('src') else thumbnail.get('data-src')

        category_data.append({
            'title': title,
            'description': description,
            'rating': rating,
            'reviews': reviews,
            'thumbnail': thumbnail
        })

    return category_data


def main():
    URL = 'https://www.google.com/travel/things-to-do/see-all?dest_mid=%2Fm%2F06c62&dest_state_type=sattd&q=Rome'

    driver = start_selenium_stealth(URL)
    google_things_to_do = scrape_google_things_to_do(driver)

    print(json.dumps(google_things_to_do, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()

Preparation

Install libraries:

pip install selenium selenium-stealth webdriver webdriver_manager selectolax

Code Explanation

Import libraries:

from selenium import webdriver
from selenium_stealth import stealth
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selectolax.lexbor import LexborHTMLParser
import time, json

Library	Purpose
`webdriver`	to drive a browser natively, as a user would, either locally or on a remote machine using the Selenium server.
`selenium-stealth`	to prevent almost all website detections (CAPTCHA and so on).
`Service`	to manage the starting and stopping of the ChromeDriver.
`By`	to set of supported locator strategies (By.ID, By.TAG_NAME, By.XPATH etc).
`LexborHTMLParser`	a fast HTML5 parser with CSS selectors using Lexbor engine.
`time`	to work with time in Python.
`json`	to convert extracted data to a JSON object.

The next part of the code is divided into functions. Each function is described in the corresponding heading below.

Top-level code environment

The URL variable contains a link to the Google "Things to do" page:

URL = 'https://www.google.com/travel/things-to-do/see-all?dest_mid=%2Fm%2F06c62&dest_state_type=sattd&q=Rome'

📌Note: This link leads to "What to see" tab, and this blog post focuses only on this link.

There are 3 different tabs on the Google "Things to do" page:

Each tab has its own link:

# Highlights
https://www.google.com/travel/things-to-do?dest_mid=%2Fm%2F06c62&q=Rome&dest_state_type=main
# What to see
https://www.google.com/travel/things-to-do/see-all?dest_mid=%2Fm%2F06c62&q=Rome&dest_state_type=sattd
# What to do
https://www.google.com/travel/things-to-do/experiences?dest_mid=%2Fm%2F06c62&q=Rome&dest_state_type=sae

Next, the URL is passed to the start_selenium_stealth(URL) function to start selenium in stealth mode. The driver returned by this function is passed to the scrape_google_things_to_do(driver) function to extract all the data. The explanation of these functions will be in the corresponding headings below.

driver = start_selenium_stealth(URL)
google_things_to_do = scrape_google_things_to_do(driver)

After the all data is retrieved, it is output in JSON format:

print(json.dumps(google_things_to_do, indent=2, ensure_ascii=False))

This code uses boilerplate __name__ == "__main__" construct that protects users from accidentally invoking the script when they didn't intend to. This indicates that the code is a runnable script:

def main():
    URL = 'https://www.google.com/travel/things-to-do/see-all?dest_mid=%2Fm%2F06c62&dest_state_type=sattd&q=Rome'

    driver = start_selenium_stealth(URL)
    google_things_to_do = scrape_google_things_to_do(driver)

    print(json.dumps(google_things_to_do, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()

This check will only be performed if the user has run this file. If the user imports this file into another, then the check will not work.

You can watch the video Python Tutorial: if name == 'main' for more details.

Start selenium stealth

The function takes the URL and returns a selenium driver.

The selenium library is used to simulate user actions in the browser. For selenium to work, you need to use ChromeDriver, which can be downloaded manually or using code. In our case, the second method is used. To control the start and stop of ChromeDriver, you need to use Service which will install browser binaries under the hood:

service = Service(ChromeDriverManager().install())

You should also add options to work correctly:

options = webdriver.ChromeOptions()
options.add_argument('start-maximized')
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_experimental_option('excludeSwitches', ['enable-automation'])
options.add_experimental_option('useAutomationExtension', False)

Now we can start webdriver:

driver = webdriver.Chrome(service=service, options=options)

After starting the webdriver, you need to pass it and other attributes to the stealth() function. This will make selenium more stealthy to be able to bypass CAPTCHA (Cloudflare one also):

stealth(
    driver,
    user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
    languages=['en-US', 'en'],
    vendor='Google Inc.',
    platform='Win32',
    webgl_vendor='Intel Inc.',
    renderer='Intel Iris OpenGL Engine',
    fix_hairline=True,
)

Pass the URL to the get() method and wait a few seconds for the site to load:

driver.get(url)
time.sleep(2)

At the end of the function, the driver is returned:

return driver

The function looks like this:

def start_selenium_stealth(url):    
    service = Service(ChromeDriverManager().install())

    options = webdriver.ChromeOptions()
    options.add_argument('start-maximized')
    options.add_argument('--headless')
    options.add_argument('--no-sandbox')
    options.add_experimental_option('excludeSwitches', ['enable-automation'])
    options.add_experimental_option('useAutomationExtension', False)

    driver = webdriver.Chrome(service=service, options=options)

    stealth(
        driver,
        user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
        languages=['en-US', 'en'],
        vendor='Google Inc.',
        platform='Win32',
        webgl_vendor='Intel Inc.',
        renderer='Intel Iris OpenGL Engine',
        fix_hairline=True,
    )

    driver.get(url)
    time.sleep(2)

    return driver

Scrape Google "Things to do" page

The function takes a selenium driver and returns the extracted data.

The data list is declared to which the extracted data will be added:

data = []

Passing HTML content to Lexbor:

parser = LexborHTMLParser(driver.page_source)

At the moment, the page displays the top sights. A dictionary with received data from the "Top sights" category is added to the data list. The get_category_data(parser) function returns a list of the extracted places:

data.append({
    'category': 'Top sights',
    'results': get_category_data(parser)
})

After that we get lists from all categories and buttons:

buttons = driver.find_elements(By.CSS_SELECTOR, '.iydyUc button')
categories = parser.root.css('.m1GHmf')

Then we iterate these lists in a loop. In each iteration of the loop, the following algorithm is executed:

Click on the button that is responsible for the special category.
Wait a few seconds for the page to load.
Pass HTML content to Lexbor.
Add a dictionary with extracted data from this category to the data list.

for button, category in zip(buttons, categories):
    driver.execute_script('arguments[0].click();', button)
    time.sleep(2)
    parser = LexborHTMLParser(driver.page_source)

    data.append({
        'category': category.text(),
        'results': get_category_data(parser)
    })

After all the operations are done, stop the driver and return the data list:

driver.quit()

return data

The function looks like this:

def scrape_google_things_to_do(driver):
    data = []

    parser = LexborHTMLParser(driver.page_source)

    data.append({
        'category': 'Top sights',
        'results': get_category_data(parser)
    })

    buttons = driver.find_elements(By.CSS_SELECTOR, '.iydyUc button')
    categories = parser.root.css('.m1GHmf')

    for button, category in zip(buttons, categories):
        driver.execute_script('arguments[0].click();', button)
        time.sleep(2)
        parser = LexborHTMLParser(driver.page_source)

        data.append({
            'category': category.text(),
            'results': get_category_data(parser)
        })

    driver.quit()

    return data

Get category data

The function takes a parser and returns the extracted data by category.

The category_data list is declared to which the extracted data will be added:

category_data = []

The data extraction itself is done with selectolax because it has Lexbor parser which is incredibly fast, like 186% faster compared to bs4 with lxml backend when parsing data with 3000 iterations 5 times.

In terms of syntax, it is very similar to both bs4 and parsel, making it easy to use. Please note that selectolax does not currently support XPath.

In order to find all the places, you need to use the css() method and pass the .T1Yjbc selector there:

for result in parser.root.css('.T1Yjbc'):
    # data extraction will be here

Next, for each place, you need to extract all the data about it. Data such as title and description are easily retrieved. Using the css_first() method, we find the desired elements. For them, the text content is retrieved, so the corresponding text() method is used:

title = result.css_first('.YmWhbc').text()
description = result.css_first('.nFoFM').text()

Sometimes it happens that in some places there are no data such as ratings and reviews. So a ternary expression is used for such cases:

rating = result.css_first('.lA0BZ')
rating = float(rating.text()) if rating else None

When retrieving reviews the user gets an element like this:

<span class="jdzyld XLC8M " aria-hidden="true"> (329,696)</span>

In order to extract a numeric value, you need to do the following operations: slice a list by removing space and brackets; remove the comma.

reviews = result.css_first('.XLC8M')
reviews = int(reviews.text()[2:-1].replace(',', '')) if reviews else None

When extracting thumbnail, I noticed that the required value is stored in different attributes:

Therefore, when extracting a thumbnail, we first check whether there is an attribute src. If this attribute exists, then retrieve its value. Otherwise, we extract the value from the data-src attribute:

thumbnail = result.css_first('.R1Ybne').attributes
thumbnail = thumbnail.get('src') if thumbnail.get('src') else thumbnail.get('data-src')

Add dictionary with all extracted data to the category_data list:

category_data.append({
    'title': title,
    'description': description,
    'rating': rating,
    'reviews': reviews,
    'thumbnail': thumbnail
})

At the end of the function, the category_data list is returned:

return category_data

The function looks like this:

def get_category_data(parser):
    category_data = []

    for result in parser.root.css('.T1Yjbc'):
        title = result.css_first('.YmWhbc').text()
        description = result.css_first('.nFoFM').text()
        rating = result.css_first('.lA0BZ')
        rating = float(rating.text()) if rating else None
        reviews = result.css_first('.XLC8M')
        reviews = int(reviews.text()[2:-1].replace(',', '')) if reviews else None
        thumbnail = result.css_first('.R1Ybne').attributes
        thumbnail = thumbnail.get('src') if thumbnail.get('src') else thumbnail.get('data-src')

        category_data.append({
            'title': title,
            'description': description,
            'rating': rating,
            'reviews': reviews,
            'thumbnail': thumbnail
        })

    return category_data

Output

[
  {
    "category": "Top sights",
    "results": [
      {
        "title": "Colosseum",
        "description": "Iconic ancient Roman gladiatorial arena",
        "rating": 4.7,
        "reviews": 329514,
        "thumbnail": "https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcSqsodn85JD0w-M7U-BpFSwvCcj_XGXejYVMoWU8KyL6KPogCN5Mw_NRY9v3kYN5L3-r1DCz_VdR1YysZta5nO5OQ"
      },
      ... other results
    ]
  },
  {
    "category": "Local favorites",
    "results": [
      {
        "title": "Centrale Montemartini",
        "description": "Sculpture museum in a former power plant",
        "rating": 4.7,
        "reviews": 3622,
        "thumbnail": "https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcTQyIoBqIledLbNakjEisXXQxB6jnUK1AHhyplqqwGDuN7UkG-U_gs6Bvb0K9pu4jkb3F1eb_46XkEGpKnKp-R99Q"
      },
      ... other results
    ]
  },
  {
    "category": "Kid-friendly",
    "results": [
      {
        "title": "Explora - Il Museo dei Bambini di Roma",
        "description": "Interactive museum for kids' learning",
        "rating": 4.4,
        "reviews": 3750,
        "thumbnail": "https://lh5.googleusercontent.com/p/AF1QipODf1pdsNaAGmRDnDzr9oL6JVML7AzxM6Ac7Nfb=w464-h260-n-k-no"
      },
      ... other results
    ]
  },
  {
    "category": "Art & Culture",
    "results": [
      {
        "title": "Trevi Fountain",
        "description": "Iconic 18th-century sculpted fountain",
        "rating": 4.8,
        "reviews": 344287,
        "thumbnail": "https://encrypted-tbn3.gstatic.com/licensed-image?q=tbn:ANd9GcQ3oYnBk3YBSnYMN5lm9LCsAYjKCu8aTUmq9IcagmcIMWUIbcp1P05qBe2ZCPIqJiGIZMrLhDC6vJqDfigCU-RYVQ"
      },
      ... other results
    ]
  },
  {
    "category": "History",
    "results": [
      {
        "title": "St. Peter's Basilica",
        "description": "World’s largest basilica of Christianity",
        "rating": 4.8,
        "reviews": 133218,
        "thumbnail": "https://encrypted-tbn2.gstatic.com/licensed-image?q=tbn:ANd9GcTN7ZkQkBKeCk_3pPWGkP89Nh1HZRsUS1VABgURN4xCbafTl3ykf3hsHzSIgeMfWV2GPjWPBB9USYfBqlIjGrCwzQ"
      },
      ... other results
    ]
  },
  {
    "category": "Outdoors",
    "results": [
      {
        "title": "Colosseum",
        "description": "Iconic ancient Roman gladiatorial arena",
        "rating": 4.7,
        "reviews": 329514,
        "thumbnail": "https://encrypted-tbn2.gstatic.com/licensed-image?q=tbn:ANd9GcT6QDJmOIFe5RKBYZeXMaqSbeby5JGoyKYgCQx_QYR_d5u_9jew-QEyMkHJm-RlphGMyeGNWSrK0mkGiRhapCdgLg"
      },
      ... other results
    ]
  },
  {
    "category": "Museums",
    "results": [
      {
        "title": "Sistine Chapel",
        "description": "Michelangelo's iconic painted ceiling",
        "rating": 4.7,
        "reviews": 57439,
        "thumbnail": "https://encrypted-tbn0.gstatic.com/licensed-image?q=tbn:ANd9GcSS5FraPHJIrRHAjLscN_heiHqyxeYk-SSfdHxBEHWSJccjR_k-3_8yQQvfw9jJ26M3OPuVFlGWi0nxmG1et__Wuw"
      },
      ... other results
    ]
  },
  {
    "category": "Amusement Parks",
    "results": [
      {
        "title": "Cinecittà World",
        "description": "Amusement park with rides & shows",
        "rating": 3.9,
        "reviews": 11920,
        "thumbnail": "https://encrypted-tbn0.gstatic.com/licensed-image?q=tbn:ANd9GcTValH23IxS-mwl6aDBgi1zC_z-o0Oy9xnJlq6KsmiJSlYCLUcrfqoyuWNXAEdan00ykkVO90Ef8wRYX8SMlbtwnQ"
      },
      ... other results
    ]
  },
  {
    "category": "Markets",
    "results": [
      {
        "title": "Porta Portese",
        "description": "Large flea market, open Sundays",
        "rating": 4.2,
        "reviews": 5381,
        "thumbnail": "https://encrypted-tbn1.gstatic.com/licensed-image?q=tbn:ANd9GcRDjn4p2s2qFdoAtywEis1F_05IvFZd7lCDVxU3ICCcGIAvyCv1mdetSjr6izoid1DjbbEgQlRxf2S6utMqqwNe_Q"
      },
      ... other results
    ]
  }
]

Dropshipping Research Tool Demo in Python

Artur Chukhrai — Thu, 09 Feb 2023 16:03:00 +0000

Intro
App Demo
Virtual Environment and Libraries Installation
Full Code
Code Explanation
- Get results from sites
- Compare products
- Create custom table
- Save results to file
Links

Intro

This blog post is created to show you how SerpApi was integrated into this small app with an explanation of how the app functionates and SerpApi's role in it.

The main idea comes from a tool called Zik Analytics, which used by dropshippers.

This demo shows a comparison between Walmart and eBay products to determine profit. The user selects a product and a place of sale, and the program finds the same product in another place and calculates the profit.

App Demo

📌Note: Use the dark Streamlit theme to view the table normally.

Virtual Environment and Libraries Installation

If you want to use your own API key, follow these steps:

Clone repository:

$ git clone https://github.com/chukhraiartur/dropshipping-tool-demo.git

Install dependencies:

$ cd dropshipping-tool-demo && pip install -r requriements.txt

Add SerpApi api key for current shell and all processes started from current shell:

# used to parse Walmart and eBay results, has a plan of 100 free searches
$ export SERPAPI_API_KEY=<your-api-key>

Run the app:

$ streamlit run main.py

Full Code

from serpapi import EbaySearch, WalmartSearch
import streamlit as st
import streamlit.components.v1 as components
import pandas as pd
import time, os, Levenshtein


def get_walmart_results(query: str):
    params = {
        'api_key': os.getenv('SERPAPI_API_KEY'),    # https://serpapi.com/manage-api-key
        'engine': 'walmart',                        # search engine
        'query': query,                             # search query
    }

    search = WalmartSearch(params)                  # data extraction on the SerpApi backend
    results = search.get_dict()                     # JSON -> Python dict

    return results.get('organic_results', [])


def get_ebay_results(query: str):
    params = {
        'api_key': os.getenv('SERPAPI_API_KEY'),    # https://serpapi.com/manage-api-key
        'engine': 'ebay',                           # search engine
        '_nkw': query,                              # search query
        'ebay_domain': 'ebay.com',                  # ebay domain
    }

    search = EbaySearch(params)                     # data extraction on the SerpApi backend
    results = search.get_dict()                     # JSON -> Python dict

    return results.get('organic_results', [])


def compare_walmart_with_ebay(query: str, number_of_products: int, percentage_of_uniqueness: float):
    data = []

    walmart_results = get_walmart_results(query)

    for walmart_result in walmart_results[:number_of_products]:
        ebay_results = get_ebay_results(walmart_result['title'])

        for ebay_result in ebay_results:
            if Levenshtein.ratio(walmart_result['title'], ebay_result['title']) < percentage_of_uniqueness:
                continue

            walmart_price = walmart_result.get('primary_offer', {}).get('offer_price')
            ebay_price = ebay_result.get('price', {}).get('extracted')

            if not ebay_price:
                ebay_price = ebay_result.get('price', {}).get('from', {}).get('extracted')

            profit = 0

            if walmart_price and ebay_price:
                profit = round(walmart_price - ebay_price, 2)

            data.append({
                'Walmart': {
                    'thumbnail': walmart_result['thumbnail'],
                    'title': walmart_result['title'],
                    'link': walmart_result['product_page_url'],
                    'price': walmart_price
                },
                'eBay': {
                    'thumbnail': ebay_result['thumbnail'],
                    'title': ebay_result['title'],
                    'link': ebay_result['link'],
                    'price': ebay_price
                },
                'Profit': profit
            })

    return data


def compare_ebay_with_walmart(query: str, number_of_products: int, percentage_of_uniqueness: float):
    data = []

    ebay_results = get_ebay_results(query)

    for ebay_result in ebay_results[:number_of_products]:
        walmart_results = get_walmart_results(ebay_result['title'])

        for walmart_result in walmart_results:
            if Levenshtein.ratio(ebay_result['title'], walmart_result['title']) < percentage_of_uniqueness:
                continue

            ebay_price = ebay_result.get('price', {}).get('extracted')
            walmart_price = walmart_result.get('primary_offer', {}).get('offer_price')

            if not ebay_price:
                ebay_price = ebay_result.get('price', {}).get('from', {}).get('extracted')

            profit = 0

            if ebay_price and walmart_price:
                profit = round(ebay_price - walmart_price, 2)

            data.append({
                'eBay': {
                    'thumbnail': ebay_result['thumbnail'],
                    'title': ebay_result['title'],
                    'link': ebay_result['link'],
                    'price': ebay_price
                },
                'Walmart': {
                    'thumbnail': walmart_result['thumbnail'],
                    'title': walmart_result['title'],
                    'link': walmart_result['product_page_url'],
                    'price': walmart_price
                },
                'Profit': profit
            })

    return data


def create_table(data: list, where_to_sell: str):
    with open('table_style.css') as file:
        style = file.read()

    products = ''

    for product in data:
        profit_color = 'lime' if product['Profit'] >= 0 else 'red'

        if where_to_sell == 'Walmart':
            products += f'''
            <tr>
                <td><div><img src="{product['Walmart']['thumbnail']}" width="50"></div></td>
                <td><div><a href="{product['Walmart']['link']}" target="_blank">{product['Walmart']['title']}</div></td>
                <td><div>{str(product['Walmart']['price'])}$</div></td>
                <td><div><img src="{product['eBay']['thumbnail']}" width="50"></div></td>
                <td><div><a href="{product['eBay']['link']}" target="_blank">{product['eBay']['title']}</div></td>
                <td><div>{str(product['eBay']['price'])}$</div></td>
                <td><div style="color:{profit_color}">{str(product['Profit'])}$</div></td>
            </tr>
            '''
        elif where_to_sell == 'eBay':
            products += f'''
            <tr>
                <td><div><img src="{product['eBay']['thumbnail']}" width="50"></div></td>
                <td><div><a href="{product['eBay']['link']}" target="_blank">{product['eBay']['title']}</div></td>
                <td><div>{str(product['eBay']['price'])}$</div></td>
                <td><div><img src="{product['Walmart']['thumbnail']}" width="50"></div></td>
                <td><div><a href="{product['Walmart']['link']}" target="_blank">{product['Walmart']['title']}</div></td>
                <td><div>{str(product['Walmart']['price'])}$</div></td>
                <td><div style="color:{profit_color}">{str(product['Profit'])}$</div></td>
            </tr>
            '''

    table = f'''
    <style>
        {style}
    </style>
    <table border="1">
        <thead>
            <tr>
                <th colspan="3"><div>{list(data[0].keys())[0]}</div></th>
                <th colspan="3"><div>{list(data[0].keys())[1]}</div></th>
                <th><div>{list(data[0].keys())[2]}</div></th>
            </tr>
        </thead>
        <tbody>{products}</tbody>
    </table>
    '''

    return table


def save_to_json(data: list):
    json_file = pd.DataFrame(data=data).to_json(index=False, orient='table')

    st.download_button(
        label='Download JSON',
        file_name='comparison-results.json',
        mime='application/json',
        data=json_file,
    )


def save_to_csv(data: list):
    csv_file = pd.DataFrame(data=data).to_csv(index=False)

    st.download_button(
        label="Download CSV",
        file_name='comparison-results.csv',
        mime='text/csv',
        data=csv_file
    )


def main():
    st.title('💸Product Comparison')
    st.markdown(body='This demo compares products from Walmart and eBay to find a profit. SerpApi Demo Project ([repository](https://github.com/chukhraiartur/dropshipping-tool-demo)). Made with [Streamlit](https://streamlit.io/) and [SerpApi](http://serpapi.com/) 🧡')

    if 'visibility' not in st.session_state:
        st.session_state.visibility = 'visible'
        st.session_state.disabled = False

    SEARCH_QUERY: str = st.text_input(
        label='Search query',
        placeholder='Search',
        help='Multiple search queries is not supported.'
    )
    WHERE_TO_SELL = st.selectbox(
        label='Where to sell',
        options=('Walmart', 'eBay'),
        help='Select the platform where you want to sell products. The program will look for the same products on another site and calculate the profit.'
    )
    NUMBER_OF_PRODUCTS: int = st.slider(
        label='Number of products to search',
        min_value=1,
        max_value=20,
        value=10,
        help='Limit the number of products to analyze.'
    )
    PERCENTAGE_OF_UNIQUENESS: int = st.slider(
        label='Percentage of uniqueness',
        min_value=1,
        max_value=100,
        value=50,
        help='The percentage of uniqueness is used to compare how similar one title is to another. The higher this parameter, the more accurate the result.'
    )
    SAVE_OPTION = st.selectbox(
        label='Choose file format to save',
        options=(None, 'JSON', 'CSV'),
        help='By default data won\'t be saved. Choose JSON or CSV format if you want to save the results.'
    )

    col1, col2, col3, col4, col5 = st.columns(5)

    with col3:
        submit_button_holder = st.empty()
        submit_search = submit_button_holder.button(label='Compare products')

    if submit_search and not SEARCH_QUERY:
        st.error(body='Looks like you click a button without a search query. Please enter a search query 👆')
        st.stop()

    if submit_search and SEARCH_QUERY and WHERE_TO_SELL:
        with st.spinner(text='Parsing Product Data...'):
            comparison_results = []

            if WHERE_TO_SELL == 'Walmart':
                comparison_results = compare_walmart_with_ebay(SEARCH_QUERY, NUMBER_OF_PRODUCTS, PERCENTAGE_OF_UNIQUENESS/100)
            elif WHERE_TO_SELL == 'eBay':
                comparison_results = compare_ebay_with_walmart(SEARCH_QUERY, NUMBER_OF_PRODUCTS, PERCENTAGE_OF_UNIQUENESS/100)

        parsing_is_success = st.success('Done parsing 🎉')
        time.sleep(1)
        parsing_is_success.empty()
        submit_button_holder.empty()

        comparison_results_header = st.markdown(body='#### Comparison results')

        if comparison_results:
            table = create_table(comparison_results, WHERE_TO_SELL)
            components.html(table, height=len(comparison_results)*62 + 40)
            time.sleep(1)

        with col3:
            start_over_button_holder = st.empty()
            start_over_button = st.button(label='Start over')  # centered button

        if SAVE_OPTION and comparison_results:
            with st.spinner(text=f'Saving data to {SAVE_OPTION}...'):
                if SAVE_OPTION == 'JSON':
                    save_to_json(comparison_results)
                elif SAVE_OPTION == 'CSV':
                    save_to_csv(comparison_results)

            saving_is_success = st.success('Done saving 🎉')

            time.sleep(1)
            saving_is_success.empty()
            submit_button_holder.empty()

            start_over_info_holder = st.empty()
            start_over_info_holder.error(body='To rerun the script, click on the "Start over" button, or refresh the page.')

            if start_over_button:
                comparison_results_header.empty()
                start_over_button_holder.empty()
                start_over_info_holder.empty()

        if SAVE_OPTION and not comparison_results:
            comparison_results_header.empty()

            no_data_holder = st.empty()
            no_data_holder.error(body='No product found. Click "Start Over" button and try different search query.')

            if start_over_button:
                no_data_holder.empty()
                start_over_button_holder.empty()

        if SAVE_OPTION is None and comparison_results:
            start_over_info_holder = st.empty()
            start_over_info_holder.error(body='To rerun the script, click on the "Start over" button, or refresh the page.')

            if start_over_button:
                comparison_results_header.empty()
                start_over_button_holder.empty()
                start_over_info_holder.empty()

        if SAVE_OPTION is None and not comparison_results:
            comparison_results_header.empty()

            no_data_holder = st.empty()
            no_data_holder.error(body='No product found. Click "Start Over" button and try different search query.')

            if start_over_button:
                comparison_results_header.empty()
                no_data_holder.empty()
                start_over_button_holder.empty()


if __name__ == '__main__':
    main()

Code Explanation

First, let's look at how the algorithm works. The items in this algorithm are functions that will be described in the respective headings:

Import libraries:

from serpapi import EbaySearch, WalmartSearch
import streamlit as st
import streamlit.components.v1 as components
import pandas as pd
import time, os, Levenshtein

Library	Purpose
`serpapi`	SerpApi's Python API wrapper that parses data from 15+ search engines.
`streamlit`	to create beautiful web apps.
`streamlit.components.v1`	to find complex or just not available components by default on streamlit.
`pandas`	to convert data to file format.
`time`	to work with time in Python.
`os`	to read secret environment variable. In this case it's SerpApi API key.
`Levenshtein`	to quickly calculate string similarity.

After a bunch of imports, we define a main function where everything happens. At the beginning of this function, the title and description of the application were added:

def main():
    st.title('💸Product Comparison')
    st.markdown(body='This demo compares products from Walmart and eBay to find a profit. SerpApi Demo Project ([repository](https://github.com/chukhraiartur/dropshipping-tool-demo)). Made with [Streamlit](https://streamlit.io/) and [SerpApi](http://serpapi.com/) 🧡')

Next is to define streamlit session state. I've used this to hide or unhide certain widgets:

if 'visibility' not in st.session_state:
    st.session_state.visibility = 'visible'
    st.session_state.disabled = False

After that, I defined an input field, two sliders and two select boxes:

SEARCH_QUERY: str = st.text_input(
    label='Search query',
    placeholder='Search',
    help='Multiple search queries is not supported.'
)
WHERE_TO_SELL = st.selectbox(
    label='Where to sell',
    options=('Walmart', 'eBay'),
    help='Select the platform where you want to sell products. The program will look for the same products on another site and calculate the profit.'
)
NUMBER_OF_PRODUCTS: int = st.slider(
    label='Number of products to search',
    min_value=1,
    max_value=20,
    value=10,
    help='Limit the number of products to analyze.'
)
PERCENTAGE_OF_UNIQUENESS: int = st.slider(
    label='Percentage of uniqueness',
    min_value=1,
    max_value=100,
    value=50,
    help='The percentage of uniqueness is used to compare how similar one title is to another. The higher this parameter, the more accurate the result.'
)
SAVE_OPTION = st.selectbox(
    label='Choose file format to save',
    options=(None, 'JSON', 'CSV'),
    help='By default data won\'t be saved. Choose JSON or CSV format if you want to save the results.'
)

Here I was creating a centered button:

col1, col2, col3, col4, col5 = st.columns(5)

with col3:
    submit_button_holder = st.empty()
    submit_search = submit_button_holder.button(label='Compare products')

if submit_search and not SEARCH_QUERY:
    st.error(body='Looks like you click a button without a search query. Please enter a search query 👆')
    st.stop()

submit_button_holder is used to hide or unhide widget.
st.stop is used to stop the script if user didn't provide any search query.

If the user enters a search query, a place to sell and clicks the "Compare Products" button, then the function for comparing products is launched. Depending on the place to sale, different functions are triggered, but they are similar:

if submit_search and SEARCH_QUERY and WHERE_TO_SELL:
    with st.spinner(text='Parsing Product Data...'):
        comparison_results = []

        if WHERE_TO_SELL == 'Walmart':
            comparison_results = compare_walmart_with_ebay(SEARCH_QUERY, NUMBER_OF_PRODUCTS, PERCENTAGE_OF_UNIQUENESS/100)
        elif WHERE_TO_SELL == 'eBay':
            comparison_results = compare_ebay_with_walmart(SEARCH_QUERY, NUMBER_OF_PRODUCTS, PERCENTAGE_OF_UNIQUENESS/100)

    parsing_is_success = st.success('Done parsing 🎉')
    time.sleep(1)
    parsing_is_success.empty()
    submit_button_holder.empty()

If matches were found, then the results must be shown. The standard streamlit table is not suitable for my purposes, so I am creating a custom table:

comparison_results_header = st.markdown(body='#### Comparison results')

if comparison_results:
    table = create_table(comparison_results, WHERE_TO_SELL)

To display the table, I used the components.html() method. In addition to the passed table, this method accepts the height of the rendered content. The table height value depends on the number of matches found multiplied by the height of each table row len(comparison_results)*62 plus the table header height value 40. Thus, the entire table is successfully displayed:

components.html(table, height=len(comparison_results)*62 + 40)

After the data has been displayed, the "Compare Products" button changes to the "Start over" button:

with col3:
    start_over_button_holder = st.empty()
    start_over_button = st.button(label='Start over')

Then I check the save option and comparison results for their presence:

if SAVE_OPTION and comparison_results:
    with st.spinner(text=f'Saving data to {SAVE_OPTION}...'):
        if SAVE_OPTION == 'JSON':
            save_to_json(comparison_results)
        elif SAVE_OPTION == 'CSV':
            save_to_csv(comparison_results)

    saving_is_success = st.success('Done saving 🎉')

    time.sleep(1)
    saving_is_success.empty()
    submit_button_holder.empty()

    start_over_info_holder = st.empty()
    start_over_info_holder.error(body='To rerun the script, click on the "Start over" button, or refresh the page.')

    if start_over_button:
        comparison_results_header.empty()
        start_over_button_holder.empty()
        start_over_info_holder.empty()

if SAVE_OPTION and not comparison_results:
    comparison_results_header.empty()

    no_data_holder = st.empty()
    no_data_holder.error(body='No product found. Click "Start Over" button and try different search query.')

    if start_over_button:
        no_data_holder.empty()
        start_over_button_holder.empty()

if SAVE_OPTION is None and comparison_results:
    start_over_info_holder = st.empty()
    start_over_info_holder.error(body='To rerun the script, click on the "Start over" button, or refresh the page.')

    if start_over_button:
        comparison_results_header.empty()
        start_over_button_holder.empty()
        start_over_info_holder.empty()

if SAVE_OPTION is None and not comparison_results:
    comparison_results_header.empty()

    no_data_holder = st.empty()
    no_data_holder.error(body='No product found. Click "Start Over" button and try different search query.')

    if start_over_button:
        comparison_results_header.empty()
        no_data_holder.empty()
        start_over_button_holder.empty()

You may have noticed that in each of the checks there is an additional check for the "Start Over" button. This is necessary to remove unnecessary information for a new comparison.

In the end I've added a if __name__ == '__main__' idiom which protects users from accidentally invoking the script when they didn't intend to, and call the main function which will run the whole script:

if __name__ == '__main__':
    main()

Get results from sites

In this application, data is retrieved from Walmart and eBay. I will analyze how the function for extracting data from Walmart works. The function for extracting data from eBay works in a similar way.

The parameters are defined for generating the URL:

def get_walmart_results(query: str):
    params = {
        'api_key': os.getenv('SERPAPI_API_KEY'),    # https://serpapi.com/manage-api-key
        'engine': 'walmart',                        # search engine
        'query': query,                             # search query
    }

Parameters	Explanation
`api_key`	Parameter defines the SerpApi private key to use. You can find it under your account -> API key
`engine`	Set parameter to `walmart` to use the Walmart API engine.
`query`	Parameter defines the search query. You can use anything that you would use in a regular Walmart search.

Then, we create a search object where the data is retrieved from the SerpApi backend. In the results dictionary we get data from JSON:

search = WalmartSearch(params)      # data extraction on the SerpApi backend
results = search.get_dict()         # JSON -> Python dict

At the end of the function, you need to return information about the products. If no such products were found, then an empty list is returned:

return results.get('organic_results', [])

Compare products

If the user has chosen Walmart as a place to sell, then the function of comparing Walmart with eBay is launched. Also works in reverse.

At the beginning of the function, a list is declared to which data will be added:

def compare_walmart_with_ebay(query: str, number_of_products: int, percentage_of_uniqueness: float):
    data = []

For a search query, we get results from Walmart. After that, for each product name from Walmart, we get results from eBay:

walmart_results = get_walmart_results(query)

for walmart_result in walmart_results[:number_of_products]:
    ebay_results = get_ebay_results(walmart_result['title'])

Every item from Walmart is compared to every match found on eBay. Comparison of goods is implemented through the title. To get how much one string is similar to another string, I used the Levenshtein.ratio() method. If the percentage of text matching in titles is less than the percentage of uniqueness, then go to the next product:

for ebay_result in ebay_results:
    if Levenshtein.ratio(walmart_result['title'], ebay_result['title']) < percentage_of_uniqueness:
        continue

If a match is found, the Walmart and eBay price of that item is retrieved. Then profit is calculated:

walmart_price = walmart_result.get('primary_offer', {}).get('offer_price')
ebay_price = ebay_result.get('price', {}).get('extracted')

if not ebay_price:
    ebay_price = ebay_result.get('price', {}).get('from', {}).get('extracted')

profit = 0

if walmart_price and ebay_price:
    profit = round(walmart_price - ebay_price, 2)

The data about the found match is added to the list:

data.append({
    'Walmart': {
        'thumbnail': walmart_result['thumbnail'],
        'title': walmart_result['title'],
        'link': walmart_result['product_page_url'],
        'price': walmart_price
    },
    'eBay': {
        'thumbnail': ebay_result['thumbnail'],
        'title': ebay_result['title'],
        'link': ebay_result['link'],
        'price': ebay_price
    },
    'Profit': profit
})

At the end of the function, a list of all matches is returned:

return data

Create custom table

This function generates a table based on the passed data.

At the beginning of the function, the style sheet for the table is included:

def create_table(data: list, where_to_sell: str):
    with open('table_style.css') as file:
        style = file.read()

Next, each row of the table is formed. The first column will contain data about the place of sale selected by the user. Also, the color of the text in the profit will change depending on the value:

products = ''

for product in data:
    profit_color = 'lime' if product['Profit'] >= 0 else 'red'

    if where_to_sell == 'Walmart':
        products += f'''
        <tr>
            <td><div><img src="{product['Walmart']['thumbnail']}" width="50"></div></td>
            <td><div><a href="{product['Walmart']['link']}" target="_blank">{product['Walmart']['title']}</div></td>
            <td><div>{str(product['Walmart']['price'])}$</div></td>
            <td><div><img src="{product['eBay']['thumbnail']}" width="50"></div></td>
            <td><div><a href="{product['eBay']['link']}" target="_blank">{product['eBay']['title']}</div></td>
            <td><div>{str(product['eBay']['price'])}$</div></td>
            <td><div style="color:{profit_color}">{str(product['Profit'])}$</div></td>
        </tr>
        '''
    elif where_to_sell == 'eBay':
        products += f'''
        <tr>
            <td><div><img src="{product['eBay']['thumbnail']}" width="50"></div></td>
            <td><div><a href="{product['eBay']['link']}" target="_blank">{product['eBay']['title']}</div></td>
            <td><div>{str(product['eBay']['price'])}$</div></td>
            <td><div><img src="{product['Walmart']['thumbnail']}" width="50"></div></td>
            <td><div><a href="{product['Walmart']['link']}" target="_blank">{product['Walmart']['title']}</div></td>
            <td><div>{str(product['Walmart']['price'])}$</div></td>
            <td><div style="color:{profit_color}">{str(product['Profit'])}$</div></td>
        </tr>
        '''

Now the table structure is formed, after which it is returned by this function:

table = f'''
<style>
    {style}
</style>
<table border="1">
    <thead>
        <tr>
            <th colspan="3"><div>{list(data[0].keys())[0]}</div></th>
            <th colspan="3"><div>{list(data[0].keys())[1]}</div></th>
            <th><div>{list(data[0].keys())[2]}</div></th>
        </tr>
    </thead>
    <tbody>{products}</tbody>
</table>
'''

return table

Save results to file

This function converts a list of matches into data corresponding to a JSON or CSV format.

To convert data to JSON format, you need to create a DataFrame object based on the passed data, and then call the to_json() method. The st.download_button() method is used to create a download button:

def save_to_json(data: list):
    json_file = pd.DataFrame(data=data).to_json(index=False, orient='table')

    st.download_button(
        label='Download JSON',
        file_name='comparison-results.json',
        mime='application/json',
        data=json_file,
    )

The process of converting data to CSV format is different in that you need to use the to_csv() method.

def save_to_csv(data: list):
    csv_file = pd.DataFrame(data=data).to_csv(index=False)

    st.download_button(
        label="Download CSV",
        file_name='comparison-results.csv',
        mime='text/csv',
        data=csv_file
    )

Scraping Apple App Store Product Info And Reviews with Python

Artur Chukhrai — Sun, 05 Feb 2023 13:12:22 +0000

What will be scraped
Why using API?
Full Code
Preparation
Code Explanation
- Top-level code environment
- Get product information
- Get product reviews
Output
Links

What will be scraped

Why using API?

There're a couple of reasons that may use API, ours in particular:

No need to create a parser from scratch and maintain it.
Bypass blocks from Google: solve CAPTCHA or solve IP blocks.
Pay for proxies, and CAPTCHA solvers.
Don't need to use browser automation.

SerpApi handles everything on the backend with fast response times under ~2.5 seconds (~1.2 seconds with Ludicrous speed) per request and without browser automation, which becomes much faster. Response times and status rates are shown under SerpApi Status page:

Apple Product:

Apple Reviews:

Head to the Apple Product Page playground and Apple App Store Reviews playground for a live and interactive demo.

Full Code

If you don't need an explanation, have a look at the full code example in the online IDE.

from serpapi import GoogleSearch
import json


def get_product_info(product_id):
    params = {
        'api_key': '...',               # https://serpapi.com/manage-api-key
        'engine': 'apple_product',      # SerpApi search engine 
        'product_id': product_id,       # ID of a product
        'type': 'app',                  # type of Apple Product
        'country': 'us',                # country for the search
    }

    search = GoogleSearch(params)       # data extraction on the SerpApi backend
    product_info = search.get_dict()    # JSON -> Python dict

    del product_info['search_metadata']
    del product_info['search_parameters']
    del product_info['search_information']

    return product_info


def get_product_reviews(product_id):
    params = {
        'api_key': '...',               # https://serpapi.com/manage-api-key
        'engine': 'apple_reviews',      # SerpApi search engine 
        'product_id': product_id,       # ID of a product
        'country': 'us',                # country for the search
        'sort': 'mostrecent',           # sorting reviews
        'page': 1,                      # pagination
    }

    product_reviews = []

    while True:
        search = GoogleSearch(params)
        new_page_results = search.get_dict()

        product_reviews.extend(new_page_results['reviews'])

        if 'next' in new_page_results.get('serpapi_pagination', {}):
            params['page'] += 1
        else:
            break

    return product_reviews


def main():
    product_id = 1507782672

    app_store_results = {
        'product_info': get_product_info(product_id),
        'product_reviews': get_product_reviews(product_id)
    }

    print(json.dumps(app_store_results, indent=2, ensure_ascii=False))


if __name__ == '__main__':
    main()

Preparation

Install library:

pip install google-search-results

google-search-results is a SerpApi API package.

Code Explanation

Import libraries:

from serpapi import GoogleSearch
import json

Library	Purpose
`GoogleSearch`	to scrape and parse Google results using SerpApi web scraping library.
`json`	to convert extracted data to a JSON object.

Top-level code environment

At the beginning of the function, the product_id variable is created that stores the ID of the desired product:

product_id = 1507782672

Next, the app_store_results dictionary is created, to which the data returned by the get_product_info(product_id) and get_product_reviews(product_id) functions are added. The explanation of these functions will be in the corresponding headings below.

app_store_results = {
    'product_info': get_product_info(product_id),
    'product_reviews': get_product_reviews(product_id)
}

After the all data is retrieved, it is output in JSON format:

print(json.dumps(app_store_results, indent=2, ensure_ascii=False))

This code uses the generally accepted rule of using the __name__ == "__main__" construct:

def main():
    product_id = 1507782672

    app_store_results = {
        'product_info': get_product_info(product_id),
        'product_reviews': get_product_reviews(product_id)
    }

    print(json.dumps(app_store_results, indent=2, ensure_ascii=False))


if __name__ == '__main__':
    main()

This check will only be performed if the user has run this file. If the user imports this file into another, then the check will not work.

You can watch the video Python Tutorial: if name == 'main' for more details.

Get product information

The function takes a specific product_id and returns a dictionary with all the information about that product.

At the beginning of the function, the params dictionary are defined for generating the URL:

params = {
    'api_key': '...',               # https://serpapi.com/manage-api-key
    'engine': 'apple_product',      # SerpApi search engine 
    'product_id': product_id,       # ID of a product
    'type': 'app',                  # type of Apple Product
    'country': 'us',                # country for the search
}

Parameters	Explanation
`api_key`	Parameter defines the SerpApi private key to use. You can find it under your account -> API key.
`engine`	Set parameter to `apple_product` to use the Apple Product engine.
`product_id`	Parameter defines the product id you want to search. You can use the specific id of a product that you would like to get the product page of.
`type`	Parameter defines the type of Apple Product to get the product page of. It defaults to `app`.
`country`	Parameter defines the country to use for the search. It's a two-letter country code. Head to the Apple Regions for a full list of supported Apple Regions.

📌Note: You can also add other API Parameters.

Then, we create a search object where the data is retrieved from the SerpApi backend. In the product_info dictionary we get data from JSON:

search = GoogleSearch(params)       # data extraction on the SerpApi backend
product_info = search.get_dict()    # JSON -> Python dict

The product_info dictionary contains information not only about the product, but also about the request. Request information is not needed, so we remove the corresponding keys using the del statement:

del product_info['search_metadata']
del product_info['search_parameters']
del product_info['search_information']

At the end of the function, the product_info dictionary with the extracted data is returned:

return product_info

The complete function to get product information would look like this:

def get_product_info(product_id):
    params = {
        'api_key': '...',               # https://serpapi.com/manage-api-key
        'engine': 'apple_product',      # SerpApi search engine 
        'product_id': product_id,       # ID of a product
        'type': 'app',                  # type of Apple Product
        'country': 'us',                # country for the search
    }

    search = GoogleSearch(params)       # data extraction on the SerpApi backend
    product_info = search.get_dict()    # JSON -> Python dict

    del product_info['search_metadata']
    del product_info['search_parameters']
    del product_info['search_information']

    return product_info

Get product reviews

The function takes a specific product_id and returns a dictionary with all the reviews about that product.

At the beginning of the function, the params dictionary are defined for generating the URL:

params = {
    'api_key': '...',               # https://serpapi.com/manage-api-key
    'engine': 'apple_reviews',      # SerpApi search engine 
    'product_id': product_id,       # ID of a product
    'country': 'us',                # country for the search
    'sort': 'mostrecent',           # sorting reviews
    'page': 1,                      # pagination
}

Parameters	Explanation
`api_key`	Parameter defines the SerpApi private key to use. You can find it under your account -> API key.
`engine`	Set parameter to `apple_reviews` to use the Apple Reviews engine.
`product_id`	Parameter defines the ID of a product you want to get the reviews for.
`country`	Parameter defines the country to use for the search. It's a two-letter country code. Head to the Apple Regions for a full list of supported Apple Regions.
`sort`	Parameter is used for sorting reviews. It can be set to `mostrecent` or `mosthelpful`.
`page`	Parameter is used to get the items on a specific page. (e.g., `1` (default) is the first page of results, `2` is the 2nd page of results, `3` is the 3rd page of results, etc.).

📌Note: You can also add other API Parameters.

Define the product_reviews list to which the retrieved reviews will be added:

product_reviews = []

The while loop is created that is needed to extract reviews from all pages:

while True:
    # data extraction will be here

Then, we create a search object where the data is retrieved from the SerpApi backend. In the new_page_results dictionary we get data from JSON:

search = GoogleSearch(params)
new_page_results = search.get_dict()

Adding new data from this page to the product_reviews list:

product_reviews.extend(new_page_results['reviews'])

# first_review = new_page_results['reviews'][0]
# title = first_review['title']
# text = first_review['text']
# rating = first_review['rating']
# review_date = first_review['review_date']
# author_name = first_review['author']['name']
# author_link = first_review['author']['link']

📌Note: In the comments above, I showed how to extract specific fields. You may have noticed the new_page_results['reviews'][0]. This is the index of a review, which means that we are extracting data from the first review. The new_page_results['reviews'][1] is from the second review and so on.

After data is retrieved from the current page, a check is made to see if the next page exists. If there is one in the serpapi_pagination dictionary, then the page parameter is incremented by 1. Else, the loop stops:

if 'next' in new_page_results.get('serpapi_pagination', {}):
    params['page'] += 1
else:
    break

At the end of the function, the product_reviews dictionary with the extracted data is returned:

return product_reviews

The complete function to get product reviews would look like this:

def get_product_reviews(product_id):
    params = {
        'api_key': '...',               # https://serpapi.com/manage-api-key
        'engine': 'apple_reviews',      # SerpApi search engine 
        'product_id': product_id,       # ID of a product
        'country': 'us',                # country for the search
        'sort': 'mostrecent',           # sorting reviews
        'page': 1,                      # pagination
    }

    product_reviews = []

    while True:
        search = GoogleSearch(params)
        new_page_results = search.get_dict()

        product_reviews.extend(new_page_results['reviews'])

        if 'next' in new_page_results.get('serpapi_pagination', {}):
            params['page'] += 1
        else:
            break

    return product_reviews

Output

{
  "product_info": {
    "title": "Pixea",
    "snippet": "The invisible image viewer",
    "id": "1507782672",
    "age_rating": "4+",
    "developer": {
      "name": "ImageTasks Inc",
      "link": "https://apps.apple.com/us/developer/imagetasks-inc/id450316587"
    },
    "rating": 4.6,
    "rating_count": "620 Ratings",
    "price": "Free",
    "in_app_purchases": "Offers In-App Purchases",
    "logo": "https://is3-ssl.mzstatic.com/image/thumb/Purple118/v4/f6/93/b6/f693b68f-9b14-3689-7521-c19a83fb0d88/AppIcon-1x_U007emarketing-85-220-6.png/320x0w.webp",
    "mac_screenshots": [
      "https://is4-ssl.mzstatic.com/image/thumb/Purple113/v4/e0/21/86/e021868d-b43b-0a78-8d4a-e4e0097a1d01/0131f1c2-3227-46bf-8328-7b147d2b1ea2_Pixea-1.jpg/643x0w.webp",
      "https://is4-ssl.mzstatic.com/image/thumb/Purple113/v4/55/3c/98/553c982d-de30-58b5-3b5a-d6b3b2b6c810/a0424c4d-4346-40e6-8cde-bc79ce690040_Pixea-2.jpg/643x0w.webp",
      "https://is3-ssl.mzstatic.com/image/thumb/Purple123/v4/77/d7/d8/77d7d8c1-4b4c-ba4b-4dde-94bdc59dfb71/6e66509c-5886-45e9-9e96-25154a22fd53_Pixea-3.jpg/643x0w.webp",
      "https://is3-ssl.mzstatic.com/image/thumb/PurpleSource113/v4/44/79/91/447991e0-518f-48b3-bb7e-c7121eb57ba4/79be2791-5b93-4c4d-b4d1-38a3599c2b2d_Pixea-4.jpg/643x0w.webp"
    ],
    "description": "Pixea is an image viewer for macOS with a nice minimal modern user interface. Pixea works great with JPEG, HEIC, PSD, RAW, WEBP, PNG, GIF, and many other formats. Provides basic image processing, including flip and rotate, shows a color histogram, EXIF, and other information. Supports keyboard shortcuts and trackpad gestures. Shows images inside archives, without extracting them.Supported formats:JPEG, HEIC, GIF, PNG, TIFF, Photoshop (PSD), BMP, Fax images, macOS and Windows icons, Radiance images, Google's WebP. RAW formats: Leica DNG and RAW, Sony ARW, Olympus ORF, Minolta MRW, Nikon NEF, Fuji RAF, Canon CR2 and CRW, Hasselblad 3FR. Sketch files (preview only). ZIP-archives.Export formats:JPEG, JPEG-2000, PNG, TIFF, BMP.Found a bug? Have a suggestion? Please, send it to support@imagetasks.comFollow us on Twitter @imagetasks!",
    "version_history": [
      {
        "release_version": "2.1",
        "release_notes": "- New \"Fixed Size and Position\" zoom mode- Fixed a bug causing crash when browsing ZIP-files- Bug fixes and improvements",
        "release_date": "2023-01-03"
      },
      ... other versions
    ],
    "ratings_and_reviews": {
      "rating_percentage": {
        "5_star": "76%",
        "4_star": "13%",
        "3_star": "4%",
        "2_star": "2%",
        "1_star": "4%"
      },
      "review_examples": [
        {
          "rating": "5 out of 5",
          "username": "MyrtleBlink182",
          "review_date": "01/18/2022",
          "review_title": "Full-Screen Perfection",
          "review_text": "This photo-viewer is by far the best in the biz. I thoroughly enjoy viewing photos with it. I tried a couple of others out, but this one is exactly what I was looking for. There is no dead space or any extra design baggage when viewing photos. Pixea knocks it out of the park keeping the design minimalistic while ensuring the functionality is through the roof"
        },
        ... other reviews examples
      ]
    },
    "privacy": {
      "description": "The developer, ImageTasks Inc, indicated that the app’s privacy practices may include handling of data as described below. For more information, see the developer’s privacy policy.",
      "privacy_policy_link": "https://www.imagetasks.com/Pixea-policy.txt",
      "cards": [
        {
          "title": "Data Not Collected",
          "description": "The developer does not collect any data from this app."
        }
      ],
      "sidenote": "Privacy practices may vary, for example, based on the features you use or your age. Learn More",
      "learn_more_link": "https://apps.apple.com/story/id1538632801"
    },
    "information": {
      "seller": "ImageTasks Inc",
      "price": "Free",
      "size": "7.1 MB",
      "categories": [
        "Photo & Video"
      ],
      "compatibility": [
        {
          "device": "Mac",
          "requirement": "Requires macOS 10.12 or later."
        }
      ],
      "supported_languages": [
        "English"
      ],
      "age_rating": {
        "rating": "4+"
      },
      "copyright": "Copyright © 2020-2023 ImageTasks Inc. All rights reserved.",
      "in_app_purchases": [
        {
          "name": "Upgrade to Pixea Plus",
          "price": "$3.99"
        }
      ],
      "developer_website": "https://www.imagetasks.com",
      "app_support_link": "https://www.imagetasks.com/pixea",
      "privacy_policy_link": "https://www.imagetasks.com/Pixea-policy.txt"
    },
    "more_by_this_developer": {
      "apps": [
        {
          "logo": "https://is3-ssl.mzstatic.com/image/thumb/Purple118/v4/f6/93/b6/f693b68f-9b14-3689-7521-c19a83fb0d88/AppIcon-1x_U007emarketing-85-220-6.png/320x0w.webp",
          "link": "https://apps.apple.com/us/app/istatistica/id1126874522",
          "serpapi_link": "https://serpapi.com/search.json?country=us&engine=apple_product&product_id=1507782672&type=app",
          "name": "iStatistica",
          "category": "Utilities"
        },
        ... other apps
      ],
      "result_type": "Full",
      "see_all_link": "https://apps.apple.com/us/app/id1507782672#see-all/developer-other-apps"
    }
  },
  "product_reviews": [
    {
      "position": 1,
      "id": "9446406432",
      "title": "Stop begging for reviews",
      "text": "Stop begging for reviews",
      "rating": 1,
      "review_date": "2022-12-28 21:42:28 UTC",
      "author": {
        "name": "stalfos_knight",
        "link": "https://itunes.apple.com/us/reviews/id41752602"
      }
    },
    ... other reviews
  ]
}

Links

Join us on Twitter | YouTube

Add a Feature Request💫 or a Bug🐞

Scraping Apple App Store Search with Python

Artur Chukhrai — Sun, 05 Feb 2023 11:51:00 +0000

What will be scraped
Why using API?
Full Code
Preparation
Code Explanation
Output
Links

What will be scraped

📌Note: In this blog post, I will show you how to scrape the Apple App Store search and get exactly the same result as on Apple iMac, because the search results on Mac are completely different from the results on PC. The screenshots below show the difference:

Mac results:

PC results:

Why using API?

There're a couple of reasons that may use API, ours in particular:

No need to create a parser from scratch and maintain it.
Bypass blocks from Google: solve CAPTCHA or solve IP blocks.
Pay for proxies, and CAPTCHA solvers.
Don't need to use browser automation.

SerpApi handles everything on the backend with fast response times under ~2.6 seconds (~0.6 seconds with Ludicrous speed) per request and without browser automation, which becomes much faster. Response times and status rates are shown under SerpApi Status page.

Full Code

If you don't need an explanation, have a look at the full code example in the online IDE.

from serpapi import GoogleSearch
import json

params = {
    'api_key': '...',               # https://serpapi.com/manage-api-key
    'engine': 'apple_app_store',    # SerpApi search engine 
    'term': 'image viewer',         # search query
    'device': 'desktop',            # device to get the results
    'country': 'us',                # country for the search
    'lang': 'en-us',                # language for the search
    'disallow_explicit': False,     # disallowing explicit apps
    'num': 20,                      # number of items per page
    'page': 0,                      # pagination
    # 'property': 'developer'       # developer of an app
}

app_store_results = []

while True:
    search = GoogleSearch(params)            # data extraction on the SerpApi backend
    new_page_results = search.get_dict()     # JSON -> Python dict

    app_store_results.extend(new_page_results['organic_results'])

    if 'next' in new_page_results.get('serpapi_pagination', {}):
        params['page'] += 1
    else:
        break

print(json.dumps(app_store_results, indent=2, ensure_ascii=False))

Preparation

Install library:

pip install google-search-results

google-search-results is a SerpApi API package.

Code Explanation

Import libraries:

from serpapi import GoogleSearch
import json

Library	Purpose
`GoogleSearch`	to scrape and parse Google results using SerpApi web scraping library.
`json`	to convert extracted data to a JSON object.

The parameters are defined for generating the URL. If you want to pass other parameters to the URL, you can do so using the params dictionary:

params = {
    'api_key': '...',               # https://serpapi.com/manage-api-key
    'engine': 'apple_app_store',    # SerpApi search engine 
    'term': 'image viewer',         # search query
    'device': 'desktop',            # device to get the results
    'country': 'us',                # country for the search
    'lang': 'en-us',                # language for the search
    'disallow_explicit': False,     # disallowing explicit apps
    'num': 20,                      # number of items per page
    'page': 0,                      # pagination
    # 'property': 'developer'       # developer of an app
}

Parameters	Explanation
`api_key`	Parameter defines the SerpApi private key to use. You can find it under your account -> API key.
`engine`	Set parameter to `apple_app_store` to use the App Store API engine.
`term`	Parameter defines the query you want to search. You can use any search term that you would use in a regular App Store search.
`device`	Parameter defines the device to use to get the results. It can be set to `desktop` to use a Mac App Store, `tablet` to use a iPad App Store, or `mobile` (default) to use a iPhone App Store.
`country`	Parameter defines the country to use for the search. It's a two-letter country code. Head to the Apple Regions for a full list of supported Apple Regions.
`lang`	Parameter defines the language to use for the search. It's a four-letter country code. Head to the Apple Languages for a full list of supported Apple Languages.
`disallow_explicit`	Parameter defines the filter for disallowing explicit apps. It defaults to `false`.
`num`	Parameter defines the number of results you want to get per each page. It defaults to `10`. Maximum number of results you can get per page is `200`.
`page`	Parameter is used to get the items on a specific page. (e.g., `0` (default) is the first page of results, `1` is the 2nd page of results, `2` is the 3rd page of results, etc.).
`property`	Parameter allows to search the property of an app. `developer` allows searching the developer title of an app ( e.g., `property=developer` and `term=Coffee` gives apps with "Coffee" in their developer's name. (Ex: `Coffee Inc.`)

📌Note: You can also add other API Parameters.

Define the app_store_results list to which the retrieved data will be added:

app_store_results = []

The while loop is created that is needed to extract data from all pages:

while True:
    # data extraction will be here

Then, we create a search object where the data is retrieved from the SerpApi backend. In the new_page_results dictionary we get data from JSON:

search = GoogleSearch(params)            # data extraction on the SerpApi backend
new_page_results = search.get_dict()     # JSON -> Python dict

Adding new data from this page to the app_store_results list:

app_store_results.extend(new_page_results['organic_results'])

# title = new_page_results['organic_results'][0]['title']
# version = new_page_results['organic_results'][0]['version']
# description = new_page_results['organic_results'][0]['description']

📌Note: In the comments above, I showed how to extract specific fields. You may have noticed the new_page_results['organic_results'][0]. This is the index of a product, which means that we are extracting data from the first product. The new_page_results['organic_results'][1] is from the second product and so on.

if 'next' in new_page_results.get('serpapi_pagination', {}):
    params['page'] += 1
else:
    break

After the all data is retrieved, it is output in JSON format:

print(json.dumps(app_store_results, indent=2, ensure_ascii=False))

Output

[
  {
    "position": 1,
    "id": 1507782672,
    "title": "Pixea",
    "bundle_id": "imagetasks.Pixea",
    "version": "2.1",
    "vpp_license": true,
    "age_rating": "4+",
    "release_note": "- New \"Fixed Size and Position\" zoom mode - Fixed a bug causing crash when browsing ZIP-files - Bug fixes and improvements",
    "seller_link": "https://www.imagetasks.com",
    "minimum_os_version": "10.12",
    "description": "Pixea is an image viewer for macOS with a nice minimal modern user interface. Pixea works great with JPEG, HEIC, PSD, RAW, WEBP, PNG, GIF, and many other formats. Provides basic image processing, including flip and rotate, shows a color histogram, EXIF, and other information. Supports keyboard shortcuts and trackpad gestures. Shows images inside archives, without extracting them. Supported formats: JPEG, HEIC, GIF, PNG, TIFF, Photoshop (PSD), BMP, Fax images, macOS and Windows icons, Radiance images, Google's WebP. RAW formats: Leica DNG and RAW, Sony ARW, Olympus ORF, Minolta MRW, Nikon NEF, Fuji RAF, Canon CR2 and CRW, Hasselblad 3FR. Sketch files (preview only). ZIP-archives. Export formats: JPEG, JPEG-2000, PNG, TIFF, BMP. Found a bug? Have a suggestion? Please, send it to support@imagetasks.com Follow us on Twitter @imagetasks!",
    "link": "https://apps.apple.com/us/app/pixea/id1507782672?mt=12&uo=4",
    "serpapi_product_link": "https://serpapi.com/search.json?country=us&engine=apple_product&product_id=1507782672&type=app",
    "serpapi_reviews_link": "https://serpapi.com/search.json?country=us&engine=apple_reviews&page=1&product_id=1507782672",
    "release_date": "2020-04-20 07:00:00 UTC",
    "price": {
      "type": "Free"
    },
    "rating": [
      {
        "type": "All Times",
        "rating": 0.0,
        "count": 0
      }
    ],
    "genres": [
      {
        "name": "Photo & Video",
        "id": 6008,
        "primary": true
      },
      {
        "name": "Graphics & Design",
        "id": 6027,
        "primary": false
      }
    ],
    "developer": {
      "name": "ImageTasks Inc",
      "id": 450316587,
      "link": "https://apps.apple.com/us/developer/id450316587"
    },
    "size_in_bytes": 7113871,
    "supported_languages": [
      "EN"
    ],
    "screenshots": {
      "general": [
        {
          "link": "https://is4-ssl.mzstatic.com/image/thumb/Purple113/v4/e0/21/86/e021868d-b43b-0a78-8d4a-e4e0097a1d01/0131f1c2-3227-46bf-8328-7b147d2b1ea2_Pixea-1.jpg/800x500bb.jpg",
          "size": "800x500"
        },
        {
          "link": "https://is4-ssl.mzstatic.com/image/thumb/Purple113/v4/55/3c/98/553c982d-de30-58b5-3b5a-d6b3b2b6c810/a0424c4d-4346-40e6-8cde-bc79ce690040_Pixea-2.jpg/800x500bb.jpg",
          "size": "800x500"
        },
        {
          "link": "https://is3-ssl.mzstatic.com/image/thumb/Purple123/v4/77/d7/d8/77d7d8c1-4b4c-ba4b-4dde-94bdc59dfb71/6e66509c-5886-45e9-9e96-25154a22fd53_Pixea-3.jpg/800x500bb.jpg",
          "size": "800x500"
        },
        {
          "link": "https://is3-ssl.mzstatic.com/image/thumb/PurpleSource113/v4/44/79/91/447991e0-518f-48b3-bb7e-c7121eb57ba4/79be2791-5b93-4c4d-b4d1-38a3599c2b2d_Pixea-4.jpg/800x500bb.jpg",
          "size": "800x500"
        }
      ]
    },
    "logos": [
      {
        "size": "60x60",
        "link": "https://is3-ssl.mzstatic.com/image/thumb/Purple113/v4/8a/a7/f7/8aa7f75f-620b-74d5-7958-35aa5d851582/AppIcon-0-0-85-220-0-0-0-0-4-0-0-0-2x-sRGB-0-0-0-0-0.png/60x60bb.png"
      },
      {
        "size": "512x512",
        "link": "https://is3-ssl.mzstatic.com/image/thumb/Purple113/v4/8a/a7/f7/8aa7f75f-620b-74d5-7958-35aa5d851582/AppIcon-0-0-85-220-0-0-0-0-4-0-0-0-2x-sRGB-0-0-0-0-0.png/512x512bb.png"
      },
      {
        "size": "100x100",
        "link": "https://is3-ssl.mzstatic.com/image/thumb/Purple113/v4/8a/a7/f7/8aa7f75f-620b-74d5-7958-35aa5d851582/AppIcon-0-0-85-220-0-0-0-0-4-0-0-0-2x-sRGB-0-0-0-0-0.png/100x100bb.png"
      }
    ]
  },
  ... other results
]

Links

Join us on Twitter | YouTube

Add a Feature Request💫 or a Bug🐞

SerpApi Changelog: December, 2022

Artur Chukhrai — Mon, 30 Jan 2023 11:12:04 +0000

Summary of December Updates

A sum-up of SerpApi's latest product improvements, and feature releases 🙂

Features

[Google Knowledge Graph API] Now supports new layout for 'Knowledge Graph'
[Google Knowledge Graph API] Support new artworks layout
[Google Knowledge Graph API] Support 'Knowledge Graph' accordion buttons
[Bing Search API] Support other kinds of expanded sitelinks
[Bing Search API] Support inline sitelinks inside expanded sitelinks
[Google Jobs API] Allow users to search with 'Work from home' location
[Google Jobs API] Added the ability to scrape 'Job highlights'
[Google Jobs Listing API] Added the ability to scrape 'Similar jobs'
[Google Search API] Answer box for hotels now parsing 'Ratings' and 'Reviews'
[Google Search API] 'Rich snippet' support in the organic results
[Google Maps Place API] Support 'Questions and Answers' about the place
[Google Maps Place API] Support 'At this place' results

Fixes

[Bing Search API] Expanded sitelinks parse wrong snippet
[Google Maps Local API] GPS Coordinates are at 0,0
[Walmart Search API] Fix search by category only
[Google Jobs Listing API] Blocks of 'Salaries' and 'Ratings' are not returned
[Youtube API] Subscribers are not parsed in channel results
[Google Answer Box API] 'About this result' returns empty hash

Slight changes

[Google Answer Box API] Added other answer box types
[Google Related Questions API] The next_page_token now correctly returns more questions
[Google Search API] Supports hotel results answer box on mobile
[Google Knowledge Graph API] The new layout for 'Knowledge Graph' is also supported on mobile
[Bing Search API] Returned cached page parsing for organic results
[Google Product API] Support rating and reviews for related products
[Ebay API] Fixed bug when prices were not extracted for some products
[YouTube API] Return parsing ads in JSON response
[Google Shopping API] Fixed display of results for mobile search for some locations
[Apple App Store API] Playground now loads the default mobile device

Projects built with SerpApi

ChatGPT Telegram Bot
Release SerpApi for JavaScript/TypeScript
WebGPT CLI
pyserpZotero

API Updates

[Feature] Google Knowledge Graph API: Now supports new layout for 'Knowledge Graph'

This update will allow you to extract all the data from the new layout for 'Knowledge Graph'. This data is under the web_results key.

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Knowledge Graph API: Support new artworks layout

Added support for new artwork layouts in 'Knowledge Graph'.

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Knowledge Graph API: Support 'Knowledge Graph' accordion buttons

Significantly increased result depth for some 'Knowledge Graph' results.

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Bing Search API: Support other kinds of expanded sitelinks

Added support for other types of extended sitelinks for the organic result.

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Bing Search API: Support inline sitelinks inside expanded sitelinks

Added support for inline sitelinks inside expanded sitelinks.

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Jobs API: Allow users to search with 'Work from home' location

Users can now use the ltype parameter if they want to select a 'Work from home' location.

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Jobs API: Added the ability to scrape 'Job highlights'

Previously, we are only scraping 'Job descriptions'. But now we are added the ability to scrape 'Job highlights'.

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Jobs Listing API: Added the ability to scrape 'Similar jobs'

The ability to parse 'Similar jobs' has been added.

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Search API: Answer box for hotels now parsing 'Ratings' and 'Reviews'

Added the ability to parse ratings and reviews in the answer box for hotels.

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Search API: 'Rich snippet' support in the organic results

Added the ability to parse a rich snippet in the organic result.

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Maps Place API: Support 'Questions and Answers' about the place

Added support for 'Questions and Answers' about the place.

Try in the Playground | Documentation | Public Roadmap Issue

[Feature] Google Maps Place API: Support 'At this place' results

Added support for 'At this place' results for a single enterprise.

Try in the Playground | Documentation | Public Roadmap Issue

Added support for 'At this place' results for multiple businesses from the same address directory.

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] Bing Search API: Expanded sitelinks parse wrong snippet

Now snippets in expanded sitelinks are extracted correctly.

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] Google Maps Local API: GPS Coordinates are at 0,0

GPS coordinates were not analyzed. Now the correct latitude and longitude values are being retrieved.

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] Walmart Search API: Fix search by category only

Fixed an issue with Walmart Search API calls hanging when searching by category only.

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] Google Jobs Listing API: Blocks of 'Salaries' and 'Ratings' are not returned

Salaries

Try in the Playground | Documentation | Public Roadmap Issue

Ratings

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] Youtube API: Subscribers are not parsed in channel results

Fixed a bug where subscribers were not parsed in the channel results.

Try in the Playground | Documentation | Public Roadmap Issue

[Fix] Google Answer Box API: 'About this result' returns empty hash

'About this result' now returns the correct content.

Try in the Playground | Documentation | Public Roadmap Issue

New Projects Made with SerpApi

ChatGPT Telegram Bot

ChatGPT Telegram Bot is a very early attempt at having chatGPT work within a telegram bot.

This is a Telegram bot that lets you chat with the chatGPT language model using your local browser. The bot uses Playwright to run chatGPT in Chromium, and can parse code and text, as well as send messages. It also includes a /draw command that allows you to generate pictures using stable diffusion. More features are coming soon.

Features:

Chat with chatGPT from your Telegram on the go
/draw pictures using stable diffusion (version 0.0.2)
/browse give chatGPT access to Google (version 0.0.3)

Release SerpApi for JavaScript/TypeScript

serpapi-javascript is a scrape and parse search engine results using SerpApi.

WebGPT CLI

webgpt-cli is a command line tool that recreates webgpt with multi shot prompting.

pyserpZotero

pyserpZotero is a Google Scholar citation download, parsing, Bibtex export, and Zotero cloud upload via SerpApi.

Conclusion

Many improvements have been made this month and interesting bugs have been found.

All this is due to the care of the development team and you, the users of our product. For example, a few issues were opened based on the StackOverflow question by one of our users.

Therefore, if you notice any bugs or possible improvements, then you can directly influence the work by creating a problem in the SerpApi Public Roadmap.

Join us on Twitter | YouTube

💛