DEV Community

Cover image for How to scrape Google Shopping results?
Serpdog
Serpdog

Posted on • Updated on • Originally published at serpdog.io

How to scrape Google Shopping results?

In this post, we will learn to scrape Google Shopping Results using Node JS.

Requirements:

Before we begin, install these libraries so we can move forward and prepare our scraper.

  1. Unirest JS
  2. Cheerio JS

To extract our HTML data we will use Unirest JS and for parsing the HTML data we will use Cheerio JS.

Target:

Google Shopping Nike shoe Results
We will target to scrape the shopping results of Nike shoes.

Process:

We have installed all the things which we will need for our scraper. Now we will hit our target URL using Unirest JS to get our HTML data and then we will parse our extracted HTML data with the help of Cheerio JS.
We will target this URL:

https://www.google.com/search?q=nike shoes&tbm=shop&gl=us
Enter fullscreen mode Exit fullscreen mode

Look at the tbm parameter and its value(shop, here). This value shop will tell Google that we are looking for shopping results.

Open this URL in your browser. Inspect the code. You will see that every organic shopping result is inside this tag .sh-dgr__gr-auto.

Nike shoes Google Shopping Results HTML

Now, we will search the tags for title, product link, price, rating, reviews, delivery, and source.

NIke Shoes Google Shopping HTML tags

Nike Shoes Google Shopping HTML tags

The above images are in the pattern of two at the top and one at the bottom.

We have completed our search for tags of organic shopping results. Now, we will search for the tags of ad results.

Google Shopping ad results

If you inspect the ad results you will see that all the ad results are inside the tag .sh-np__click-target. This tag contains all the information about the title, link, price, and source.

All the above things make our code look like this:

const unirest = require("unirest");
const cheerio = require("cheerio");

const getShoppingData = () => {

  try
  {
  return unirest
    .get("https://www.google.com/search?q=nike shoes&tbm=shop&gl=us")
    .headers({
      "User-Agent":
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36",
    })
    .then((response) => {
      let $ = cheerio.load(response.body);


    let ads = [];

    $(".sh-np__click-target").each((i,el) => {
     ads.push({
        title: $(el).find(".sh-np__product-title").text(),
        link: "https://google.com" + $(el).attr("href"),
        source: $(el).find(".sh-np__seller-container").text(),
        price: $(el).find(".hn9kf").text(),
        delivery: $(el).find(".U6puSd").text(),
     })
     if($(el).find(".rz2LD").length)
     {
      let extensions = []
        extensions = $(el).find(".rz2LD").text()
        ads[i].extensions = extensions
     }
     })

    for (let i = 0; i < ads.length; i++) {
        Object.keys(ads[i]).forEach(key => ads[i][key] === "" ? delete ads[i][key] : {});  
    }

    let shopping_results = [];

    $(".sh-dgr__gr-auto").each((i,el) => {
        shopping_results.push({
            title: $(el).find(".Xjkr3b").text(),
            link: $(el).find(".zLPF4b .eaGTj a.shntl").attr("href").substring($(el).find("a.shntl").attr("href").indexOf("=")+1),
            source: $(el).find(".IuHnof").text(),
            price: $(el).find(".XrAfOe .a8Pemb").text(),
            rating: $(el).find(".Rsc7Yb").text(),
            reviews: $(el).find(".NzUzee div").attr("aria-label") ? $(el).find(".NzUzee div").attr("aria-label").substring(0,$(el).find(".NzUzee div").attr("aria-label").indexOf(" ")) : "",
            delivery: $(el).find(".vEjMR").text()
        })
        if($(el).find(".Ib8pOd").length)
        {
            let extensions = [];
            extensions = $(el).find(".Ib8pOd").text();
            shopping_results[i].extensions = extensions
        }
    })

    for (let i = 0; i < shopping_results.length; i++) {
        Object.keys(shopping_results[i]).forEach(key => shopping_results[i][key] === "" ? delete shopping_results[i][key] : {});  
     }

     console.log(ads)
     console.log(shopping_results)
    })
  }
 catch(e)
 {
    console.log(e)
 }
}


getShoppingData();

Enter fullscreen mode Exit fullscreen mode

Result:

Google-Shopping-Results

Our result should look like this 👆🏻.

With Google Shopping API

If you don't want to code and maintain the scraper in the long run then you can definitely try Serpdog | Google Search API.
Serpdog also offer 100 free requests on the first sign-up.

const axios = require('axios');

axios.get('https://api.serpdog.io/shopping?api_key=APIKEY&q=shoes&gl=us')
  .then(response => {
    console.log(response.data);
  })
  .catch(error => {
    console.log(error);
  });
Enter fullscreen mode Exit fullscreen mode

Result:

Google Shopping API docs

Conclusion:

In this tutorial, we learned to scrape Google Shopping Results using Node JS. Feel free to comment anything you need clarification on. Follow me on Twitter Thanks for reading!

Additional Resources

  1. How to scrape Google Organic Search Results using Node JS?
  2. Scrape Google Images Results
  3. Scrape Google News Results
  4. Scrape Google Maps Reviews

Top comments (3)

Collapse
 
gabrielsilveira2026 profile image
Gabriel Silveira

Hello, can you help me? I need to get the images displayed in the google shopping ads, I tried to use the other attributes as an example, but I still wasn't successful, can you tell me how it would be possible, I identified the class "SirUVb sh-img__image" but I couldn't access the tag

Collapse
 
serpdogapi profile image
Serpdog

Here is the correct way of doing it: div.sh-img__image img

Collapse
 
serpdogapi profile image
Serpdog • Edited

Hey,

Can you message me on this link: drift.me/darshankhandelwal12