Scrape Google News Results

Introduction

In this article, I will teach you to scrape Google News Results using Node JS.

Requirements

Before we begin this tutorial, we need to install some NPM packages, which we will use further in this tutorial.

1. Unirest JS
2. Cheerio JS

We will use NPM libraries Unirest to extract our target HTML data and Cheerio JS for the parsing the extracted HTML data.

Target

We will scrape the Google News Results of query Football.

Procedure

We have installed and set up all the things to prepare our scraper. Now we will use Unirest to make a get request to our target URL, which is: https://www.google.com/search?q=football&gl=us
, and Cheerio JS to parse the HTML.

If you inspect the Google News page you will find that all the main articles are contained inside this div.BGxR7d container. By searching in this container we got the tag for title as .MBeuO, for the snippet as .GI74Re, for the thumbnail as div.NUnG9d img, for the date as div.ZE0LJdand for the link as .WlydOe.

This makes our code as:

const unirest = require("unirest");
const cheerio = require("cheerio");

const getNewsData = () => {
  return unirest
    .get("https://www.google.com/search?q=football&gl=us&tbm=nws")
    .headers({
      "User-Agent":
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36",
    })
    .then((response) => {
      let $ = cheerio.load(response.body);

      let news_results = []; 

      $(".BGxR7d").each((i,el) => {
        news_results.push({
         link: $(el).find("a").attr('href'),
         title: $(el).find("div.mCBkyc").text(),
         snippet: $(el).find(".GI74Re").text(),
         date: $(el).find(".ZE0LJd span").text(),
         thumbnail: $(el).find(".NUnG9d img").attr("src")
        })
      })

    console.log(news_results)
    });
};

getNewsData();

Results:

By using Google News API

If you want to scrape Google News Results easily, without making a scraper, as scraping can take a lot of time sometimes, you can try this Google Search API.

Serpdog also provides 100 free searches per month, and if you want to scale your requests quota, you can buy paid plans also.

Example request code:

Results:

Conclusion

In this tutorial we learned to scrape Google News Results. If you have any questions, feel free to ask me in the comments. Follow me on Twitter. Thanks for reading!

Additional Resources

1. Web Scraping Google With Node JS - A Complete Guide
2. Web Scraping Google Without Getting Blocked
3. Google Maps Reviews Scraper
4. Web Scraping Google Organic Search Results