How to Scrape Glassdoor Reviews

Introduction

In this article, you will learn how to scrape company reviews from Glassdoor with Page2API

Glassdoor.com is an American website where current and former employees anonymously review companies.

‎

Disclaimer:

We highly recommend you to scrape Glassdoor only for personal use.

For example: let's say you are looking for a new job and you want to quickly analyze the reviews for the company that you are interested in.

‎

Prerequisites

To start scraping Glassdoor reviews, we will need the following things:

• A Page2API account
• A company name that we are interested in. In our case, the company that we are interested in is... Glassdoor. (Which also has reviews on its website)

‎

How to scrape Glassdoor Reviews

First what we need is to open glassdoor.com and type Glassdoor reviews into the search input.

This will change the browser URL to something similar to:

https://www.glassdoor.com/Reviews/Glassdoor-Reviews-E100431.htm

We will use this URL as the first parameter we need to start the scraping process.

The page that you see must look like the following one:

If you inspect the page HTML, you will find out that a single review looks like the following:

From the Glassdoor Reviews page, we will scrape the following attributes from each review:

• Title
• Author Info
• Rating
• Pros
• Cons
• Helpful

Now, let's define the selectors for each attribute.

/* Parent: */
div.gdReview

/* Title */
a.reviewLink

/* Author Info */
.authorInfo

/* Rating */
span.ratingNumber

/* Pros */
span[data-test=pros]

/* Cons */
span[data-test=cons]

/* Helpful */
div.common__EiReviewDetailsStyle__socialHelpfulcontainer

Let's take a look at the pagination handling.

To go to the next page, we must click on the next page button if it's present on the page:

document.querySelector(".nextButton").click()

The scraping will continue while the next page button is present on the page, and stop if it disappears.

The stop condition for the scraper will be the following javascript snippet:

document.querySelector(".nextButton") === null

// but to avoid timeouts, we will scrape a fixed amount of pages (see the payload below)

Now it's time to build the request that will scrape Glassdoor reviews.

The payload for our scraping request will be:

{
  "url": "https://www.glassdoor.com/Reviews/Glassdoor-Reviews-E100431.htm",
  "real_browser": true,
  "merge_loops": true,
  "premium_proxy": "us",
  "scenario": [
    {
      "loop": [
        { "wait_for": "div.gdReview" },
        { "execute": "parse" },
        { "execute_js": "document.querySelector(\".nextButton\").click()" }
      ],
      "iterations": 2
    }
  ],
  "parse": {
    "reviews": [
      {
        "_parent": "div.gdReview",
        "title": "a.reviewLink >> text",
        "author_info": ".authorInfo >> text",
        "rating": "span.ratingNumber >> text",
        "pros": "span[data-test=pros] >> text",
        "cons": "span[data-test=cons] >> text",
        "helpful": "div.common__EiReviewDetailsStyle__socialHelpfulcontainer >> text"
      }
    ]
  }
}

Setting the api_key as an environment variable:

export API_KEY=YOUR_PAGE2API_KEY

Running the scraping request with cURL:

curl -v -XPOST -H "Content-type: application/json" -d '{
  "api_key": "'"$API_KEY"'",
  "url": "https://www.glassdoor.com/Reviews/Glassdoor-Reviews-E100431.htm",
  "real_browser": true,
  "merge_loops": true,
  "premium_proxy": "us",
  "scenario": [
    {
      "loop": [
        { "wait_for": "div.gdReview" },
        { "execute": "parse" },
        { "execute_js": "document.querySelector(\".nextButton\").click()" }
      ],
      "iterations": 2
    }
  ],
  "parse": {
    "reviews": [
      {
        "_parent": "div.gdReview",
        "title": "a.reviewLink >> text",
        "author_info": ".authorInfo >> text",
        "rating": "span.ratingNumber >> text",
        "pros": "span[data-test=pros] >> text",
        "cons": "span[data-test=cons] >> text",
        "helpful": "div.common__EiReviewDetailsStyle__socialHelpfulcontainer >> text"
      }
    ]
  }
}' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result:

{
  "result": {
    "reviews": [
      {
        "title": "Glassdoor Walks the Walk",
        "author_info": "Jan 7, 2022 - Senior Manager",
        "rating": "5.0",
        "pros": "Glassdoor creates a positive environment for employees to learn and grow. ...",
        "cons": "At any organization, there is always room for improvement. ...",
        "helpful": "1 person found this review helpful"
      },
      {
        "title": "Great Company To Work For",
        "author_info": "Jan 5, 2022 - Customer Success Manager",
        "rating": "4.0",
        "pros": "I absolutely love working at Glassdoor. ...",
        "cons": "While we do have more of an extensive career growth plan, ...",
        "helpful": "2 people found this review helpful"
      }, ...
    ]
  }, ...
}

‎

Conclusion

Done!

We just finished scraping the reviews from Glassdoor, and it turned to be easy and fun if we have the proper scraping tool.

The original article can be found here:

page2api.com/blog/how-to-scrape-glassdoor-reviews/