DEV Community: Nicolae Rotaru

How to Scrape Reddit Data: Posts, Images, Comments, and more.

Nicolae Rotaru — Fri, 27 May 2022 10:39:40 +0000

Introduction

Reddit.com is a social news website and forum where content is socially curated and promoted by site members through voting.

In this article, you will read about the easiest way to scrape Reddit posts and comments with Page2API, and find some code examples in Python.

To scrape Reddit, we will use Page2API - a powerful and delightful API that will help you scrape the web like a PRO.

‎

Prerequisites

To start scraping Reddit posts, we will need the following things:

A Page2API account
The link to a subreddit that we are about to scrape. To make the scraping process easier - we will use the old Reddit UI since it has a simpler HTML structure. In this article, we will scrape the BMW subreddit.

‎

How to scrape Reddit Posts

The first thing we need is to open the subreddit URL:

https://old.reddit.com/r/BMW/

The URL is the first parameter we need to perform the scraping.

The page that you see must look like the following one:

If you inspect the page HTML, you will find out that a single result is wrapped into an element that looks like the following:

From this page, we will scrape the following attributes from each Reddit post:

Title
URL
Thumbnail
Submitted
Author
Author URL
Comments
Rank
Score

Now, let's define the selectors for each attribute.

/* Parent: */
.thing[data-promoted=false]

/* Title */
a.title

/* URL */
a.title

/* Thumbnail */
a.thumbnail img

/* Submitted */
time

/* Author */
a.author

/* Author URL */
a.author

/* Comments */
(parent div, attribute: data-comments-count)

/* Rank */
(parent div, attribute: data-rank

/* Score */
(parent div, attribute: data-score

Now, let's handle the pagination.
We will iterate through the Reddit pages by clicking on the Next page button.

To go to the next page, we must click on the next page link if it's present on the page:

document.querySelector('.next-button a').click()

The scraping will continue while the Next link is present on the page, and stop if it disappears.
The stop condition for the scraper will be the following javascript snippet:

document.querySelector('.next-button a') === null

Let's build the request that will scrape the needed pages from this subreddit.

The following examples will show how to scrape 5 pages of posts from Reddit.com

Our payload will look like this

{
  "api_key": "YOUR_PAGE2API_KEY",
  "url": "https://old.reddit.com/r/BMW/",
  "merge_loops": true,
  "real_browser": true,
  "scenario": [
    {
      "loop": [
        { "wait_for": ".thing" },
        { "execute": "parse" },
        { "click": ".next-button a" }
      ],
      "stop_condition": "document.querySelector('.next-button a') === null",
      "iterations": 5
    }
  ],
  "parse": {
    "posts": [
      {
        "_parent": ".thing[data-promoted=false]",
        "title": "a.title >> text",
        "url": "a.title >> href",
        "thumbnail": "a.thumbnail img >> src",
        "submitted": "time >> datetime",
        "author": "a.author >> text",
        "author_url": "a.author >> href",
        "comments": "_parent >> data-comments-count",
        "rank": "_parent >> data-rank",
        "score": "_parent >> data-score"
      }
    ]
  }
}

Python code example

import requests
import json

api_url = 'https://www.page2api.com/api/v1/scrape'

The following example will show how to scrape 5 pages of posts from Reddit.com

payload = {
  "api_key": "YOUR_PAGE2API_KEY",
  "url": "https://old.reddit.com/r/BMW/",
  "merge_loops": True,
  "real_browser": True,
  "scenario": [
    {
      "loop": [
        { "wait_for": ".thing" },
        { "execute": "parse" },
        { "click": ".next-button a" }
      ],
      "stop_condition": "document.querySelector('.next-button a') === null",
      "iterations": 5
    }
  ],
  "parse": {
    "posts": [
      {
        "_parent": ".thing[data-promoted=false]",
        "title": "a.title >> text",
        "url": "a.title >> href",
        "thumbnail": "a.thumbnail img >> src",
        "submitted": "time >> datetime",
        "author": "a.author >> text",
        "author_url": "a.author >> href",
        "comments": "_parent >> data-comments-count",
        "rank": "_parent >> data-rank",
        "score": "_parent >> data-score"
      }
    ]
  }
}

headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
response = requests.post(api_url, data=json.dumps(payload), headers=headers)
result = json.loads(response.text)

print(result)

The result

{
  "result": {
    "posts": [
      {
          "title": "Thought you would enjoy this",
          "url": "https://old.reddit.com/r/BMW/comments/uu50bl/thought_you_would_enjoy_this/",
          "thumbnail": "https://b.thumbs.redditmedia.com/1OMVuKzvOeVi6IkOMc1K94HzPOLs3InozhCPlxU8K7E.jpg",
          "submitted": "2022-05-20T18:55:11+00:00",
          "author": "L9FanboyXD",
          "author_url": "https://old.reddit.com/user/L9FanboyXD",
          "comments": "206",
          "rank": "1",
          "score": "1234"
      },
      {
          "title": "Totalled..",
          "url": "https://old.reddit.com/r/BMW/comments/uu7yih/totalled/",
          "thumbnail": "https://b.thumbs.redditmedia.com/QQ7_MbJqhB5t4QfKJjbTFU4sEK0Cd9Cy5uFCSzzeLzM.jpg",
          "submitted": "2022-05-20T21:16:40+00:00",
          "author": "Conscious_Champion35",
          "author_url": "https://old.reddit.com/user/Conscious_Champion35",
          "comments": "130",
          "rank": "2",
          "score": "375"
      },
      {
          "title": "Joined the squad. Yes there was a flat bed at the dealer as I was closing this deal today.",
          "url": "https://www.reddit.com/gallery/uucgb5",
          "thumbnail": "https://b.thumbs.redditmedia.com/g8vAVq0N59R0a-IX1mBtM1YEy2gouhhGTTS7m4Nr8us.jpg",
          "submitted": "2022-05-21T01:16:08+00:00",
          "author": "restloy",
          "author_url": "https://old.reddit.com/user/restloy",
          "comments": "22",
          "rank": "3",
          "score": "157"
      }, ...
    ]
  }, ...
}

‎

How to scrape Reddit Comments

We need to open any post URL from the previous step.

This will change the browser URL to something similar to:

https://old.reddit.com/r/BMW/comments/uu50bl/thought_you_would_enjoy_this/

This URL is the first parameter we need to scrape the Reddit post data and comments.

The comments section that you see must look like the following one:

From this page, we will scrape the following attributes:

Post data:

Title
Image

Comments data:

Author
Author URL
Posted
Score
ID
Content

Now, let's define the selectors for each attribute.

Post data

/* Title */
a.title

/* Image */
.media-preview-content img.preview

Comments data

/* Parent */
.entry

/* Author */
a.author

/* Author URL */
a.author

/* Posted */
time

/* Score */
.score.unvoted

/* ID */
input[name=thing_id]

/* Content */
.usertext-body

It's time to prepare the request that will scrape the data from a Reddit post page.

The payload for our scraping request will be:

{
  "api_key": "YOUR_PAGE2API_KEY",
  "url": "https://old.reddit.com/r/BMW/comments/urv3mr/i_saw_it_in_prague_the_red_is_much_prettier_in/",
  "parse": {
    "title": "a.title >> text",
    "image": ".media-preview-content img.preview >> src",
    "comments": [
      {
        "_parent": ".entry",
        "author": "a.author >> text",
        "author_url": "a.author >> href",
        "posted": "time >> datetime",
        "score": ".score.unvoted >> text",
        "id": "input[name=thing_id] >> value",
        "content": ".usertext-body >> text"
      }
    ]
  }
}

Python code example:

import requests
import json

api_url = 'https://www.page2api.com/api/v1/scrape'
payload = {
  "api_key": "YOUR_PAGE2API_KEY",
  "url": "https://old.reddit.com/r/BMW/comments/urv3mr/i_saw_it_in_prague_the_red_is_much_prettier_in/",
  "parse": {
    "title": "a.title >> text",
    "image": ".media-preview-content img.preview >> src",
    "comments": [
      {
        "_parent": ".entry",
        "author": "a.author >> text",
        "author_url": "a.author >> href",
        "posted": "time >> datetime",
        "score": ".score.unvoted >> text",
        "id": "input[name=thing_id] >> value",
        "content": ".usertext-body >> text"
      }
    ]
  }
}

headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
response = requests.post(api_url, data=json.dumps(payload), headers=headers)
result = json.loads(response.text)

print(result)

The result:

{
  "result": {
    "title": "I saw it in prague. the red is much prettier in real life",
    "image": "https://preview.redd.it/91x8qz36f3091.jpg?width=1023&auto=png&s=9c0aced50d6743989f053cf9cc78c896f928d8a7",
    "comments": [
      {
        "author": "DarkMeta81",
        "author_url": "https://old.reddit.com/user/DarkMeta81",
        "posted": "2022-05-17T20:12:31+00:00",
        "score": "15 points",
        "id": "t1_i8zlp77",
        "content": "i was on vacation and prague is full of beautiful bmw this m5 is great"
      },
      {
        "author": "Id-atl",
        "author_url": "https://old.reddit.com/user/Id-atl",
        "posted": "2022-05-18T01:06:15+00:00",
        "score": "5 points",
        "id": "t1_i90on16",
        "content": "Wow imagine being in Prague ripping this around!"
      }, ...
    ]
  }, ...
}

‎

Conclusion

That's it!

In this article, you've learned how to scrape the data from Reddit.com with Page2API - a Web Scraping API that handles all the hassle, and lets you get the data you need with ease.

The original article can be found here:

https://www.page2api.com/blog/how-to-scrape-reddit/

How to Scrape Glassdoor Reviews

Nicolae Rotaru — Wed, 12 Jan 2022 23:56:48 +0000

Introduction

In this article, you will learn how to scrape company reviews from Glassdoor with Page2API

Glassdoor.com is an American website where current and former employees anonymously review companies.

‎

Disclaimer:

We highly recommend you to scrape Glassdoor only for personal use.

For example: let's say you are looking for a new job and you want to quickly analyze the reviews for the company that you are interested in.

‎

Prerequisites

To start scraping Glassdoor reviews, we will need the following things:

A Page2API account
A company name that we are interested in. In our case, the company that we are interested in is... Glassdoor. (Which also has reviews on its website)

‎

How to scrape Glassdoor Reviews

First what we need is to open glassdoor.com and type Glassdoor reviews into the search input.

This will change the browser URL to something similar to:

https://www.glassdoor.com/Reviews/Glassdoor-Reviews-E100431.htm

We will use this URL as the first parameter we need to start the scraping process.

The page that you see must look like the following one:

If you inspect the page HTML, you will find out that a single review looks like the following:

From the Glassdoor Reviews page, we will scrape the following attributes from each review:

Title
Author Info
Rating
Pros
Cons
Helpful

Now, let's define the selectors for each attribute.

/* Parent: */
div.gdReview

/* Title */
a.reviewLink

/* Author Info */
.authorInfo

/* Rating */
span.ratingNumber

/* Pros */
span[data-test=pros]

/* Cons */
span[data-test=cons]

/* Helpful */
div.common__EiReviewDetailsStyle__socialHelpfulcontainer

Let's take a look at the pagination handling.

To go to the next page, we must click on the next page button if it's present on the page:

document.querySelector(".nextButton").click()

The scraping will continue while the next page button is present on the page, and stop if it disappears.

The stop condition for the scraper will be the following javascript snippet:

document.querySelector(".nextButton") === null

// but to avoid timeouts, we will scrape a fixed amount of pages (see the payload below)

Now it's time to build the request that will scrape Glassdoor reviews.

The payload for our scraping request will be:

{
  "url": "https://www.glassdoor.com/Reviews/Glassdoor-Reviews-E100431.htm",
  "real_browser": true,
  "merge_loops": true,
  "premium_proxy": "us",
  "scenario": [
    {
      "loop": [
        { "wait_for": "div.gdReview" },
        { "execute": "parse" },
        { "execute_js": "document.querySelector(\".nextButton\").click()" }
      ],
      "iterations": 2
    }
  ],
  "parse": {
    "reviews": [
      {
        "_parent": "div.gdReview",
        "title": "a.reviewLink >> text",
        "author_info": ".authorInfo >> text",
        "rating": "span.ratingNumber >> text",
        "pros": "span[data-test=pros] >> text",
        "cons": "span[data-test=cons] >> text",
        "helpful": "div.common__EiReviewDetailsStyle__socialHelpfulcontainer >> text"
      }
    ]
  }
}

Setting the api_key as an environment variable:

export API_KEY=YOUR_PAGE2API_KEY

Running the scraping request with cURL:

curl -v -XPOST -H "Content-type: application/json" -d '{
  "api_key": "'"$API_KEY"'",
  "url": "https://www.glassdoor.com/Reviews/Glassdoor-Reviews-E100431.htm",
  "real_browser": true,
  "merge_loops": true,
  "premium_proxy": "us",
  "scenario": [
    {
      "loop": [
        { "wait_for": "div.gdReview" },
        { "execute": "parse" },
        { "execute_js": "document.querySelector(\".nextButton\").click()" }
      ],
      "iterations": 2
    }
  ],
  "parse": {
    "reviews": [
      {
        "_parent": "div.gdReview",
        "title": "a.reviewLink >> text",
        "author_info": ".authorInfo >> text",
        "rating": "span.ratingNumber >> text",
        "pros": "span[data-test=pros] >> text",
        "cons": "span[data-test=cons] >> text",
        "helpful": "div.common__EiReviewDetailsStyle__socialHelpfulcontainer >> text"
      }
    ]
  }
}' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result:

{
  "result": {
    "reviews": [
      {
        "title": "Glassdoor Walks the Walk",
        "author_info": "Jan 7, 2022 - Senior Manager",
        "rating": "5.0",
        "pros": "Glassdoor creates a positive environment for employees to learn and grow. ...",
        "cons": "At any organization, there is always room for improvement. ...",
        "helpful": "1 person found this review helpful"
      },
      {
        "title": "Great Company To Work For",
        "author_info": "Jan 5, 2022 - Customer Success Manager",
        "rating": "4.0",
        "pros": "I absolutely love working at Glassdoor. ...",
        "cons": "While we do have more of an extensive career growth plan, ...",
        "helpful": "2 people found this review helpful"
      }, ...
    ]
  }, ...
}

‎

Conclusion

Done!

We just finished scraping the reviews from Glassdoor, and it turned to be easy and fun if we have the proper scraping tool.

The original article can be found here:

page2api.com/blog/how-to-scrape-glassdoor-reviews/

How to Scrape Indeed Data: Jobs, Salaries, and more.

Nicolae Rotaru — Sun, 09 Jan 2022 10:30:44 +0000

Introduction

Indeed.com is a job board that aggregates postings, allowing users to search for specific positions.

Why may we need to web scrape Indeed?
Collecting job listings from Indeed will help us to:

search for hirings
analyze the demand for specific job positions
analyze the average salaries

‎

Prerequisites

To start scraping Indeed jobs, we will need the following things:

A Page2API account
A job position in a specific location that we are about to scrape. In our case, we will search for Ruby On Rails Software Engineer in Redwood City, CA, and set the area to within 10 miles.

‎

How to scrape Indeed Jobs

First what we need is to open indeed.com and type Ruby On Rails Software Engineer into the search input from the Indeed home page and pick the location we need.

This will change the browser URL to something similar to:

https://www.indeed.com/jobs?q=Ruby%20On%20Rails%20Software%20Engineer&l=Redwood%20City%2C%20CA&radius=10

The URL is the first parameter we need to perform the scraping.

The page that you see must look like the following one:

If you inspect the page HTML, you will find out that a single result is wrapped into an element that looks like the following:

From this page, we will scrape the following attributes from each Indeed job posting:

Title
URL
Company
Location
Rating
Additional info

Now, let's define the selectors for each attribute.

/* Parent: */
a.result

/* Title */
h2.jobTitle

/* URL */
a.result

/* Company */
.companyName

/* Location */
.companyLocation

/* Rating */
.ratingNumber span[aria-hidden=true]

/* Additional info */
.metadata div

It's time to handle the pagination.

To go to the next page, we must click on the next page link if it's present on the page:

 var next = document.querySelector('a[aria-label=Next]'); if(next) { next.click() }

  // we have this simple check to avoid any javascript errors (in case the Next page button is missing)

The scraping will continue while the Next link is present on the page, and stop if it disappears.
The stop condition for the scraper will be the following javascript snippet:

document.querySelector('a[aria-label=Next]') == null

Let's build the request that will scrape all the results that the search page returned.

The payload for our scraping request will be:

{
  "url": "https://www.indeed.com/jobs?q=Ruby%20On%20Rails%20Software%20Engineer&l=Redwood%20City%2C%20CA&radius=10",
  "real_browser": true,
  "merge_loops": true,
  "scenario": [
    {
      "loop": [
        {
          "wait_for": "a.result"
        },
        {
          "execute": "parse"
        },
        {
          "execute_js": "var next = document.querySelector('a[aria-label=Next]'); if(next) { next.click() }"
        }
      ],
      "stop_condition": "document.querySelector('a[aria-label=Next]') == null"
    }
  ],
  "parse": {
    "jobs": [
      {
        "_parent": "a.result",
        "url": "_parent >> href",
        "title": "h2.jobTitle >> text",
        "company": ".companyName >> text",
        "location": ".companyLocation >> text",
        "rating": ".ratingNumber span[aria-hidden=true] >> text",
        "additional_info": [
          ".metadata div >> text"
        ]
      }
    ]
  }
}

Setting the api_key as an environment variable:

 export API_KEY=YOUR_PAGE2API_KEY

Running the scraping request with cURL:

curl -v -XPOST -H "Content-type: application/json" -d '{
  "api_key": "'"$API_KEY"'",
  "url": "https://www.indeed.com/jobs?q=Ruby%20On%20Rails%20Software%20Engineer&l=Redwood%20City%2C%20CA&radius=10",
  "merge_loops": true,
  "real_browser": true,
  "scenario": [
    {
      "loop": [
        { "wait_for": "a.result" },
        { "execute": "parse" },
        { "execute_js": "var next = document.querySelector(\"a[aria-label=Next]\"); if(next) { next.click() }" }
      ],
      "stop_condition": "document.querySelector(\"a[aria-label=Next]\") == null"
    }
  ],
  "parse": {
    "jobs": [
      {
        "_parent": "a.result",
        "url": "_parent >> href",
        "title": "h2.jobTitle >> text",
        "company": ".companyName >> text",
        "location": ".companyLocation >> text",
        "rating": ".ratingNumber span[aria-hidden=true] >> text",
        "additional_info": [
            ".metadata div >> text"
        ]
      }
    ]
  }
}' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result:

{
  "result": {
    "places": [
      {
        "url": "https://www.indeed.com/company/Coupa/jobs/Senior-Lead-Software-Engineer-fa676bc66ad1daae?fccid=c6a1779d65543307&vjs=3",
        "title": "Senior/Lead Software Engineer, Ruby on Rails",
        "company": "Coupa Software",
        "location": "San Mateo, CA 94402 (Nineteenth Avenue area)+1 location",
        "rating": "3.9",
        "additional_info": [
          "$145,000 - $165,000 a year",
          "Full-time",
          "8 hour shift"
        ]
      },
      {
        "url": "https://www.indeed.com/company/Poshmark/jobs/Software-Engineer-e55c033766067a6c?fccid=0f4f2d112db7d324&vjs=3",
        "title": "Software Engineer, Web Applications",
        "company": "Poshmark",
        "location": "Redwood City, CA",
        "rating": "4.6",
        "additional_info": [
          "Full-time",
        ]
      },
      {
        "url": "https://www.indeed.com/rc/clk?jk=0d91d46d5d05c6ad&fccid=c6a1779d65543307&vjs=3",
        "title": "Software Engineer, Ruby on Rails",
        "company": "Coupa Software",
        "location": "San Mateo, CA",
        "rating": "3.9",
        "additional_info": [
          "Remote",
        ]
      }, ...
    ]
  }, ...
}

‎

How to scrape Indeed Job Page

We need to open any URL from the previous step with the job listing in a new tab.

This will change the browser URL to something similar to:

https://www.indeed.com/viewjob?jk=0d91d46d5d05c6ad

This URL is the first parameter we need to scrape all the information about a job.

The page that you see must look like the following one:

From this page, we will scrape the following attributes:

Title
Company
Rating
Reviews count
Description

Now, let's define the selectors for each attribute.

/* Title */
h1

/* Company */
.jobsearch-InlineCompanyRating a

/* Rating */
meta[itemprop=ratingValue]

/* Reviews count */
meta[itemprop=ratingCount]

/* Description */
#jobDescriptionText

It's time to prepare the request that will scrape Indeed Job Page.

The payload for our scraping request will be:

{
  "url": "https://www.indeed.com/viewjob?jk=0d91d46d5d05c6ad",
  "parse": {
    "title": "h1 >> text",
    "company": ".jobsearch-InlineCompanyRating a >> text",
    "rating": "meta[itemprop=ratingValue] >> content",
    "reviews_count": "meta[itemprop=ratingCount] >> content",
    "description": "#jobDescriptionText >> text"
  }
}

Running the scraping request with cURL:

curl -v -XPOST -H "Content-type: application/json" -d '{
  "api_key": "'"$API_KEY"'",
  "url": "https://www.indeed.com/viewjob?jk=0d91d46d5d05c6ad",
  "parse": {
    "title": "h1 >> text",
    "company": ".jobsearch-InlineCompanyRating a >> text",
    "rating": "meta[itemprop=ratingValue] >> content",
    "reviews_count": "meta[itemprop=ratingCount] >> content",
    "description": "#jobDescriptionText >> text"
  }
}' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result:

{
  "result": {
    "title": "Software Engineer, Ruby on Rails",
    "company": "Coupa Software",
    "rating": "3.9",
    "reviews_count": "27",
    "description": "Coupa Software (NASDAQ: COUP), a leader in business spend management (BSM), ..."
  }, ...
}

‎

Conclusion

That's pretty much of it!

In this article, you've learned how to scrape the data from a job board such as Indeed.com with Page2API - a Web Scraping API that handles all the hassle, and lets you get the data you need with ease.

The original article can be found here:

page2api.com/blog/how-to-scrape-indeed/

How to scrape Zillow in no time

Nicolae Rotaru — Mon, 22 Nov 2021 15:51:40 +0000

Introduction

Zillow is a real estate company that offers various products targeted at both home buyers and sellers.

You can scrape listings from Zillow, with data including addresses, prices, descriptions, photos, URLs to perform:

price monitoring
trends analysis
competitor analysis

Challenges

At first look, scraping Zillow doesn't seem to be a trivial task because of the following aspects:

The content from the listing page is returned dynamically, based on scrolling events.
The names of the CSS selectors are dynamically generated and cannot be used to pick the needed content, and we will use XPath selectors instead.

For this purpose, we will use Page2API - the scraping API that overtakes the challenges mentioned above with ease.

In this article, we will describe how to scrape two types of Zillow pages:

Property listings page
Property overview page

Prerequisites

To start scraping, you will need the following things:

A Page2API account
A location, in which we want to search for listed properties, let's use for example Redwood City
A property overview page from Zillow. We will pick a random property link from the page mentioned above.

Scraping Zillow listings page

First what we need is to open the Homes page and type the name of the city that will show the properties we are searching for.

In our case we will open this page:

https://www.zillow.com/homes/

and search for Redwood City

It will change the browser URL to something similar to:

https://www.zillow.com/homes/Redwood-City,-CA_rb/

The resulted URL is the first parameter we need to start scraping the listings page.

The listings page must look similar to the following one:

If you inspect the page HTML, you will find out that a single result is wrapped into an element that looks like the following:

The HTML for a single result element will look like this:

From the listing page, we will scrape the following attributes from each property:

Price
URL
Bedrooms
Bathrooms
Living area
Status
Address

Each property container is wrapped in an article element with the following class: list-card.

Now, let's define the selectors for each attribute.

/* Parent: */
article.list-card

/* Price: */
.list-card-price

/* URL: */
a

/* Bedrooms: */
ul.list-card-details li:nth-child(1)

/* Bathrooms: */
ul.list-card-details li:nth-child(2)

/* Living area: */
ul.list-card-details li:nth-child(3)

/* Status: */
ul.list-card-details li:nth-child(4)

/* Address: */
a address.list-card-addr

Next is the pagination handling.

In our case, we must click on the next page link while the link will be active:

var next = document.querySelector('.search-pagination a[rel=next]'); if(next){ next.click() }

And stop our scraping request when the next page link became disabled.

In our case, a new attribute (disabled) is assigned to the pagination link.

The stop condition for the pagination will look like this:

var next = document.querySelector('.search-pagination a[rel=next]'); next === null || next.getAttributeNames().includes('disabled')

// returns true if there is no next page

The last thing is handling the content that loads dynamically when we scroll down.

Usually, there are 40 items on the page, but when the page loads - it has only about 8 items.

To load all items we will do the next trick:

Wait for the page to load
Scroll down 3 times slowly, (with a short delay) until we see the last item
Start scraping the page

Now let's build the request that will scrape all properties that the search page returned.

Setting the api_key as an environment variable:

export API_KEY=YOUR_PAGE2API_KEY

The payload for our scraping request will be:

{
  "url": "https://www.zillow.com/homes/Redwood-City,-CA_rb/",
  "real_browser": true,
  "merge_loops": true,
  "premium_proxy": "us",
  "scenario": [
    {
      "loop": [
        { "wait_for": ".search-pagination a[rel=next]" },
        { "execute_js": "var articles = document.querySelectorAll('article')"},
        { "execute_js": "articles[Math.round(articles.length/4)].scrollIntoView({behavior: 'smooth'})"},
        { "wait": 1 },
        { "execute_js": "articles[Math.round(articles.length/2)].scrollIntoView({behavior: 'smooth'})"},
        { "wait": 1 },
        { "execute_js": "articles[Math.round(articles.length/1.5)].scrollIntoView({behavior: 'smooth'})"},
        { "wait": 1 },
        { "execute": "parse"},
        { "execute_js": "var next = document.querySelector('.search-pagination a[rel=next]'); if(next){ next.click() }" }
      ],
    "stop_condition": "var next = document.querySelector('.search-pagination a[rel=next]'); next === null || next.getAttributeNames().includes('disabled')"
  }

],
  "parse": {
    "properties": [
      {
        "_parent": "article.list-card",
        "price": ".list-card-price >> text",
        "url": "a >> href",
        "bedrooms": "ul.list-card-details li:nth-child(1) >> text",
        "bathrooms": "ul.list-card-details li:nth-child(2) >> text",
        "living_area": "ul.list-card-details li:nth-child(3) >> text",
        "status": "ul.list-card-details li:nth-child(4) >> text",
        "address": "a address.list-card-addr >> text"
      }
    ]
  }
}

We have to encode our JS snippets in base64 to run the request in the terminal with cURL.

Running the scraping request with cURL:

curl -v -XPOST -H "Content-type: application/json" -d '{
  "api_key": "'"$API_KEY"'",
  "url": "https://www.zillow.com/homes/Redwood-City,-CA_rb/",
  "real_browser": true,
  "merge_loops": true,
  "premium_proxy": "us",
  "scenario": [
    {
      "loop" : [
        { "wait_for": ".search-pagination a[rel=next]" },
        { "execute_js": "dmFyIGFydGljbGVzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCgnYXJ0aWNsZScp"},
        { "execute_js": "YXJ0aWNsZXNbTWF0aC5yb3VuZChhcnRpY2xlcy5sZW5ndGgvNCldLnNjcm9sbEludG9WaWV3KHtiZWhhdmlvcjogJ3Ntb290aCd9KQ=="},
        { "wait": 1 },
        { "execute_js": "YXJ0aWNsZXNbTWF0aC5yb3VuZChhcnRpY2xlcy5sZW5ndGgvMildLnNjcm9sbEludG9WaWV3KHtiZWhhdmlvcjogJ3Ntb290aCd9KQ=="},
        { "wait": 1 },
        { "execute_js": "YXJ0aWNsZXNbTWF0aC5yb3VuZChhcnRpY2xlcy5sZW5ndGgvMS41KV0uc2Nyb2xsSW50b1ZpZXcoe2JlaGF2aW9yOiAnc21vb3RoJ30p"},
        { "wait": 1 },
        { "execute": "parse"},
        { "execute_js": "dmFyIG5leHQgPSBkb2N1bWVudC5xdWVyeVNlbGVjdG9yKCcuc2VhcmNoLXBhZ2luYXRpb24gYVtyZWw9bmV4dF0nKTsgaWYobmV4dCl7IG5leHQuY2xpY2soKSB9" }
      ],
    "stop_condition": "dmFyIG5leHQgPSBkb2N1bWVudC5xdWVyeVNlbGVjdG9yKCcuc2VhcmNoLXBhZ2luYXRpb24gYVtyZWw9bmV4dF0nKTsgbmV4dCA9PT0gbnVsbCB8fCBuZXh0LmdldEF0dHJpYnV0ZU5hbWVzKCkuaW5jbHVkZXMoJ2Rpc2FibGVkJyk="
    }
  ],
  "parse": {
    "properties": [
      {
        "_parent": "article.list-card",
        "price": ".list-card-price >> text",
        "url": "a >> href",
        "bedrooms": "ul.list-card-details li:nth-child(1) >> text",
        "bathrooms": "ul.list-card-details li:nth-child(2) >> text",
        "living_area": "ul.list-card-details li:nth-child(3) >> text",
        "status": "ul.list-card-details li:nth-child(4) >> text",
        "address": "a address.list-card-addr >> text"
      }
    ]
  }
}' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result:

{
  "result": {
    "properties": [
      {
        "price": "$600,000",
        "url": "https://www.zillow.com/homedetails/464-Clinton-St-APT-211-Redwood-City-CA-94062/15638802_zpid/",
        "bedrooms": "1 bd",
        "bathrooms": "1 ba",
        "living_area": "761 sqft",
        "status": "- Condo for sale",
        "address": "464 Clinton St APT 211, Redwood City, CA 94062"
      },
      {
        "price": "$2,498,000",
        "url": "https://www.zillow.com/homedetails/3618-Midfield-Way-Redwood-City-CA-94062/15571874_zpid/",
        "bedrooms": "4 bds",
        "bathrooms": "4 ba",
        "living_area": "2,960 sqft",
        "status": "- House for sale",
        "address": "3618 Midfield Way, Redwood City, CA 94062"
      },
      ...
    ]
  }, ...
}

Scraping Zillow property overview page

From the Homes page, we click on any property.

This will change the browser URL to something similar to:

https://www.zillow.com/homedetails/464-Clinton-St-APT-211-Redwood-City-CA-94062/15638802_zpid/

We will see something like this when we will inspect the page source:

From this page, we will scrape the following attributes:

Price
Address
Bedrooms
Bathrooms
Living area
Status
Overview
Time on Zillow
Views
Saves
Images

Let's define the selectors for each attribute.

/* Price: */
.ds-summary-row span

/* Address */
h1

/* Bedrooms: */
.ds-bed-bath-living-area-container span span

/* Bathrooms: */
.ds-bed-bath-living-area-container button span span

/* Living area: */
.ds-bed-bath-living-area-container span:nth-child(5) span

/* Status: */
.ds-status-details

/* Overview: */
//*[@id='home-details-content']/div/div/div[1]/div[2]/div[2]/div[3]/div/div/div/ul/li[2]/div[3]/div[1]/div[1]

/* Time on Zillow: */
//*[contains(text(),'Time on Zillow')]/../dd[1]/strong

/* Views: */
//*[contains(text(),'Time on Zillow')]/../dd[2]/strong

/* Saves: */
//*[contains(text(),'Time on Zillow')]/../dd[3]/strong

/* Images: */
.media-stream-tile picture img

The payload for our scraping request will be:

{
  "url": "https://www.zillow.com/homedetails/31-Northumberland-Ave-Redwood-City-CA-94063/15567306_zpid/",
  "real_browser": true,
  "wait_for": ".ds-status-details",
  "parse": {
    "price": ".ds-summary-row span >> text",
    "address": "h1 >> text",
    "bedrooms": ".ds-bed-bath-living-area-container span span >> text",
    "bathrooms": ".ds-bed-bath-living-area-container button span span >> text",
    "living_area": ".ds-bed-bath-living-area-container span:nth-child(5) span >> text",
    "status": ".ds-status-details >> text",
    "overview": "//*[@id='home-details-content']/div/div/div[1]/div[2]/div[2]/div[3]/div/div/div/ul/li[2]/div[3]/div[1]/div[1] >> text",
    "time_on_zillow": "//*[contains(text(),'Time on Zillow')]/../dd[1]/strong >> text",
    "views": "//*[contains(text(),'Time on Zillow')]/../dd[2]/strong >> text",
    "saves": "//*[contains(text(),'Time on Zillow')]/../dd[3]/strong >> text",
    "images": [".media-stream-tile picture img >> src"]
  }
}

Running the scraping request with cURL:

curl -v -XPOST -H "Content-type: application/json" -d '{
  "api_key": "'"$API_KEY"'",
  "url": "https://www.zillow.com/homedetails/31-Northumberland-Ave-Redwood-City-CA-94063/15567306_zpid/",
  "real_browser": true,
  "wait_for": ".ds-status-details",
  "parse": {
    "price": ".ds-summary-row span >> text",
    "address": "h1 >> text",
    "bedrooms": ".ds-bed-bath-living-area-container span span >> text",
    "bathrooms": ".ds-bed-bath-living-area-container button span span >> text",
    "living_area": ".ds-bed-bath-living-area-container span:nth-child(5) span >> text",
    "status": ".ds-status-details >> text",
    "overview": "//*[@id=\"home-details-content\"]/div/div/div[1]/div[2]/div[2]/div[3]/div/div/div/ul/li[2]/div[3]/div[1]/div[1] >> text",
    "time_on_zillow": "//*[contains(text(), \"Time on Zillow\")]/../dd[1]/strong >> text",
    "views": "//*[contains(text(),\"Time on Zillow\")]/../dd[2]/strong >> text",
    "saves": "//*[contains(text(),\"Time on Zillow\")]/../dd[3]/strong >> text",
    "images": [".media-stream-tile picture img >> src"]
  }
}' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result:

{
  "result": {
    "price": "$1,299,000",
    "address": "31 Northumberland Ave, Redwood City, CA 94063",
    "bedrooms": "3",
    "bathrooms": "3",
    "living_area": "2,400",
    "status": "For sale",
    "overview": "This move-in ready home with hardwood floors throughout has a large living room, dining room, and spacious kitchen with 2 large bedrooms downstairs and 2 full baths. Upstairs has a spacious loft area, bedroom and bathroom. The backyard is accessible off the dining and kitchen area which has 2 additional units. This home is in a great location near public transportation, shopping, and downtown.Read moreLoft area",
    "time_on_zillow": "33 days",
    "views": "5,291",
    "saves": "96",
    "images": [
      "https://photos.zillowstatic.com/fp/c8a9884ca0f563fd38288ee61a219ec6-cc_ft_960.jpg",
      "https://photos.zillowstatic.com/fp/f9d7a99109da7a7369cdd731e6dc1de1-cc_ft_576.jpg",
      "https://photos.zillowstatic.com/fp/48a0adb1b3d34c0b1f5959ed4474aba5-cc_ft_576.jpg",
      "https://photos.zillowstatic.com/fp/a730702c85aac3d69e6af9b184d0e1c2-cc_ft_576.jpg",
      "https://photos.zillowstatic.com/fp/4e94c21078360505a68d4633237e48b3-cc_ft_576.jpg",
      "https://photos.zillowstatic.com/fp/4559a516478e95d3218095a6b8dd3f64-cc_ft_576.jpg",
      "https://photos.zillowstatic.com/fp/084a2b97777c80766dcd7f27746d18db-cc_ft_576.jpg"
    ]
  }, ...
}

Conclusion

That's it!

In this article, you've discovered the easiest way to scrape a real estate website, such as Zillow, with Page2API - a Web Scraping API that handles any challenges for you.

The original article can be found here:

https://www.page2api.com/blog/how-to-scrape-zillow/

The easiest way to scrape eBay products

Nicolae Rotaru — Mon, 01 Nov 2021 09:58:11 +0000

Introduction

‎

This article will describe the easiest way to scrape eBay products with Page2API

eBay is an online shopping site that's best known for its auctions and consumer-to-consumer sales.

You can scrape eBay if you intend to perform tasks such as:

analyzing products
predicting market trends
price monitoring

Compared to Amazon, scraping eBay can be a little more challenging, but the product page doesn't require a real browser, which leads to faster scraping speed.

For scraping eBay, we will use Page2API - a powerful and delightful API that makes web scraping easy and fun.

We will scrape 2 types of eBay pages:

Products list page
Product page

‎

Prerequisites

‎
To perform this simple task, you will need the following things:

A Page2API account

The free trial offers a credit that covers up to 1000 web pages to scrape.

A product or a category of products that we are about to scrape.

In our case, we will search for 'fitbit 4 silicone strap'
and then scrape the product page for a random product.

‎

Scraping eBay Products List Page

‎
First what we need is to type 'fitbit 4 silicone strap' into the search input from eBay's search page change the view type to Gallery View.

This will change the browser URL to something similar to:

https://www.ebay.com/sch/i.html?_nkw=fitbit+4+silicone+strap&_sacat=0&_dmd=2&rt=nc

The URL is the first parameter we need to perform the scraping.

The page that you see must look like the following one:

‎
If you inspect the page HTML, you will find out that a single result is wrapped into a div that looks like the following:

‎

The HTML for a single result element will look like this:

‎
The last part is the pagination handling.

In our case, we must click on the Next (→) button while the list item's class will be active:

‎

And stop our scraping request when the Next (→) button became disabled.

In our case, a new attribute (aria-disabled="true") is assigned to the Next (→) button:

‎

Now it's time to prepare the request that will scrape all products that the search page returned.

Setting the api_key as an environment variable

export API_KEY=YOUR_PAGE2API_KEY

Running the scraping request with cURL

    curl -v -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.ebay.com/sch/i.html?_nkw=fitbit+4+silicone+strap&_sacat=0&_dmd=2&rt=nc",
      "real_browser": true,
      "merge_loops": true,
      "scenario": [
        {
          "loop" : [
            { "wait_for": ".pagination__next" },
            { "execute": "parse" },
            { "execute_js": "document.querySelector(\"a.pagination__next\").click()" }
          ],
          "stop_condition": "document.querySelector(\"a.pagination__next\").ariaDisabled == \"true\""
        }
      ],
      "parse": {
        "items": [
          {
            "_parent":"ul.srp-grid li.s-item",
            "title":"h3.s-item__title >> text",
            "link":"a.s-item__link >> href",
            "price":".s-item__price >> text",
            "shipping":".s-item__shipping >> text"
          }
        ]
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result

    {
      "result": {
        "items": [
          {
            "title": "For Fitbit Charge 3/4 Replacement Silicone Wristband Straps Sports Watch Band",
            "link": "https://www.ebay.com/itm/174485572199?hash=item28a0268a67:g:1QMAAOSwLA9fjTwQ",
            "price": "$3.89",
            "shipping": "Free shipping"
          },
          {
            "title": "Unisex Silicone Sports Bracelet Strap Fashion Band for Fitbit Inspire/Inspire HR",
            "link": "https://www.ebay.com/itm/324593039446?hash=item4b9340b856:g:5jsAAOSwRLJgh62j",
            "price": "$3.49",
            "shipping": "Free shipping"
          },
          ...
        ]
      },
      ...
    }

‎

Scraping eBay Product Page

‎
From the products list page, we need to click on any product.

The browser URL will look like this:

 https://www.ebay.com/itm/114503614713

The URL is the first parameter we need to perform the product data scraping.

Let's prepare the request that will scrape the needed information from the page.

Setting the api_key as an environment variable (if needed)

export API_KEY=YOUR_PAGE2API_KEY

Running the scraping request with cURL

    curl -v -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.ebay.com/itm/114503614713",
      "parse": {
        "name": "h1 >> text",
        "price": "span[itemprop=price] >> text",
        "condition": "div[itemprop=itemCondition] >> text",
        "categories": [
          "#vi-VR-brumb-lnkLst span >> text"
        ],
        "amount_sold": ".soldwithfeedback a >> text",
        "seller_name": "#RightSummaryPanel span.mbg-nw >> text",
        "item_location": "span[itemprop=availableAtOrFrom] >> text",
        "feedback_score": "#RightSummaryPanel span.mbg-l a >> text",
        "approximate_price": "#convbidPrice >> text"
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result

    {
      "result": {
        "name": "Details about Fitbit Charge 2 3 4 Bracelet Stainless Steel Spare Band Nylon Milanese Sport",
        "price": "GBP 5.66",
        "condition": "New with tags",
        "categories": [
          "Jewelry & Watches",
          "Watches, Parts & Accessories",
          "Watch Accessories",
          "Wristwatch Bands"
        ],
        "amount_sold": "",
        "seller_name": "fro-shop",
        "item_location": "Mülheim, Germany",
        "feedback_score": "28166",
        "approximate_price": "US $7.76(including shipping)"
      },
      ...
    }

‎

Conclusion

‎
That's it!

Scraping eBay product data is a bit tricky because of the HTML structure, but that is not a problem if you have the proper scraping tool, such as Page2API that makes web scraping something you can enjoy.

The original article can be found here:

https://www.page2api.com/blog/scrape-ebay-products/

The Easiest Way to Scrape Amazon Products and Reviews

Nicolae Rotaru — Thu, 21 Oct 2021 14:07:22 +0000

Introduction

Amazon is a vast Internet-based enterprise that sells all kinds of goods either directly or as the middleman between other retailers and Amazon's millions of customers.

You can scrape their data to run different types of tasks such as:

competitor analysis
improving your products and value proposition
identifying market trends and what influences them
price monitoring

Luckily, Amazon is a website that is pretty easy to scrape if you have the right tools.

For this purpose, we will use Page2API - a powerful and delightful API that makes web scraping easy and fun.

In this case, we will scrape 2 types of amazon pages:

Search results page
Product reviews page

Prerequisites

To perform this simple task, you will need the following things:

A Page2API account

The free trial offers a credit that covers up to 1000 web pages to scrape, and it takes under 10 seconds to create the account if you sign up with Google.

A product or a category of products that we are about to scrape.

In our case, we will search for 'luminox watches'
and then scrape the reviews for a random product.

Scraping all products from the search page

First what we need is to type 'luminox watches' into the search input from amazon's search page.

This will change the browser URL to something similar to:

https://www.amazon.com/s?k=luminox+watches

The URL is the first parameter we need to perform the scraping.

The page that you see must look like the following one:

If you inspect the page HTML, you will find out that a single result is wrapped into a div that looks like the following:

The HTML for a single result element will look like this:

The last part is the pagination handling.

In our case, we must click on the Next → button while the list item's class will be active:

And stop our scraping request when the Next → button became disabled.

In our case, a new class (a-disabled) is assigned to the list element where the button is located:

Now it's time to prepare the request that will scrape all products that the search page returned.

Setting the api_key as an environment variable

export API_KEY=YOUR_PAGE2API_KEY

Running the scraping request with cURL

    curl -v -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.amazon.com/s?k=luminox+watches",
      "real_browser": true,
      "merge_loops": true,
      "scenario": [
        {
          "loop" : [
            { "wait_for": ".a-pagination li.a-last" },
            { "execute": "parse" },
            { "execute_js": "document.querySelector(\".a-pagination li.a-last a\").click()" }
          ],
          "stop_condition": "document.querySelector(\".a-last.a-disabled\") !== null"
        }
      ],
      "parse": {
        "watches": [
          {
            "_parent": "[data-component-type='s-search-result']",
            "title": "h2 >> text",
            "link": ".a-link-normal >> href",
            "price": ".a-price-whole >> text",
            "stars": ".a-icon-alt >> text"
          }
        ]
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result

  {
    "result": {
      "watches": [
        {
          "title": "Men's Luminox Leatherback Sea Turtle 44mm Watch",
          "link": "https://www.amazon.com/Luminox-Leatherback-Turtle-Giant-Black/dp/B07CVFWXMR/ref=sr_1_2?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-2",
          "price": "$223.47",
          "stars": "4.5 out of 5 stars"
        },
        {
          "title": "The Original Navy Seal Mens Watch Black Display (XS.3001.F/Navy Seal Series): 200 Meter Water Resistant + Light Weight Case + Constant Night Visibility",
          "link": "https://www.amazon.com/Luminox-Wrist-Watch-Navy-Original/dp/B07NYXV77C/ref=sr_1_3?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-3",
          "price": "$254.12",
          "stars": "4.3 out of 5 stars"
        },
        {
          "title": "Leatherback SEA Turtle Giant - 0323",
          "link": "https://www.amazon.com/Luminox-Leatherback-SEA-Turtle-Giant/dp/B07PBC31N8/ref=sr_1_4?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-4",
          "price": "$179.00",
          "stars": "4.3 out of 5 stars"
        },
        ...
      ]
    },
    ...
  }

Scraping product reviews

First what we need is to click on the See all reviews link from the Product page.

This will change the browser URL to something similar to:

https://www.amazon.com/product-reviews/B072FNJLBC

The URL is the first parameter we need to perform the reviews scraping.

The HTML from a single review will look like this:

Luckily, the pagination handling is similar to the one described above, so we will use the same flow.

Now it's time to prepare the request that will scrape all reviews.

Setting the api_key as an environment variable (if needed)

export API_KEY=YOUR_PAGE2API_KEY

Running the scraping request with cURL

    curl -v -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.amazon.com/product-reviews/B072FNJLBC",
      "real_browser": true,
      "merge_loops": true,
      "scenario": [
        {
          "loop" : [
            { "wait_for": ".a-pagination li.a-last" },
            { "execute": "parse" },
            { "execute_js": "document.querySelector(\".a-pagination li.a-last a\").click()" }
          ],
          "stop_condition": "document.querySelector(\".a-last.a-disabled\") !== null"
        }
      ],
      "parse": {
        "reviews": [
          {
            "_parent": "[data-hook='review']",
            "title": ".review-title >> text",
            "author": ".a-profile-name >> text",
            "stars": ".review-rating >> text",
            "content": ".review-text >> text"
          }
        ]
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result

  {
    "result": {
      "reviews": [
          {
            "title": "Great watch & easy to read in low light conditions",
            "author": "Paul E. Papas",
            "stars": "5.0 out of 5 stars",
            "content": "I'm a 60+ year old equestrian and outdoorsman. I was looking for a watch that could take the shock of firearm discharge ..."
          },
          {
            "title": "Not Water Resistant, impossible to get amazon help",
            "author": "Benjamin H. Curry",
            "stars": "2.0 out of 5 stars",
            "content": "This watch has a 2 year warranty from date of purchase however after not even on full year my adult son went swimming with it ..."
          },
        ...
      ]
    },
    ...
  }

Conclusion

That's pretty much of it!

As mentioned, Amazon is a website that is pretty easy to scrape if you have the right tools.
This is the place where Page2API shines, making web scraping super easy and fun.

The original article can be found here:

https://www.page2api.com/blog/scrape-amazon-product-data