DEV Community: Nicolae Rotaru

How to Monitor Competitors with AI Agents: MCP Setup

Nicolae Rotaru — Mon, 27 Jul 2026 18:18:30 +0000

You still have twelve competitor tabs open. Someone on the team checks them on Monday, forgets by Wednesday, and finds out about a rival's price cut from a prospect on a demo call.

Competitor monitoring should not depend on whoever remembered to look. If you already work with an AI agent in your editor, you can give it structured tools to watch rivals, read interpreted changes, and summarize what needs a human decision.

This guide walks through competitor monitoring with the Competiflow MCP server: create an account, connect your agent, add one competitor on the free trial, and run the five calls that cover setup through weekly triage.

What you'll learn

Why field-level diffs beat bookmark folders and pixel monitors
The five MCP tools that cover setup, review, evidence, digest, and triage
How to create a Competiflow account and API key
How to install the MCP server (with a Cursor example)
How to add Linear as your first competitor and poll async discovery
How to filter changes, pull evidence, and acknowledge what you have reviewed
Three agent workflows PMMs and founders actually keep running

TL;DR: Sign up, create an API key, add the Competiflow MCP server to your agent client, then ask your agent to create a workspace and add a competitor homepage. Discovery runs in the background. After monitors check pages on cadence, your agent reads interpreted changes through list_changes and get_digest instead of you opening tabs.

Why give your agent competitor monitoring tools?

Give your agent competitor monitoring tools when manual bookmark checks stop scaling across pricing, changelog, docs, and homepage copy. The agent polls structured APIs on your schedule instead of you opening tabs.

Most teams start with bookmarks. That works until the list grows. Generic website monitors fill the gap, but they diff pixels or raw HTML. Cookie banners, timestamps, and layout shifts trigger alerts that have nothing to do with pricing or positioning.

Enterprise competitive intelligence platforms solve interpretation, but they often die in a dashboard nobody opens daily. You still end up triaging noise, just in a more expensive inbox.

Competiflow sits in the middle: structured competitor monitoring with field-level diffs, severity, a plain-language summary, why it matters, and evidence you can inspect. One successful page read costs one check, whether or not the page changed.

Your agent does not replace judgment on what to do about a move. It removes the manual polling. Ask for unreviewed pricing changes this week, and it calls the API instead of you opening five pricing pages.

The five MCP calls that cover the workflow

The Competiflow MCP server exposes 18 tools. You do not need all of them on day one. Five calls cover the loop we see teams keep:

Step	Tool	What it does
1	`add_competitor`	Point at a homepage; discovery finds pricing, blog, changelog, docs, and homepage monitors
2	`list_changes`	Filter the change feed by time, severity, and monitor type
3	`get_change`	Pull field diffs and optional capture evidence for one change
4	`get_digest`	Synthesize the workspace: what needs review, who moved, coverage stats
5	`update_change`	Mark a change acknowledged or dismissed so the inbox stays honest

Everything else (create_workspace, get_competitor, refresh_workspace, list_monitors) supports setup and async polling. The MCP docs list the full tool table.

Async by design: add_competitor, check_monitor, and refresh_workspace queue work. They return acknowledgements, not final results. Poll get_competitor during discovery, then read outcomes with list_changes or get_digest.

The same flow is demoed on the homepage agent section. The steps below use live tool names and response shapes from the Competiflow API.

Create your account and API key

You need a Competiflow account and a sk_live_ API key before your agent can call the MCP server. The steps below take about two minutes.

Sign up for Competiflow (free trial includes enough checks to track one competitor across multiple monitors).
Open Settings → API keys in the dashboard.
Create a key. It is shown once, starts with sk_live_, and is stored as a hash on our side. Copy it to a password manager.

If you lose a key, revoke it and create a new one. Full details are in Authentication.

Every MCP and API call sends:

Authorization: Bearer sk_live_your_key_here

Never commit that key to git, a public repo, or a screenshot.

Install the Competiflow MCP server

Install the server globally, or let npx fetch it on each run:

npx -y @competiflow/mcp-server

Set one environment variable:

Variable	Description
`COMPETIFLOW_API_KEY`	Your `sk_live_...` key (required)

The server calls https://api.competiflow.com/v1/.... Claude Desktop, Windsurf, and other MCP clients use the same package with their own config file path. See MCP server docs for client-specific paths.

Cursor

Add to .cursor/mcp.json in your project (or home directory):

{
  "mcpServers": {
    "competiflow": {
      "command": "npx",
      "args": ["-y", "@competiflow/mcp-server"],
      "env": {
        "COMPETIFLOW_API_KEY": "sk_live_your_key_here"
      }
    }
  }
}

Restart Cursor so the server loads. Do not commit .cursor/mcp.json if it contains a real key. Use an env reference or local-only config.

Add your first competitor with your agent

On the free trial you can track one competitor. This walkthrough uses Linear as a public B2B SaaS example. On paid plans, repeat add_competitor for each rival you care about.

1. Create a workspace

Prompt: "Create a Competiflow workspace called 'Acme PM' for our product."

Your agent calls create_workspace:

{
  "workspace": {
    "id": 42,
    "name": "Acme PM",
    "slug": "acme-pm",
    "created_at": "2026-07-19T10:00:00.000Z"
  }
}

If you already have a workspace, list_workspaces is enough.

2. Add Linear

Prompt: "Add https://linear.app to workspace 42. Watch pricing and changelog."

Your agent calls add_competitor:

{
  "url": "https://linear.app",
  "monitor_types": ["pricing", "changelog"]
}

Response (202 Accepted):

{
  "status": "discovering",
  "url": "https://linear.app",
  "competitor": {
    "id": 8,
    "name": "Linear",
    "discovery_status": "running",
    "homepage_url": "https://linear.app"
  }
}

Discovery is asynchronous. Competiflow finds monitor-worthy URLs and seeds baseline checks. That usually takes a few minutes, not seconds.

3. Poll until discovery completes

Prompt: "Check whether discovery finished for competitor 8."

Your agent calls get_competitor until discovery_status is completed. Then list_monitors shows what it found (pricing, blog, changelog, docs, homepage depending on the site).

If you add a second competitor while discovery is still running on another URL, the API returns 409 discovery_in_progress with active_competitor_id. Poll that competitor until it finishes, then retry.

4. Queue fresh checks (optional)

Prompt: "Refresh all monitors in workspace 42."

refresh_workspace queues checks for active monitors. Like discovery, it acknowledges immediately. Read results through list_changes after checks finish.

Day one is often quiet. Right after setup, list_changes may return an empty array. Monitors are running; interpreted changes appear after Competiflow detects a real diff on a future check. The shape is still useful:
{
  "changes": [],
  "has_more": false,
  "meta": {
    "counts": { "unreviewed": 0, "acknowledged": 0, "dismissed": 0 },
    "max_unreviewed_severity": null
  }
}

More detail on competitors and discovery: Competitors API.

Review interpreted changes

Once monitors have baselines and at least one diff, your agent can triage the inbox. The JSON below uses the same response shapes as the live API (verified against MCP calls). Competitor names in the examples match our homepage demos.

Filter the feed

Prompt: "What competitor moves need review this week? Medium severity or higher."

Your agent calls list_changes with since=7d, min_severity=medium, and your workspace_id:

{
  "changes": [
    {
      "id": 901,
      "competitor": "Northwind",
      "monitor_type": "pricing",
      "severity": "high",
      "summary": "Team plan raised from $49 to $79/mo",
      "why_it_matters": "Update pricing battlecard and brief sales",
      "review_status": "unreviewed",
      "category": "pricing",
      "direction": "advanced",
      "delta": { "added": 0, "removed": 0, "modified": 1 },
      "created_at": "2026-07-14T12:30:00.000Z"
    }
  ],
  "has_more": false,
  "meta": {
    "counts": { "unreviewed": 3, "acknowledged": 12, "dismissed": 5 },
    "max_unreviewed_severity": "high"
  }
}

Use monitor_type=pricing when you only want pricing surfaces. Use review_status=unreviewed to skip what you already handled. See Changes API for filters and pagination.

Pull evidence for one change

Prompt: "Show me the before/after for change 901."

get_change with include=capture_markdown returns interpreted field diffs plus page context:

{
  "change": {
    "summary": "Team plan raised from $49 to $79/mo",
    "why_it_matters": "Northwind is 60% more expensive on its most popular plan",
    "field_diffs": [
      {
        "op": "modified",
        "path": "plans[team].price",
        "tier": "alertable",
        "change_type": "price_increase",
        "old": "$49/mo",
        "new": "$79/mo"
      }
    ],
    "why_it_matters": "Update pricing battlecard and brief sales",
    "source_url": "https://northwind.io/pricing"
  }
}

That is field-level evidence, not a pixel diff. You see which plan row moved and what the interpreter recommends, without squinting at a screenshot.

Acknowledge what you reviewed

Prompt: "Mark change 901 as acknowledged."

{
  "change": {
    "id": 901,
    "review_status": "acknowledged"
  }
}

Use dismissed for noise you never want to revisit. Prefer update_competitor or update_monitor with status: paused over delete when you want to keep history.

Three workflows that stick

The three patterns teams keep running are a Monday workspace digest, pricing-only triage before reviews, and a battlecard refresh loop after high-severity changes. Each maps to one or two MCP calls your agent can repeat on a schedule.

Monday workspace digest

Prompt: "Summarize my Competiflow workspace for the past 7 days."

get_digest returns synthesis, not a raw dump:

{
  "digest": {
    "period": "7d",
    "needs_review": { "count": 3, "max_severity": "high" },
    "this_week": {
      "bullets": [
        { "text": "Northwind raised Team pricing to $79/mo", "change_ids": [901] }
      ]
    },
    "whos_moving": [
      { "name": "Northwind", "change_count": 4, "max_severity": "high" }
    ],
    "coverage": { "competitors_count": 1, "monitors_healthy": 5, "monitors_failing": 0 }
  }
}

Paste the bullet list into Slack or email. One call replaces opening every competitor tab.

Pricing-only triage

Prompt: "List unreviewed high-severity pricing changes from the last 30 days."

Combine min_severity=high, monitor_type=pricing, and since=30d. Your agent returns a short list with summaries and why it matters. PMMs use this before pipeline reviews or board prep.

Battlecard refresh loop

When get_change flags a positioning or pricing move, ask your agent to draft a battlecard bullet from summary, why_it_matters, and field_diffs. After you edit the doc, call update_change so the inbox reflects what you handled.

Full digest fields: Digest API.

Agent monitoring vs checking tabs yourself

Agent plus MCP wins when you want interpreted changes on demand inside the editor you already use, without living in a CI dashboard or an email alert queue. Bookmarks and generic page monitors still fit when you track very few rivals or need arbitrary DOM diffs on any URL.

Agent-ready scraping MCPs (Firecrawl, general /monitor APIs) watch arbitrary pages. Competiflow focuses on CI surfaces with field diffs, severity, and why it matters instead of raw HTML noise.

	Bookmark folder	Generic page monitor	Agent + Competiflow MCP
Setup	Free, manual	URL list, selector tuning	Homepage URL; discovery seeds monitors
What changes	Whatever you notice	Pixels or DOM noise	Normalized fields (price, plan, headline)
Interpretation	You write it	You write it	Summary, severity, why it matters
Delivery	You open tabs	Email alert flood	Agent queries on demand or digest
Best for	2–3 rivals	"Tell me any change"	Teams who want signal without a CI dashboard habit

Visualping and Distill are fine when you need arbitrary page diffs. Klue and Crayon fit teams with dedicated CI analysts and budget. Competiflow targets teams who want competitor monitoring tools that stay quiet until something worth reviewing appears, then meet you in the agent you already use.

Frequently Asked Questions

What is competitor monitoring?

Competitor monitoring is the practice of watching rival websites and product surfaces (pricing, changelogs, docs, homepages) for meaningful updates. Good monitoring filters noise and explains why a change might matter to your positioning, pricing, or sales motion. For the full category definition, see What is competitor monitoring?.

What is competitive intelligence?

Competitive intelligence is the broader discipline of turning competitor and market signals into decisions. Monitoring supplies the raw change feed; intelligence is what you do with it in product, marketing, and sales strategy.

How does competitor monitoring work in Competiflow?

You add a competitor homepage. Discovery finds monitor-worthy URLs and runs scheduled checks. Each check compares normalized fields to the last snapshot. When alertable fields change, Competiflow creates an interpreted change with severity, summary, and evidence. Your agent reads those changes through MCP or the REST API.

How do I set up and monitor competitors with an AI agent?

Create a Competiflow account and API key, install the MCP server, and configure your MCP client. Ask your agent to create_workspace, add_competitor with a homepage URL, poll get_competitor until discovery completes, then use list_changes and get_digest for ongoing review.

How do I monitor competitor pricing automatically?

Add the competitor's homepage (discovery usually finds the pricing page). Set cadence to daily on the trial, or faster on paid plans. Filter with list_changes using monitor_type=pricing and min_severity so your agent only surfaces pricing moves.

How do I reduce competitor alert noise?

Use severity filters (min_severity=medium or high), triage with update_change, and prefer field-level monitoring over pixel diffs. Competiflow ignores most layout churn by diffing structured fields, not raw HTML.

How do I track competitors automatically?

Add a rival homepage with add_competitor (one competitor on the free trial; repeat on paid plans). Discovery seeds pricing, changelog, and other monitors on a daily cadence. Your agent calls list_changes or get_digest on a schedule you define instead of you revisiting bookmark folders.

How many competitors can I track on the trial?

The free trial tracks one competitor across multiple monitors (pricing, blog, changelog, etc.). Paid plans raise monitor and check limits. See Billing & checks.

Competitor monitoring gets easier when your agent can call structured tools instead of you polling bookmarks. Sign up, connect the MCP server, add Linear (or whichever rival matters most on your trial), and teach your agent the five-call loop: add, list, inspect, digest, triage.

When you are ready for a second competitor, upgrade your plan and ask your agent to run add_competitor again. Same tools, same inbox shape, less Monday-morning tab sprawl.

Start free · MCP docs · API quickstart

How to Scrape Reddit Data: Posts, Images, Comments, and more.

Nicolae Rotaru — Fri, 27 May 2022 10:39:40 +0000

Introduction

Reddit.com is a social news website and forum where content is socially curated and promoted by site members through voting.

In this article, you will read about the easiest way to scrape Reddit posts and comments with Page2API, and find some code examples in Python.

To scrape Reddit, we will use Page2API - a powerful and delightful API that will help you scrape the web like a PRO.

‎

Prerequisites

To start scraping Reddit posts, we will need the following things:

A Page2API account
The link to a subreddit that we are about to scrape. To make the scraping process easier - we will use the old Reddit UI since it has a simpler HTML structure. In this article, we will scrape the BMW subreddit.

‎

How to scrape Reddit Posts

The first thing we need is to open the subreddit URL:

https://old.reddit.com/r/BMW/

The URL is the first parameter we need to perform the scraping.

The page that you see must look like the following one:

If you inspect the page HTML, you will find out that a single result is wrapped into an element that looks like the following:

From this page, we will scrape the following attributes from each Reddit post:

Title
URL
Thumbnail
Submitted
Author
Author URL
Comments
Rank
Score

Now, let's define the selectors for each attribute.

/* Parent: */
.thing[data-promoted=false]

/* Title */
a.title

/* URL */
a.title

/* Thumbnail */
a.thumbnail img

/* Submitted */
time

/* Author */
a.author

/* Author URL */
a.author

/* Comments */
(parent div, attribute: data-comments-count)

/* Rank */
(parent div, attribute: data-rank

/* Score */
(parent div, attribute: data-score

Now, let's handle the pagination.
We will iterate through the Reddit pages by clicking on the Next page button.

To go to the next page, we must click on the next page link if it's present on the page:

document.querySelector('.next-button a').click()

The scraping will continue while the Next link is present on the page, and stop if it disappears.
The stop condition for the scraper will be the following javascript snippet:

document.querySelector('.next-button a') === null

Let's build the request that will scrape the needed pages from this subreddit.

The following examples will show how to scrape 5 pages of posts from Reddit.com

Our payload will look like this

{
  "api_key": "YOUR_PAGE2API_KEY",
  "url": "https://old.reddit.com/r/BMW/",
  "merge_loops": true,
  "real_browser": true,
  "scenario": [
    {
      "loop": [
        { "wait_for": ".thing" },
        { "execute": "parse" },
        { "click": ".next-button a" }
      ],
      "stop_condition": "document.querySelector('.next-button a') === null",
      "iterations": 5
    }
  ],
  "parse": {
    "posts": [
      {
        "_parent": ".thing[data-promoted=false]",
        "title": "a.title >> text",
        "url": "a.title >> href",
        "thumbnail": "a.thumbnail img >> src",
        "submitted": "time >> datetime",
        "author": "a.author >> text",
        "author_url": "a.author >> href",
        "comments": "_parent >> data-comments-count",
        "rank": "_parent >> data-rank",
        "score": "_parent >> data-score"
      }
    ]
  }
}

Python code example

import requests
import json

api_url = 'https://www.page2api.com/api/v1/scrape'

The following example will show how to scrape 5 pages of posts from Reddit.com

payload = {
  "api_key": "YOUR_PAGE2API_KEY",
  "url": "https://old.reddit.com/r/BMW/",
  "merge_loops": True,
  "real_browser": True,
  "scenario": [
    {
      "loop": [
        { "wait_for": ".thing" },
        { "execute": "parse" },
        { "click": ".next-button a" }
      ],
      "stop_condition": "document.querySelector('.next-button a') === null",
      "iterations": 5
    }
  ],
  "parse": {
    "posts": [
      {
        "_parent": ".thing[data-promoted=false]",
        "title": "a.title >> text",
        "url": "a.title >> href",
        "thumbnail": "a.thumbnail img >> src",
        "submitted": "time >> datetime",
        "author": "a.author >> text",
        "author_url": "a.author >> href",
        "comments": "_parent >> data-comments-count",
        "rank": "_parent >> data-rank",
        "score": "_parent >> data-score"
      }
    ]
  }
}

headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
response = requests.post(api_url, data=json.dumps(payload), headers=headers)
result = json.loads(response.text)

print(result)

The result

{
  "result": {
    "posts": [
      {
          "title": "Thought you would enjoy this",
          "url": "https://old.reddit.com/r/BMW/comments/uu50bl/thought_you_would_enjoy_this/",
          "thumbnail": "https://b.thumbs.redditmedia.com/1OMVuKzvOeVi6IkOMc1K94HzPOLs3InozhCPlxU8K7E.jpg",
          "submitted": "2022-05-20T18:55:11+00:00",
          "author": "L9FanboyXD",
          "author_url": "https://old.reddit.com/user/L9FanboyXD",
          "comments": "206",
          "rank": "1",
          "score": "1234"
      },
      {
          "title": "Totalled..",
          "url": "https://old.reddit.com/r/BMW/comments/uu7yih/totalled/",
          "thumbnail": "https://b.thumbs.redditmedia.com/QQ7_MbJqhB5t4QfKJjbTFU4sEK0Cd9Cy5uFCSzzeLzM.jpg",
          "submitted": "2022-05-20T21:16:40+00:00",
          "author": "Conscious_Champion35",
          "author_url": "https://old.reddit.com/user/Conscious_Champion35",
          "comments": "130",
          "rank": "2",
          "score": "375"
      },
      {
          "title": "Joined the squad. Yes there was a flat bed at the dealer as I was closing this deal today.",
          "url": "https://www.reddit.com/gallery/uucgb5",
          "thumbnail": "https://b.thumbs.redditmedia.com/g8vAVq0N59R0a-IX1mBtM1YEy2gouhhGTTS7m4Nr8us.jpg",
          "submitted": "2022-05-21T01:16:08+00:00",
          "author": "restloy",
          "author_url": "https://old.reddit.com/user/restloy",
          "comments": "22",
          "rank": "3",
          "score": "157"
      }, ...
    ]
  }, ...
}

‎

How to scrape Reddit Comments

We need to open any post URL from the previous step.

This will change the browser URL to something similar to:

https://old.reddit.com/r/BMW/comments/uu50bl/thought_you_would_enjoy_this/

This URL is the first parameter we need to scrape the Reddit post data and comments.

The comments section that you see must look like the following one:

From this page, we will scrape the following attributes:

Post data:

Title
Image

Comments data:

Author
Author URL
Posted
Score
ID
Content

Now, let's define the selectors for each attribute.

Post data

/* Title */
a.title

/* Image */
.media-preview-content img.preview

Comments data

/* Parent */
.entry

/* Author */
a.author

/* Author URL */
a.author

/* Posted */
time

/* Score */
.score.unvoted

/* ID */
input[name=thing_id]

/* Content */
.usertext-body

It's time to prepare the request that will scrape the data from a Reddit post page.

The payload for our scraping request will be:

{
  "api_key": "YOUR_PAGE2API_KEY",
  "url": "https://old.reddit.com/r/BMW/comments/urv3mr/i_saw_it_in_prague_the_red_is_much_prettier_in/",
  "parse": {
    "title": "a.title >> text",
    "image": ".media-preview-content img.preview >> src",
    "comments": [
      {
        "_parent": ".entry",
        "author": "a.author >> text",
        "author_url": "a.author >> href",
        "posted": "time >> datetime",
        "score": ".score.unvoted >> text",
        "id": "input[name=thing_id] >> value",
        "content": ".usertext-body >> text"
      }
    ]
  }
}

Python code example:

import requests
import json

api_url = 'https://www.page2api.com/api/v1/scrape'
payload = {
  "api_key": "YOUR_PAGE2API_KEY",
  "url": "https://old.reddit.com/r/BMW/comments/urv3mr/i_saw_it_in_prague_the_red_is_much_prettier_in/",
  "parse": {
    "title": "a.title >> text",
    "image": ".media-preview-content img.preview >> src",
    "comments": [
      {
        "_parent": ".entry",
        "author": "a.author >> text",
        "author_url": "a.author >> href",
        "posted": "time >> datetime",
        "score": ".score.unvoted >> text",
        "id": "input[name=thing_id] >> value",
        "content": ".usertext-body >> text"
      }
    ]
  }
}

headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
response = requests.post(api_url, data=json.dumps(payload), headers=headers)
result = json.loads(response.text)

print(result)

The result:

{
  "result": {
    "title": "I saw it in prague. the red is much prettier in real life",
    "image": "https://preview.redd.it/91x8qz36f3091.jpg?width=1023&auto=png&s=9c0aced50d6743989f053cf9cc78c896f928d8a7",
    "comments": [
      {
        "author": "DarkMeta81",
        "author_url": "https://old.reddit.com/user/DarkMeta81",
        "posted": "2022-05-17T20:12:31+00:00",
        "score": "15 points",
        "id": "t1_i8zlp77",
        "content": "i was on vacation and prague is full of beautiful bmw this m5 is great"
      },
      {
        "author": "Id-atl",
        "author_url": "https://old.reddit.com/user/Id-atl",
        "posted": "2022-05-18T01:06:15+00:00",
        "score": "5 points",
        "id": "t1_i90on16",
        "content": "Wow imagine being in Prague ripping this around!"
      }, ...
    ]
  }, ...
}

‎

Conclusion

That's it!

In this article, you've learned how to scrape the data from Reddit.com with Page2API - a Web Scraping API that handles all the hassle, and lets you get the data you need with ease.

The original article can be found here:

https://www.page2api.com/blog/how-to-scrape-reddit/

How to Scrape Glassdoor Reviews

Nicolae Rotaru — Wed, 12 Jan 2022 23:56:48 +0000

Introduction

In this article, you will learn how to scrape company reviews from Glassdoor with Page2API

Glassdoor.com is an American website where current and former employees anonymously review companies.

‎

Disclaimer:

We highly recommend you to scrape Glassdoor only for personal use.

For example: let's say you are looking for a new job and you want to quickly analyze the reviews for the company that you are interested in.

‎

Prerequisites

To start scraping Glassdoor reviews, we will need the following things:

A Page2API account
A company name that we are interested in. In our case, the company that we are interested in is... Glassdoor. (Which also has reviews on its website)

‎

How to scrape Glassdoor Reviews

First what we need is to open glassdoor.com and type Glassdoor reviews into the search input.

This will change the browser URL to something similar to:

https://www.glassdoor.com/Reviews/Glassdoor-Reviews-E100431.htm

We will use this URL as the first parameter we need to start the scraping process.

The page that you see must look like the following one:

If you inspect the page HTML, you will find out that a single review looks like the following:

From the Glassdoor Reviews page, we will scrape the following attributes from each review:

Title
Author Info
Rating
Pros
Cons
Helpful

Now, let's define the selectors for each attribute.

/* Parent: */
div.gdReview

/* Title */
a.reviewLink

/* Author Info */
.authorInfo

/* Rating */
span.ratingNumber

/* Pros */
span[data-test=pros]

/* Cons */
span[data-test=cons]

/* Helpful */
div.common__EiReviewDetailsStyle__socialHelpfulcontainer

Let's take a look at the pagination handling.

To go to the next page, we must click on the next page button if it's present on the page:

document.querySelector(".nextButton").click()

The scraping will continue while the next page button is present on the page, and stop if it disappears.

The stop condition for the scraper will be the following javascript snippet:

document.querySelector(".nextButton") === null

// but to avoid timeouts, we will scrape a fixed amount of pages (see the payload below)

Now it's time to build the request that will scrape Glassdoor reviews.

The payload for our scraping request will be:

{
  "url": "https://www.glassdoor.com/Reviews/Glassdoor-Reviews-E100431.htm",
  "real_browser": true,
  "merge_loops": true,
  "premium_proxy": "us",
  "scenario": [
    {
      "loop": [
        { "wait_for": "div.gdReview" },
        { "execute": "parse" },
        { "execute_js": "document.querySelector(\".nextButton\").click()" }
      ],
      "iterations": 2
    }
  ],
  "parse": {
    "reviews": [
      {
        "_parent": "div.gdReview",
        "title": "a.reviewLink >> text",
        "author_info": ".authorInfo >> text",
        "rating": "span.ratingNumber >> text",
        "pros": "span[data-test=pros] >> text",
        "cons": "span[data-test=cons] >> text",
        "helpful": "div.common__EiReviewDetailsStyle__socialHelpfulcontainer >> text"
      }
    ]
  }
}

Setting the api_key as an environment variable:

export API_KEY=YOUR_PAGE2API_KEY

Running the scraping request with cURL:

curl -v -XPOST -H "Content-type: application/json" -d '{
  "api_key": "'"$API_KEY"'",
  "url": "https://www.glassdoor.com/Reviews/Glassdoor-Reviews-E100431.htm",
  "real_browser": true,
  "merge_loops": true,
  "premium_proxy": "us",
  "scenario": [
    {
      "loop": [
        { "wait_for": "div.gdReview" },
        { "execute": "parse" },
        { "execute_js": "document.querySelector(\".nextButton\").click()" }
      ],
      "iterations": 2
    }
  ],
  "parse": {
    "reviews": [
      {
        "_parent": "div.gdReview",
        "title": "a.reviewLink >> text",
        "author_info": ".authorInfo >> text",
        "rating": "span.ratingNumber >> text",
        "pros": "span[data-test=pros] >> text",
        "cons": "span[data-test=cons] >> text",
        "helpful": "div.common__EiReviewDetailsStyle__socialHelpfulcontainer >> text"
      }
    ]
  }
}' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result:

{
  "result": {
    "reviews": [
      {
        "title": "Glassdoor Walks the Walk",
        "author_info": "Jan 7, 2022 - Senior Manager",
        "rating": "5.0",
        "pros": "Glassdoor creates a positive environment for employees to learn and grow. ...",
        "cons": "At any organization, there is always room for improvement. ...",
        "helpful": "1 person found this review helpful"
      },
      {
        "title": "Great Company To Work For",
        "author_info": "Jan 5, 2022 - Customer Success Manager",
        "rating": "4.0",
        "pros": "I absolutely love working at Glassdoor. ...",
        "cons": "While we do have more of an extensive career growth plan, ...",
        "helpful": "2 people found this review helpful"
      }, ...
    ]
  }, ...
}

‎

Conclusion

Done!

We just finished scraping the reviews from Glassdoor, and it turned to be easy and fun if we have the proper scraping tool.

The original article can be found here:

page2api.com/blog/how-to-scrape-glassdoor-reviews/

How to Scrape Indeed Data: Jobs, Salaries, and more.

Nicolae Rotaru — Sun, 09 Jan 2022 10:30:44 +0000

Introduction

Indeed.com is a job board that aggregates postings, allowing users to search for specific positions.

Why may we need to web scrape Indeed?
Collecting job listings from Indeed will help us to:

search for hirings
analyze the demand for specific job positions
analyze the average salaries

‎

Prerequisites

To start scraping Indeed jobs, we will need the following things:

A Page2API account
A job position in a specific location that we are about to scrape. In our case, we will search for Ruby On Rails Software Engineer in Redwood City, CA, and set the area to within 10 miles.

‎

How to scrape Indeed Jobs

First what we need is to open indeed.com and type Ruby On Rails Software Engineer into the search input from the Indeed home page and pick the location we need.

This will change the browser URL to something similar to:

https://www.indeed.com/jobs?q=Ruby%20On%20Rails%20Software%20Engineer&l=Redwood%20City%2C%20CA&radius=10

The URL is the first parameter we need to perform the scraping.

The page that you see must look like the following one:

If you inspect the page HTML, you will find out that a single result is wrapped into an element that looks like the following:

From this page, we will scrape the following attributes from each Indeed job posting:

Title
URL
Company
Location
Rating
Additional info

Now, let's define the selectors for each attribute.

/* Parent: */
a.result

/* Title */
h2.jobTitle

/* URL */
a.result

/* Company */
.companyName

/* Location */
.companyLocation

/* Rating */
.ratingNumber span[aria-hidden=true]

/* Additional info */
.metadata div

It's time to handle the pagination.

To go to the next page, we must click on the next page link if it's present on the page:

 var next = document.querySelector('a[aria-label=Next]'); if(next) { next.click() }

  // we have this simple check to avoid any javascript errors (in case the Next page button is missing)

The scraping will continue while the Next link is present on the page, and stop if it disappears.
The stop condition for the scraper will be the following javascript snippet:

document.querySelector('a[aria-label=Next]') == null

Let's build the request that will scrape all the results that the search page returned.

The payload for our scraping request will be:

{
  "url": "https://www.indeed.com/jobs?q=Ruby%20On%20Rails%20Software%20Engineer&l=Redwood%20City%2C%20CA&radius=10",
  "real_browser": true,
  "merge_loops": true,
  "scenario": [
    {
      "loop": [
        {
          "wait_for": "a.result"
        },
        {
          "execute": "parse"
        },
        {
          "execute_js": "var next = document.querySelector('a[aria-label=Next]'); if(next) { next.click() }"
        }
      ],
      "stop_condition": "document.querySelector('a[aria-label=Next]') == null"
    }
  ],
  "parse": {
    "jobs": [
      {
        "_parent": "a.result",
        "url": "_parent >> href",
        "title": "h2.jobTitle >> text",
        "company": ".companyName >> text",
        "location": ".companyLocation >> text",
        "rating": ".ratingNumber span[aria-hidden=true] >> text",
        "additional_info": [
          ".metadata div >> text"
        ]
      }
    ]
  }
}

Setting the api_key as an environment variable:

 export API_KEY=YOUR_PAGE2API_KEY

Running the scraping request with cURL:

curl -v -XPOST -H "Content-type: application/json" -d '{
  "api_key": "'"$API_KEY"'",
  "url": "https://www.indeed.com/jobs?q=Ruby%20On%20Rails%20Software%20Engineer&l=Redwood%20City%2C%20CA&radius=10",
  "merge_loops": true,
  "real_browser": true,
  "scenario": [
    {
      "loop": [
        { "wait_for": "a.result" },
        { "execute": "parse" },
        { "execute_js": "var next = document.querySelector(\"a[aria-label=Next]\"); if(next) { next.click() }" }
      ],
      "stop_condition": "document.querySelector(\"a[aria-label=Next]\") == null"
    }
  ],
  "parse": {
    "jobs": [
      {
        "_parent": "a.result",
        "url": "_parent >> href",
        "title": "h2.jobTitle >> text",
        "company": ".companyName >> text",
        "location": ".companyLocation >> text",
        "rating": ".ratingNumber span[aria-hidden=true] >> text",
        "additional_info": [
            ".metadata div >> text"
        ]
      }
    ]
  }
}' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result:

{
  "result": {
    "places": [
      {
        "url": "https://www.indeed.com/company/Coupa/jobs/Senior-Lead-Software-Engineer-fa676bc66ad1daae?fccid=c6a1779d65543307&vjs=3",
        "title": "Senior/Lead Software Engineer, Ruby on Rails",
        "company": "Coupa Software",
        "location": "San Mateo, CA 94402 (Nineteenth Avenue area)+1 location",
        "rating": "3.9",
        "additional_info": [
          "$145,000 - $165,000 a year",
          "Full-time",
          "8 hour shift"
        ]
      },
      {
        "url": "https://www.indeed.com/company/Poshmark/jobs/Software-Engineer-e55c033766067a6c?fccid=0f4f2d112db7d324&vjs=3",
        "title": "Software Engineer, Web Applications",
        "company": "Poshmark",
        "location": "Redwood City, CA",
        "rating": "4.6",
        "additional_info": [
          "Full-time",
        ]
      },
      {
        "url": "https://www.indeed.com/rc/clk?jk=0d91d46d5d05c6ad&fccid=c6a1779d65543307&vjs=3",
        "title": "Software Engineer, Ruby on Rails",
        "company": "Coupa Software",
        "location": "San Mateo, CA",
        "rating": "3.9",
        "additional_info": [
          "Remote",
        ]
      }, ...
    ]
  }, ...
}

‎

How to scrape Indeed Job Page

We need to open any URL from the previous step with the job listing in a new tab.

This will change the browser URL to something similar to:

https://www.indeed.com/viewjob?jk=0d91d46d5d05c6ad

This URL is the first parameter we need to scrape all the information about a job.

The page that you see must look like the following one:

From this page, we will scrape the following attributes:

Title
Company
Rating
Reviews count
Description

Now, let's define the selectors for each attribute.

/* Title */
h1

/* Company */
.jobsearch-InlineCompanyRating a

/* Rating */
meta[itemprop=ratingValue]

/* Reviews count */
meta[itemprop=ratingCount]

/* Description */
#jobDescriptionText

It's time to prepare the request that will scrape Indeed Job Page.

The payload for our scraping request will be:

{
  "url": "https://www.indeed.com/viewjob?jk=0d91d46d5d05c6ad",
  "parse": {
    "title": "h1 >> text",
    "company": ".jobsearch-InlineCompanyRating a >> text",
    "rating": "meta[itemprop=ratingValue] >> content",
    "reviews_count": "meta[itemprop=ratingCount] >> content",
    "description": "#jobDescriptionText >> text"
  }
}

Running the scraping request with cURL:

curl -v -XPOST -H "Content-type: application/json" -d '{
  "api_key": "'"$API_KEY"'",
  "url": "https://www.indeed.com/viewjob?jk=0d91d46d5d05c6ad",
  "parse": {
    "title": "h1 >> text",
    "company": ".jobsearch-InlineCompanyRating a >> text",
    "rating": "meta[itemprop=ratingValue] >> content",
    "reviews_count": "meta[itemprop=ratingCount] >> content",
    "description": "#jobDescriptionText >> text"
  }
}' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result:

{
  "result": {
    "title": "Software Engineer, Ruby on Rails",
    "company": "Coupa Software",
    "rating": "3.9",
    "reviews_count": "27",
    "description": "Coupa Software (NASDAQ: COUP), a leader in business spend management (BSM), ..."
  }, ...
}

‎

Conclusion

That's pretty much of it!

In this article, you've learned how to scrape the data from a job board such as Indeed.com with Page2API - a Web Scraping API that handles all the hassle, and lets you get the data you need with ease.

The original article can be found here:

page2api.com/blog/how-to-scrape-indeed/

How to scrape Zillow in no time

Nicolae Rotaru — Mon, 22 Nov 2021 15:51:40 +0000

Introduction

Zillow is a real estate company that offers various products targeted at both home buyers and sellers.

You can scrape listings from Zillow, with data including addresses, prices, descriptions, photos, URLs to perform:

price monitoring
trends analysis
competitor analysis

Challenges

At first look, scraping Zillow doesn't seem to be a trivial task because of the following aspects:

The content from the listing page is returned dynamically, based on scrolling events.
The names of the CSS selectors are dynamically generated and cannot be used to pick the needed content, and we will use XPath selectors instead.

For this purpose, we will use Page2API - the scraping API that overtakes the challenges mentioned above with ease.

In this article, we will describe how to scrape two types of Zillow pages:

Property listings page
Property overview page

Prerequisites

To start scraping, you will need the following things:

A Page2API account
A location, in which we want to search for listed properties, let's use for example Redwood City
A property overview page from Zillow. We will pick a random property link from the page mentioned above.

Scraping Zillow listings page

First what we need is to open the Homes page and type the name of the city that will show the properties we are searching for.

In our case we will open this page:

https://www.zillow.com/homes/

and search for Redwood City

It will change the browser URL to something similar to:

https://www.zillow.com/homes/Redwood-City,-CA_rb/

The resulted URL is the first parameter we need to start scraping the listings page.

The listings page must look similar to the following one:

If you inspect the page HTML, you will find out that a single result is wrapped into an element that looks like the following:

The HTML for a single result element will look like this:

From the listing page, we will scrape the following attributes from each property:

Price
URL
Bedrooms
Bathrooms
Living area
Status
Address

Each property container is wrapped in an article element with the following class: list-card.

Now, let's define the selectors for each attribute.

/* Parent: */
article.list-card

/* Price: */
.list-card-price

/* URL: */
a

/* Bedrooms: */
ul.list-card-details li:nth-child(1)

/* Bathrooms: */
ul.list-card-details li:nth-child(2)

/* Living area: */
ul.list-card-details li:nth-child(3)

/* Status: */
ul.list-card-details li:nth-child(4)

/* Address: */
a address.list-card-addr

Next is the pagination handling.

In our case, we must click on the next page link while the link will be active:

var next = document.querySelector('.search-pagination a[rel=next]'); if(next){ next.click() }

And stop our scraping request when the next page link became disabled.

In our case, a new attribute (disabled) is assigned to the pagination link.

The stop condition for the pagination will look like this:

var next = document.querySelector('.search-pagination a[rel=next]'); next === null || next.getAttributeNames().includes('disabled')

// returns true if there is no next page

The last thing is handling the content that loads dynamically when we scroll down.

Usually, there are 40 items on the page, but when the page loads - it has only about 8 items.

To load all items we will do the next trick:

Wait for the page to load
Scroll down 3 times slowly, (with a short delay) until we see the last item
Start scraping the page

Now let's build the request that will scrape all properties that the search page returned.

Setting the api_key as an environment variable:

export API_KEY=YOUR_PAGE2API_KEY

The payload for our scraping request will be:

{
  "url": "https://www.zillow.com/homes/Redwood-City,-CA_rb/",
  "real_browser": true,
  "merge_loops": true,
  "premium_proxy": "us",
  "scenario": [
    {
      "loop": [
        { "wait_for": ".search-pagination a[rel=next]" },
        { "execute_js": "var articles = document.querySelectorAll('article')"},
        { "execute_js": "articles[Math.round(articles.length/4)].scrollIntoView({behavior: 'smooth'})"},
        { "wait": 1 },
        { "execute_js": "articles[Math.round(articles.length/2)].scrollIntoView({behavior: 'smooth'})"},
        { "wait": 1 },
        { "execute_js": "articles[Math.round(articles.length/1.5)].scrollIntoView({behavior: 'smooth'})"},
        { "wait": 1 },
        { "execute": "parse"},
        { "execute_js": "var next = document.querySelector('.search-pagination a[rel=next]'); if(next){ next.click() }" }
      ],
    "stop_condition": "var next = document.querySelector('.search-pagination a[rel=next]'); next === null || next.getAttributeNames().includes('disabled')"
  }

],
  "parse": {
    "properties": [
      {
        "_parent": "article.list-card",
        "price": ".list-card-price >> text",
        "url": "a >> href",
        "bedrooms": "ul.list-card-details li:nth-child(1) >> text",
        "bathrooms": "ul.list-card-details li:nth-child(2) >> text",
        "living_area": "ul.list-card-details li:nth-child(3) >> text",
        "status": "ul.list-card-details li:nth-child(4) >> text",
        "address": "a address.list-card-addr >> text"
      }
    ]
  }
}

We have to encode our JS snippets in base64 to run the request in the terminal with cURL.

Running the scraping request with cURL:

curl -v -XPOST -H "Content-type: application/json" -d '{
  "api_key": "'"$API_KEY"'",
  "url": "https://www.zillow.com/homes/Redwood-City,-CA_rb/",
  "real_browser": true,
  "merge_loops": true,
  "premium_proxy": "us",
  "scenario": [
    {
      "loop" : [
        { "wait_for": ".search-pagination a[rel=next]" },
        { "execute_js": "dmFyIGFydGljbGVzID0gZG9jdW1lbnQucXVlcnlTZWxlY3RvckFsbCgnYXJ0aWNsZScp"},
        { "execute_js": "YXJ0aWNsZXNbTWF0aC5yb3VuZChhcnRpY2xlcy5sZW5ndGgvNCldLnNjcm9sbEludG9WaWV3KHtiZWhhdmlvcjogJ3Ntb290aCd9KQ=="},
        { "wait": 1 },
        { "execute_js": "YXJ0aWNsZXNbTWF0aC5yb3VuZChhcnRpY2xlcy5sZW5ndGgvMildLnNjcm9sbEludG9WaWV3KHtiZWhhdmlvcjogJ3Ntb290aCd9KQ=="},
        { "wait": 1 },
        { "execute_js": "YXJ0aWNsZXNbTWF0aC5yb3VuZChhcnRpY2xlcy5sZW5ndGgvMS41KV0uc2Nyb2xsSW50b1ZpZXcoe2JlaGF2aW9yOiAnc21vb3RoJ30p"},
        { "wait": 1 },
        { "execute": "parse"},
        { "execute_js": "dmFyIG5leHQgPSBkb2N1bWVudC5xdWVyeVNlbGVjdG9yKCcuc2VhcmNoLXBhZ2luYXRpb24gYVtyZWw9bmV4dF0nKTsgaWYobmV4dCl7IG5leHQuY2xpY2soKSB9" }
      ],
    "stop_condition": "dmFyIG5leHQgPSBkb2N1bWVudC5xdWVyeVNlbGVjdG9yKCcuc2VhcmNoLXBhZ2luYXRpb24gYVtyZWw9bmV4dF0nKTsgbmV4dCA9PT0gbnVsbCB8fCBuZXh0LmdldEF0dHJpYnV0ZU5hbWVzKCkuaW5jbHVkZXMoJ2Rpc2FibGVkJyk="
    }
  ],
  "parse": {
    "properties": [
      {
        "_parent": "article.list-card",
        "price": ".list-card-price >> text",
        "url": "a >> href",
        "bedrooms": "ul.list-card-details li:nth-child(1) >> text",
        "bathrooms": "ul.list-card-details li:nth-child(2) >> text",
        "living_area": "ul.list-card-details li:nth-child(3) >> text",
        "status": "ul.list-card-details li:nth-child(4) >> text",
        "address": "a address.list-card-addr >> text"
      }
    ]
  }
}' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result:

{
  "result": {
    "properties": [
      {
        "price": "$600,000",
        "url": "https://www.zillow.com/homedetails/464-Clinton-St-APT-211-Redwood-City-CA-94062/15638802_zpid/",
        "bedrooms": "1 bd",
        "bathrooms": "1 ba",
        "living_area": "761 sqft",
        "status": "- Condo for sale",
        "address": "464 Clinton St APT 211, Redwood City, CA 94062"
      },
      {
        "price": "$2,498,000",
        "url": "https://www.zillow.com/homedetails/3618-Midfield-Way-Redwood-City-CA-94062/15571874_zpid/",
        "bedrooms": "4 bds",
        "bathrooms": "4 ba",
        "living_area": "2,960 sqft",
        "status": "- House for sale",
        "address": "3618 Midfield Way, Redwood City, CA 94062"
      },
      ...
    ]
  }, ...
}

Scraping Zillow property overview page

From the Homes page, we click on any property.

This will change the browser URL to something similar to:

https://www.zillow.com/homedetails/464-Clinton-St-APT-211-Redwood-City-CA-94062/15638802_zpid/

We will see something like this when we will inspect the page source:

From this page, we will scrape the following attributes:

Price
Address
Bedrooms
Bathrooms
Living area
Status
Overview
Time on Zillow
Views
Saves
Images

Let's define the selectors for each attribute.

/* Price: */
.ds-summary-row span

/* Address */
h1

/* Bedrooms: */
.ds-bed-bath-living-area-container span span

/* Bathrooms: */
.ds-bed-bath-living-area-container button span span

/* Living area: */
.ds-bed-bath-living-area-container span:nth-child(5) span

/* Status: */
.ds-status-details

/* Overview: */
//*[@id='home-details-content']/div/div/div[1]/div[2]/div[2]/div[3]/div/div/div/ul/li[2]/div[3]/div[1]/div[1]

/* Time on Zillow: */
//*[contains(text(),'Time on Zillow')]/../dd[1]/strong

/* Views: */
//*[contains(text(),'Time on Zillow')]/../dd[2]/strong

/* Saves: */
//*[contains(text(),'Time on Zillow')]/../dd[3]/strong

/* Images: */
.media-stream-tile picture img

The payload for our scraping request will be:

{
  "url": "https://www.zillow.com/homedetails/31-Northumberland-Ave-Redwood-City-CA-94063/15567306_zpid/",
  "real_browser": true,
  "wait_for": ".ds-status-details",
  "parse": {
    "price": ".ds-summary-row span >> text",
    "address": "h1 >> text",
    "bedrooms": ".ds-bed-bath-living-area-container span span >> text",
    "bathrooms": ".ds-bed-bath-living-area-container button span span >> text",
    "living_area": ".ds-bed-bath-living-area-container span:nth-child(5) span >> text",
    "status": ".ds-status-details >> text",
    "overview": "//*[@id='home-details-content']/div/div/div[1]/div[2]/div[2]/div[3]/div/div/div/ul/li[2]/div[3]/div[1]/div[1] >> text",
    "time_on_zillow": "//*[contains(text(),'Time on Zillow')]/../dd[1]/strong >> text",
    "views": "//*[contains(text(),'Time on Zillow')]/../dd[2]/strong >> text",
    "saves": "//*[contains(text(),'Time on Zillow')]/../dd[3]/strong >> text",
    "images": [".media-stream-tile picture img >> src"]
  }
}

Running the scraping request with cURL:

curl -v -XPOST -H "Content-type: application/json" -d '{
  "api_key": "'"$API_KEY"'",
  "url": "https://www.zillow.com/homedetails/31-Northumberland-Ave-Redwood-City-CA-94063/15567306_zpid/",
  "real_browser": true,
  "wait_for": ".ds-status-details",
  "parse": {
    "price": ".ds-summary-row span >> text",
    "address": "h1 >> text",
    "bedrooms": ".ds-bed-bath-living-area-container span span >> text",
    "bathrooms": ".ds-bed-bath-living-area-container button span span >> text",
    "living_area": ".ds-bed-bath-living-area-container span:nth-child(5) span >> text",
    "status": ".ds-status-details >> text",
    "overview": "//*[@id=\"home-details-content\"]/div/div/div[1]/div[2]/div[2]/div[3]/div/div/div/ul/li[2]/div[3]/div[1]/div[1] >> text",
    "time_on_zillow": "//*[contains(text(), \"Time on Zillow\")]/../dd[1]/strong >> text",
    "views": "//*[contains(text(),\"Time on Zillow\")]/../dd[2]/strong >> text",
    "saves": "//*[contains(text(),\"Time on Zillow\")]/../dd[3]/strong >> text",
    "images": [".media-stream-tile picture img >> src"]
  }
}' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result:

{
  "result": {
    "price": "$1,299,000",
    "address": "31 Northumberland Ave, Redwood City, CA 94063",
    "bedrooms": "3",
    "bathrooms": "3",
    "living_area": "2,400",
    "status": "For sale",
    "overview": "This move-in ready home with hardwood floors throughout has a large living room, dining room, and spacious kitchen with 2 large bedrooms downstairs and 2 full baths. Upstairs has a spacious loft area, bedroom and bathroom. The backyard is accessible off the dining and kitchen area which has 2 additional units. This home is in a great location near public transportation, shopping, and downtown.Read moreLoft area",
    "time_on_zillow": "33 days",
    "views": "5,291",
    "saves": "96",
    "images": [
      "https://photos.zillowstatic.com/fp/c8a9884ca0f563fd38288ee61a219ec6-cc_ft_960.jpg",
      "https://photos.zillowstatic.com/fp/f9d7a99109da7a7369cdd731e6dc1de1-cc_ft_576.jpg",
      "https://photos.zillowstatic.com/fp/48a0adb1b3d34c0b1f5959ed4474aba5-cc_ft_576.jpg",
      "https://photos.zillowstatic.com/fp/a730702c85aac3d69e6af9b184d0e1c2-cc_ft_576.jpg",
      "https://photos.zillowstatic.com/fp/4e94c21078360505a68d4633237e48b3-cc_ft_576.jpg",
      "https://photos.zillowstatic.com/fp/4559a516478e95d3218095a6b8dd3f64-cc_ft_576.jpg",
      "https://photos.zillowstatic.com/fp/084a2b97777c80766dcd7f27746d18db-cc_ft_576.jpg"
    ]
  }, ...
}

Conclusion

That's it!

In this article, you've discovered the easiest way to scrape a real estate website, such as Zillow, with Page2API - a Web Scraping API that handles any challenges for you.

The original article can be found here:

https://www.page2api.com/blog/how-to-scrape-zillow/

The easiest way to scrape eBay products

Nicolae Rotaru — Mon, 01 Nov 2021 09:58:11 +0000

Introduction

‎

This article will describe the easiest way to scrape eBay products with Page2API

eBay is an online shopping site that's best known for its auctions and consumer-to-consumer sales.

You can scrape eBay if you intend to perform tasks such as:

analyzing products
predicting market trends
price monitoring

Compared to Amazon, scraping eBay can be a little more challenging, but the product page doesn't require a real browser, which leads to faster scraping speed.

For scraping eBay, we will use Page2API - a powerful and delightful API that makes web scraping easy and fun.

We will scrape 2 types of eBay pages:

Products list page
Product page

‎

Prerequisites

‎
To perform this simple task, you will need the following things:

A Page2API account

The free trial offers a credit that covers up to 1000 web pages to scrape.

A product or a category of products that we are about to scrape.

In our case, we will search for 'fitbit 4 silicone strap'
and then scrape the product page for a random product.

‎

Scraping eBay Products List Page

‎
First what we need is to type 'fitbit 4 silicone strap' into the search input from eBay's search page change the view type to Gallery View.

This will change the browser URL to something similar to:

https://www.ebay.com/sch/i.html?_nkw=fitbit+4+silicone+strap&_sacat=0&_dmd=2&rt=nc

The URL is the first parameter we need to perform the scraping.

The page that you see must look like the following one:

‎
If you inspect the page HTML, you will find out that a single result is wrapped into a div that looks like the following:

‎

The HTML for a single result element will look like this:

‎
The last part is the pagination handling.

In our case, we must click on the Next (→) button while the list item's class will be active:

‎

And stop our scraping request when the Next (→) button became disabled.

In our case, a new attribute (aria-disabled="true") is assigned to the Next (→) button:

‎

Now it's time to prepare the request that will scrape all products that the search page returned.

Setting the api_key as an environment variable

export API_KEY=YOUR_PAGE2API_KEY

Running the scraping request with cURL

    curl -v -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.ebay.com/sch/i.html?_nkw=fitbit+4+silicone+strap&_sacat=0&_dmd=2&rt=nc",
      "real_browser": true,
      "merge_loops": true,
      "scenario": [
        {
          "loop" : [
            { "wait_for": ".pagination__next" },
            { "execute": "parse" },
            { "execute_js": "document.querySelector(\"a.pagination__next\").click()" }
          ],
          "stop_condition": "document.querySelector(\"a.pagination__next\").ariaDisabled == \"true\""
        }
      ],
      "parse": {
        "items": [
          {
            "_parent":"ul.srp-grid li.s-item",
            "title":"h3.s-item__title >> text",
            "link":"a.s-item__link >> href",
            "price":".s-item__price >> text",
            "shipping":".s-item__shipping >> text"
          }
        ]
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result

    {
      "result": {
        "items": [
          {
            "title": "For Fitbit Charge 3/4 Replacement Silicone Wristband Straps Sports Watch Band",
            "link": "https://www.ebay.com/itm/174485572199?hash=item28a0268a67:g:1QMAAOSwLA9fjTwQ",
            "price": "$3.89",
            "shipping": "Free shipping"
          },
          {
            "title": "Unisex Silicone Sports Bracelet Strap Fashion Band for Fitbit Inspire/Inspire HR",
            "link": "https://www.ebay.com/itm/324593039446?hash=item4b9340b856:g:5jsAAOSwRLJgh62j",
            "price": "$3.49",
            "shipping": "Free shipping"
          },
          ...
        ]
      },
      ...
    }

‎

Scraping eBay Product Page

‎
From the products list page, we need to click on any product.

The browser URL will look like this:

 https://www.ebay.com/itm/114503614713

The URL is the first parameter we need to perform the product data scraping.

Let's prepare the request that will scrape the needed information from the page.

Setting the api_key as an environment variable (if needed)

export API_KEY=YOUR_PAGE2API_KEY

Running the scraping request with cURL

    curl -v -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.ebay.com/itm/114503614713",
      "parse": {
        "name": "h1 >> text",
        "price": "span[itemprop=price] >> text",
        "condition": "div[itemprop=itemCondition] >> text",
        "categories": [
          "#vi-VR-brumb-lnkLst span >> text"
        ],
        "amount_sold": ".soldwithfeedback a >> text",
        "seller_name": "#RightSummaryPanel span.mbg-nw >> text",
        "item_location": "span[itemprop=availableAtOrFrom] >> text",
        "feedback_score": "#RightSummaryPanel span.mbg-l a >> text",
        "approximate_price": "#convbidPrice >> text"
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result

    {
      "result": {
        "name": "Details about Fitbit Charge 2 3 4 Bracelet Stainless Steel Spare Band Nylon Milanese Sport",
        "price": "GBP 5.66",
        "condition": "New with tags",
        "categories": [
          "Jewelry & Watches",
          "Watches, Parts & Accessories",
          "Watch Accessories",
          "Wristwatch Bands"
        ],
        "amount_sold": "",
        "seller_name": "fro-shop",
        "item_location": "Mülheim, Germany",
        "feedback_score": "28166",
        "approximate_price": "US $7.76(including shipping)"
      },
      ...
    }

‎

Conclusion

‎
That's it!

Scraping eBay product data is a bit tricky because of the HTML structure, but that is not a problem if you have the proper scraping tool, such as Page2API that makes web scraping something you can enjoy.

The original article can be found here:

https://www.page2api.com/blog/scrape-ebay-products/

The Easiest Way to Scrape Amazon Products and Reviews

Nicolae Rotaru — Thu, 21 Oct 2021 14:07:22 +0000

Introduction

Amazon is a vast Internet-based enterprise that sells all kinds of goods either directly or as the middleman between other retailers and Amazon's millions of customers.

You can scrape their data to run different types of tasks such as:

competitor analysis
improving your products and value proposition
identifying market trends and what influences them
price monitoring

Luckily, Amazon is a website that is pretty easy to scrape if you have the right tools.

For this purpose, we will use Page2API - a powerful and delightful API that makes web scraping easy and fun.

In this case, we will scrape 2 types of amazon pages:

Search results page
Product reviews page

Prerequisites

To perform this simple task, you will need the following things:

A Page2API account

The free trial offers a credit that covers up to 1000 web pages to scrape, and it takes under 10 seconds to create the account if you sign up with Google.

A product or a category of products that we are about to scrape.

In our case, we will search for 'luminox watches'
and then scrape the reviews for a random product.

Scraping all products from the search page

First what we need is to type 'luminox watches' into the search input from amazon's search page.

This will change the browser URL to something similar to:

https://www.amazon.com/s?k=luminox+watches

The URL is the first parameter we need to perform the scraping.

The page that you see must look like the following one:

If you inspect the page HTML, you will find out that a single result is wrapped into a div that looks like the following:

The HTML for a single result element will look like this:

The last part is the pagination handling.

In our case, we must click on the Next → button while the list item's class will be active:

And stop our scraping request when the Next → button became disabled.

In our case, a new class (a-disabled) is assigned to the list element where the button is located:

Now it's time to prepare the request that will scrape all products that the search page returned.

Setting the api_key as an environment variable

export API_KEY=YOUR_PAGE2API_KEY

Running the scraping request with cURL

    curl -v -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.amazon.com/s?k=luminox+watches",
      "real_browser": true,
      "merge_loops": true,
      "scenario": [
        {
          "loop" : [
            { "wait_for": ".a-pagination li.a-last" },
            { "execute": "parse" },
            { "execute_js": "document.querySelector(\".a-pagination li.a-last a\").click()" }
          ],
          "stop_condition": "document.querySelector(\".a-last.a-disabled\") !== null"
        }
      ],
      "parse": {
        "watches": [
          {
            "_parent": "[data-component-type='s-search-result']",
            "title": "h2 >> text",
            "link": ".a-link-normal >> href",
            "price": ".a-price-whole >> text",
            "stars": ".a-icon-alt >> text"
          }
        ]
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result

  {
    "result": {
      "watches": [
        {
          "title": "Men's Luminox Leatherback Sea Turtle 44mm Watch",
          "link": "https://www.amazon.com/Luminox-Leatherback-Turtle-Giant-Black/dp/B07CVFWXMR/ref=sr_1_2?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-2",
          "price": "$223.47",
          "stars": "4.5 out of 5 stars"
        },
        {
          "title": "The Original Navy Seal Mens Watch Black Display (XS.3001.F/Navy Seal Series): 200 Meter Water Resistant + Light Weight Case + Constant Night Visibility",
          "link": "https://www.amazon.com/Luminox-Wrist-Watch-Navy-Original/dp/B07NYXV77C/ref=sr_1_3?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-3",
          "price": "$254.12",
          "stars": "4.3 out of 5 stars"
        },
        {
          "title": "Leatherback SEA Turtle Giant - 0323",
          "link": "https://www.amazon.com/Luminox-Leatherback-SEA-Turtle-Giant/dp/B07PBC31N8/ref=sr_1_4?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-4",
          "price": "$179.00",
          "stars": "4.3 out of 5 stars"
        },
        ...
      ]
    },
    ...
  }

Scraping product reviews

First what we need is to click on the See all reviews link from the Product page.

This will change the browser URL to something similar to:

https://www.amazon.com/product-reviews/B072FNJLBC

The URL is the first parameter we need to perform the reviews scraping.

The HTML from a single review will look like this:

Luckily, the pagination handling is similar to the one described above, so we will use the same flow.

Now it's time to prepare the request that will scrape all reviews.

Setting the api_key as an environment variable (if needed)

export API_KEY=YOUR_PAGE2API_KEY

Running the scraping request with cURL

    curl -v -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.amazon.com/product-reviews/B072FNJLBC",
      "real_browser": true,
      "merge_loops": true,
      "scenario": [
        {
          "loop" : [
            { "wait_for": ".a-pagination li.a-last" },
            { "execute": "parse" },
            { "execute_js": "document.querySelector(\".a-pagination li.a-last a\").click()" }
          ],
          "stop_condition": "document.querySelector(\".a-last.a-disabled\") !== null"
        }
      ],
      "parse": {
        "reviews": [
          {
            "_parent": "[data-hook='review']",
            "title": ".review-title >> text",
            "author": ".a-profile-name >> text",
            "stars": ".review-rating >> text",
            "content": ".review-text >> text"
          }
        ]
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool

The result

  {
    "result": {
      "reviews": [
          {
            "title": "Great watch & easy to read in low light conditions",
            "author": "Paul E. Papas",
            "stars": "5.0 out of 5 stars",
            "content": "I'm a 60+ year old equestrian and outdoorsman. I was looking for a watch that could take the shock of firearm discharge ..."
          },
          {
            "title": "Not Water Resistant, impossible to get amazon help",
            "author": "Benjamin H. Curry",
            "stars": "2.0 out of 5 stars",
            "content": "This watch has a 2 year warranty from date of purchase however after not even on full year my adult son went swimming with it ..."
          },
        ...
      ]
    },
    ...
  }

Conclusion

That's pretty much of it!

As mentioned, Amazon is a website that is pretty easy to scrape if you have the right tools.
This is the place where Page2API shines, making web scraping super easy and fun.

The original article can be found here:

https://www.page2api.com/blog/scrape-amazon-product-data