DEV Community: Darshan Khandelwal

A Complete Guide to Web Scraping - 2023

Darshan Khandelwal — Fri, 14 Jul 2023 06:19:37 +0000

Web Scraping, also known as data extraction or data scraping, is the process of extracting or collecting data from websites or other sources in the form of text, images, videos, links, etc.

Web Scraping is helpful when a particular website does not have an official API or has a limit on the access of data. It has various uses like price monitoring, media monitoring, sentimental analysis, etc.

It is helpful for businesses that make decisions based on large amounts of public data available on the internet, which can be extracted easily with the help of data scraping.

Data has now become the new Oil in the market right now. If used correctly, businesses can achieve their targets by getting ahead of their competitors. This way, they can leverage this advantage over their competitors. “The more relevant data you have, the better-informed decisions you make.”

In this blog, we will learn everything about web scraping, its methods and uses, the correct way of doing it, and various other information related to it.

What is Web Scraping?

Web Scraping is the process of extracting data from a single or bunch of websites with the help of HTTP requests on the website’s server to access the raw HTML of a particular webpage and then convert it into a format you want.

We sometimes copy content from a web page and embed it into an Excel file or some other file. It is none other than web scraping but at a tiny scale. For large-scale scraping, developers use web scraping API, which can gather a vast amount of data rapidly.

The benefit of using a web scraping API is that you don’t have to copy data from the websites regularly, but you can use an API that will automate the process and will save your valuable time and effort.

Uses Of Web Scraping

Web scraping is a powerful and useful tool that can be used for a variety of purposes:

SEO

Web scraping can be used to extract a large amount of data from search engines like Google, and then this scraped information can be used, to track keywords, website rankings, and much more. This can be useful for your business, as with the help of data-driven research, you can increase your product visibility in the market.

You can use various dedicated [Google Search APIs](https://serpdog.io) available in the market for scraping Google search results. They scrape every inch of information from Google webpage and also convert the raw HTML code into JSON format, giving you the results in the structured format.

Read More: [How to Scrape Google Search Results](https://serpdog.io/blog/scrape-google-organic-search-results)

Data Mining

With the help of web scraping, one can gather a vast amount of data about their competitors, and products, uncover their strategy, and can make informed decisions with insights based on the data available in the market.

Price Monitoring

It is one of the most popular uses of web scraping. Price monitoring can be used to gather pricing data from competitors or multiple online retailers present in the market and can help consumers with saving money by finding the best deal in the market.

News and Media Monitoring

Web scraping can be used to track current news and events taking place in the world. You can access a large number of articles from big news agencies like the New York Times, the Washington Post, the Economic Times, etc with the help of web scraping.

If you run a company, that time to time appears in the news and you want to know who is saying what about your company or brand then scraping news data can be a beneficial thing for you.

Lead Generation

Web scraping can help your company to generate leads for your company’s potential customers from various online sources. You can target a specific set of people instead of making mass emails which can be beneficial for your product sales.

So, web scraping has various uses depending on the user’s specifications and requirements. From SEO to Lead Generation, web scraping can help businesses make data-driven decisions.

Web Scraping can help you to extract a large amount of data without any time and effort. It is much more efficient to use a web scraper instead of manually copying a piece of data for every website.

Property Analysis

Realtors can use web scraping as a tool to scrape property data from various platforms like Zillow, Realtors.com, and Idealista.com to predict future trends based on pricing, public sentiment, location, and other factors. It allows them to make an informed analysis and make strategically correct decisions to stay ahead of the competition.

Methods Of Web Scraping

There are several web scraping methods you can use to scrape a website. Here are some of these methods which help in scraping a website efficiently:

Designing Your Scraper:

Designing your scraper involves writing your code in a certain programming language, which will automate the process of navigating to a website and extracting the required data. You can write your script in various programming languages like Python, Javascript, C++, etc. Python is the most popular language for web scraping right now, but there are some powerful libraries in Javascript also like Unirest, Cheerio, and Puppeteer which have very high-performance capabilities.

While designing your scraper, you have to first search for certain element tags you want to scrape by inspecting the HTML code and then embedding them into your code when you start with the parsing of HTML.

Parsing is the process of extracting structured data from an HTML document. Beautiful Soup (Python), Cheerio (JavaScript), and Group (Java) are some of the preferred libraries for web parsing.

After identifying the required tags, you can send an HTTP request to a particular website with the help of a web scraping library in your chosen programming language and then parse the extracted data by using a web parsing library.

It is also important to note that while designing your scraper, you have to keep in mind that your scraping bot doesn’t violate the website’s terms of conditions. It is also advisable not to make a large number of requests on a smaller website, everybody doesn’t have a high budget like big enterprises used to have.

Advantages: Full control over your scraper allows you to customize the scraper according to your scraping needs.

Disadvantages: Designing Your scraper can sometimes become a time-consuming task if you were not able to scrape data because of an incorrect process.

Manual Web Scraping:

Manual Web Scraping is the process of navigating to a particular website in your web browser and copying the required data from the website into an Excel or any other file. This process is done manually and no script or data extraction service is used in this type of web scraping.

There are quite different ways you can do manual web scraping. You can download a whole web page as an HTML file and then filter out the required data from the HTML file with the help of any text editor you use into the spreadsheet or any other file.

Another way you can manually scrape a website is by using a browser inspection tool, where you can identify and select the element that consists of the data that you want to extract.

This method is good for small-scale web data extraction but can produce errors when done on a large scale, also it takes more time and effort than automated web scraping.

Advantages: Copy and pasting is a basic skill. You don’t have to require any type of technical skills here.

Disadvantages: This method requires heavy effort and is time-consuming if you are planning to scrape data from a large number of websites.

Web Scraping Services:

Many companies and freelancers offer web scraping services to their clients, where you can just provide them with URLs and they will send you the data in the required format.

It is one of the best methods if you want to scrape large amounts of data and don’t want to mess with the complex scraping process.
Generally, the companies which offer web scraping services to their clients have a ready-made script with them already, and they also have a team who are experts in handling any errors that can come while scraping the URLs like IP bans, CAPTCHAs, timeout errors, etc. They can handle a large amount of data more efficiently and can complete the task much faster than you can do your own.

Advantages: Web Scraping Services can be cost-effective in the long run, as they can scrape the data with their ready-made infrastructure much faster than you can do your own.

Disadvantages: No control over the scraping process.

Another important thing is that one should trust only reputable services for these big tasks that can deliver the high-quality data you want.

Web Scraping API:

Web Scraping API is an API that can scrape the data from a website using an API call. You don’t have to directly access the HTML code of the web page but the API will handle the whole scraping process.

API (Application Programming Interface) is a set of definitions and protocols that allows one software system to communicate with another software system.

Web Scraping API is easy to use and requires no such technical knowledge, one just has to pass the URL at their endpoint and it will return the result in a well-structured format. They are highly scalable means you can scrape a large amount of data without fearing any IP ban or CAPTCHAs.

Advantages: They are highly scalable and the data you receive is accurate, complete, and of high quality.

Disadvantages: Some Web Scraping APIs can limit the number of requests you can send per unit of time, thus limiting the amount of data you can collect.

So, there are a wide variety of web scraping methods you can apply according to your scraping needs. If you want to save money then method one and method two are best for you. These two methods also give you complete control over the scraping process. While if you don’t want to mess with the IP bans, CAPTCHAs, and handling large amounts of data then the last two methods are the best choice for you.

Is Web Scraping Legal?

Web Scraping legality is still an evolving process, but the judgment depends on various factors like how you scrape any specific data and how you use it.

In general, web scraping can be considered legal if you want to use the data for research purposes, educational projects, price comparisons, etc. But the legality can be affected if the website in its terms of conditions, strictly prohibits any kind of web scraping without its permission.

Web scraping can also be considered illegal if it is used to gain any unfair advantage over competitors or for unauthorized purposes like stealing sensitive data from the website. You can also get blocked in the process of extracting the data from the website, and get sued for violating any copyright laws.

Overall, web scraping is a valuable tool if used correctly, but one has to keep in mind the legal consequences if it is carried out maliciously. It is also important to respect the website’s terms of service and not to harm its services or functionality in any way.

Best Languages For Web Scraping

There are various programming languages that you can use for web scraping, depending on your needs. Let us discuss these:

Python: Python is the most popular language among developers for web scraping, thanks to its simplicity and a large number of libraries and frameworks including Scrapy and Beautiful Soup. Also, the community support is quite good in terms of web scraping when we talk about Python.

Javascript: Javascript is also becoming one of the preferable choices for web scraping, because of its capability to scrape data from websites that use JavaScript to dynamically load the web page. Libraries like Unirest, Puppeteer, and Cheerio are making data scraping in JavaScript easier.

Java: Java is another popular language widely used in large-scale projects. Libraries like Jsoup makes it easier to scrape data from websites.

Ruby: An high-level programming language with libraries like Nokogiri and Mechanize makes it easier to scrape data from websites.

There can be more such examples like C#, R, PHP, etc, which can be used for web scraping, but in the end, it depends on the requirements of the project.

How Can I Learn Web Scraping?

Web Scraping is nowadays becoming an important skill that can earn you money, almost every website requires leads to expand their business which is only possible because of web scraping, every active website wants to track its rankings on Google which is only possible because of Google Scraping. So, Web Scraping has become one of the main pillars in the growth of businesses.

In this section, we are going to discuss various ways to get started with web scraping:

Learn it by yourself: You can also learn web scraping by creating small projects on your own. First, start researching smaller projects. When you get comfortable with them, try to extract data from websites that are harder to scrape.

Online Tutorials: You can also take various online courses available on educational platforms like Udemy, Coursera, etc. The teachers are well experienced and will take you from beginner to advance level in a structured manner.
But it will also require you to learn the programming language you want to start with web scraping. Learn the language, from basic to intermediate level first, then when you can gain enough experience, join these courses to kickstart your web scraping journey.

Join online communities: It is advisable to join communities related to your programming language or web scraping, so you can ask any question if you are stuck on an error while making a scraper. You can join various communities on platforms like Reddit, Discord, etc. They have some very highly experienced guys on their server who can solve even a high-level problem easily.

Read Articles: There are tons of articles available on the internet on web scraping, which can take you from level zero to an expert in web scraping. You can learn to scrape advanced websites like Google, Amazon, and LinkedIn in these tutorials with a complete explanation.
Hence, there are many ways to start with learning web scraping, but the ultimate key is to be consistent and focused while learning new things. You can start by giving at least 1 hour per day and then increase it slowly to give your 100%. This will give you a good hand in scraping and will make you a proficient learner.

Conclusion

In this tutorial, we learned about web scraping, some methods to scrape websites, and at last how you can kickstart your web scraping journey.

We also learned web scraping is a valuable skill that allows you to scrape data from different websites, which can be used for research-based purposes like price monitoring, media monitoring, SEO, etc. We can also generate tons of leads for our business with the help of web scraping to stay ahead of the competition.

I hope this tutorial gave you a complete overview of web scraping. Please do not hesitate to message me if I missed something. If you think we can complete your custom scraping projects feel free to contact us. Follow me on Twitter. Thanks for reading!

Additional Resources

I have prepared a complete list of blogs on scraping Google, which can give you an idea about web scraping:

Scrape Yelp Business Reviews

Darshan Khandelwal — Tue, 10 Jan 2023 08:37:30 +0000

Introduction

Yelp is one of the biggest crowd-sourced rating and review websites for local businesses. It is a trusted review website because of the limited amount of spam and ads. With more than 200 million public reviews on its platform makes Yelp, a data-rich website for data miners.

In this tutorial, we are going to scrape Yelp Business Reviews using different Node JS libraries. And at the end, to make things simple, I will suggest you a Yelp Search API, which you can use to scrape data from Yelp easily.

Why Scrape Yelp?

Yelp has a mighty base of 90 million visitors per month across its website and mobile app, with users and businesses contributing to this platform day-to-day.

Scraping Yelp can help you to get information about your competitors, their ratings, and reviews. And using that way, you can analyze where your business stands in the market and the weak points that are becoming problems in your business expansion.

Yelp's ultra-big size business directory can also help you to generate quality leads for your business. You can also collect addresses, phone numbers, and other details by scraping Yelp.

Before we start with the tutorial, let me explain some requirements for this tutorial.

Requirements:

Web Parsing with CSS selectors

Searching the tags from the HTML files is not only a difficult thing to do but also a time-consuming process. It is better to use the CSS Selectors Gadget for selecting the perfect tags to make your web scraping journey easier.

This gadget can help you to come up with the perfect CSS selector for your need. Here is the link to the tutorial, which will teach you to use this gadget for selecting the best CSS selectors according to your needs.

User Agents

User-Agent is used to identify the application, operating system, vendor, and version of the requesting user agent, which can save help in making a fake visit to Google by acting as a real user.

You can also rotate User Agents, read more about this in this article: How to fake and rotate User Agents using Python 3.

If you want to further safeguard your IP from being blocked by Google, you can try these 10 Tips to avoid getting Blocked while Scraping Websites.

Install Libraries

To start scraping Yelp Reviews, we need to install some NPM libraries to move forward.

So before starting, we have to ensure that we have set up our Node JS project and installed both Puppeteer JS. You can install this package from the above link.

Scraping Yelp Reviews

We will scrape the overall rating and the public reviews given by the customers of this restaurant. Here is a list of data that we are gonna scrape in this tutorial:

Average Rating of the business
Name of the person
Location of the person
Review gave by the person

Scraping Yelp Business Reviews:

The Yelp Reviews scraping can be divided into two parts:

Making the HTTP request on the target URL to extract the raw HTML data.
Parsing the HTML data to extract the required data.

Now, let's start scraping Yelp Business Reviews by making a GET request on the target URL using Puppeteer to get the raw HTML data.

const yelpReviewScraper = async() => {


        browser = await puppeteer.launch({
            headless: false,
            args: ["--disabled-setuid-sandbox", "--no-sandbox"],
            });
            const [page] = await browser.pages();        

            await page.goto("https://www.yelp.com/biz/hard-rock-cafe-san-francisco-5" , {
                waitUntil: 'domcontentloaded',
            })
            await page.waitForTimeout(5000)  
            let html = await page.content();

            await browser.close();
            getData(html);
    };

Step-by-step explanation:

puppeteer.launch() - This will launch the Chromium browser with the options we have set in our code. In our case, we are launching our browser in non-headless mode.
browser.newPage() - This will open a new page or tab in the browser.
page.goto() - This will navigate the page to the specified target URL.
page.waitForTimeout() - It will cause the page to wait for 3 seconds to do further operations.
page.content() - It will return the raw HTML scraped from the URL page.

Now, we have completed the part for scraping raw HTML.
Then, we will parse this raw HTML using Cheerio.

const $ = cheerio.load(html)

If you use the CSS selector Gadget correctly, you will find all these reviews are under the tag .review__09f24__oHr9V.

We will run a loop for this selector and extract every possible information inside this individual review.

Similarly, we can find the tags for other respective data also. For example, the tag for the name is .css-ux5mu6 .css-1m051bw, the tag for the location is .responsive-hidden-small__09f24__qQFtj .css-qgunke, etc.

And this is what our parser looks like:

  const getData = (html) => {
        const $ = cheerio.load(html)

        const avg_rating = $(".five-stars__09f24__mBKym").attr("aria-label");
        const reviews = $(".padding-t0-5__09f24__lDQoQ .css-foyide").text();
        let user_reviews = [];
        $(".review__09f24__oHr9V").each((i,el) => {
            user_reviews.push({
                name: $(el).find(".css-ux5mu6 .css-1m051bw").text(),
                location: $(el).find(".responsive-hidden-small__09f24__qQFtj .css-qgunke").text(),
                review: $(el).find(".comment__09f24__gu0rG").text(),
                date: $(el).find(".css-chan6m").text(),
                rating: $(el).find(".five-stars__09f24__mBKym").attr("aria-label"),
                friends: $(el).find("[aria-label='Friends'] span > span").text(),
                reviews: $(el).find("[aria-label='Reviews'] span > span").text(),
                photos: $(el).find("[aria-label='Photos'] span > span").text(),
                thumbnail: $(el).find(".css-1pz4y59").attr("src")
            })
            let images = [];
            if($(el).find(".photo-container-small__09f24__obhgq"))
            {
            $(el).find(".photo-container-small__09f24__obhgq").each((i,el) => {
                images[i] = $(el).find("img").attr("src")
            })
            user_reviews[i].images = images
            }
        })
        console.log(avg_rating)
        console.log(reviews)
        console.log(user_reviews)
    }

Here are the results:

3 star rating
    991 reviews
    {
      name: 'Jacqueline B.',
      location: 'Chicago, IL',
      review: 'We flew into San Francisco on our way to Napa and decided to add to our Hard Rock SHIRT collection since it was too early to check in to the hotel. The area near Alcatraz was cool! The Pier 39 area is right off the water, with shops, rides, sightseeing, etc. The store for Hard Rock was smallish but nice, and the restaurant seated us quickly. It was busy for a Monday at 2:00, service was a tad slow but the waiter was friendly. I got iced tea/lemonade and the salmon salad. Very good, a nice size too.I know this chain can be over priced and a tourist trap, but the rock/music memorabilia on the walls are just so fascinating, no two restaurants are alike! I re-joined the points program and earned a free shot glass with my shirt purchase!',
      date: '11/30/2022',
      rating: '4 star rating',
      friends: '51',
      reviews: '454',
      photos: '873',
      thumbnail: 'https://s3-media0.fl.yelpcdn.com/photo/1zU0fO63-mrEBNFHM5UrRQ/60s.jpg',
      images: [
        'https://s3-media0.fl.yelpcdn.com/bphoto/E5wwDryXExDLMIUw0TYyaQ/180s.jpg',
        'https://s3-media0.fl.yelpcdn.com/bphoto/TvXYc1t1H223B6xTv1uHaw/180s.jpg',
        'https://s3-media0.fl.yelpcdn.com/bphoto/zcHaS-DIidDXQyqvnbTsvw/180s.jpg',
        'https://s3-media0.fl.yelpcdn.com/bphoto/YIBW-vgdHWZyXfPuywm4jg/180s.jpg'
        ]
    }

Here is the complete code:

const cheerio = require("cheerio")
    const puppeteer = require("puppeteer")

    const getData = (html) => {
        const $ = cheerio.load(html)

        const avg_rating = $(".five-stars__09f24__mBKym").attr("aria-label");
        const reviews = $(".padding-t0-5__09f24__lDQoQ .css-foyide").text();
        let user_reviews = [];
        $(".review__09f24__oHr9V").each((i,el) => {
            user_reviews.push({
                name: $(el).find(".css-ux5mu6 .css-1m051bw").text(),
                location: $(el).find(".responsive-hidden-small__09f24__qQFtj .css-qgunke").text(),
                review: $(el).find(".comment__09f24__gu0rG").text(),
                date: $(el).find(".css-chan6m").text(),
                rating: $(el).find(".five-stars__09f24__mBKym").attr("aria-label"),
                friends: $(el).find("[aria-label='Friends'] span > span").text(),
                reviews: $(el).find("[aria-label='Reviews'] span > span").text(),
                photos: $(el).find("[aria-label='Photos'] span > span").text(),
                thumbnail: $(el).find(".css-1pz4y59").attr("src")
            })
            let images = [];
            if($(el).find(".photo-container-small__09f24__obhgq").length)
            {
            $(el).find(".photo-container-small__09f24__obhgq").each((i,el) => {
                images[i] = $(el).find("img").attr("src")
            })
            user_reviews[i].images = images
            }
        })
        console.log(avg_rating)
        console.log(reviews)
        console.log(user_reviews[0])
    }

    const yelpReviewScraper = async() => {


            browser = await puppeteer.launch({
            headless: false,
            args: ["--disabled-setuid-sandbox", "--no-sandbox"],
            });
            const [page] = await browser.pages();        

            await page.goto("https://www.yelp.com/biz/hard-rock-cafe-san-francisco-5" , {
                waitUntil: 'domcontentloaded',
            })
            await page.waitForTimeout(5000)  
            let html = await page.content();

            await browser.close();
            getData(html);


        };

        yelpReviewScraper()

And that's what a basic scraper of Yelp Reviews looks like. Similarly, this same process of scraping and selecting tags can be followed in other programming languages.

The second way you can scrape Yelp data is by using Serpdog's Yelp Search API.

With Yelp Search API

If you don't want to code and maintain the scraper in the long run and don't want to work with complex URLs and HTML, then you can try this Yelp Search API.

Serpdog solves all the problem of captchas and proxies and allow developers to scrape Yelp Search Results smoothly. Also, the pre-cooked structured JSON data can save you a lot of time.

Get your API Key and 100 free credits by registering at Serpdog.

  const axios = require('axios');
    axios.get('https://api.serpdog.io/yelp?api_key=APIKEY&find_desc=burger&find_loc=San+Francisco,CA')
        .then(response => {
        console.log(response.data);
        })
        .catch(error => {
        console.log(error);
        });

Results:

{
    "filters": [
        {
            "title": "Suggested",
            "options": [
                {
                    "text": "Open Now -:-- am"
                },
                {
                    "text": "Offers Delivery"
                },
                {
                    "text": "Offers Takeout"
                },
                {
                    "text": "Reservations"
                },
                {
                    "text": "Good for Late Night"
                },
                {
                    "text": "Good for Lunch"
                }
            ]
        },
        {
            "title": "Category",
            "options": [
                {
                    "text": "Burgers"
                },
                {
                    "text": "Restaurants"
                },
                {
                    "text": "Food"
                },
                {
                    "text": "Bars"
                }
            ]
        }
        ......
    ],
    "inline_ads": [
        {
            "title": "Burger King",
            "reviews": 92,
            "categories": "Burgers, Fast Food",
            "button": {
                "text": "Get Directions",
                "link": "https://www.yelp.com/adredir?ad_business_id=wvJvLx0y-O9OMC67HtBrkA&click_origin=search_results_direction&placement=carousel_0&placement_slot=0&redirect_url=https%3A%2F%2Fwww.yelp.com%2Fmap%2Fburger-king-san-francisco-10&request_id=14c56576b87012dc&signature=de18fac9d95909f62ebfe265dee17e6f2a1447277e0dd7af18695cb2d41e9f97&slot=0"
            },
            "thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/5RiI4XXVY_7uylnuMv5TnA/348s.jpg"
        },
        {
            "title": "Taco Bell",
            "reviews": 61,
            "categories": "Fast Food, Burgers, Mexican",
            "button": {
                "text": "Get Directions",
                "link": "https://www.yelp.com/adredir?ad_business_id=Q02CIzGUxyLKx5hdzHAIGA&click_origin=search_results_direction&placement=carousel_0&placement_slot=0&redirect_url=https%3A%2F%2Fwww.yelp.com%2Fmap%2Ftaco-bell-san-francisco-12&request_id=14c56576b87012dc&signature=7097c7348afc5aec13e435f6f663b103cc606b763182a9a95774fc166ba20b4a&slot=1"
            },
            "thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/G8awa-z4PEKlvZNuDtxJng/348s.jpg"
        },
        {
            "title": "Applebee’s Grill + Bar",
            "reviews": 654,
            "categories": "American (Traditional), Sports Bars, Burgers",
            "button": {
                "text": "Get Directions",
                "link": "https://www.yelp.com/adredir?ad_business_id=EOM6CItD6sI5P3a-JV61_Q&click_origin=search_results_direction&placement=carousel_0&placement_slot=0&redirect_url=https%3A%2F%2Fwww.yelp.com%2Fmap%2Fapplebees-grill-bar-san-francisco&request_id=14c56576b87012dc&signature=dd3a0dff97cfe3f1e2daf6b42a776bf65218f934fc207420620577372a99de08&slot=2"
            },
            "thumbnail": "https://s3-media0.fl.yelpcdn.com/adphoto/qeBOAProDYlA6ivNjmxGvA/ls.jpg"
        }
    ],
    "results": [
        {
            "title": "Applebee’s Grill + Bar",
            "snippet": "“Try one of our NEW Sizzlin' Skillets, like the savory new Chicken & Shrimp Scampi Skillet!” more",
            "neighbourhood": "Fisherman's Wharf",
            "thumbnail": "https://s3-media0.fl.yelpcdn.com/adphoto/CVMY-qKbwkkVkklT9x0y5g/ls.jpg",
            "button": {
                "text": "Get Directions",
                "link": "https://www.yelp.com/adredir?ad_business_id=EOM6CItD6sI5P3a-JV61_Q&click_origin=search_results_direction&placement=above_search&placement_slot=1&redirect_url=https%3A%2F%2Fwww.yelp.com%2Fmap%2Fapplebees-grill-bar-san-francisco&request_id=14c56576b87012dc&signature=e66820c182bbbf6b0b17860344825ce6e6d8269505379207fbe070f54d847732&slot=0"
            },
            "categories": [
                {
                    "title": "American (Traditional)",
                    "link": "https://www.yelp.com/search?cflt=tradamerican&find_loc=San+Francisco%2C+CA"
                },
                {
                    "title": "Sports Bars",
                    "link": "https://www.yelp.com/search?cflt=sportsbars&find_loc=San+Francisco%2C+CA"
                },
                {
                    "title": "Burgers",
                    "link": "https://www.yelp.com/search?cflt=burgers&find_loc=San+Francisco%2C+CA"
                }
            ],
            "service_options": [
                "Outdoor seating",
                "Delivery",
                "Takeout"
            ]
        },
        {
            "title": "8MilePi Detroit Style Pizza",
            "reviews": "18",
            "snippet": "“I got firehouse special and Motown Meat lover- both Detroit style! They provided ranch and honey which I ended up drizzling on my pizza and OMG that was a game changer hahah. From…” more",
            "neighbourhood": "SoMa",
            "thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/sqvilUnyxXZF2geJA1Mdkg/348s.jpg",
            "highlights": [
                "undefinedCatering service",
                "undefinedLarge group friendly"
            ],
            "button": {
                "text": "View Website",
                "link": "https://www.yelp.com/adredir?ad_business_id=N-mE8rxOpW4F5mwEkU7dxQ&campaign_id=MqcxEAopgcGNc4zTtAu85A&click_origin=search_results_visit_website&placement=above_search&placement_slot=1&redirect_url=https%3A%2F%2Fwww.yelp.com%2Fbiz_redir%3Fcachebuster%3D1670182514%26s%3D134b83fe1187f3d5d32148fb4a5819a31d42ccf9a6c688193a9f743fb22aec24%26src_bizid%3DN-mE8rxOpW4F5mwEkU7dxQ%26url%3Dhttps%253A%252F%252Fwww.8milepidetroitstylepizza.com%26website_link_type%3Dwebsite&request_id=14c56576b87012dc&signature=ab927641ab9f4403ece3cf423555f96b5b6be6fd4724606e92d6676316fcbe62&slot=1"
            },
            "categories": [
                {
                    "title": "Pizza",
                    "link": "https://www.yelp.com/search?cflt=pizza&find_loc=San+Francisco%2C+CA"
                }
            ],
            "service_options": [
                "Delivery",
                "Takeout"
            ]
        },
        {
            "title": "Native Burger",
            "reviews": "305",
            "snippet": "“will kick my flip flops off for? In-N-Out. IMO, they make the best burgers in its price category. Now” more",
            "neighbourhood": "Laurel Heights",
            "thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/ZNQwC9CeaVwLYkIx-qWbmg/348s.jpg",
            "button": {
                "text": "Start Order",
                "link": "https://www.yelp.com/biz/AonBwLuprSoZ1dx9iYaDBw?show_platform_modal=True"
            },
            "categories": [
                {
                    "title": "Burgers",
                    "link": "https://www.yelp.com/search?cflt=burgers&find_loc=San+Francisco%2C+CA"
                }
            ],
            "service_options": [
                "Outdoor seating",
                "Delivery",
                "Takeout"
            ]
        },
        {
            "title": "Foodee Burger",
            "reviews": "193",
            "snippet": "“Thought the burger was very tasty but the grease and fries were really greasy. My mouth still feels greasy. Cut the fat please!” more",
            "neighbourhood": "Forest Hill",
            "thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/2AvlfZlwXNlk1pGVqktKqQ/348s.jpg",
            "button": {
                "text": "Start Order",
                "link": "https://www.yelp.com/biz/DqJjIACQF6T85MR4ZYTrEA?show_platform_modal=True"
            },
            "categories": [
                {
                    "title": "Burgers",
                    "link": "https://www.yelp.com/search?cflt=burgers&find_loc=San+Francisco%2C+CA"
                }
            ],
            "service_options": [
                "Outdoor seating",
                "Delivery",
                "Takeout"
            ]
        },
        {
            "title": "Uncle Boy’s",
            "reviews": "731",
            "snippet": "“Extremely rude manager and over priced burgers. Do I need to leave a long story about it? I don't” more",
            "neighbourhood": "Inner Richmond",
            "thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/qYAqCmJ21xBKa1skFoSRXA/348s.jpg",
            "categories": [
                {
                    "title": "Burgers",
                    "link": "https://www.yelp.com/search?cflt=burgers&find_loc=San+Francisco%2C+CA"
                },
                {
                    "title": "Fast Food",
                    "link": "https://www.yelp.com/search?cflt=hotdogs&find_loc=San+Francisco%2C+CA"
                }
            ],
            "service_options": [
                "Outdoor seating",
                "Delivery",
                "Takeout"
            ]
        },
        .....
    ],
    "serpdog_pagination": {
        "next": "https://api.serpdog.io/yelp?api_key=APIKEY&find_desc=burger&find_loc=San+Francisco%2CCA&start=10"
    }
}

Frequently Asked Questions

Q. How to scrape Yelp data for free?

You can use Serpdog's Yelp Search API for scraping Yelp Data for free. Serpdog provides 100 credits to users on their first sign-up.

Q. Can you scrape Yelp data?

Yes, you can scrape Yelp data with the help of any scraping API present in the market.

Q. Is it legal to scrape Yelp data?

You can scrape any data publicly available on the internet without any legal problem.

Conclusion:

In this tutorial, we learned to scrape Yelp Buessiness Reviews by making a basic scraper with the help of Node JS. We also saw how Serpdog | Yelp Search API can help you with scraping the results.

I hope you enjoyed the tutorial. Feel free to message me if I missed something. Follow me on Twitter. Thanks for reading!

Additional Resources

Want to learn more about web scraping? Not a problem! We have already prepared the list of tutorials so you can kickstart your web scraping journey.

Author:

My name is Darshan, and I am the founder of serpdog.io. I love to create scrapers. I am working for several MNCs to provide them with Google Search Data through a seamless pipeline.

Also published here.