Introduction
Yelp is one of the biggest crowd-sourced rating and review websites for local businesses. It is a trusted review website because of the limited amount of spam and ads. With more than 200 million public reviews on its platform makes Yelp, a data-rich website for data miners.
In this tutorial, we are going to scrape Yelp Business Reviews using different Node JS libraries. And at the end, to make things simple, I will suggest you a Yelp Search API, which you can use to scrape data from Yelp easily.
Why Scrape Yelp?
Yelp has a mighty base of 90 million visitors per month across its website and mobile app, with users and businesses contributing to this platform day-to-day.
Scraping Yelp can help you to get information about your competitors, their ratings, and reviews. And using that way, you can analyze where your business stands in the market and the weak points that are becoming problems in your business expansion.
Yelp's ultra-big size business directory can also help you to generate quality leads for your business. You can also collect addresses, phone numbers, and other details by scraping Yelp.
Before we start with the tutorial, let me explain some requirements for this tutorial.
Requirements:
Web Parsing with CSS selectors
Searching the tags from the HTML files is not only a difficult thing to do but also a time-consuming process. It is better to use the CSS Selectors Gadget for selecting the perfect tags to make your web scraping journey easier.
This gadget can help you to come up with the perfect CSS selector for your need. Here is the link to the tutorial, which will teach you to use this gadget for selecting the best CSS selectors according to your needs.
User Agents
User-Agent is used to identify the application, operating system, vendor, and version of the requesting user agent, which can save help in making a fake visit to Google by acting as a real user.
You can also rotate User Agents, read more about this in this article: How to fake and rotate User Agents using Python 3.
If you want to further safeguard your IP from being blocked by Google, you can try these 10 Tips to avoid getting Blocked while Scraping Websites.
Install Libraries
To start scraping Yelp Reviews, we need to install some NPM libraries to move forward.
So before starting, we have to ensure that we have set up our Node JS project and installed both Puppeteer JS. You can install this package from the above link.
Scraping Yelp Reviews
We will scrape the overall rating and the public reviews given by the customers of this restaurant. Here is a list of data that we are gonna scrape in this tutorial:
- Average Rating of the business
- Name of the person
- Location of the person
- Review gave by the person
Scraping Yelp Business Reviews:
The Yelp Reviews scraping can be divided into two parts:
- Making the HTTP request on the target URL to extract the raw HTML data.
- Parsing the HTML data to extract the required data.
Now, let's start scraping Yelp Business Reviews by making a GET request on the target URL using Puppeteer to get the raw HTML data.
const yelpReviewScraper = async() => {
browser = await puppeteer.launch({
headless: false,
args: ["--disabled-setuid-sandbox", "--no-sandbox"],
});
const [page] = await browser.pages();
await page.goto("https://www.yelp.com/biz/hard-rock-cafe-san-francisco-5" , {
waitUntil: 'domcontentloaded',
})
await page.waitForTimeout(5000)
let html = await page.content();
await browser.close();
getData(html);
};
Step-by-step explanation:
-
puppeteer.launch()- This will launch the Chromium browser with the options we have set in our code. In our case, we are launching our browser in non-headless mode. -
browser.newPage()- This will open a new page or tab in the browser. -
page.goto()- This will navigate the page to the specified target URL. -
page.waitForTimeout()- It will cause the page to wait for 3 seconds to do further operations. -
page.content()- It will return the raw HTML scraped from the URL page.
Now, we have completed the part for scraping raw HTML.
Then, we will parse this raw HTML using Cheerio.
const $ = cheerio.load(html)
If you use the CSS selector Gadget correctly, you will find all these reviews are under the tag .review__09f24__oHr9V.
We will run a loop for this selector and extract every possible information inside this individual review.
Similarly, we can find the tags for other respective data also. For example, the tag for the name is .css-ux5mu6 .css-1m051bw, the tag for the location is .responsive-hidden-small__09f24__qQFtj .css-qgunke, etc.
And this is what our parser looks like:
const getData = (html) => {
const $ = cheerio.load(html)
const avg_rating = $(".five-stars__09f24__mBKym").attr("aria-label");
const reviews = $(".padding-t0-5__09f24__lDQoQ .css-foyide").text();
let user_reviews = [];
$(".review__09f24__oHr9V").each((i,el) => {
user_reviews.push({
name: $(el).find(".css-ux5mu6 .css-1m051bw").text(),
location: $(el).find(".responsive-hidden-small__09f24__qQFtj .css-qgunke").text(),
review: $(el).find(".comment__09f24__gu0rG").text(),
date: $(el).find(".css-chan6m").text(),
rating: $(el).find(".five-stars__09f24__mBKym").attr("aria-label"),
friends: $(el).find("[aria-label='Friends'] span > span").text(),
reviews: $(el).find("[aria-label='Reviews'] span > span").text(),
photos: $(el).find("[aria-label='Photos'] span > span").text(),
thumbnail: $(el).find(".css-1pz4y59").attr("src")
})
let images = [];
if($(el).find(".photo-container-small__09f24__obhgq"))
{
$(el).find(".photo-container-small__09f24__obhgq").each((i,el) => {
images[i] = $(el).find("img").attr("src")
})
user_reviews[i].images = images
}
})
console.log(avg_rating)
console.log(reviews)
console.log(user_reviews)
}
Here are the results:
3 star rating
991 reviews
{
name: 'Jacqueline B.',
location: 'Chicago, IL',
review: 'We flew into San Francisco on our way to Napa and decided to add to our Hard Rock SHIRT collection since it was too early to check in to the hotel. The area near Alcatraz was cool! The Pier 39 area is right off the water, with shops, rides, sightseeing, etc. The store for Hard Rock was smallish but nice, and the restaurant seated us quickly. It was busy for a Monday at 2:00, service was a tad slow but the waiter was friendly. I got iced tea/lemonade and the salmon salad. Very good, a nice size too.I know this chain can be over priced and a tourist trap, but the rock/music memorabilia on the walls are just so fascinating, no two restaurants are alike! I re-joined the points program and earned a free shot glass with my shirt purchase!',
date: '11/30/2022',
rating: '4 star rating',
friends: '51',
reviews: '454',
photos: '873',
thumbnail: 'https://s3-media0.fl.yelpcdn.com/photo/1zU0fO63-mrEBNFHM5UrRQ/60s.jpg',
images: [
'https://s3-media0.fl.yelpcdn.com/bphoto/E5wwDryXExDLMIUw0TYyaQ/180s.jpg',
'https://s3-media0.fl.yelpcdn.com/bphoto/TvXYc1t1H223B6xTv1uHaw/180s.jpg',
'https://s3-media0.fl.yelpcdn.com/bphoto/zcHaS-DIidDXQyqvnbTsvw/180s.jpg',
'https://s3-media0.fl.yelpcdn.com/bphoto/YIBW-vgdHWZyXfPuywm4jg/180s.jpg'
]
}
Here is the complete code:
const cheerio = require("cheerio")
const puppeteer = require("puppeteer")
const getData = (html) => {
const $ = cheerio.load(html)
const avg_rating = $(".five-stars__09f24__mBKym").attr("aria-label");
const reviews = $(".padding-t0-5__09f24__lDQoQ .css-foyide").text();
let user_reviews = [];
$(".review__09f24__oHr9V").each((i,el) => {
user_reviews.push({
name: $(el).find(".css-ux5mu6 .css-1m051bw").text(),
location: $(el).find(".responsive-hidden-small__09f24__qQFtj .css-qgunke").text(),
review: $(el).find(".comment__09f24__gu0rG").text(),
date: $(el).find(".css-chan6m").text(),
rating: $(el).find(".five-stars__09f24__mBKym").attr("aria-label"),
friends: $(el).find("[aria-label='Friends'] span > span").text(),
reviews: $(el).find("[aria-label='Reviews'] span > span").text(),
photos: $(el).find("[aria-label='Photos'] span > span").text(),
thumbnail: $(el).find(".css-1pz4y59").attr("src")
})
let images = [];
if($(el).find(".photo-container-small__09f24__obhgq").length)
{
$(el).find(".photo-container-small__09f24__obhgq").each((i,el) => {
images[i] = $(el).find("img").attr("src")
})
user_reviews[i].images = images
}
})
console.log(avg_rating)
console.log(reviews)
console.log(user_reviews[0])
}
const yelpReviewScraper = async() => {
browser = await puppeteer.launch({
headless: false,
args: ["--disabled-setuid-sandbox", "--no-sandbox"],
});
const [page] = await browser.pages();
await page.goto("https://www.yelp.com/biz/hard-rock-cafe-san-francisco-5" , {
waitUntil: 'domcontentloaded',
})
await page.waitForTimeout(5000)
let html = await page.content();
await browser.close();
getData(html);
};
yelpReviewScraper()
And that's what a basic scraper of Yelp Reviews looks like. Similarly, this same process of scraping and selecting tags can be followed in other programming languages.
The second way you can scrape Yelp data is by using Serpdog's Yelp Search API.
With Yelp Search API
If you don't want to code and maintain the scraper in the long run and don't want to work with complex URLs and HTML, then you can try this Yelp Search API.
Serpdog solves all the problem of captchas and proxies and allow developers to scrape Yelp Search Results smoothly. Also, the pre-cooked structured JSON data can save you a lot of time.
Get your API Key and 100 free credits by registering at Serpdog.
const axios = require('axios');
axios.get('https://api.serpdog.io/yelp?api_key=APIKEY&find_desc=burger&find_loc=San+Francisco,CA')
.then(response => {
console.log(response.data);
})
.catch(error => {
console.log(error);
});
Results:
{
"filters": [
{
"title": "Suggested",
"options": [
{
"text": "Open Now -:-- am"
},
{
"text": "Offers Delivery"
},
{
"text": "Offers Takeout"
},
{
"text": "Reservations"
},
{
"text": "Good for Late Night"
},
{
"text": "Good for Lunch"
}
]
},
{
"title": "Category",
"options": [
{
"text": "Burgers"
},
{
"text": "Restaurants"
},
{
"text": "Food"
},
{
"text": "Bars"
}
]
}
......
],
"inline_ads": [
{
"title": "Burger King",
"reviews": 92,
"categories": "Burgers, Fast Food",
"button": {
"text": "Get Directions",
"link": "https://www.yelp.com/adredir?ad_business_id=wvJvLx0y-O9OMC67HtBrkA&click_origin=search_results_direction&placement=carousel_0&placement_slot=0&redirect_url=https%3A%2F%2Fwww.yelp.com%2Fmap%2Fburger-king-san-francisco-10&request_id=14c56576b87012dc&signature=de18fac9d95909f62ebfe265dee17e6f2a1447277e0dd7af18695cb2d41e9f97&slot=0"
},
"thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/5RiI4XXVY_7uylnuMv5TnA/348s.jpg"
},
{
"title": "Taco Bell",
"reviews": 61,
"categories": "Fast Food, Burgers, Mexican",
"button": {
"text": "Get Directions",
"link": "https://www.yelp.com/adredir?ad_business_id=Q02CIzGUxyLKx5hdzHAIGA&click_origin=search_results_direction&placement=carousel_0&placement_slot=0&redirect_url=https%3A%2F%2Fwww.yelp.com%2Fmap%2Ftaco-bell-san-francisco-12&request_id=14c56576b87012dc&signature=7097c7348afc5aec13e435f6f663b103cc606b763182a9a95774fc166ba20b4a&slot=1"
},
"thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/G8awa-z4PEKlvZNuDtxJng/348s.jpg"
},
{
"title": "Applebee’s Grill + Bar",
"reviews": 654,
"categories": "American (Traditional), Sports Bars, Burgers",
"button": {
"text": "Get Directions",
"link": "https://www.yelp.com/adredir?ad_business_id=EOM6CItD6sI5P3a-JV61_Q&click_origin=search_results_direction&placement=carousel_0&placement_slot=0&redirect_url=https%3A%2F%2Fwww.yelp.com%2Fmap%2Fapplebees-grill-bar-san-francisco&request_id=14c56576b87012dc&signature=dd3a0dff97cfe3f1e2daf6b42a776bf65218f934fc207420620577372a99de08&slot=2"
},
"thumbnail": "https://s3-media0.fl.yelpcdn.com/adphoto/qeBOAProDYlA6ivNjmxGvA/ls.jpg"
}
],
"results": [
{
"title": "Applebee’s Grill + Bar",
"snippet": "“Try one of our NEW Sizzlin' Skillets, like the savory new Chicken & Shrimp Scampi Skillet!” more",
"neighbourhood": "Fisherman's Wharf",
"thumbnail": "https://s3-media0.fl.yelpcdn.com/adphoto/CVMY-qKbwkkVkklT9x0y5g/ls.jpg",
"button": {
"text": "Get Directions",
"link": "https://www.yelp.com/adredir?ad_business_id=EOM6CItD6sI5P3a-JV61_Q&click_origin=search_results_direction&placement=above_search&placement_slot=1&redirect_url=https%3A%2F%2Fwww.yelp.com%2Fmap%2Fapplebees-grill-bar-san-francisco&request_id=14c56576b87012dc&signature=e66820c182bbbf6b0b17860344825ce6e6d8269505379207fbe070f54d847732&slot=0"
},
"categories": [
{
"title": "American (Traditional)",
"link": "https://www.yelp.com/search?cflt=tradamerican&find_loc=San+Francisco%2C+CA"
},
{
"title": "Sports Bars",
"link": "https://www.yelp.com/search?cflt=sportsbars&find_loc=San+Francisco%2C+CA"
},
{
"title": "Burgers",
"link": "https://www.yelp.com/search?cflt=burgers&find_loc=San+Francisco%2C+CA"
}
],
"service_options": [
"Outdoor seating",
"Delivery",
"Takeout"
]
},
{
"title": "8MilePi Detroit Style Pizza",
"reviews": "18",
"snippet": "“I got firehouse special and Motown Meat lover- both Detroit style! They provided ranch and honey which I ended up drizzling on my pizza and OMG that was a game changer hahah. From…” more",
"neighbourhood": "SoMa",
"thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/sqvilUnyxXZF2geJA1Mdkg/348s.jpg",
"highlights": [
"undefinedCatering service",
"undefinedLarge group friendly"
],
"button": {
"text": "View Website",
"link": "https://www.yelp.com/adredir?ad_business_id=N-mE8rxOpW4F5mwEkU7dxQ&campaign_id=MqcxEAopgcGNc4zTtAu85A&click_origin=search_results_visit_website&placement=above_search&placement_slot=1&redirect_url=https%3A%2F%2Fwww.yelp.com%2Fbiz_redir%3Fcachebuster%3D1670182514%26s%3D134b83fe1187f3d5d32148fb4a5819a31d42ccf9a6c688193a9f743fb22aec24%26src_bizid%3DN-mE8rxOpW4F5mwEkU7dxQ%26url%3Dhttps%253A%252F%252Fwww.8milepidetroitstylepizza.com%26website_link_type%3Dwebsite&request_id=14c56576b87012dc&signature=ab927641ab9f4403ece3cf423555f96b5b6be6fd4724606e92d6676316fcbe62&slot=1"
},
"categories": [
{
"title": "Pizza",
"link": "https://www.yelp.com/search?cflt=pizza&find_loc=San+Francisco%2C+CA"
}
],
"service_options": [
"Delivery",
"Takeout"
]
},
{
"title": "Native Burger",
"reviews": "305",
"snippet": "“will kick my flip flops off for? In-N-Out. IMO, they make the best burgers in its price category. Now” more",
"neighbourhood": "Laurel Heights",
"thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/ZNQwC9CeaVwLYkIx-qWbmg/348s.jpg",
"button": {
"text": "Start Order",
"link": "https://www.yelp.com/biz/AonBwLuprSoZ1dx9iYaDBw?show_platform_modal=True"
},
"categories": [
{
"title": "Burgers",
"link": "https://www.yelp.com/search?cflt=burgers&find_loc=San+Francisco%2C+CA"
}
],
"service_options": [
"Outdoor seating",
"Delivery",
"Takeout"
]
},
{
"title": "Foodee Burger",
"reviews": "193",
"snippet": "“Thought the burger was very tasty but the grease and fries were really greasy. My mouth still feels greasy. Cut the fat please!” more",
"neighbourhood": "Forest Hill",
"thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/2AvlfZlwXNlk1pGVqktKqQ/348s.jpg",
"button": {
"text": "Start Order",
"link": "https://www.yelp.com/biz/DqJjIACQF6T85MR4ZYTrEA?show_platform_modal=True"
},
"categories": [
{
"title": "Burgers",
"link": "https://www.yelp.com/search?cflt=burgers&find_loc=San+Francisco%2C+CA"
}
],
"service_options": [
"Outdoor seating",
"Delivery",
"Takeout"
]
},
{
"title": "Uncle Boy’s",
"reviews": "731",
"snippet": "“Extremely rude manager and over priced burgers. Do I need to leave a long story about it? I don't” more",
"neighbourhood": "Inner Richmond",
"thumbnail": "https://s3-media0.fl.yelpcdn.com/bphoto/qYAqCmJ21xBKa1skFoSRXA/348s.jpg",
"categories": [
{
"title": "Burgers",
"link": "https://www.yelp.com/search?cflt=burgers&find_loc=San+Francisco%2C+CA"
},
{
"title": "Fast Food",
"link": "https://www.yelp.com/search?cflt=hotdogs&find_loc=San+Francisco%2C+CA"
}
],
"service_options": [
"Outdoor seating",
"Delivery",
"Takeout"
]
},
.....
],
"serpdog_pagination": {
"next": "https://api.serpdog.io/yelp?api_key=APIKEY&find_desc=burger&find_loc=San+Francisco%2CCA&start=10"
}
}
Frequently Asked Questions
Q. How to scrape Yelp data for free?
You can use Serpdog's Yelp Search API for scraping Yelp Data for free. Serpdog provides 100 credits to users on their first sign-up.
Q. Can you scrape Yelp data?
Yes, you can scrape Yelp data with the help of any scraping API present in the market.
Q. Is it legal to scrape Yelp data?
You can scrape any data publicly available on the internet without any legal problem.
Conclusion:
In this tutorial, we learned to scrape Yelp Buessiness Reviews by making a basic scraper with the help of Node JS. We also saw how Serpdog | Yelp Search API can help you with scraping the results.
I hope you enjoyed the tutorial. Feel free to message me if I missed something. Follow me on Twitter. Thanks for reading!
Additional Resources
Want to learn more about web scraping? Not a problem! We have already prepared the list of tutorials so you can kickstart your web scraping journey.
- Web Scraping Google With Node JS - A Complete Guide
- Web Scraping Google Maps Results
- Scrape Google Shopping Results
- Scrape Google Maps Reviews
Author:
My name is Darshan, and I am the founder of serpdog.io. I love to create scrapers. I am working for several MNCs to provide them with Google Search Data through a seamless pipeline.
Also published here.
Top comments (0)