DEV Community: Vlad

How to scrape Google Maps local businesses (+emails)

Vlad — Tue, 23 Mar 2021 09:18:59 +0000

Scrape local businesses from Google

Google Maps is a perfect source of local businesses for marketers as it has millions of companies with contact information about them. You can get phone numbers, emails, social profiles, etc.

Automate the scraping process

Fetching data manually is unhumanistic job to do even for 25 companies! Therefore, there are many companies that provide such services to extract the data automatically. In the following post, I'll describe the best way to scrape all certified accountants from the US by using Outscraper.

Let's get started

First of all, sign up to Outscraper.
Open Google Maps scraper, enter categories, and select locations.
Check advanced parameters.
Press "Scrape data" and validate a few queries by opening them.
Press "Confirm" and you will be offered to verify your payment method. Go to billing information page by clicking "Verify" and connect a valid credit card. There is a Free Tier that you can use each month. You will not be charged for the usage belove the Free Tier.
Once the pavement method is verified, move back to our tab and click "I've performed the action". The task should start and you should see the notification about it.
Open tasks page to monitor the progress of your task. For our case, the scraping task will take about 19 hours to be finished.
After any task is finished you will receive an email with the link to S3 storage where the results are saved for 30 days. Alternatively, you can download results from the tasks page.

Bonus

Use the promotion code GOOGLEMAPSSCRAPER on the profile page to receive a 10% discount.

How to scrape all the Google Play reviews from any app

Vlad — Fri, 19 Mar 2021 18:11:05 +0000

Scrape Google Play review

Manual extracting even for 25 reviews is a really boring job. Fortunately, there are some services that will do with a decent price or even with a Free tier.

In the following post, I'll describe the extraction process of any amount of reviews from any app/movie/book into csv/excel files (check out my other posts on how to do it via API).

Getting started

Create an account at Outscraper - the service we are going to use to extract the data.
Open Profile and connect a valid credit card. There is a Free Tier that will be refreshed each month. It should be enough for some basic usage and you will not be charged for the usage belove the Free Tier.
Copy link to any app, movie, or book.
Open Google Play scraper service and insert the link or app ID to the textbox like this. Be aware that you can use IDs only for apps (like Facebook app ID: com.facebook.katana).
Select language, sorting, and other parameters:
Start the extraction.
Go to the tasks page and check for results.
Download the results:
Open the file:

Some extra

Use the promotion code SCRAPER2021 on profile page to receive a 10% discount.

How to Scrape Google Maps

Vlad — Wed, 06 Jan 2021 20:32:32 +0000

Scrape Google Maps (Places)

Professional marketers, researchers, and data scientists use the data from Google Maps to build models or reach clients. There is a vast amount of company contacts stored on Google that can be exported into a file. In the following article, I'll show the quickest and simplest way of doing so.

Getting started

Create an account at Outscraper - the service we are going to use to extract the data and emails from companies' websites. Doing so manually even for 25 sites is unhumanistic.
Open Profile page and connect a valid credit card. There is a Free Tier that we can use each month. You will not be charged for the usage belove the Free Tier.
Open Google Maps scraper and enter categories, locations, and other parameters.
Check the queries before starting a task.
Make sure your task is limited to the amount of the data that you are comfortable with (don't forget to check prices for the usage above the Free Tier).
Run the task.
After any task is finished you will receive an email with the link to S3 storage where the results are saved. Alternatively, you can download results and see tasks progress from the Tasks page.

Tasks page:

Use this promotion code on the Profile page to receive a 10% discount: GOOGLEMAPSSCRAPER.

How to scrape Google Maps in Python

Vlad — Wed, 06 Jan 2021 19:33:28 +0000

Creating the scraper from scratch

Developing your own Google Maps scraping tool might be a big challenge in case you don't have a few years of experience behind you. You will need to be prepared for plenty of challenges from Google - IP protection (proxies), cookies & sessions, browser emulation, site updates, etc.

More quick and stable way

Fortunately, there are some good 3-rd party tools that you can easily integrate with your code and start scraping Google just within 5 minutes.

Getting started

Installation (Python 3+)

pip install google-services-api

Usage

from outscraper import ApiClient

api_cliet = ApiClient(api_key='SECRET_API_KEY_FROM_OUTSCRAPER')
response = api_cliet.google_maps_search('restaurants brooklyn usa', language='en', region='US', limit=100)

Response

{
  'id': '3f6f8bd0-4073-4ca3-b92f-4caff9cb0456',
  'status': 'Success',
  'data': [
    {
      'name': 'The Loft Steakhouse',
      'full_address': '1306 40th St, Brooklyn, NY 11218',
      'borough': 'Borough Park',
      'street': '1306 40th St',
      'city': 'Brooklyn',
      'postal_code': '11218',
      'country_code': 'US',
      'country': 'United States of America',
      'us_state': 'New York',
      'state': 'New York',
      'plus_code': None,
      'latitude': 40.639734499999996,
      'longitude': -73.9868193,
      'time_zone': 'America/New_York',
      'site': 'http://www.theloftsteakhouse.com/',
      'phone': '+1 718-475-5600',
      'type': 'Steak house',
      'subtypes': 'Steak house, Fine dining restaurant, Kosher restaurant, Delivery Restaurant, Takeout Restaurant, Restaurant',
      'posts': None,
      'rating': 4.9,
      'reviews': 1638,
      'photos_count': 1054,
      'google_id': '0x89c25ad4507a6e2f:0x135c0e38302054ac',
      'place_id': 'ChIJL256UNRawokRrFQgMDgOXBM',
      ...
    }
   ...
  ]
}

Python package page

API docs

How to scrape Google Maps reviews (more than 5) in python

Vlad — Mon, 28 Dec 2020 21:14:55 +0000

Scrape reviews with Google Places API?

The official way of getting reviews from Google Places API is limited to 5 reviews and it's barely enough even for the most simplest task.

“predictions” contains an array of places, with information about the
place. The Places API returns up to 5 results.

Google Maps Reviews API by Outscraper

Professional marketers and data-scientists employ many 3-d party services like Outscraper for their needs. It allows you to get started within 5 minutes.

You can use UI version and extract reviews to CSV file or there is API and SDK that you can use from your code.
In the following example, I'll show some demo using Python 3.

Extracting Google Maps Reviews using python SDK

We will need python 3+ and Outscraper SDK
pip install google-services-api

Extracting 100 reviews for Golden Unicorn


from outscraper import ApiClient

api_cliet = ApiClient(api_key='SECRET_API_KEY')
reviews_response = api_cliet.google_maps_business_reviews(
    'Golden Unicorn, East Broadway, USA', limit=100, language='en')

The SECRET_API_KEY can be found on the profile page after registration. There is a Free Tier that you can use each month.

Response

{
  "id": "373f43de-7394-479b-a0bd-d2666def8ddf",
  "status": "Success",
  "data": {
    "name": "Golden Unicorn",
    "full_address": "18 E Broadway, New York, NY 10002",
    "borough": "Manhattan",
    "street": "18 E Broadway",
    "city": "New York",
    "postal_code": "10002",
    "country_code": "US",
    "country": "United States of America",
    "us_state": "New York",
    "state": "New York",
    "plus_code": null,
    "latitude": 40.7136857,
    "longitude": -73.99718659999999,
    "time_zone": "America/New_York",
    "site": "http://www.goldenunicornrestaurant.com/",
    "phone": "+1 212-941-0911",
    "type": "Cantonese restaurant",
    "subtypes": "Cantonese restaurant, Bar, Chinese restaurant, Dim sum restaurant",
    "posts": null,
    "rating": 4,
    "reviews": 2051,
    "photos_count": 2383,
    "google_id": "0x89c25a2665c2cc5d:0xede58acf7633b27a",
    "place_id": "ChIJXczCZSZawokRerIzds-K5e0",
    "reviews_link": "https://www.google.com/search?q=Golden+Unicorn,+18+E+Broadway,+New+York,+NY+10002&ludocid=17142260180301623930#lrd=0x89c25a2665c2cc5d:0xede58acf7633b27a,1",
    "reviews_id": "-1304483893407927686",
    "photo": "https://lh5.googleusercontent.com/p/AF1QipOilwEHwjKXdH9PAaq490iCTrpfaEfvfO3vNsxq",
    "working_hours": {
      "Monday": "9:30AM–3:30PM",
      "Tuesday": "9:30AM–3:30PM",
      "Wednesday": "9:30AM–3:30PM",
      "Thursday": "9:30AM–3:30PM",
      "Friday": "9:30AM–3:30PM",
      "Saturday": "9:30AM–4:30PM",
      "Sunday": "9:30AM–4:30PM"
    },
    "temporarily_closed": null,
    "about": {
      "Service options": {
        "Delivery": true,
        "Takeout": true
      },
      "Health & safety": {
        "Mask required": true,
        "Staff required to disinfect surfaces between visits": true
      },
      "Highlights": {
        "Great tea selection": true
      },
      "Popular for": {
        "Breakfast": true,
        "Lunch": true,
        "Dinner": true
      },
      "Accessibility": {
        "Wheelchair accessible entrance": true,
        "Wheelchair accessible seating": true,
        "Wheelchair accessible parking lot": false
      },
      "Offerings": {
        "Comfort food": true,
        "Small plates": true
      },
      "Dining options": {
        "Catering": true,
        "Dessert": true
      },
      "Amenities": {
        "Good for kids": true,
        "High chairs": true
      },
      "Atmosphere": {
        "Casual": true
      },
      "Crowd": {
        "College students": true,
        "Groups": true,
        "Tourists": true
      },
      "Planning": {
        "Dinner reservations recommended": true,
        "Accepts reservations": true,
        "Usually a wait": true
      }
    },
    "price_range": "$$",
    "reviews_per_score": {
      "1": 152,
      "2": 86,
      "3": 307,
      "4": 655,
      "5": 851
    },
    "verified": true,
    "reserving_table_link": null,
    "booking_appointment_link": "http://www.goldenunicornnyc.com/menu.aspx",
    "owner_id": "109794426950456075294",
    "owner_title": "Golden Unicorn",
    "owner_link": "https://www.google.com/maps/contrib/109794426950456075294",
    "location_link": "https://www.google.com/maps/place/Golden+Unicorn/@40.7136857,-73.99718659999999,14z/data=!4m8!1m2!2m1!1sGolden+Unicorn!3m4!1s0x89c25a2665c2cc5d:0xede58acf7633b27a!8m2!3d40.7136857!4d-73.99718659999999",
    "reviews_data": [
      {
        "google_id": "0x89c25a2665c2cc5d:0xede58acf7633b27a",
        "autor_link": "https://www.google.com/maps/contrib/109132427106491791258?hl=en-US",
        "autor_name": "Benjamin Qiu",
        "autor_id": "109132427106491791258",
        "review_text": "Keeping it real out here in CT\n\nWish I get to go inside but nonetheless had my first outdoor dimsum experience and was quite a positive experience.  Despite limited seating we were able to be seated with very little wait, the ordering process was simple with menu in both English and Chinese. Staff were very patient and professional, even made sure we were warm enough from the heating lamp. We ordered our favorite dishes and they were simply amazing.\nTimes are tough during COVID but this is one place that's doing things right. Will love to visit again soon.",
        "review_img_url": "https://lh5.googleusercontent.com/p/AF1QipPlgEuTFN8m_lePXzdcOCC0PA-fzV5Htc0Tk2eW",
        "owner_answer": null,
        "owner_answer_timestamp": null,
        "owner_answer_timestamp_datetime_utc": null,
        "review_link": "https://www.google.com/maps/reviews/data=!4m5!14m4!1m3!1m2!1s109132427106491791258!2s0x0:0xede58acf7633b27a?hl=en-US",
        "review_rating": 5,
        "review_timestamp": 1609111047,
        "review_datetime_utc": "12/27/2020 23:17:27",
        "review_likes": 0
      },
      {
        "google_id": "0x89c25a2665c2cc5d:0xede58acf7633b27a",
        "autor_link": "https://www.google.com/maps/contrib/100406104884094884994?hl=en-US",
        "autor_name": "Cynthia Gomez",
        "autor_id": "100406104884094884994",
        "review_text": null,
        "review_img_url": null,
        "owner_answer": null,
        "owner_answer_timestamp": null,
        "owner_answer_timestamp_datetime_utc": null,
        "review_link": "https://www.google.com/maps/reviews/data=!4m5!14m4!1m3!1m2!1s100406104884094884994!2s0x0:0xede58acf7633b27a?hl=en-US",
        "review_rating": 5,
        "review_timestamp": 1609018749,
        "review_datetime_utc": "12/26/2020 21:39:09",
        "review_likes": 0
      },
      {
        "google_id": "0x89c25a2665c2cc5d:0xede58acf7633b27a",
        "autor_link": "https://www.google.com/maps/contrib/112779066937104499139?hl=en-US",
        "autor_name": "Steven Magallanes",
        "autor_id": "112779066937104499139",
        "review_text": null,
        "review_img_url": null,
        "owner_answer": null,
        "owner_answer_timestamp": null,
        "owner_answer_timestamp_datetime_utc": null,
        "review_link": "https://www.google.com/maps/reviews/data=!4m5!14m4!1m3!1m2!1s112779066937104499139!2s0x0:0xede58acf7633b27a?hl=en-US",
        "review_rating": 4,
        "review_timestamp": 1608797697,
        "review_datetime_utc": "12/24/2020 08:14:57",
        "review_likes": 0
      },
      {
        "google_id": "0x89c25a2665c2cc5d:0xede58acf7633b27a",
        "autor_link": "https://www.google.com/maps/contrib/103001143875467131446?hl=en-US",
        "autor_name": "Sandra",
        "autor_id": "103001143875467131446",
        "review_text": "(Translated by Google) I've been here before and have never forgotten my experience. I want to go back so bad. The meal was excellent.\n\n(Original)\nIch war schon einmal hier und habe meine Erfahrung nie vergessen. Ich möchte so sehr zurückgehen. Das Essen war ausgezeichnet.",
        "review_img_url": null,
        "owner_answer": null,
        "owner_answer_timestamp": null,
        "owner_answer_timestamp_datetime_utc": null,
        "review_link": "https://www.google.com/maps/reviews/data=!4m5!14m4!1m3!1m2!1s103001143875467131446!2s0x0:0xede58acf7633b27a?hl=en-US",
        "review_rating": 5,
        "review_timestamp": 1608058639,
        "review_datetime_utc": "12/15/2020 18:57:19",
        "review_likes": 0
      },
...
      {
        "google_id": "0x89c25a2665c2cc5d:0xede58acf7633b27a",
        "autor_link": "https://www.google.com/maps/contrib/107401903433772909404?hl=en-US",
        "autor_name": "Chris Clarke",
        "autor_id": "107401903433772909404",
        "review_text": "One of the best places for dim sum. Delicious",
        "review_img_url": null,
        "owner_answer": null,
        "owner_answer_timestamp": null,
        "owner_answer_timestamp_datetime_utc": null,
        "review_link": "https://www.google.com/maps/reviews/data=!4m5!14m4!1m3!1m2!1s107401903433772909404!2s0x0:0xede58acf7633b27a?hl=en-US",
        "review_rating": 5,
        "review_timestamp": 1605630176,
        "review_datetime_utc": "11/17/2020 16:22:56",
        "review_likes": 0
      }
    ]
  }
}

That's it!

Python package page

API docs