DEV Community

The Power of Scraping Google Maps

Did you know that nearly 80% of online searchers use Google Maps to find local businesses? For businesses, market analysts, and developers, this wealth of data can be a goldmine. Scraping it with Python is a powerful way to harness all that valuable location-based information.
From understanding customer preferences to pinpointing new venue locations, scraping Google Maps data gives you insights that can directly influence strategic decisions. In this guide, we’ll walk you through the essentials of extracting data from Google Maps using Python. You'll learn how to pull location details, business information, and customer trends — and even export it all into a clean, usable CSV file.

Step 1: Preparing the Environment

Before we get into the code, make sure you have these Python libraries installed:
requests
lxml
csv (comes with Python by default)
If you’re missing any of these, just install them via pip:

pip install requests
pip install lxml
Enter fullscreen mode Exit fullscreen mode

Step 2: Building Your Target URL

The first thing you need is the URL from Google Maps or a Google search results page. This URL will be the foundation of your scraping task.
Here’s an example URL for local businesses in your area:

url = "https://www.google.com/search?q=restaurants+near+me"
Enter fullscreen mode Exit fullscreen mode

Step 3: Setting Up Headers and Proxies

Google is smart — it doesn’t want bots scraping its pages. To avoid getting flagged, you need to mimic a legitimate user. That’s where headers come in.
Headers essentially tell the website that the request is coming from a regular browser (not a script). Along with headers, using proxies helps mask your IP and prevents rate-limiting.
Here’s how you can set them up:

headers = {
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
}

proxies = {
    "http": "http://your_proxy_ip:port",
    "https": "https://your_proxy_ip:port",
}
Enter fullscreen mode Exit fullscreen mode

If you're unsure about proxies, think of them as your anonymity shield when scraping.

Step 4: Extracting Page Content

Now, it’s time to send the request and pull the page content. Here's how to get the raw HTML:

import requests

response = requests.get(url, headers=headers, proxies=proxies)
if response.status_code == 200:
    page_content = response.content
else:
    print(f"Failed to retrieve the page. Status code: {response.status_code}")
    exit()
Enter fullscreen mode Exit fullscreen mode

If the request is successful, you'll have the raw page content, which is crucial for scraping.

Step 5: Parsing the HTML Content

We need to extract specific pieces of data from the page. This is where lxml comes in handy. It helps you navigate the HTML tree and pinpoint the data you need using XPath.
Let’s load the page content into an lxml object:

from lxml import html

parser = html.fromstring(page_content)
Enter fullscreen mode Exit fullscreen mode

Step 6: Discovering Data XPaths

To scrape meaningful data, you must locate it within the HTML structure. Use browser developer tools to inspect the elements you're after (like business names, addresses, and ratings). For example, here are a few XPaths to locate relevant data points:
Restaurant Name:

  //div[@class="VkpGBb"]/span/text()
Enter fullscreen mode Exit fullscreen mode

Address:

  //div[@class="Io6YTe"]/text()
Enter fullscreen mode Exit fullscreen mode

Geo-coordinates:

  //div[@class="VkpGBb"]/@data-lat
Enter fullscreen mode Exit fullscreen mode

Step 7: Getting Data

With your XPaths ready, you can start scraping the data. Here's how you pull the information and store it in a list:

results = parser.xpath('//div[@class="VkpGBb"]')
data = []

for result in results:
    restaurant_name = result.xpath('.//span/text()')[0]
    address = result.xpath('.//div[@class="Io6YTe"]/text()')[0]
    latitude = result.xpath('.//@data-lat')[0]
    longitude = result.xpath('.//@data-lng')[0]

    data.append({
        "restaurant_name": restaurant_name,
        "address": address,
        "latitude": latitude,
        "longitude": longitude
    })
Enter fullscreen mode Exit fullscreen mode

Step 8: Saving the Data to a CSV File

Once you have all the data you need, it’s time to save it in a structured format like CSV. Here's the Python code to do that:

import csv

with open("google_maps_data.csv", "w", newline='', encoding='utf-8') as csv_file:
    writer = csv.DictWriter(csv_file, fieldnames=["restaurant_name", "address", "latitude", "longitude"])
    writer.writeheader()
    for entry in data:
        writer.writerow(entry)
Enter fullscreen mode Exit fullscreen mode

This will output your data in a neat, readable format that can be easily imported into Excel, databases, or other tools.

Final Thoughts

Scraping Google Maps with Python is a straightforward process, but it requires attention to detail. From setting up the right headers to selecting the proxies, every step plays a crucial role in making your scraping task successful.
Proxies are vital for smooth scraping. Rotate IPs and use residential proxies to improve trust factors. It is also important to remember that scraping Google Maps violates their Terms of Service, so always use this information responsibly and consider alternatives like the Google Places API for more legitimate access to Google Maps data.

Top comments (0)