DEV Community

Cover image for How to Scrape Homes.com Property Data
Crawlbase
Crawlbase

Posted on • Originally published at crawlbase.com

How to Scrape Homes.com Property Data

#scrapehomescom #scrapinghomescom #scrapinghomescomwithpython #homescomscraper

This blog was originally posted to Crawlbase Blog

Major cities around the world have recently reported a spike in house prices due to the several reasons. Property data is one of the most-sought after information today as more people embrace technology to solve this challenge. Homes.com stands as a useful resource in the real estate sector, with vast database of property listings across the United States. Most prospective use the website to gather important information like prices, locations and other specifics at their comfort.

However, browsing through hundreds of pages on Homes.com can be a daunting task. That’s why scraping homes.com is a good opportunity for buyers, investors and sellers to gain valuable insights on the housing prices in the United States.

This blog will teach you how to scrape homes.com using Python and Crawlbase. It will explore the fundamentals of setting up your environment to handling anti-scraping measures, enabling you to create a good homes.com scraper.

Table of Contents

  1. Why Scrape homes.com Property Data?
  2. What can we Scrape from homes.com?
  3. Bypass Homes.com Blocking with Crawlbase
  • Overview of Homes.com Anti-Scraping Measures
  • Using Crawlbase Crawling API for Smooth Scraping
  1. Environment Setup for homes.com Scraping
  2. How to Scrape homes.com Search Pages
  3. How to Scrape homes.com Property Pages
  4. Final Thoughts
  5. Frequently Asked Questions (FAQs)

Why Scrape homes.com Property Data?

There are many reasons you might want to scrape homes.com. Suppose you are a real estate professional or analyst. In that case, you can gather homes.com data to stay ahead of the market and get great insight into property values, rent prices, neighborhood statistics, etc. This information is crucial to making an investment decision and marketing strategy.

If you are a developer or a data scientist, scraping homes.com with Python allows you to construct a powerful application that uses data as the foundation. By creating a homes.com scraper, you can automate the process of collecting and analyzing property data, saving time and effort. Additionally, having access to up-to-date property listings can help you identify emerging trends and opportunities in the real estate market.

Overall, scrapping homes.com can bring many benefits to anyone who works in the real estate industry, whether it is investors, agents, data scientists, or developers.

What can we Scrape from homes.com?

Here's a glimpse of what you can scrape from homes.com:

  1. Property Listings: Homes.com property listings provide information about available homes, apartments, condos, and more. Scraping these listings provide data about important features, amenities, and images of properties.
  2. Pricing Information: Knowledge of the real estate market price trends is key to being in an advantageous position. Scraping pricing information from homes.com allows you to analyze price variations over time and across different locations.
  3. Property Details: Apart from the basic details, homes.com makes available to customers explicit details about the property, which includes square footage, number of bedrooms and bathrooms, property type, and so forth. You can scrape all this information for a better understanding of each listing.
  4. Location Data: Location plays a significant role in real estate. Scraping location data from homes.com provides insights into neighborhood amenities, schools, transportation options, and more, helping you evaluate the desirability of a property.
  5. Market Trends: By scraping homes.com regularly, you can track market trends and fluctuations in supply and demand. This data enables you to identify emerging patterns and predict future market movements.
  6. Historical Data: Holding data about the history of the real estate market, historical data is useful for studying past trends and patterns in real estate. Presuming to have scraped historical listing and pricing data from homes.com, you can now conduct longitudinal studies, and understand long term trends.
  7. Comparative Analysis: Using Homes.com data, you can do comparative analysis, comparing the properties within the same neighborhood versus across town or in multiple locations where you want to buy or sell property. You can quickly ascertain who your competition is with this data, and use it to determine price strategies.
  8. Market Dynamics: Understanding market dynamics is essential for navigating the real estate landscape. Scraping data from homes.com allows you to monitor factors such as inventory levels, time on market, and listing frequency, providing insights into market health and stability.

Bypass Homes.com Blocking with Crawlbase

Homes.com, like many other websites, employs JavaScript rendering and anti-scraping measures to prevent automated bots from accessing and extracting data from its pages.

Overview of Homes.com Anti-Scraping Measures

Here's what you need to know about how Homes.com tries to stop scraping:

  1. JS Rendering: Homes.com, like many other websites, uses JavaScript (JS) rendering to dynamically load content, making it more challenging for traditional scraping methods that rely solely on HTML parsing.
  2. IP Blocking: Homes.com may block access to its website from specific IP addresses if it suspects automated scraping activity.
  3. CAPTCHAs: To verify that users are human and not bots, Homes.com may display CAPTCHAs, which require manual interaction to proceed.
  4. Rate Limiting: Homes.com may limit the number of requests a user can make within a certain time frame to prevent scraping overload.

These measures make it challenging to scrape data from Homes.com using traditional methods.

Use Crawlbase Crawling API for Smooth Scraping

Crawlbase offers a reliable solution for scraping data from Homes.com while bypassing its blocking mechanisms. By utilizing Crawlbase's Crawling API, you gain access to a pool of residential IP addresses, ensuring seamless scraping operations without interruptions. Its parameters allow you to handle any kind of scraping problem with ease.

Crawling API can handle JavaScript rendering, which allows you to scrape dynamic content that wouldn't be accessible with simple requests. Moreover, Crawlbase manages user-agent rotation and CAPTCHA solving, further improving the scraping process.

Crawlbase provides its own Python library for easy integration. The following steps demonstrate how you can use the Crawlbase library in your Python projects:

  1. Installation: Install the Crawlbase Python library by running the following command. 
pip install crawlbase
Enter fullscreen mode Exit fullscreen mode
  1. Authentication: Obtain an access token by creating an account on Crawlbase. This token will be used to authenticate your requests. For homes.com, we need JS token.

Here's an example function demonstrating the usage of the Crawling API from the Crawlbase library to send requests:

from crawlbase import CrawlingAPI

# Initialize Crawlbase API with your access token
crawling_api = CrawlingAPI({ 'token': 'YOUR_CRAWLBASE_TOKEN' })

# Function to make a request using Crawlbase API
def make_crawlbase_request(url):
    # Send request using Crawlbase API
    response = crawling_api.get(url)

    # Check if request was successful
    if response['headers']['pc_status'] == '200':
        html_content = response['body'].decode('utf-8')
        return html_content
    else:
        print(f"Failed to fetch the page. Crawlbase status code: {response['headers']['pc_status']}")
        return None
Enter fullscreen mode Exit fullscreen mode

Note: The first 1000 requests through the Crawling API are free of cost, and no credit card is required. You can refer to the API documentation for more details.

Environment Setup for homes.com Scraping

Before diving into scraping homes.com, it's essential to set up your environment to ensure a smooth and efficient process. Here's a step-by-step guide to help you get started:

  1. Install Python: First, make sure you have Python installed on your computer. You can download and install the latest version of Python from the official website.
  2. Virtual Environment: It's recommended to create a virtual environment to manage project dependencies and avoid conflicts with other Python projects. Navigate to your project directory in the terminal and execute the following command to create a virtual environment named "homes_scraping_env": 
python -m venv homes_scraping_env
Enter fullscreen mode Exit fullscreen mode

Activate the virtual environment by running the appropriate command based on your operating system:

  • On Windows: 
  homes_scraping_env\Scripts\activate
Enter fullscreen mode Exit fullscreen mode
  • On macOS/Linux: 
  source homes_scraping_env/bin/activate
Enter fullscreen mode Exit fullscreen mode
  1. Install Required Libraries: Next, install the necessary libraries for web scraping. You'll need libraries like BeautifulSoup and Crawlbase to scrape homes.com efficiently. You can install these libraries using pip, the Python package manager. Simply open your command prompt or terminal and run the following commands: 
pip install beautifulsoup4
pip install Crawlbase
Enter fullscreen mode Exit fullscreen mode

  1. Code Editor: Choose a code editor or Integrated Development Environment (IDE) for writing and running your Python code. Popular options include PyCharm, Visual Studio Code, and Jupyter Notebook. Install your preferred code editor and ensure it's configured to work with Python.

  2. Create a Python Script: Create a new Python file in your chosen IDE where you'll write your scraping code. You can name this file something like "homes_scraper.py". This script will contain the code to scrape homes.com and extract the desired data.

By following these steps, you'll have a well-configured environment for scraping homes.com efficiently. With the right tools and techniques, you'll be able to gather valuable data from homes.com to support your real estate endeavors.

How to Scrape homes.com Search Pages

Scraping property listings from Homes.com can give you valuable insights into the housing market.

In this section, we will show you how to scrape Homes.com search pages using Python straightforward approach.

Importing Libraries

We need to import the required libraries: CrawlingAPI for making HTTP requests and BeautifulSoup for parsing HTML content.

from crawlbase import CrawlingAPI
from bs4 import BeautifulSoup
Enter fullscreen mode Exit fullscreen mode

Initialize Crawling API

Get your JS token form Crawlbase and initialize the CrawlingAPI class using it.

# Initialize Crawlbase API with your access token
crawling_api = CrawlingAPI({ 'token': 'CRAWLBASE_JS_TOKEN' })
Enter fullscreen mode Exit fullscreen mode

Defining Constants

Set the base URL for Homes.com search pages and the output JSON file. To overcome the JS rendering issue, we can use ajax_wait and page_wait parameters provided by Crawling API. We can also provide a custom user_agent like in below options. We will set a limit on the number of pages to scrape from the pagination.

BASE_URL = 'https://www.homes.com/los-angeles-ca/homes-for-rent'
OUTPUT_FILE = 'properties.json'
MAX_PAGES = 2

options = {
    'ajax_wait': 'true',
    'page_wait': 10000,
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
}
Enter fullscreen mode Exit fullscreen mode

Scraping Function

Create a function to scrape property listings from Homes.com. This function will loop through the specified number of pages, make requests to Homes.com, and parse the HTML content to extract property details.

We have to inspect the page and find CSS selector through which we can get all the listing elements.

Each listing is inside a div with class for-rent-content-container.

def scrape_listings():
    properties = []  # List to store the properties' information

    # Loop through the pages
    for page in range(1, MAX_PAGES + 1):
        url = f'{BASE_URL}/p{page}/'
        print(f"Scraping page {page} of {url}")

        try:
            html_content = make_crawlbase_request(url)
            if html_content:
                soup = BeautifulSoup(html_content, 'html.parser')
                properties_list = soup.select('div.for-rent-content-container')
                properties.extend(properties_list)
        except Exception as e:
            print(f"Request failed on page {page}: {e}")

    return properties
Enter fullscreen mode Exit fullscreen mode

Parsing Data

To extract relevant details from the HTML content, we need a function that processes the soup object and retrieves specific information. We can inspect the page and find the selectors of elements that hold the information we need.

def parse_property_details(properties):
    property_list = []
    for property in properties:
        title_elem = property.select_one('p.property-name')
        address_elem = property.select_one('p.address')
        info_container = property.select_one('ul.detailed-info-container')
        extra_info = info_container.find_all('li') if info_container else []
        description_elem = property.select_one('p.property-description')
        url_elem = property.select_one('a')

        title = title_elem.text.strip() if title_elem else 'N/A'
        address = address_elem.text.strip() if address_elem else 'N/A'
        price = extra_info[0].text.strip() if extra_info else 'N/A'
        beds = extra_info[1].text.strip() if len(extra_info) > 1 else 'N/A'
        baths = extra_info[2].text.strip() if len(extra_info) > 2 else 'N/A'
        description = description_elem.text.strip() if description_elem else 'N/A'
        url = BASE_URL + url_elem.get('href') if url_elem else 'N/A'

        property_data = {
            "title": title,
            "address": address,
            "price": price,
            "beds": beds,
            "baths": baths,
            "description": description,
            "url": url
        }
        property_list.append(property_data)

    return property_list
Enter fullscreen mode Exit fullscreen mode

This function processes the list of property elements and extracts relevant details. It returns a list of dictionaries containing the property details.

Storing Data

Next, we need a function to store the parsed property details into a JSON file.

import json

def save_property_details_to_json(property_list, filename):
    with open(filename, 'w') as json_file:
        json.dump(property_list, json_file, indent=4)
Enter fullscreen mode Exit fullscreen mode

This function writes the collected property data to a JSON file for easy analysis.

Running the Script

Finally, combine the scraping and parsing functions, and run the script to start collecting data from Homes.com search page.

if __name__ == '__main__':
    properties = scrape_listings()
    property_list = parse_property_details(properties)
    save_property_details_to_json(property_list, OUTPUT_FILE)
Enter fullscreen mode Exit fullscreen mode

Complete Code

Below is the complete code for scraping property listing for homes.com search page.

from bs4 import BeautifulSoup
from crawlbase import CrawlingAPI
import json

# Initialize Crawlbase API with your access token
crawling_api = CrawlingAPI({ 'token': 'CRAWLBASE_JS_TOKEN' })

BASE_URL = 'https://www.homes.com/los-angeles-ca/homes-for-rent'
OUTPUT_FILE = 'properties.json'
MAX_PAGES = 2

options = {
    'ajax_wait': 'true',
    'page_wait': 10000,
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
}

# Function to make a request using Crawlbase API
def make_crawlbase_request(url):
    # Send request using Crawlbase API
    response = crawling_api.get(url, options)
    # Check if request was successful
    if response['headers']['pc_status'] == '200':
        html_content = response['body'].decode('utf-8')
        return html_content
    else:
        print(f"Failed to fetch the page. Crawlbase status code: {response['headers']['pc_status']}")
        return None

def scrape_listings():
    properties = []  # List to store the properties' information

    # Loop through the pages
    for page in range(1, MAX_PAGES + 1):
        url = f'{BASE_URL}/p{page}/'
        print(f"Scraping page {page} of {url}")

        try:
            html_content = make_crawlbase_request(url)
            if html_content:
                soup = BeautifulSoup(html_content, 'html.parser')
                properties_list = soup.select('div.for-rent-content-container')
                properties.extend(properties_list)
        except Exception as e:
            print(f"Request failed on page {page}: {e}")

    return properties

def parse_property_details(properties):
    property_list = []
    for property in properties:
        title_elem = property.select_one('p.property-name')
        address_elem = property.select_one('p.address')
        info_container = property.select_one('ul.detailed-info-container')
        extra_info = info_container.find_all('li') if info_container else []
        description_elem = property.select_one('p.property-description')
        url_elem = property.select_one('a')

        title = title_elem.text.strip() if title_elem else 'N/A'
        address = address_elem.text.strip() if address_elem else 'N/A'
        price = extra_info[0].text.strip() if extra_info else 'N/A'
        beds = extra_info[1].text.strip() if len(extra_info) > 1 else 'N/A'
        baths = extra_info[2].text.strip() if len(extra_info) > 2 else 'N/A'
        description = description_elem.text.strip() if description_elem else 'N/A'
        url = BASE_URL + url_elem.get('href') if url_elem else 'N/A'

        property_data = {
            "title": title,
            "address": address,
            "price": price,
            "beds": beds,
            "baths": baths,
            "description": description,
            "url": url
        }
        property_list.append(property_data)

    return property_list

def save_property_details_to_json(property_list, filename):
    with open(filename, 'w') as json_file:
        json.dump(property_list, json_file, indent=4)


if __name__ == '__main__':
    properties = scrape_listings()
    property_list = parse_property_details(properties)
    save_property_details_to_json(property_list, OUTPUT_FILE)
Enter fullscreen mode Exit fullscreen mode

Example Output:

[
    {
        "title": "Condo for Rent",
        "address": "3824 Keystone Ave Unit 2, Culver City, CA 90232",
        "price": "$3,300 per month",
        "beds": "2 Beds",
        "baths": "1.5 Baths",
        "description": "Fully remodeled and spacious apartment with 2 Bedrooms and 1.5 Bathrooms in an amazing Culver City location. Walking distance to Downtown Culver City plus convenient access to the 405 and the 10 freeways. Open concept kitchen with breakfast bar overlooking the living room and the large private",
        "url": "https://www.homes.com/los-angeles-ca/homes-for-rent/property/3824-keystone-ave-culver-city-ca-unit-2/2er2mwklw8zq6/"
    },
    {
        "title": "House for Rent",
        "address": "3901 Alonzo Ave, Encino, CA 91316",
        "price": "$17,000 per month",
        "beds": "4 Beds",
        "baths": "3.5 Baths",
        "description": "Tucked away in the hills of Encino on a quiet cul-de-sac, resides this updated Spanish home that offers sweeping panoramic views of the Valley. Double doors welcome you into an open concept floor plan that features a spacious formal living and dining room, sleek modern kitchen equipped with",
        "url": "https://www.homes.com/los-angeles-ca/homes-for-rent/property/3901-alonzo-ave-encino-ca/879negnf45nee/"
    },
    {
        "title": "House for Rent",
        "address": "13463 Chandler Blvd, Sherman Oaks, CA 91401",
        "price": "$30,000 per month",
        "beds": "5 Beds",
        "baths": "4.5 Baths",
        "description": "A one-story stunner, this completely and newly remodeled home resides in the highly desirable Chandler Estates neighborhood of Sherman Oaks.A expansive floor plan that utilizes all 3,600 sq ft to its best advantage, this 5 BR - 4.5 BA home is a true expression of warmth and beauty, with",
        "url": "https://www.homes.com/los-angeles-ca/homes-for-rent/property/13463-chandler-blvd-sherman-oaks-ca/mnrh1cw3fn92b/?t=forrent"
    },
    {
        "title": "House for Rent",
        "address": "4919 Mammoth Ave, Sherman Oaks, CA 91423",
        "price": "$19,995 per month",
        "beds": "5 Beds",
        "baths": "6.5 Baths",
        "description": "Gorgeous new home on a gated lot in prime Sherman Oaks, lovely neighborhood! Featuring 5 BR \u2013 6.5 BA in main home and spacious accessory dwelling unit with approx. 4,400 sq ft. Open floor plan features living room with custom backlit accent wall, as well as dining room with custom wine display and",
        "url": "https://www.homes.com/los-angeles-ca/homes-for-rent/property/4919-mammoth-ave-sherman-oaks-ca/yv8l136ks5f2e/"
    },
    {
        "title": "House for Rent",
        "address": "12207 Valleyheart Dr, Studio City, CA 91604",
        "price": "$29,500 per month",
        "beds": "6 Beds",
        "baths": "6.5 Baths",
        "description": "Graceful and Spacious Modern farmhouse, with stunning curb appeal, a luxurious cozy retreat on one of the most charming streets in the valley.   Located in coveted and convenient Studio City, this property boosts an open welcoming floor plan and complete with ADU, providing enough room and space",
        "url": "https://www.homes.com/los-angeles-ca/homes-for-rent/property/12207-valleyheart-dr-studio-city-ca/6104hnkegbnx3/"
    },
    ..... more
]
Enter fullscreen mode Exit fullscreen mode

How to Scrape Homes.com Property Pages

Scraping Homes.com property pages can provide detailed insights into individual listings.

In this section, we will guide you through the process of scraping specific property pages using Python.

Importing Libraries

We need to import the required libraries: crawlbase for making HTTP requests and BeautifulSoup for parsing HTML content.

from crawlbase import CrawlingAPI
from bs4 import BeautifulSoup
Enter fullscreen mode Exit fullscreen mode

Initialize Crawling API

Initialize the CrawlingAPI class using your Crawlbase JS token like below.

# Initialize Crawlbase API with your access token
crawling_api = CrawlingAPI({ 'token': 'CRAWLBASE_JS_TOKEN' })
Enter fullscreen mode Exit fullscreen mode

Defining Constants

Set the target URL for the property page you want to scrape and define the output JSON file. To overcome the JS rendering issue, we can use ajax_wait and page_wait parameters provided by Crawling API. We can also provide a custom user_agent like in below options.

URL = 'https://www.homes.com/property/14710-greenleaf-st-sherman-oaks-ca/fylqz9clgbzd2/'
OUTPUT_FILE = 'property_details.json'

options = {
    'ajax_wait': 'true',
    'page_wait': 10000,
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
}
Enter fullscreen mode Exit fullscreen mode

Scraping Function

Create a function to scrape the details of a single property from Homes.com. This function will make a request to the property page, parse the HTML content, and extract the necessary details.

def scrape_property(url):
    try:
        html_content = make_crawlbase_request(url)
        if html_content:
            soup = BeautifulSoup(html_content, 'html.parser')
            property_details = extract_property_details(soup)
            return property_details
    except Exception as e:
        print(f"Request failed: {e}")

    return None
Enter fullscreen mode Exit fullscreen mode

Extracting Property Details

Create a function to extract specific details from the property page. This function will parse the HTML and extract information such as the title, address, price, number of bedrooms, bathrooms, and description.

We can use “Inspect” tool in the browser to find CSS selectors of elements holding the information we need like we did in previous section.

def extract_property_details(soup):
    address_elem = soup.select_one('div.property-info-address')
    price_elem = soup.select_one('span#price')
    beds_elem = soup.select_one('span.property-info-feature > span.feature-beds')
    baths_elem = soup.select_one('span.property-info-feature > span.feature-baths')
    area_elem = soup.select_one('span.property-info-feature.lotsize')
    description_elem = soup.select_one('div#ldp-description-text')
    agent_elem = soup.select_one('div.agent-name')
    agent_phone_elem = soup.select_one('div.agent-phone')

    address = address_elem.text.strip() if address_elem else 'N/A'
    price = price_elem.text.strip() if price_elem else 'N/A'
    beds = beds_elem.text.strip() if beds_elem else 'N/A'
    baths = baths_elem.text.strip() if baths_elem else 'N/A'
    area = area_elem.text.strip() if area_elem else 'N/A'
    description = description_elem.text.strip() if description_elem else 'N/A'
    agent = agent_elem.text.strip() if agent_elem else 'N/A'
    agent_phone = agent_phone_elem.text.strip() if agent_phone_elem else 'N/A'

    property_data = {
        'address': address,
        'price': price,
        'beds': beds,
        'baths': baths,
        'area': area,
        'description': description,
        'agent': agent,
        'agent_phone': agent_phone
    }

    return property_data
Enter fullscreen mode Exit fullscreen mode

Storing Data

Create a function to store the scraped data in a JSON file. This function takes the extracted property data and saves it into a JSON file.

import json

def save_property_details_to_json(property_data, filename):
    with open(filename, 'w') as json_file:
        json.dump(property_data, json_file, indent=4)
Enter fullscreen mode Exit fullscreen mode

Running the Script

Combine the functions and run the script to scrape multiple property pages. Provide the property IDs you want to scrape in a list.

if __name__ == '__main__':
    property_data = scrape_property(URL)

    if property_data:
        save_property_details_to_json(property_data, OUTPUT_FILE)
Enter fullscreen mode Exit fullscreen mode

Complete Code

Below is the complete code for scraping property listing for homes.com property page.

from crawlbase import CrawlingAPI
from bs4 import BeautifulSoup
import json

# Initialize Crawlbase API with your access token
crawling_api = CrawlingAPI({ 'token': 'CRAWLBASE_JS_TOKEN' })

URL = 'https://www.homes.com/property/14710-greenleaf-st-sherman-oaks-ca/fylqz9clgbzd2/'
OUTPUT_FILE = 'property_details.json'

options = {
    'ajax_wait': 'true',
    'page_wait': 10000,
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
}

# Function to make a request using Crawlbase API
def make_crawlbase_request(url):
    # Send request using Crawlbase API
    response = crawling_api.get(url, options)
    # Check if request was successful
    if response['headers']['pc_status'] == '200':
        html_content = response['body'].decode('utf-8')
        return html_content
    else:
        print(f"Failed to fetch the page. Crawlbase status code: {response['headers']['pc_status']}")
        return None

def scrape_property(url):
    try:
        html_content = make_crawlbase_request(url)
        if html_content:
            soup = BeautifulSoup(html_content, 'html.parser')
            property_details = extract_property_details(soup)
            return property_details
    except Exception as e:
        print(f"Request failed: {e}")

    return None

def extract_property_details(soup):
    address_elem = soup.select_one('div.property-info-address')
    price_elem = soup.select_one('span#price')
    beds_elem = soup.select_one('span.property-info-feature > span.feature-beds')
    baths_elem = soup.select_one('span.property-info-feature > span.feature-baths')
    area_elem = soup.select_one('span.property-info-feature.lotsize')
    description_elem = soup.select_one('div#ldp-description-text')
    agent_elem = soup.select_one('div.agent-name')
    agent_phone_elem = soup.select_one('div.agent-phone')

    address = address_elem.text.strip() if address_elem else 'N/A'
    price = price_elem.text.strip() if price_elem else 'N/A'
    beds = beds_elem.text.strip() if beds_elem else 'N/A'
    baths = baths_elem.text.strip() if baths_elem else 'N/A'
    area = area_elem.text.strip() if area_elem else 'N/A'
    description = description_elem.text.strip() if description_elem else 'N/A'
    agent = agent_elem.text.strip() if agent_elem else 'N/A'
    agent_phone = agent_phone_elem.text.strip() if agent_phone_elem else 'N/A'

    property_data = {
        'address': address,
        'price': price,
        'beds': beds,
        'baths': baths,
        'area': area,
        'description': description,
        'agent': agent,
        'agent_phone': agent_phone
    }

    return property_data

def save_property_details_to_json(property_data, filename):
    with open(filename, 'w') as json_file:
        json.dump(property_data, json_file, indent=4)

if __name__ == '__main__':
    property_data = scrape_property(URL)

    if property_data:
        save_property_details_to_json(property_data, OUTPUT_FILE)
Enter fullscreen mode Exit fullscreen mode

Example Output:

{
  "address": "14710 Greenleaf St Sherman Oaks, CA 91403",
  "price": "$11,000 per month",
  "beds": "Beds",
  "baths": "Baths",
  "area": "10,744 Sq Ft Lot",
  "description": "N/A",
  "agent": "Myles Lewis",
  "agent_phone": "(747) 298-7020"
}
Enter fullscreen mode Exit fullscreen mode

Final Thoughts

Scraping data from Homes.com is useful for market research, investment analysis, and marketing strategies. Using Python with libraries like BeautifulSoup or services like Crawlbase, you can efficiently collect data from Homes.com listings.

Crawlbase's Crawling API executes scraping tasks confidently, ensuring that your requests mimic genuine user interactions. This approach enhances scraping efficiency while minimizing the risk of detection and blocking by Homes.com's anti-scraping measures.

If you're interested in learning how to scrape data from other real estate websites, check out our helpful guides below.

📜 How to Scrape Realtor.com
📜 How to Scrape Zillow
📜 How to Scrape Airbnb
📜 How to Scrape Booking.com
📜 How to Scrape Redfin

If you have any questions or feedback, our support team is always available to assist you on your web scraping journey. Remember to follow ethical guidelines and respect the website's terms of service. Happy scraping!

Frequently Asked Questions (FAQs)

Q. Is scraping data from Homes.com legal?

Yes, scraping data from Homes.com is legal as long as you abide by their terms of service and do not engage in any activities that violate their policies. It's essential to use scraping responsibly and ethically, ensuring that you're not causing any harm or disruption to the website or its users.

Q. Can I scrape Homes.com without getting blocked?

While scraping Homes.com without getting blocked can be challenging due to its anti-scraping measures, there are techniques and tools available to help mitigate the risk of being blocked. Leveraging APIs like Crawlbase, rotating IP addresses, and mimicking human behavior can help improve your chances of scraping smoothly without triggering blocking mechanisms.

Q. How often should I scrape data from Homes.com?

The frequency of scraping data from Homes.com depends on your specific needs and objectives. It's essential to strike a balance between gathering timely updates and avoiding overloading the website's servers or triggering anti-scraping measures. Regularly monitoring changes in listings, market trends, or other relevant data can help determine the optimal scraping frequency for your use case.

Top comments (0)

Read next

ryoichihomma profile image

How to Customize Your GitHub Profile: Part 2

Ryoichi Homma -

smy profile image

TypeScript SDK Development: A 5-year-old could follow this step-by-step ~ Part 3, Making Test Apps

Syed Muhammad Yaseen -

smy profile image

TypeScript SDK Development: A 5-year-old could follow this step-by-step ~ Part 4, Publishing to NPM

Syed Muhammad Yaseen -

awwsamu profile image

Software Engineer in Dubai: Jobs, Salary & Challenges Explained for Fresh Graduates and Entry-Level Professionals

Sam -