Scraping real estate data with Python to find opportunities

Philippe Greenleaf — Wed, 15 Jan 2025 15:19:41 +0000

In this tutorial, we'll explore how to scrape real estate data from an API using Python's requests library. We'll also learn how to apply filters to retrieve potential bargain properties whose prices have recently dropped.

Introduction

When hunting for great real estate opportunities, one of the best indicators can be a recent price drop. Having a tool that quickly shows you only these properties can save you tons of time—and might help you scoop up a deal before everyone else notices!

In this post, we’ll:

Discuss the basics of using requests to interact with a real estate API.
Learn how to filter results using query parameters—particularly focusing on price variation queries.
Parse and display the returned data in a concise format.

Requirements

Python 3 installed
A terminal or command-line prompt
Basic familiarity with the Python requests library
An API key (if required by the API)

Step 1: Understanding the API

The API we use might respond with data such as:

Property ID
Title or Address
Price
Location
Historical price changes
Other relevant information

Key Query Parameters

This API supports several query parameters that help us filter the results:

Parameter	Type	Description
includedDepartments[]	array	Filter by department(s). Example: `departments/77`
fromDate	date	Only retrieve properties listed (or updated) after this date.
propertyTypes[]	array	Filter by property type. Example: `0` for apartments, `1` for houses, etc.
transactionType	string	`0` for sale, `1` for rent, etc.
withCoherentPrice	bool	Only retrieve properties whose price is coherent with the market.
budgetMin	number	Minimum budget threshold.
budgetMax	number	Maximum budget threshold.
eventPriceVariationFromCreatedAt	date	Date from which an event of type price is created — inclusive.
eventPriceVariationMin	number	Minimum percentage of price variation (negative or positive).

We’ll especially focus on the eventPriceVariation parameters to find properties that have decreased in price.

Step 2: Crafting the Request

Below is a sample script using Python's requests library to query the endpoint. Adjust the parameters and headers as needed, especially if an X-API-KEY is required.

import requests
import json

# 1. Define the endpoint URL
url = "https://api.stream.estate/documents/properties"

# 2. Create the parameters
params = {
    'includedDepartments[]': 'departments/77',
    'fromDate': '2025-01-10',
    'propertyTypes[]': '1',    # 1 might represent 'apartment'
    'transactionType': '0',    # 0 might represent 'sale'
    'withCoherentPrice': 'true',
    'budgetMin': '100000',
    'budgetMax': '500000',
    # Focusing on price variation
    'eventPriceVariationFromCreatedAt': '2025-01-01',  # since the beginning of the year    
    'eventPriceVariationMin': '10',  # at least a 10% drop    
}

# 3. Define headers with the API key
headers = {
  'Content-Type': 'application/json',
  'X-API-KEY': '<your_api_key_here>'
}

# 4. Make the GET request
response = requests.get(url, headers=headers, params=params)

# 5. Handle the response
if response.status_code == 200:
    data = response.json()
    print(json.dumps(data, indent=2))
else:
    print(f"Request failed with status code {response.status_code}")

Explanation of Important Parameters

eventPriceVariationMin = '-10'

This means you’re looking for at least a 10% price decrease.

eventPriceVariationMax = '0'

Setting this to 0 ensures you don’t include properties that have had any price increase or any variation above 0%. Essentially, you’re capturing negative or zero changes.

💡 Tip: Adjust the min/max values to suit your strategy. For instance, -5 and 5 would include price changes within a ±5% range.

Potential Pitfalls & Considerations

Authentication: Always ensure you’re using valid API keys. Some APIs also have rate limits or usage quotas.
Error Handling: Handle cases where the API is down or parameters are invalid.
Data Validation: The API might return incomplete data for some listings. Always check for missing fields.
Date Formats: Make sure your fromDate and toDate are in a format the API recognizes (e.g., YYYY-MM-DD).
Large Datasets: If the API returns hundreds or thousands of listings, you might need pagination. Check the API docs for pagination parameters like page or limit.

Wrap-Up

Now you have a basic Python script to scrape real estate data, focusing on properties that have seen a drop in price. This approach can be extremely powerful if you’re looking to invest in real estate or if you simply want to track market trends.

As always, tailor the parameters to your specific needs. You can expand this script to sort results by price, integrate advanced analytics, or even plug the data into a machine learning model for deeper insights.

Happy scraping, and may you find that hidden gem!

DEV Community: Philippe Greenleaf