UPDATE Annoyingly, whilst continuing to work on this, I have gotten stuck due to the details listed here: https://medium.com/@mariomenti/how-not-to-run-an-api-looking-at-you-zoopla-bda247e27d15. To cut a long story short, Zoopla no longer actively support the API, which means that the API keys just randomly stop working with no warning. Back to the drawing board I suppose...
Introduction
For a little holiday project, I wanted to try and build something to help with a non-work thing I've found has been inefficient - property searching. My partner and I are hoping to buy a place soon, and we've found it frustrating trying to keep up with property pages, plus who doesn't like finding something that annoys them a little, then spend hours trying to get rid of that itch!
This article is the first of a few (depending on far I take it), and I'll add links to subsequent docs at the bottom. Things that will be covered later will include integrating with the Airtable API for a good spreadsheet + images view, using Lambda and Cloudformation to make the work repeatable for others, then using Cloudwatch Events and Eventbridge to automate the process!
Initial Research
My first thought was a web scraper might be a a good option, and the site I use most frequently when looking at properties is Rightmove.
However, a little googling quickly lead me to this section of Rightmove's T's & C's:
You must not use or attempt to use any automated program (including, without limitation, any spider or other web crawler) to access our system or this Site. You must not use any scraping technology on the Site. Any such use or attempted use of an automated program shall be a misuse of our system and this Site. Obtaining access to any part of our system or this Site by means of any such automated programs is strictly unauthorised.
So that was the end of that train of thought...
Zoopla is another commonly-used property site in the UK, so this was the next option. They have a refreshing opinion on third party development, and have an extensive API with what I've seen so far to be a good set of docs, so this seemed like a good place to start!
Broad Plan
The first version for this won't be serverless, just to make sure I'm getting things right. I want to be able to query Zoopla using a script, via the Node REPL, and then be able to send the results to a suitable place for viewing. I've toyed with a few ideas, but I think to begin with Airtable (on the free version!) should be good enough. This does mean this won't be a totally serverless solution, but we can potentially replace it further down the line. From there, I'll use Lambda to do the communicating with Zoopla/Airtable, and then set it to run at a certain frequency using a Cloudwatch cron-event.
Development
Step 1: Zoopla Test
First step here was to register for the Zoopla Developer API for the API key required to make queries. This was fairly simple, just needing some standard details and some inclination of what you want to do with it.
Then, to see what results I would get I quickly tested the Property Listings Endpoint using Postman.
Adding a bunch of fields which I felt would be useful (postcode, radius, listing_status, maximum_price, minimum_beds and of course the api_key), revealed a quite extensive result (see below).
Step 2: Prototype Code
Though I want to get this into Lambda, I thought it best to try with running a Javascript-script locally to repeat what i've done with Postman, and allowing me to separate out the params for the request.
In order to separate the logic of the params being used for a specific query and the querying itself, for now I have written the params I want to test with to a local params.json file that looks like this:
{
"api_key": "INSERTAPIKEYHERE",
"radius": 1,
"listing_status": "sale",
"maximum_price": 1221000,
"minimum_beds": 1,
"postcode": "NW1 6XE",
"page_size": 10
}
(I have of course changed the demo params here to be those for Sherlock Holmes, who due to house price rises will now have to do with a budget of 0.001221b£ for a property on Baker St.)
These params can then be used, along with axios, to query the Zoopla endpoint like so:
const axios = require("axios");
const fs = require("fs");
const propertyListingUrl = "https://api.zoopla.co.uk/api/v1/property_listings.js";
const getParams = async () => {
const params = await fs.readFileSync("./params.json");
return JSON.parse(params);
};
const buildConfig = async () => {
return {
params: await getParams(),
url: propertyListingUrl,
headers: {},
method: "get"
};
};
const axiosRequest = async config => {
const result = axios(config);
return result;
};
const queryZooplaPropertyListing = async () => {
const config = await buildConfig();
const result = await axiosRequest(config);
return result.data;
};
module.exports = {
queryZoopla: queryZooplaPropertyListing
};
Here, in the main queryZooplaPropertyListing function we are building our config, which involves reading in our params from ./params.json, then we are using this built config with axios to request the property listings from the Zoopla url.(Note I've appended .js to the url in order to receive a JSON response!)
This uses Node's async-await functionality, as both the file-reading, and the Zoopla request itself are asynchronous processes.
After the promises have resolved, the exported queryZoopla function should then returns an object which looks like this:
{
"country": "England",
"result_count": 196,
"longitude": -0.158541,
"area_name": " NW1",
"listing": [
{
"country_code": "gb",
"num_floors": 0,
"image_150_113_url": "https://lid.zoocdn.com/150/113/2cd80711fb52d57e85068b025920836abb906b89.jpg",
"listing_status": "sale",
"num_bedrooms": 2,
"location_is_approximate": 0,
"image_50_38_url": "https://lid.zoocdn.com/50/38/2cd80711fb52d57e85068b025920836abb906b89.jpg",
"latitude": 51.525627,
"furnished_state": null,
"agent_address": "24 Charlotte Street, London",
"category": "Residential",
"property_type": "Flat",
"longitude": -0.162988,
"thumbnail_url": "https://lid.zoocdn.com/80/60/2cd80711fb52d57e85068b025920836abb906b89.jpg",
"description": "360' virtual tour available. A very well presented second floor apartment set within a popular gated development located just moments away from the open spaces of Regent's Park, Baker Street & Marylebone stations and numerous shops, bars and restaurants. The property is quietly located overlooking the courtyard gardens comprising two bedrooms, two bathrooms, a reception room, seperate kitchen with integrated appliances and 2 x private balconys. The apartment is sold with an underground parking space. As a resident of the building benefits include concierge, access to a communal gym, a swimming pool and landscaped communal gardens. Alberts Court is set within the modern Palgrave Gardens development to the west of Regent's Park. The Leaseholders are currently in the process of purchasing the freehold.The building provides easy access to the West End, The City and various transport links around and out of London.",
"post_town": "London",
"details_url": "https://www.zoopla.co.uk/for-sale/details/55172443?utm_source=v1:8aaVEj3AGALC-xWzf7867y2rJwMs0-2Y&utm_medium=api",
"short_description": "360' virtual tour available. A very well presented second floor apartment set within a popular gated development located just moments away from the open spaces of Regent's Park, Baker Street & Marylebone stations and numerous shops, bars (truncated)",
"outcode": "NW1",
"image_645_430_url": "https://lid.zoocdn.com/645/430/2cd80711fb52d57e85068b025920836abb906b89.jpg",
"county": "London",
"price": "1200000",
"listing_id": "55172443",
"image_caption": "Picture No. 13",
"image_80_60_url": "https://lid.zoocdn.com/80/60/2cd80711fb52d57e85068b025920836abb906b89.jpg",
"status": "for_sale",
"agent_name": "Hudsons Property",
"num_recepts": 1,
"country": "England",
"first_published_date": "2020-07-09 08:44:51",
"displayable_address": "Alberts Court, 2 Palgrave Gardens, Regent's Park, London NW1",
"floor_plan": [
"https://lc.zoocdn.com/4cb0366075b14e99efe3a1a7b24a608f4c7a92f0.jpg"
],
"street_name": "Alberts Court",
"num_bathrooms": 2,
"agent_logo": "https://st.zoocdn.com/zoopla_static_agent_logo_(62918).jpeg",
"price_change": [
{
"direction": "",
"date": "2020-06-28 22:30:07",
"percent": "0%",
"price": 1200000
}
],
"agent_phone": "020 3641 7089",
"image_354_255_url": "https://lid.zoocdn.com/354/255/2cd80711fb52d57e85068b025920836abb906b89.jpg",
"image_url": "https://lid.zoocdn.com/354/255/2cd80711fb52d57e85068b025920836abb906b89.jpg",
"last_published_date": "2020-07-09 08:44:51"
}
],
"street": "",
"radius": "0.5",
"town": "",
"latitude": 51.523659,
"county": "London",
"bounding_box": {
"longitude_min": "-0.170158861045769",
"latitude_min": "51.5164304665016",
"longitude_max": "-0.146923138954231",
"latitude_max": "51.5308875334984"
},
"postcode": "NW1 6XE"
}
And voila. A swanky 2 bed, 2 bath property near Baker St for Sherlock to relocate to! With a whole heap of extra data to boot. Evaluating this will be the first part of the next step, as we aim to get this data into Airtable, so stay tuned!
You can see this code in full at https://github.com/jcts3/serverlessPropertyHelper/tree/workingLocalQuery
Oldest comments (2)
Hi James,
How did you manage to get the Zoopla API working? I have seen that it is no longer usable. My API key is stuck on 'waiting' on their website.
Hey,
I've found it's a bit hit and miss with getting an account to work. I've come across the issues explained here medium.com/@mariomenti/how-not-to-... a lot. Sometimes an API key works, sometimes it doesn't / it stops working after a bit of time, which I guess is due to them stopping development on it.