Devil Scrapes

Posted on Sep 2

Four levels of JSON to say the word Leasehold

#python #api #opendata #realestate

Quick answer

HM Land Registry publishes every property sale in England and Wales since 1995, free and keyless. It also publishes it as linked data, which means a single sale price arrives looking like this:

"estateType": {
  "_about": "http://landregistry.data.gov.uk/def/common/leasehold",
  "label":     [{"_value": "Leasehold", "_datatype": "langString", "_lang": "en"}],
  "prefLabel": [{"_value": "Leasehold", "_datatype": "langString", "_lang": "en"}]
}

Four levels of nesting to say the word Leasehold. The UK Land Registry Sold Prices Scraper flattens all of it into one row per sale. That flattening is the product.

Three things the shape will do to you 🪤

1. The rows are not where you think. They are at result.items, not at the top level. The top level is format, version, and a result object full of paging links.

2. The date is not a date. transactionDate comes back as "Thu, 29 Jul 2004" — an RFC-822-style string with a day name. Sort that lexically and you get Fridays before Mondays. It needs parsing and re-emitting as ISO-8601, and it is the single most likely field to ship silently wrong.

3. Half the address is optional. saon (the flat or unit) and locality are frequently absent. Declare them as bare strings in your dataset schema and the first terraced house without a flat number kills the write — and a test sample of city-centre apartments hides it perfectly, because they all have a saon.

The parameter check we now always run ✅

Before building, we asked the API to prove each filter actually does something. This is not paranoia; it is a lesson with a receipt. On a different Actor last week we found a documented, HTTP-200-returning pagination parameter that the server silently ignored — every page identical to page one, and under pay-per-event that is duplicate rows the customer pays for.

So, measured live rather than assumed:

Check	Result
`_page=0` vs `_page=1`	different transaction ids — paging is real
`min-pricePaid=500000`	returned £550,000 / £500,000 / £750,000 — filter is real
`min/max-transactionDate` across 2024	returned Jun / Mar / Jan 2024 — filter is real

All three genuinely filter. Now we can build on them.

The empty result that is not an error 🕳️

Postcode SW1A 1AA — Buckingham Palace — returns HTTP 200 with items: []. Nobody has sold it lately.

That is a legitimate answer to a legitimate question, and it must not look like a failure. But it also must not become the demo. An earlier version of this Actor's example input would have shipped a first-run experience of "success, zero rows" to every new customer, which is the worst possible first impression: nothing is broken, nothing is wrong, and nothing is there.

The prefill is M1 1AE — Manchester city centre, verified live to return real transactions, and re-verified immediately before shipping.

The deeper rule: "no results" and "never reached the API" must be different outcomes. A run that legitimately matches nothing should say so. A run that never got an answer must fail loudly. A SUCCEEDED run with zero rows scores 100% on every health dashboard while delivering nothing, and we have been burned by exactly that.

The bug we caught before you could 🐛

During the build, postcode had picked up a default of "M1 1AE" in the Pydantic model — not just in the input schema's prefill.

Look at what that does. A customer searches by town, or by district, or by a date range with no postcode at all. The model helpfully fills in M1 1AE. Every one of those searches silently narrows to one Manchester postcode, returns plausible-looking rows, and nothing anywhere reports an error.

A prefill belongs in input_schema.json, where it is a suggestion the customer can clear. A default in the model is a value they cannot see and cannot remove.

What you get per row

transaction_id, transaction_url, price_paid_gbp, transaction_date (ISO-8601), property_type, estate_type, new_build, transaction_category, record_status, the address split into paon / saon / street / locality / town / district / county / postcode, plus a single assembled address line and a scrape timestamp.

Filter by postcode, town, district, county, price band and sale-date range.

Who this is for

Estate agents and valuers — comparables for a street or postcode, as a spreadsheet.
Proptech and mortgage teams — price history by area without a data licence.
Investors and analysts — new-build versus existing, leasehold versus freehold, by district over time.

The honest limitations 🚧

England and Wales only. Scotland and Northern Ireland keep separate registers.
Residential sales as registered — not asking prices, not rental data.
Registration lags completion, so the most recent weeks are always incomplete.

Pricing

$0.05 per run plus $0.005 per transaction — $5.05 per 1,000. Pay for rows that land.

→ UK Land Registry Sold Prices Scraper on Apify

Built by Devil Scrapes. We handle the linked-data envelope, the day-name dates and the addresses that are missing half their fields, so you get a table instead of an ontology.

DEV Community