Real Estate Data Collection at Scale: Compliance, Anti-Bot Tactics, and Event-Driven Schemas

Ask anyone who has built a real estate data pipeline what the hard part is, and nobody says "parsing the HTML." The hard part is that listings data is a moving target wrapped in a legal and ethical minefield, served from infrastructure that aggressively fingerprints visitors, and where the value of the data decays by the hour — a listing that showed "Active, $620,000" this morning may be "Pending" by lunch. I've spent a fair amount of time on pipelines like this, and this post is the condensed version of what actually matters: how to stay within defensible bounds legally, how to keep a scraper alive against portal-grade anti-bot systems, and how to design the data model so a collection system becomes a market intelligence system.

First, the Compliance Layer — Before You Write a Scraper

Real estate is unusual among scraping targets because the data itself has a well-defined ownership structure. In the US, listing data mostly originates from MLSes, flows to portals (Zillow, Realtor.com, Redfin) under licensing agreements, and — critically — a large slice of the facts (sale prices, ownership, tax assessments, transaction dates) is public record held by county offices. In the UK, HM Land Registry publishes sold-price data openly. That split between facts and compiled presentation should drive your whole architecture.

A few rules of thumb I've adopted:

1. Check for an official data channel first. Many MLSes license data via RETS or the newer RESO Web API standard. Some portals offer partner APIs. Public records are increasingly available as bulk downloads or open-data portals. Any route that gets you a licensed feed beats scraping.
2. Read the robots.txt and terms, and take them seriously. Several major portals explicitly prohibit scraping in their ToS; courts have been mixed but increasingly receptive to claims built on them. If your use case is commercial, the calculus of "can I technically do it" is the wrong question.
3. Facts are safer than compilations. "This property sold for $620,000 on March 3" is a public-record fact. "This property's Zestimate is $615,000" is a proprietary derived value — scraping and redistributing derived estimates is where projects get into real trouble.
4. Rate-limit as if the site were your own. Portals serve millions of human users; a scraper that hammers listing pages at 50 requests/second is both conspicuous and rude. Politeness isn't just ethics, it's the cheapest anti-detection strategy there is.

With that foundation, let's talk mechanics.

Why Listing Pages Are Technically Annoying

Modern listing portals concentrate most of the interesting data in a few dense endpoints — search results pages, individual listing detail, and the price/tax history blocks. They also run some of the most aggressive bot mitigation on the consumer web, because listing data is valuable enough that everyone from hedge funds to individual flippers is pulling it. What you'll hit, roughly in order:

• TLS and HTTP/2 fingerprinting that flags default requests/aiohttp clients before a single byte of page content is served.
• Per-IP rate thresholds that are low — often a couple of dozen listing pages per IP per hour before a captcha interstitial or a soft 403.
• Geo-gating: listings, pre-foreclosure data, and especially rental inventories vary by the apparent location of the request. Pulling UK listings from a US datacenter IP gets you degraded or blocked results.
• Heavy client-side rendering on search results, with the actual inventory in XHR JSON rather than the HTML.

The practical architecture that survives all four: geo-matched residential proxies, one session per logical task, and — wherever possible — reading the XHR JSON instead of the rendered DOM.

Session Hygiene: One Cookie Jar, One IP, One Task

The single biggest self-inflicted failure in real estate scraping is IP rotation that's too aggressive. A human browsing listings has a consistent IP for a whole session, accumulates cookies, warms up on a couple of pages, then searches. A scraper that swaps exit IP on every request while reusing the same cookie jar (or vice versa) looks exactly like nothing a real user does.

The pattern that works: bind a proxy session and a cookie jar together for the lifetime of a logical task — say, "collect all listings in zip code 78704." Here's a worker skeleton:

import requests, random, time

class ListingWorker:
    """One listing-collection task = one sticky residential IP + one cookie jar."""

    def __init__(self, proxy_session_id: str):
        self.session = requests.Session()
        self.session.proxies = {
            # Thordata residential: sticky session via username suffix
            "http": f"http://thor-account-sessid-{proxy_session_id}-geo-us:@proxy.thordata.com:24125",
            "https": f"http://thor-account-sessid-{proxy_session_id}-geo-us:@proxy.thordata.com:24125",
        }
        self.session.headers.update({
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                          "AppleWebKit/537.36 (KHTML, like Gecko) "
                          "Chrome/126.0.0.0 Safari/537.36",
            "Accept-Language": "en-US,en;q=0.9",
        })

    def warm_up(self, portal_home: str):
        """Load the homepage first like a human would."""
        self.session.get(portal_home, timeout=30)
        time.sleep(random.uniform(1.5, 3.5))

    def collect_zip(self, search_url: str, max_pages: int = 20):
        for page in range(1, max_pages + 1):
            r = self.session.get(search_url.format(page=page), timeout=30)
            if r.status_code == 403 or "captcha" in r.text[:2000].lower():
                raise BlockedError(f"blocked on page {page}")
            yield from parse_search_results(r.json())
            time.sleep(random.uniform(4, 9))  # polite crawl delay

Each zip code gets a fresh session ID — and therefore a fresh sticky IP and a fresh cookie jar. Rate per IP stays low, the traffic pattern per session is coherent, and a block costs you one zip code, not the fleet.

Finding the JSON Under the Page

Most portal search UIs are SPAs that fetch inventory from an internal search API. Spend twenty minutes in devtools' network tab on a search page before writing any selector. If you find a clean XHR returning JSON with listing cards — and you usually will — you've turned a fragile Playwright pipeline into a plain HTTP one, an order of magnitude cheaper. When the endpoint requires tokens minted by a browser session, use the hybrid approach: one Playwright run harvests cookies and tokens, then requests spends them over the same sticky IP.

The Data Model Is Where Real Estate Pipelines Win or Lose

Here's the thing that separates a toy scraper from a real system: listings are not rows, they're timelines. The interesting signal in real estate is almost entirely in transitions — price cuts, days-on-market, status changes from Active to Pending to Sold, re-listings after withdrawal. If your schema stores "current state of each listing," you have a snapshot; if you store events, you have a market.

CREATE TABLE listing_events (
    id           BIGSERIAL PRIMARY KEY,
    listing_uid  TEXT NOT NULL,        -- stable per-listing identity across portals
    portal       TEXT NOT NULL,
    event_type   TEXT NOT NULL,        -- new | price_change | status_change | relisted
    price_usd    NUMERIC,
    status       TEXT,
    observed_at  TIMESTAMPTZ NOT NULL, -- when YOU saw it
    raw_snapshot JSONB                 -- full payload, immutable
);
CREATE INDEX ON listing_events (listing_uid, observed_at);

Every crawl appends events; nothing is overwritten. raw_snapshot as JSONB saves you every time a portal changes its schema — you can re-derive fields retroactively without re-scraping.

Two data-quality problems deserve explicit handling. First, identity resolution: the same property appears on multiple portals with different IDs. Normalize on a property key (address normalization + geocode to a parcel or building-level identifier), and treat portal listing IDs as aliases of a canonical property. Second, re-listing detection: withdrawn-and-relisted is a classic tactic to reset days-on-market. Your event timeline catches it automatically — same property key, new event, prior status_change to withdrawn within N days.

Scheduling for Freshness

Not everything needs the same crawl frequency. Status and price events cluster in the first weeks of a listing's life and on weekends; stale listings sit unchanged for months. A tiered schedule keeps you polite and cheap: hot listings (new in the last 14 days) every few hours, warm (15–60 days) daily, cold weekly. That's the same tiering-by-change-frequency approach I use in ML data pipelines generally, and in real estate it also happens to keep your request rate per portal under the radar.

Finally, a note on geography. If you aggregate across regions — say, US metros plus UK postcodes — route each region's workers through proxies in that country. It isn't just about blocks: some portals genuinely return different inventory and different price-history depth depending on where the request seems to come from, so a wrong-geo proxy silently corrupts your data even when nothing "fails."

Wrapping Up

Real estate data collection done properly is a compliance exercise, a session-management exercise, and a data-modeling exercise — in roughly that order of importance. Prefer licensed feeds and public records where they exist; scrape politely with sticky, geo-matched sessions bound to logical tasks; read the JSON under the page instead of rendering everything; and build an event log, not a snapshot table. Get the identity resolution right and even a modest pipeline becomes a genuine market-intelligence asset.

Disclosure: I use Thordata's residential proxies for the geo-distributed listing collection described in this post. If you want to try them, they're at thordata.com, and the code **thor020* gets you 10% off.*