A 200 that isn't reachability: SpareRoom's soft-redirect and the empty run nobody notices

Quick answer: SpareRoom does not 404 an unknown location. It soft-redirects to the generic /flatshare/search.pl form page and returns HTTP 200. A scraper that checks the status code sees success, parses a form, finds zero cards, and reports a clean empty run. The fix is not a better selector — it's refusing to treat a status code as evidence that you landed where you asked to land.

Why does a misspelled location return 200?

Because the redirect is a product decision, not an error path. Ask SpareRoom for /flatshare/not-a-real-place-xyz and you end up at https://www.spareroom.co.uk/flatshare/search.pl — the search form — with a perfectly healthy 200 and a full page of HTML. Live-confirmed 2026-09-16.

That is the single most expensive bug shape in scraping, because it is invisible. A run that returns zero rows and exits SUCCEEDED scores 100% on every health dashboard. Nothing alerts. The customer gets an empty dataset and a plausible explanation ("no matches in that area this week").

So the reachability question has to be asked with three independent signals instead of one:

def passes_location_guard(final_url: str, html: str) -> bool:
    """All three signals must pass for the page to be trusted as parseable."""
    if not host_matches(final_url):
        logger.warning("location_guard: host mismatch final_url=%s", final_url)
        return False
    if not location_resolved(final_url):
        logger.warning("location_guard: location did not resolve, redirected to search form")
        return False
    if not container_present(html):
        logger.error("location_guard: listing-results container missing — possible selector drift")
        return False
    return True

Host stayed on www.spareroom.co.uk. Final path is a real /flatshare/<location> page rather than the generic form. The class="listing-results container markup is still in the body.

What does each of the three signals actually catch?

They fail differently on purpose, and the log line tells you which one fired:

• Host mismatch catches an interstitial or a proxy captive portal — you are not on SpareRoom at all.
• Path check catches the soft-redirect above — you are on SpareRoom, but not on the page you asked for.
• Container check catches selector drift — right page, right host, markup changed under you.

Collapsing them into one boolean would throw away exactly the information you need at 2am.

How do you tell "zero results" apart from "blocked"?

This is the distinction that decides whether the run should retry or move on, and it is worth getting right rather than guessing.

A location that resolves but genuinely has no matches still carries the listing-results container. That was confirmed against a live page with an absurd rent filter applied — real page, real container, zero cards. A block or a soft-redirect does not carry the container.

So: container present + zero cards = an honest empty result, report it and move on. Container absent = something is wrong, rotate and retry. One is data, the other is an incident, and a len(rows) == 0 check cannot tell them apart.

Why isn't the listing URL the one in data-listing-url?

Because it's a click-tracking redirect through fad_click.pl, not the canonical page.

Every result card is a server-rendered <li class="listing-result"> carrying 23 distinct data-listing-* attributes — id, neighbourhood, postcode, rent, period, photo count, verification flags. It is a genuinely generous markup surface, and the obvious move is to read the URL out of it like everything else.

That gives your customers a dataset full of tracking redirects. The canonical detail URL is on the covering anchor inside the same <li>:

CARD_SELECTOR = "li.listing-result"
LINK_SELECTOR = "a.listing-card__link"

The general rule: an attribute named like the thing you want is not necessarily the thing you want. Open one in a browser before you ship a hundred thousand of them.

Why ship both a raw rent and a normalised one?

Because "£900" means nothing without its period, and flatshare listings mix pcm (per calendar month) and pw (per week) freely on the same results page.

The rows therefore carry four fields rather than two — rent_amount + rent_period_unit as advertised, and rent_normalised_amount + rent_normalised_period_unit for comparison. Both, always. A scraper that silently normalises has destroyed the advertised figure; one that only passes through the raw value has handed the customer the comparison problem. Keeping both costs two columns.

What keeps one bad location from killing the rest?

Parsing raises exactly one exception — UnparseableListingError, when neither listing_id nor url can be extracted from a card at all. Every other missing field degrades to None.

Per-location fault isolation lives one level up, in the scraper loop. A location that soft-redirects, a page whose container vanished, a card with mangled markup — each is contained to its own scope, and the other cities still ship. The number one cause of a low-success scraper is not a hard target; it's a recoverable error crashing the whole run.

What a row looks like

{
  "listing_id": "2720947",
  "url": "https://www.spareroom.co.uk/flatshare/london/acton/2720947",
  "title": "Spacious Double Room Acton W3",
  "location": "london",
  "neighbourhood": "Acton",
  "postcode": "W3",
  "rooms_in_property": 2,
  "advertiser_role": "live out landlord",
  "rent_amount": 900,
  "rent_period_unit": "pcm",
  "ad_verified": true,
  "photo_count": 8
}

Measured on a live 60-row run across London and Manchester: 60 rows, 60 unique listing IDs, and zero always-empty fields — every column in the schema carried data somewhere in the sample.

---

😈 SpareRoom Flatshare Listings Scraper pulls UK flatshare and room-rental listings by location — id, canonical URL, title, neighbourhood, postcode, property type, advertiser role, advertised and normalised rent, availability, verification and photo counts — paginated, deduped and geo-guarded. We handle the blocks, the retries, the soft-redirects that pretend to be results, and the selector drift, so you get rows instead of a debugging session. $1.40 per 1,000 results.

FAQ

Does an unknown location fail loudly?
No, and that's the trap this Actor is built around. SpareRoom soft-redirects an unknown location slug to /flatshare/search.pl at HTTP 200. We check the final host, the final path and the results-container markup before trusting a page, and log which of the three failed.

How do you tell an empty search from a block?
A genuinely empty result set still renders the listing-results container; a block or soft-redirect does not. Container present with zero cards is reported as an honest zero. Container missing is treated as an incident.

Is the rent per week or per month?
Both are on the row. rent_amount + rent_period_unit are exactly as advertised, and rent_normalised_amount + rent_normalised_period_unit give you a comparable figure. Listings mix pcm and pw on the same page.

Can I scrape more than one city per run?
Yes — pass a list of location slugs with a per-location page cap. Each location is fault-isolated, so one that fails doesn't take the others down.

Does it need a login?
No. These are public search-results pages, served rendered. No account, no session, no cookie jar.