Your proxy returned 200 OK and the wrong country's data

The worst scraping bug we have shipped did not throw an exception. It returned HTTP 200, a full page of well-formed listings, every field populated, every assertion green — and the data was from the wrong country.

Quick answer

A rotating residential proxy pool will happily give you an exit IP in a country you did not ask for, and a geo-aware site will then serve you plausible, correct-looking data for that country instead. There is no error to catch. The defence is two-part: pin the exit country explicitly in your proxy configuration, and then verify the country in the response body before you emit a single row. If you only do the first, you are trusting a config flag with no feedback loop.

How a 200 OK lies to you

Geo-targeted sites do not usually reject a foreign visitor. They localise for them. Ask a big job board for "developer jobs in Sydney" from a German exit IP and you may well get a working page — correct HTML, real listings, sensible salaries — for an entirely different market. Or a market splash page. Or the right search, silently region-filtered.

Every failure signal your scraper watches for is absent:

• Status code: 200
• Content length: normal
• Parse: succeeds
• Required fields: all present
• Row count: as expected

The only thing wrong is the answer. And because rotation is random, it is wrong intermittently — maybe one page in eight, depending on your pool's composition. That is the signature that makes this expensive: intermittent, silent, and invisible to every test that checks structure rather than meaning.

We now treat this as its own bug class. A wrong-country row is worse than a failed request, because a failed request is loud and a wrong row gets into someone's dataset.

Step 1 — pin the country, do not hope for it

Most proxy providers let you constrain the exit country. On Apify Proxy, that is countryCode on the residential group:

proxy = await Actor.create_proxy_configuration(
    groups=["RESIDENTIAL"],
    country_code="AU",
)

Leave that off and you get "some residential IP, somewhere". For a national job board, that is not a proxy configuration, it is a lottery ticket.

Worth knowing: requesting a tier is not the same as using one. We have measured runs that passed a residential configuration and billed zero residential bytes — one preferred an environment variable over the input we handed it, another's proxy helper failed and silently degraded to a direct connection. The only honest check is the run's billing record. If the residential transfer line is zero, residential was not used, whatever your config said.

Step 2 — verify geography in the response

Pinning is necessary and not sufficient. The response itself has to be interrogated before it is trusted. For an Australian job board that means checking the site's own market signal and each posting's country code, and treating a mismatch as a retryable failure, not as data:

if not looks_like_target_market(response):
    logger.warning("geo mismatch — rotating session and retrying")
    rotate_session()
    raise RetryableError("wrong market")

That is the whole guard. Rotate the session ID so you get a fresh exit IP, retry, and only emit rows once the market check passes. It costs one extra comparison per response and it converts an invisible data-quality defect into an ordinary, visible retry.

The general form, for any geo-sensitive target: find the field in the response that names the market, and assert on it. Currency symbol, country code, locale string, market identifier, a phone format — something the page states about itself. If a target does not expose one, that absence is itself a finding, and you should be far more conservative about what you claim the data represents.

Why not just use one static IP?

Because you will get blocked, and then you will have neither correctness nor coverage. Rotation is not the enemy here; unconstrained rotation is. You want a pool that rotates within a country, plus a check that confirms it did. Rotation gives you resilience, pinning gives you correctness, and verification gives you proof — you need all three, and most implementations stop after the first.

What else Seek actually needs

The geo guard is the interesting part, but it is not the only part:

• Real browser TLS. We use curl-cffi impersonation so the handshake and HTTP/2 fingerprint look like Chrome, Firefox or Safari rather than Python's. Fingerprint mismatch is one of the quieter reasons a scraper gets nothing back.
• Backoff that honours Retry-After. On 408 / 429 / 503, exponential backoff up to five attempts, and when the server tells you how long to wait, wait that long. Ignoring Retry-After is how a temporary throttle becomes a permanent block.
• Taxonomy preservation. Seek classifies roles with its own classification and subclassification scheme. Flattening that to a free-text string throws away the most useful filtering dimension in the dataset, so we keep both levels intact alongside the suburb → state → country location hierarchy.

What the scraper actually does

Seek Australia Jobs Scraper searches au.seek.com by keyword and Australian location and returns one normalized row per posting — title, company, classification and subclassification, flattened location hierarchy, work type and arrangement, salary label, listing date and teaser. No login, no API key. Pay-per-result at $1.83 per 1,000 rows, so a search that returns nothing costs nothing beyond the start fee.

We are not going to tell you Seek is easy to scrape. Job boards actively defend their listings, they change those defences without notice, and the honest framing is that absorbing the blocks, the rotation, the retries and the geo verification is our job, not yours. When it does start pushing back, that is our problem to fix, not a support ticket for you to write.

If you need the same treatment on other markets, the neighbours are SmartRecruiters, Workable, Greenhouse and the multi-ATS aggregator.

The rule we wrote down

Geo-random residential exits return plausible wrong data with a 200 status, not errors. Pin the country, then verify the country in the body. A proxy config you never validate against the response is a hypothesis, not a control.