DEV Community

Devil Scrapes
Devil Scrapes

Posted on

InfoJobs Spain Scraper: the proxy default that 407'd every run

Quick answer

We shipped a proxy default that read as reasonable and failed every single run. The config was {"useApifyProxy": true, "apifyProxyCountry": "ES"} — a bare useApifyProxy with no named group. That resolves to Apify's datacenter tier, and our datacenter pack is a static US-only pool, so pinning ES against it asked for an exit country the pack simply doesn't own. Cloud QA caught it before any customer did: CONNECT 407, proxy auth rejected. The fix was naming the group — apifyProxyGroups: ["RESIDENTIAL"] alongside the country pin — and the InfoJobs Spain Jobs Scraper now ships with both, not just one.

If you pin a proxy country, do you also need to name the group?

Yes, and skipping it isn't a degraded result — it's a guaranteed tunnel failure. A country pin tells Apify Proxy which exit to hand you; the group tells it which pool to hand it from. Name only the country and Apify defaults you into datacenter, because that's the cheaper tier absent other instructions. If your account's datacenter pack has no IPs in the country you pinned — ours is a static US-only 27-IP pack — the tunnel refuses to open at all. Every InfoJobs run against the old default would have failed before fetching a single page. This is a config-shape bug, not a scraping-difficulty bug, and it's exactly the class of thing a five-minute local test never surfaces because local runs don't route through Apify's proxy infrastructure.

Did the shipped defaults actually get exercised in the cloud before going live?

No — and that was the second, separate finding on this Actor. The prefill that ships with every new Actor is what Apify's own automated QA runs, and what a customer sees pre-filled if they click Start without touching the form. Ours was maxResults: 5, maxPages: 1 — a single page. Pagination is the entire point of a job-board scraper, and it had never actually run in the cloud, by us or by Apify. We widened it to maxResults: 40, maxPages: 2 and re-ran: 40 rows landed, status message read "2/2 page(s) parsed". Two separate bugs, same root cause — a config nobody had actually pointed at the live target.

A prefill that returns one page is a prefill that never tests pagination — no matter how much the pagination code itself has been reviewed.

What the Actor gives you

One row per deduplicated job posting: job_id, title, description, city, url, contract_type, salary_min/salary_max/salary_period/salary_currency (null when unpublished), workday, teleworking, published_at (ISO-8601), company_name, company_logo_url, company_url, states, and is_executive. Province filters accept either a plain name ("Madrid") or InfoJobs' own numeric id ("33") — both resolve against the site's own province table, not a hand-typed lookup.

Honest limitations 🚧

One keyword plus an optional province per run — no multi-keyword batching yet. maxResults and maxPages are independent caps; whichever fires first stops the run, and the status message says which. List-page fields only, no detail-page follow-through. A narrow search returning zero rows is a successful run, not a failure.

FAQ

Why did the Actor 407 on every run before this fix?
Because the default proxy config named a country but not a group, which silently resolved to a US-only datacenter pack that has no Spanish exits. Naming apifyProxyGroups: ["RESIDENTIAL"] fixed it.

Do I need an InfoJobs account or API key?
No — this reads InfoJobs' public search result pages, parsed from the page's own window.__INITIAL_PROPS__ JSON blob.

Can I filter by province?
Yes — pass a plain Spanish province name or its numeric InfoJobs id; the input param is provinceIds under the hood, resolved by testing live requests rather than guessed from the UI.

$0.20 per run plus $0.0025 per deduplicated job row — $2.70 per 1,000 results. A zero-match search still succeeds and costs only the start fee.

→ InfoJobs Spain Jobs Scraper on Apify


Built by Devil Scrapes. We rotate browser fingerprints, retry with backoff, and route every request through Apify Proxy — and we test our own proxy configuration against the pack we actually own before we ship it as a default.

Top comments (0)