DEV Community

Devil Scrapes
Devil Scrapes

Posted on

XING's job data ships as a JS object, not JSON. Here's how we parse it without breaking on the first undefined.

Quick answer

XING ships a complete Apollo GraphQL cache inline in every search-result page — <script id="runtime-config">window.crate={...} — so the job data never needs DOM scraping. The catch: that blob is a JavaScript object literal, not JSON, so a naive json.loads throws on it. We normalize it first, then resolve ROOT_QUERY's jobSearchByQuery collection into typed VisibleJob/Company nodes. Verified cloud run: 50 deduplicated rows across two keywords ("developer" 40, "marketing" 10), Berlin, pagination confirmed across 2 pages per keyword. The XING Jobs Scraper is built on that resolved graph, not a screen-scrape.

Isn't window.crate={...} just JSON you can json.loads?

No, and treating it like JSON is the fastest way to ship a scraper that dies on page two. window.crate is a live JavaScript object literal assigned in a <script> tag — it's what the browser's own runtime evaluates, not a JSON serialization of it. It contains bare undefined tokens in place of missing fields, which are valid JavaScript but not valid JSON syntax; json.loads throws on the first one it hits. We normalize the blob — swapping the JS-only tokens for their JSON equivalents — before we ever try to parse it, which is the difference between "this works on the one listing I tested" and "this works on the thousandth listing that happens to have a hole in it."

Where does the company name actually come from?

Not where you'd expect. The Apollo cache is normalized: every job entity holds a reference like Company:<id> instead of the company's data inline, and resolving that reference on its own gets you a logo URL and nothing else — no name. The name lives on a different node entirely, companyInfo.companyNameOverride, which we join back onto the resolved job. Skip that join and you ship a scraper that returns a logo and a blank company column — technically a "success," practically useless to a recruiter filtering by employer.

Why is salaryMedian sparser than salaryMin/salaryMax?

Because XING serves two structurally different salary shapes under one field name, and only one of them carries a median. Its own SalaryEstimate type includes a computed median; an employer-stated SalaryRange gives you a min and max only, no median, because the employer never published one. Counted live on a single result page: 44 SalaryEstimate nodes against 28 SalaryRange nodes. So on a page where roughly two-thirds of salaried postings carry a median, expect that ratio to hold in your dataset too — the sparsity isn't a scraping gap, it's what XING itself published.

The blob at window.crate looks like JSON and isn't — bare undefined tokens break json.loads before you ever get to the graph you actually want.

What the Actor gives you

One deduplicated row per job across one or more keywords: id, slug, title, url, companyName, companyLogoUrl, locationCity, locations, employmentType, salaryMin/salaryMax, salaryMedian (when the posting is a SalaryEstimate), salaryCurrency, keyResponsibilities, refreshedAt, activeUntil, paid, topJob, plus sourceKeyword/sourceLocation tagging which search found each row. Run-wide dedup means the same job id never appears twice, even across pages or across keywords batched into one run.

Honest limitations 🚧

List-page fields only — full job-description text isn't fetched; follow the returned url for a detail-page job if you need it. XING ranks by relevance, not strict keyword match, so a niche keyword can surface adjacent roles instead of an empty page. A narrow search legitimately returning zero rows is a successful run, not a failure.

FAQ

Do I need a XING account or API key?
No — this reads XING's public search result pages.

Can I search multiple keywords in one run?
Yes — pass an array to keywords; every row is tagged via sourceKeyword, and dedup applies across the whole run.

Why did I get fewer rows than maxResults?
Either the search genuinely has fewer matches, or maxPagesPerKeyword stopped the run first — the run's status message says which.

$0.20 per run plus $0.0025 per deduplicated job row — $2.70 per 1,000 results. A zero-match search still succeeds and costs only the start fee.

→ XING Jobs Scraper on Apify


Built by Devil Scrapes. We rotate Chrome/Firefox TLS fingerprints, retry with backoff, and route every request through residential proxy pinned to Germany — and we normalize XING's inline cache into a real object graph instead of guessing at a screen-scrape.

Top comments (0)