DEV Community

Devil Scrapes
Devil Scrapes

Posted on

Arbeitnow's Pagination Link Lies About Its Own Filter

#webscraping #python #api #apify

Quick answer

Arbeitnow's job-board API accepts a remote=true query parameter, echoes it right back in the pagination links.next URL on every page, and then ignores it completely when building the response. We pulled page 1 twice in the same minute — once plain, once with ?remote=true tacked on — and got the same 250 jobs in the same order both times, of which only 12 actually had remote: true. If your client trusts the URL it's told to follow next, it will happily paginate through an entire "filtered" feed that was never filtered at all.

The parameter that lied twice 🔎

Here's the exact comparison, curl against the live endpoint minutes apart:

curl -s "https://www.arbeitnow.com/api/job-board-api?page=1"              -o plain.json
curl -s "https://www.arbeitnow.com/api/job-board-api?page=1&remote=true"  -o filtered.json
Enter fullscreen mode Exit fullscreen mode 
plain.json    slugs (250) == filtered.json slugs (250)   -> True
filtered.json jobs actually remote: true                 -> 12 / 250
filtered.json "links": {"next": ".../job-board-api?remote=true&page=2"}
Enter fullscreen mode Exit fullscreen mode

That last line is the trap. The API takes your query string, folds it into the links.next field of its own pagination envelope, and hands it back like a receipt confirming the filter is live. It isn't. Nothing about the response other than that one string suggests the parameter did anything, and if you write a paginator that just follows links.next blindly — which is the normal, correct thing to do with a Laravel-style paginated API — you will walk the entire unfiltered feed, one remote=true-branded page at a time, and never notice.

The second thing: the same job returns two different HTML strings 🎭

We also compared the description field of every job that appeared in both pulls, by slug. 113 of 250 (45%) had a byte-different description_html on the second call than the first — same job, same slug, same title, called seconds apart.

The diff is always in the same place: a footer Arbeitnow appends to the description HTML, alternating between two backlink variants:

A: <p>Find <a href="https://www.arbeitnow.com">Jobs in Germany</a> on Arbeitnow</p>
B: <p>Find more <a href="https://www.arbeitnow.com/english-speaking-jobs">English Speaking Jobs in Germany</a> on Arbeitnow</p>
Enter fullscreen mode Exit fullscreen mode

Arbeitnow's own API terms ask API consumers to "appreciate linking back to the site" — this looks like the server doing exactly that for itself, injecting a rotating self-promotional link into every description at serve time. Fair enough. But it means description_html is not a stable value for a given slug. If you hash the description to detect "did this posting change" between two scheduled runs, roughly half your postings will register a false change every single time, for a reason that has nothing to do with the actual job content.

What this means for a client you'd write yourself

Two failure modes, both invisible until you check the wire:

  1. Trusting the pagination envelope over the data. A links.next URL that carries your filter forward is not proof the filter was applied upstream — only counting the actual field values in the payload is. We never send query parameters this API is known to ignore; every filter (remoteOnly, tags, locationContains) runs client-side against the fetched page, so what you configure is what you get regardless of what the server claims to support.
  2. Hashing a field for change detection that the source rewrites on every call. A content hash needs a source that's actually content-stable. On this feed, description_html isn't one field you can trust for that — which is a small, specific thing to know before you build a "notify me when this posting changes" pipeline on top of it.

Fault isolation matters here too: a single job with a shape the parser doesn't expect is logged and skipped, and the rest of that page's postings still land. A 250-row page shouldn't cost you the whole run because of one entry.

Output

One row per job posting, JSON/CSV/Excel:

slug, title, company_name, description_html, remote, url,
tags[], job_types[], location, posted_at, page
Enter fullscreen mode Exit fullscreen mode 
from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("DevilScrapes/arbeitnow-jobs-scraper").call(
    run_input={"maxResults": 50, "maxPages": 2, "remoteOnly": True, "tags": ["Engineering"]}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], "—", item["company_name"])
Enter fullscreen mode Exit fullscreen mode

Pricing is $0.20 to start a run plus $0.0015 per job row — $1.70 for a thousand postings. A run that finds nothing within maxPages still succeeds, and costs only the start fee.

→ Arbeitnow Jobs Scraper on Apify

Built by Devil Scrapes. We rotate Chrome/Firefox TLS fingerprints and retry with backoff on every request, and we never trust a query parameter — or the pagination link it comes back wrapped in — until we've checked the actual field values against it.

Top comments (0)