DEV Community: Devil Scrapes

Naver blog search: page 2 only exists if you keep page 1's token

Devil Scrapes — Wed, 14 Oct 2026 03:24:49 +0000

Quick answer

The Naver Blog Search Scraper turns a Naver blog-tab search, in Korean or any UTF-8 keyword, into a deduplicated table of blog posts: post URL, blogId and logNo, title, snippet, blog name and URL, publish date, thumbnail and the Naver influencer profile when there is one. Naver has no public search API for its blog tab. Up to 50 keywords per run and up to 600 posts per keyword. Pricing is pay-per-result, $0.20 per run plus $0.002 per post, $2.20 per 1,000 posts.

Does Naver have a public blog search API?

Not for the search page you see in a browser. Results for something like 강릉 맛집 come from the blog tab of Naver's web search. The Actor reads that tab, skips ad cards, and returns only organic blog posts, with position counting the de-duplicated organic posts per keyword.

Why does page 2 need a token from page 1?

Only the first page of the blog tab is plain HTML. Every later page is a JSON response from a separate pager endpoint, and that endpoint is addressed by a loader object embedded in the page-1 source: a next-page URL plus an X-Prs-Query-Info header token. Request the pager URL without that header and you don't get the next page. And each response carries the URL of the page after it, so pages must be fetched in order, as a chain.

Naver's blog tab only serves page 1 as HTML. Pages 2 onward come from a pager URL that must be requested with a token header embedded in page 1.

So the Actor reads the loader URL and token out of the page-1 source, then walks the chain: each JSON envelope holds the next page's card HTML and the next URL, reused with the same token. Pages are paced 3.5 seconds apart. If the page-1 markup lacks the blog-tab marker, or a pager response isn't the expected JSON, that counts as a block and the session restarts rather than silently returning nothing.

What else we handle

Dedupe by blogId + logNo, parsed from the post URL, within each keyword.
Restarts don't repeat rows. After a block the Actor opens a fresh session with the next browser fingerprint, replays the early pages to catch up, and emits only unseen posts, up to two restarts per keyword.
Dates are honest. An absolute date like 2026.07.24. becomes an ISO publishedAt. A relative one like 3주 전 is kept as publishedAtRelative and not converted to a guess. publishedAtRaw always has the original text.
Ads are skipped, and cards without a usable post link, title or blog name are dropped.

What the Actor gives you

One row per post, 15 fields: query, position, postUrl, blogId, logNo, title, snippet, blogName, blogUrl, influencerUrl, publishedAtRaw, publishedAt, publishedAtRelative, thumbnailUrl and scrapedAt.

{
  "query": "강릉 맛집",
  "position": 1,
  "postUrl": "https://blog.naver.com/104204303/224424493170",
  "blogId": "104204303",
  "logNo": "224424493170",
  "title": "강릉 물회 맛집 장안횟집 풍자 또간집 미역국이 킥",
  "blogName": "매아",
  "blogUrl": "https://blog.naver.com/104204303",
  "influencerUrl": "https://in.naver.com/104204303",
  "publishedAtRaw": "2주 전",
  "publishedAt": null,
  "publishedAtRelative": "2주 전",
  "scrapedAt": "2026-10-11T10:00:00Z"
}

Input: queries (1 to 50 keywords) and maxResultsPerQuery (default 60, up to 600).

Honest limitations 🚧

Blog tab only: no cafe, news or Knowledge-iN results, and no post bodies, comments or likes. Results follow Naver's default order, with no sort or date filter, and depend on the proxy exit region, so the same keyword can return a slightly different set from different regions. Depth is capped at 600 posts per keyword. A keyword with no matches is a successful run that costs only the start fee.

FAQ

Can I search in Korean?
Yes. Keywords are sent UTF-8 encoded, with a Korean Accept-Language.

Why do two keywords return the same post?
Rows are de-duplicated within a keyword, not across keywords. Each row carries its query so you can group as you like.

What about blocks and rate limits?
That's our job. We rotate Firefox browser fingerprints through curl-cffi, use a fresh Apify Proxy session on every restart, and retry with exponential backoff on 408, 429 and 5xx, honouring Retry-After, up to 5 attempts per request.

How does pricing work?
$0.20 per run plus $0.002 per post: $2.20 per 1,000 posts.

→ Naver Blog Search Scraper on Apify

Built by Devil Scrapes. We handle the pager tokens, the fingerprints and the retries, and we don't turn "3 weeks ago" into a date we can't prove. 😈

Scraping HelloWork: why 45 000 EUR is not 45 and 000

Devil Scrapes — Wed, 14 Oct 2026 03:24:11 +0000

Quick answer

The HelloWork Jobs Scraper turns hellowork.com searches, by keyword and French city, into a deduplicated table of job offers: title, company, location split into city and department code, contract type (CDI, CDD, Intérim and so on), working time, remote policy, salary text plus parsed minimum, maximum, currency and period, posting date and apply method. HelloWork is a major French jobs board with no public API. Pricing is pay-per-result, $0.20 per run plus $0.002 per job, $2.20 per 1,000 jobs.

Does HelloWork have a public jobs API?

No. The search page is built for browsers, and the structured facts are scattered between visible nodes, an accessible label on each offer link and an analytics attribute. The Actor reads all three, lets the visible nodes win and uses the label to fill gaps, then returns one validated row per offer.

Why is parsing a French salary harder than it looks?

French number formatting breaks naive regexes. Thousands are separated by spaces, and the site uses non-breaking and narrow non-breaking spaces (U+00A0 and U+202F), so 45 000 - 90 000 € / an may not contain a single ordinary space. The decimal separator is a comma, and the period is written in French (an, jour, mois, heure).

A salary like "45 000 - 90 000 € / an" can be separated by narrow no-break spaces, so splitting on a normal space reads 45 and 000 as two numbers.

So we normalise every space variant first, then parse the range into salaryMin and salaryMax (equal for a single value), salaryCurrency (EUR) and salaryPeriod (year, day, month, hour). If the employer discloses nothing, or the text is not numeric, the numbers are null, never 0, and salaryText keeps what the site showed.

What else we handle

Titles can contain commas. The offer label reads like "title à city, chez company, pour un CDI, ...". We split on the last ", pour un" because the tail segments never contain commas but titles can.
Remote versus contract duration. The same tag slot carries "2 ans" (a contract length) and "Télétravail partiel". Only text containing "travail", accent-folded, becomes remotePolicy.
Anonymous employers. One paragraph in the heading means no company name, so companyName is null instead of repeating the title.
Relative dates. "il y a 5 heures" is kept in postedText and converted to an ISO postedAt from the run time. A month counts as 30 days, and "plus de 1 mois" is read as one month back.
Approximate matches are flagged, not hidden. HelloWork mixes near matches after exact ones; isApproximateMatch lets you filter either way.
Dedupe by job id, and paging stops using the "Afficher N offres" total, 30 jobs per page.

What the Actor gives you

One row per job, 25 fields, including jobId, title, companyName, location, city, departmentCode, contractType, workingTime, remotePolicy, salaryText, salaryMin, salaryMax, salaryCurrency, salaryPeriod, postedText, postedAt, applyMethod, isSuperRecruiter, isApproximateMatch, url, query, searchLocation and scrapedAt.

{
  "jobId": "80459780",
  "title": "Python Developer - Paris H/F",
  "companyName": "Approach People Recruitment",
  "location": "Paris - 75",
  "city": "Paris",
  "departmentCode": "75",
  "contractType": "CDI",
  "workingTime": "temps plein",
  "remotePolicy": null,
  "salaryText": "45 000 - 90 000 € / an",
  "salaryMin": 45000,
  "salaryMax": 90000,
  "salaryCurrency": "EUR",
  "salaryPeriod": "year",
  "postedText": "il y a 5 heures",
  "postedAt": "2026-10-08",
  "applyMethod": "UrlWithForm",
  "url": "https://www.hellowork.com/fr-fr/emplois/80459780.html",
  "query": "python",
  "searchLocation": "Paris",
  "scrapedAt": "2026-10-08T09:30:00Z"
}

Input: queries (keywords), locations (optional French cities; each query runs once per city, empty means all of France) and maxItems per query and city combination.

Honest limitations 🚧

Listing data only: no full job descriptions, no contract or salary filters, no company pages. isSponsored is best effort, and we have not seen a sponsored card yet. Titles, contract names and remote labels stay in French as the site writes them. A search that finds nothing is a successful run that costs only the start fee.

FAQ

Do I need a HelloWork account or API key?
No.

Why is salary empty on many rows?
Many employers don't disclose pay. We leave it empty rather than returning a misleading 0.

What about blocks and retries?
That's our job. We send browser-grade requests through curl-cffi and retry failed pages with exponential backoff, up to 5 attempts, honouring Retry-After. HelloWork answers without a proxy, so Apify Proxy is off by default and optional.

How does pricing work?
$0.20 per run plus $0.002 per job: $2.20 per 1,000 jobs.

→ HelloWork Jobs Scraper on Apify

Built by Devil Scrapes. We handle the French number formats, the retries and the messy labels, so an empty salary means undisclosed, not zero. 😈

BizQuest listings: the anchors lie, the JSON-LD doesn't

Devil Scrapes — Wed, 14 Oct 2026 03:23:33 +0000

Quick answer

The BizQuest Business-for-Sale Scraper turns BizQuest, the US small-business marketplace, into a deduplicated table of businesses for sale: headline, teaser, asking price in USD, city or county, state code and name, lead photo, listing URL, and three card tags (Financing Available, Includes Real Estate, SBA Loan Eligible). Scrape the nationwide feed, a state (Texas, TX) or a city (Houston, TX), or paste a BizQuest URL. BizQuest has no export. Pricing is pay-per-result, $0.20 per run plus $0.006 per listing, $6.20 per 1,000 listings.

Does BizQuest have an API or export?

Not a public one. The site shows about fifty businesses per page and expects you to click through. The Actor walks the listing pages for every route you choose and returns one row per business, with the route, page and position each row was found at so you can trace any figure back to the page it came from.

Why do we read the JSON-LD and ignore the page links?

Because the links lie. BizQuest repeats listing ids in sidebar widgets, and its result list mixes in franchise advertisements and broker promo cards that look like listings. If you collect every anchor on the page you get duplicates and things that are not businesses for sale.

On a BizQuest results page the reliable list of businesses is the inline JSON-LD, not the anchors: the anchors repeat ids from sidebar widgets.

So the parser reads the page's SearchResultsPage JSON-LD list and keeps an entry only if its URL has the shape of a real listing (/business-for-sale/<slug>/BW<id>/). Franchise ads and broker cards have a different URL shape and are dropped. The tags are not in the JSON-LD, so they are read from the server-rendered card for the same BW id.

What else we handle

Out-of-range pages clamp to page 1. Ask BizQuest for a page past the end and it serves page 1 again. A page whose listings were all already seen on that route ends the route, so you are never charged for page 1 twice.
One dedupe set for the whole run. The same listing id never appears twice, across pages or across routes (nationwide plus Texas overlap heavily).
Cash flow and EBITDA are gated. BizQuest shows them only to signed-in visitors. Rather than guess, rows carry financialsGated, which is true when the card says "Sign In to View".
Undisclosed prices stay null. askingPrice is a whole-dollar integer, or null, never 0.
A blocked route doesn't end the others. The run fails only if a route was blocked and nothing at all was collected.

What the Actor gives you

One row per business, 18 fields: listingId, url, title, description, askingPrice, currency, locality, state, stateName, financialsGated, financingAvailable, includesRealEstate, sbaEligible, imageUrl, searchRoute, pageNumber, position and scrapedAt.

{
  "listingId": "2540957",
  "url": "https://www.bizquest.com/business-for-sale/private-gym/BW2540957/",
  "title": "Private gym",
  "description": "Members-only fitness club offering 24-hour gym access.",
  "askingPrice": 400000,
  "currency": "USD",
  "locality": "Monument",
  "state": "CO",
  "stateName": "Colorado",
  "financialsGated": true,
  "financingAvailable": false,
  "includesRealEstate": false,
  "sbaEligible": false,
  "searchRoute": "/businesses-for-sale/",
  "pageNumber": 1,
  "position": 1,
  "scrapedAt": "2026-10-12T08:00:00Z"
}

Input: includeNationwide (default on), locations, startUrls, maxPages (default 3, up to 200, about 50 listings per page) and maxItems (default 150, up to 10,000).

Honest limitations 🚧

Search-card fields only: no detail pages, broker contact details or confidential financials, and no cash flow, EBITDA or revenue. Query-string filters on a pasted URL are dropped, so paste the route you want. BizQuest serves up to about 200 pages for the nationwide feed. A route with no listings is a successful run that costs only the start fee.

FAQ

Do I need a BizQuest account or API key?
No. The Actor reads public listing pages.

What about blocks?
That's our job. We rotate curl-cffi browser fingerprints (Chrome, Firefox, Safari), retry with exponential backoff and Retry-After, up to 5 attempts per page, and rotate US residential Apify Proxy sessions on every block.

Why did I get fewer rows than maxItems?
The route has fewer listings than your cap, or maxPages stopped the run first. The status message says which.

How does pricing work?
$0.20 per run plus $0.006 per unique listing: $6.20 per 1,000 listings.

→ BizQuest Business-for-Sale Scraper on Apify

Built by Devil Scrapes. We handle the fingerprints, the proxies and the retries, and we keep franchise ads out of your deal flow. 😈

AutoTrader.ca Listings Scraper: when an HTTP 200 is a block, not an empty search

Devil Scrapes — Wed, 14 Oct 2026 03:10:18 +0000

Quick answer

The AutoTrader.ca Listings Scraper pulls Canadian car listings by make, model and condition into JSON, CSV or Excel: price in CAD, mileage in kilometres, year, trim, body type, transmission, fuel, seller name and type, city, province, photos and the full seller description. One run covers the whole country, up to 4,000 listings per make, model and condition. AutoTrader.ca has no public listings API and actively blocks automated traffic, so we run it through residential exits and treat every kind of soft block as a block. Pricing is pay-per-result, $0.20 per run plus $0.0025 per listing, $2.70 per 1,000 listings.

Does AutoTrader.ca have an API?

Not for the public. Dealer feeds go in through partner integrations, and nothing comes out. If you want "every used Toyota Corolla for sale in Canada with price and mileage", the only source is the search pages themselves. Each page is a Next.js render with 20 listings inside a __NEXT_DATA__ script tag, along with the total result count and the number of pages.

When is an HTTP 200 not a page of listings?

When the site has decided you are a bot. AutoTrader.ca's protection doesn't always answer with a clean 403. Sometimes it returns HTTP 202 with a challenge body. Sometimes it returns HTTP 200 with a page that has no listing data in it at all. A scraper that only checks the status code reads both as "this search has no cars" and finishes green with an empty dataset.

On AutoTrader.ca, a 202 or a 200 without __NEXT_DATA__ is a block, not an empty search.

So we treat all three as blocks: a non-200, a 202, and a 200 that lacks the listing payload. Each one triggers a retry on a fresh residential session. The only "zero results" we believe is a page that parses cleanly and reports a total of zero. If a page still fails after five attempts we log it, keep every other page, and end the run with a status like "Scraped 380/400 results; 1 page blocked", so you can see exactly what you got.

What else we clean up

Numbers become numbers. "$ 13,990" becomes price_cad: 13990, and the odometer becomes mileage_km, while price_formatted keeps the original string.
Descriptions become plain text. Seller HTML with line breaks and entities comes back as readable text with paragraph breaks kept.
Make and model are forgiving. "Land Rover", "land rover" and "land_rover" all work.
Duplicates are dropped. A listing that shows up on two pages is emitted once: we dedupe on listing_id, and result_type tells you whether it was Organic or Sponsored.
The pages are heavy. Each search page weighs roughly 0.78 MB for 20 listings, and that egress is part of the price you see.

What the Actor gives you

One row per listing, 29 fields, including listing_id, url, title, year, make, model, trim, body_type, price_cad, price_formatted, price_evaluation, mileage_km, transmission, fuel, engine_cc, condition, city, province, postal_code, street, seller_name, seller_type, seller_phone, ad_tier, result_type, image_urls, image_count, description and scraped_at.

{
  "listing_id": "79df9408-002a-44d1-90a3-3355eb9b1770",
  "url": "https://www.autotrader.ca/offers/toyota-corolla-le-79df9408-002a-44d1-90a3-3355eb9b1770",
  "title": "2018 Toyota Corolla LE",
  "year": 2018,
  "make": "Toyota",
  "model": "Corolla",
  "trim": "LE",
  "body_type": "Sedan",
  "price_cad": 13990,
  "price_formatted": "$ 13,990",
  "mileage_km": 144425,
  "transmission": "Automatic",
  "fuel": "Gasoline",
  "condition": "used",
  "city": "North York",
  "province": "ON",
  "seller_name": "First Choice Motors",
  "seller_type": "Dealer",
  "result_type": "Organic",
  "image_count": 36,
  "scraped_at": "2026-10-07T12:00:00+00:00"
}

Input: make, model, condition (used or new) and maxResults up to 4,000.

Honest limitations 🚧

Search is by make, model and condition only: no postal-code radius, province, price, year or mileage filter, so filter the dataset afterwards. One search returns at most 4,000 listings (200 pages of 20). Listing pages only: no VIN, full spec sheet, price history or sold listings. Phone numbers appear only when the listing itself shows one.

FAQ

Do I need an AutoTrader.ca account?
No. The Actor reads public search pages.

What about blocks and captchas?
That's our job. We rotate real-browser TLS fingerprints through curl-cffi, route through Apify residential proxies with a fresh session on every block, and retry with exponential backoff up to 5 attempts per page, honouring Retry-After. One blocked page never sinks the run.

Can I track prices for one model over time?
Yes. Schedule the same make and model daily and diff price_cad per listing_id.

How does pricing work?
$0.20 per run plus $0.0025 per listing: $2.70 per 1,000 listings.

→ AutoTrader.ca Listings Scraper on Apify

Built by Devil Scrapes. We handle the residential proxies, the fingerprints, the soft blocks and the retries, so an empty dataset always means an empty search. 😈

Yandex Maps Places Scraper: why two pages of 25 give you 49 places

Devil Scrapes — Wed, 14 Oct 2026 03:09:41 +0000

Quick answer

The Yandex Maps Places Scraper turns a Yandex Maps search such as cafe in Moscow into a deduplicated table of places: name, address, latitude and longitude, rating, rating and review counts, phone numbers, websites, categories and opening hours, plus the page and rank each place appeared at. Yandex Maps is the main local-business directory across Russia and the CIS, and it has no public places API. Pricing is pay-per-result, $0.20 per run plus $0.004 per place, $4.20 per 1,000 places.

Does Yandex Maps have a places API?

Not one that returns business listings for search. Yandex's geocoder and static-map APIs turn addresses into coordinates and render map tiles, but the "find me every pharmacy in this city with its phone number and rating" data lives only in the Maps web app. That app talks to its own internal endpoints, and every request it makes carries a signature computed from the exact query string. Change a parameter, a parameter's order or how a space is encoded, and the signature no longer matches. We rebuild that signing step for every page, so you just give us a query and a city.

Why do two pages of 25 give you 49 places, not 50?

This is the detail that decides whether your counts are honest. Yandex serves 25 places per page, but the pages overlap at the seam: the last place on page 1 can reappear as the first place on page 2. Some result slots are not businesses at all but curated "collection" cards. Count raw items and you over-report. Fail to dedupe and the same cafe lands in your CRM twice.

Yandex Maps search pages overlap at the seam, so two pages of 25 typically yield 49 unique places, not 50.

So we dedupe by Yandex's organization id within each search, skip the non-business items, record the page each place was first seen on and its position after dedupe, and fetch a couple of extra pages when needed so maxResults: 50 really returns 50 places when Yandex has them.

What else we normalise

Coordinates two ways. Yandex serves [lon, lat], which trips up everyone who expects latitude first. We keep coordinates exactly as served and also split out named lat and lon fields.
Opening hours two ways. workingTime is the human string ("Mon-Thu 12:00–02:00"). workingHours is the structured per-weekday version for code, with from and to hours and minutes.
Ratings rounded to two decimals, with ratingCount and reviewCount kept separate, because they are different numbers.

What the Actor gives you

One row per place, 20 fields: id, name, address, lat, lon, coordinates, rating, ratingCount, reviewCount, phones, urls, categories, workingTime, workingHours, sourceUrl, query, geoId, page, position and scrapedAt.

{
  "id": "243088808182",
  "name": "Alkobufet",
  "address": "Moscow, Sretenka Street, 36",
  "lat": 55.771647,
  "lon": 37.632628,
  "coordinates": [37.632628, 55.771647],
  "rating": 4.9,
  "ratingCount": 2757,
  "reviewCount": 1321,
  "phones": ["+79152776884"],
  "urls": ["https://taplink.cc/alkobufet.shashlik"],
  "categories": ["Cafe", "the glass room"],
  "workingTime": "Mon-Thu 12:00–02:00; Fri,Sat 12:00–05:00; Sun 12:00–02:00",
  "sourceUrl": "https://yandex.com/maps/org/alkobufet_shashlychnaya/243088808182",
  "query": "cafe",
  "geoId": 213,
  "page": 1,
  "position": 1,
  "scrapedAt": "2026-10-09T00:00:00Z"
}

Input: searches, each one an object ({"query": "cafe", "geoId": 213, "city": "moscow"}), a pasted Yandex Maps search URL, or the short text form cafe | 213 moscow. Up to 50 searches per run and maxResults up to 500 places per search.

Honest limitations 🚧

Review text, photos and per-place detail pages are not included: you get the rating and the counts. Moscow is built in; any other city needs its geoId and city slug, both visible in the Yandex Maps URL (or just paste the URL). English (US) is the supported locale for now, and place names come back as Yandex serves them. A search with no matches is a successful run that costs only the start fee.

FAQ

Do I need a Yandex account or API key?
No. The Actor reads the same public search the Maps web app uses.

What about blocks and captchas?
That's our job. We rotate real-browser TLS fingerprints (Firefox and Safari profiles) through curl-cffi, use a fresh Apify Proxy session for every search and on every block, pace requests at least three seconds apart, and retry with exponential backoff on 408, 429 and 503, up to 5 attempts per page.

How do I scrape a city other than Moscow?
Open the search on yandex.com/maps and paste the URL. It contains the geoId and city slug, and the Actor reads both from it.

How does pricing work?
$0.20 per run plus $0.004 per unique place: $4.20 per 1,000 places.

→ Yandex Maps Places Scraper on Apify

Built by Devil Scrapes. We handle the request signing, the fingerprints, the proxies and the retries, and we count unique places, not raw slots. 😈

ESPN Scores Scraper: a game that has not started still shows 0-0

Devil Scrapes — Mon, 12 Oct 2026 20:40:25 +0000

Quick answer

The ESPN Scores Scraper pulls ESPN scoreboards for seven leagues (English Premier League, La Liga, Serie A, NBA, NFL, MLB and NHL) into JSON, CSV or Excel: one row per game with teams, scores, game state, start time in UTC, the winner, venue, city, broadcasters and a link to the ESPN summary page. Pick leagues and a date range of up to 31 days and get every game in one run. ESPN has no official public scores API, so we read its public scoreboard data and return it as a tidy dataset. Pricing is pay-per-result, $0.20 per run plus $0.0015 per game, $1.70 per 1,000 games.

Does ESPN have a public scores API?

Not an official, documented one. ESPN's own pages load scoreboards from internal endpoints that are undocumented, shaped differently per sport, and free to change without notice. Every "ESPN API" guide you find is someone's reverse-engineered notes. We keep that layer working, normalise it into one row shape across all seven leagues, and absorb the changes so your pipeline doesn't have to.

Why does a game that hasn't started show 0-0?

Because ESPN's scoreboard reports a score of 0 for both sides before kickoff. Read home_score and away_score alone and every upcoming fixture looks like a goalless draw. Feed that into a model and you've invented results.

On ESPN's scoreboard, a 0-0 before kickoff and a 0-0 at full time look the same. Only the game state tells them apart.

So every row carries status_state: pre (not started), in (in progress) or post (finished), plus a completed flag and ESPN's own status_detail text. Filter on completed: true before you treat a score as a result. We also fill winner from the final score when ESPN leaves it out, and keep it null for draws and unfinished games.

What else we handle

Multi-day ranges overlap. A game near midnight UTC can appear on two days' scoreboards. We dedupe on event_id per league, so you get each game once.
One bad day doesn't sink the run. A league-day that keeps failing is logged and skipped, and the rest still land.
Odds are sparse. odds_details and over_under are filled only when ESPN's scoreboard carries a line, which is often not the case. Treat them as a bonus, not a betting feed: for many games, and for whole leagues, they come back null.

What the Actor gives you

One row per game, 21 fields: event_id, league, league_name, event_name, short_name, start_time, status_state, status_detail, completed, home_team, away_team, home_score, away_score, winner, venue, city, odds_details, over_under, broadcasts, url and scraped_at.

{
  "event_id": "401879268",
  "league": "soccer/eng.1",
  "league_name": "English Premier League",
  "event_name": "Leeds United at Arsenal",
  "short_name": "LEE @ ARS",
  "start_time": "2026-10-10T11:30:00Z",
  "status_state": "pre",
  "completed": false,
  "home_team": "Arsenal",
  "away_team": "Leeds United",
  "home_score": 0,
  "away_score": 0,
  "winner": null,
  "venue": "Emirates Stadium",
  "city": "London",
  "broadcasts": ["USA Net", "Universo"],
  "scraped_at": "2026-10-07T12:00:00Z"
}

Note the 0-0 on a pre game. Input: leagues as sport/league keys (soccer/eng.1, soccer/esp.1, soccer/ita.1, basketball/nba, football/nfl, baseball/mlb, hockey/nhl), startDate and endDate (empty means today, UTC) and maxEvents up to 10,000.

Honest limitations 🚧

Scoreboards only: no standings, rosters, box scores or play-by-play. Seven leagues in this version. Odds columns are frequently empty. Up to 31 days per run. Each run is a snapshot, so schedule it for live updates. A day with no games is a successful run that costs only the start fee.

FAQ

Is this an official ESPN API?
No. It is an independent scraper that reads ESPN's public scoreboard data.

What about blocks and rate limits?
That's our job. We retry with exponential backoff on 408, 429 and 5xx, honour Retry-After, and isolate failures per league-day so one hiccup never costs you the whole run. Apify Proxy is available if you want it.

Does it return live scores?
Yes, for games in progress at the moment of the run (status_state: "in"). Schedule it every few minutes for a live ticker.

How does pricing work?
$0.20 per run plus $0.0015 per game: $1.70 per 1,000 games.

→ ESPN Scores Scraper on Apify

Built by Devil Scrapes. We handle the undocumented endpoints, the retries and the game-state traps, so a 0-0 in your dataset means a 0-0 on the pitch. 😈

2GIS Places Scraper: page 6 of every search silently returns page 1

Devil Scrapes — Mon, 12 Oct 2026 20:39:45 +0000

Quick answer

The 2GIS Places Scraper turns 2GIS searches into a clean table of local businesses for any 2GIS city, including Moscow, Saint Petersburg, Novosibirsk, Almaty and Dubai: 2GIS id, name, categories, street address, building, district, latitude and longitude, rating, review count, opening hours per weekday, last-updated date and a direct 2GIS link. Switch on includeContacts and it also opens each place page to add business phones, website and e-mail. Give it lists of cities and queries and it runs every combination, deduplicated by place. Pricing is pay-per-result, $0.20 per run plus $0.003 per place, $3.20 per 1,000 places.

Does 2GIS have an API for this?

2GIS sells API access for maps, routing and its places catalogue under commercial keys and quotas. If you want a one-off lead list of every pharmacy in Almaty with its rating and opening hours, that means a sales conversation and an integration project before the first row. The same data is already printed on 2GIS's public search pages. We read those pages and hand you the rows.

Why does page 6 of a 2GIS search give you page 1 again?

This is the trap that inflates naive scrapers. 2GIS serves at most five result pages of 12 places for any single search: 60 places. Ask for page 6 and it doesn't return an error or an empty page. It silently serves page 1 again. A scraper that pages "until the results run out" never sees them run out. It loops, re-downloading the same 12 places, and if it counts rows instead of ids it reports a city far bigger than it is.

2GIS caps every search at 5 pages of 12 places, and page 6 silently returns page 1, so "page until empty" never ends.

We measured that limit and stop cleanly at page 5. We also dedupe on placeId across every page, query and city in the run, so a cafe that matches both кафе and кофейня is emitted and billed once. To cover more ground than 60 places, use several specific queries and more cities.

What else we handle

Contacts are opt-in. Phones, website and e-mail live on each place's own page, not in search results. With includeContacts: false (the default) those fields stay empty and the run is fast. Turn it on and we make one extra request per place to fill phone, phones, website and email, when the business publishes them.
Branches vs companies. placeId is the branch. organizationId is shared by every branch of a chain, so you can group a chain's locations.
Country sites. Kazakhstan and UAE cities live on their own 2GIS country domains. Give the city slug (almaty, dubai) and we route to the right one.
Paid placement is flagged. isPromoted tells you which rows are ads, so promotion doesn't skew your rankings.

What the Actor gives you

One row per place, 26 fields: placeId, name, organizationId, category, categories, address, addressComment, buildingName, postcode, city, district, latitude, longitude, rating, reviewCount, phone, phones, website, email, workingHours, updatedAt, isPromoted, url, query, searchCity and scrapedAt.

{
  "placeId": "70000001007179642",
  "name": "Му-Му, кафе",
  "organizationId": "4504136498368318",
  "category": "Кафе",
  "categories": ["Кафе", "Доставка еды", "Быстрое питание"],
  "address": "Манежная площадь, 1 ст2",
  "buildingName": "Охотный ряд",
  "city": "Москва",
  "district": "Тверской",
  "latitude": 55.755301,
  "longitude": 37.614586,
  "rating": 4.8,
  "reviewCount": 1107,
  "phone": "+74957818781",
  "website": "http://cafemumu.ru",
  "workingHours": { "Mon": "08:00-23:00", "Sun": "08:00-23:00" },
  "url": "https://2gis.ru/moscow/firm/70000001007179642",
  "query": "кафе",
  "searchCity": "moscow",
  "scrapedAt": "2026-10-01T10:00:00+00:00"
}

That sample was run with includeContacts: true. Input: cities (2GIS slugs), queries (in the city's language works best), maxItemsPerQuery up to 60, maxPages up to 5, a run-wide maxResults up to 50,000, and includeContacts.

Honest limitations 🚧

60 places per city and query, because that is all 2GIS serves. includeContacts makes runs slower, with one extra page per place. Rating, review count and opening hours appear only when 2GIS lists them. Search-result data only: no review text, menus or photos. A query with no matches is a successful run that costs only the start fee.

FAQ

Do I need a 2GIS account or API key?
No. Add cities and queries and start the run.

What about blocks and rate limits?
That's our job. We rotate real-browser TLS fingerprints (Chrome, Firefox, Safari) through curl-cffi, route through Apify Proxy with a fresh exit on every failed attempt, and retry with exponential backoff on 408, 429 and 5xx, up to 5 attempts per page. One failing page or place never kills the run.

Are the phone numbers personal data?
No. includeContacts reads only the business contacts a company publishes on its own 2GIS page.

How does pricing work?
$0.20 per run plus $0.003 per unique place: $3.20 per 1,000 places. Contacts enrichment costs no extra per row.

→ 2GIS Places Scraper on Apify

Built by Devil Scrapes. We handle the pager trap, the fingerprints, the proxies and the retries, and we count unique places, not repeated pages. 😈

Companies House UK Scraper: lead lists by town, SIC code and incorporation date

Devil Scrapes — Mon, 12 Oct 2026 19:11:59 +0000

Quick answer

The Companies House UK Scraper turns the UK company register into lead lists: give it a town, one or more SIC codes, a status and an incorporation date window, and it returns one row per company with number, name, status, incorporation date, company type, SIC codes and registered office address. Optionally it adds SIC descriptions and the officers list. It also does plain name search for KYC checks. No Companies House API key, no registration, no rate-limit maths. Pricing is pay-per-result, $0.20 per run plus $0.003 per company, $3.20 per 1,000 companies, and the details and officers enrichment costs nothing extra.

Isn't Companies House data already free?

The data is free. Getting it into a spreadsheet is not. The official REST API needs a registered developer account and an API key, and it is rate limited, so a lead list of a few thousand companies with officers turns into a small engineering project: auth, paging, throttling, retries and stitching three endpoints together per company. The bulk data product is a monthly snapshot of every company in the UK, which is the wrong shape when you want "active software companies in Leeds incorporated this year".

The register's own website already answers that question. Its advanced search filters by registered-office town, SIC code, status and incorporation dates in one go. We drive that search and return the rows.

Companies House's advanced search filters by town, SIC code, status and incorporation date at once, which is exactly the shape of a UK lead list.

What makes the register awkward to scrape well?

Status is free text. The site shows "Active", "Dissolved", "Liquidation", "Active - Proposal to Strike off" and more. We normalise to active, dissolved, liquidation or other in status and keep the original in statusDetail, so you can filter cleanly and still see the nuance.
Search matches previous names. Search "Acme" and you get companies that were once called Acme. We return those names in previousNames, so you can tell a match on today's name from a historical one.
Results come 20 per page and stop at 10,000. We page automatically and stop when results end or your maxResults is reached.
One company can surface in several queries. We dedupe on companyNumber across every query and page.
Officers are personal data. We return what the register publishes: name, role, appointment and resignation dates, nationality, country of residence and date of birth as month and year only (dobMonthYear).

What the Actor gives you

One row per company: companyNumber, name, status, statusDetail, incorporatedOn, dissolvedOn, companyType, sicCodes, sicDescriptions, address, previousNames, url, officers and scrapedAt. Each officer carries name, officerId, role, appointedOn, resignedOn, nationality, countryOfResidence, dobMonthYear and address.

{
  "companyNumber": "13106211",
  "name": "ISOROBOT LIMITED",
  "status": "active",
  "statusDetail": "Active",
  "incorporatedOn": "2020-12-31",
  "dissolvedOn": null,
  "companyType": "Private limited company",
  "sicCodes": ["62012", "62020", "70229"],
  "address": "20 Syke Lane, Scarcroft, Leeds, England LS14 3BQ",
  "previousNames": [],
  "url": "https://find-and-update.company-information.service.gov.uk/company/13106211",
  "officers": null,
  "scrapedAt": "2026-10-11T09:00:00Z"
}

Input: mode (search for company-name queries, advanced for filters), town, sicCodes (4 or 5 digits, for example 62020), status (active, dissolved, liquidation or any), incorporatedFrom and incorporatedTo, the includeDetails and includeOfficers switches, and maxResults up to 10,000.

Honest limitations 🚧

Public search and company-page data only: no accounts, filing history, charges or persons with significant control. Officers cover the first page of appointments (up to 35). The incorporation end-date filter is unit-tested but has not yet been confirmed against the live site. A filter that matches nothing is a successful run that costs only the start fee.

FAQ

Do I need a Companies House API key?
No. Give us names or filters and we do the rest.

What about blocks and rate limits?
That's our job. Every request carries a browser-grade TLS fingerprint through curl-cffi, retries with exponential backoff on 408, 429 and 5xx up to 5 attempts per page, honours Retry-After, and rotates the Apify Proxy session on every block.

Do details and officers cost more?
No. Each adds one request per company on our side and no extra charge on yours.

How does pricing work?
$0.20 per run plus $0.003 per company: $3.20 per 1,000 companies.

→ Companies House UK Scraper on Apify

Built by Devil Scrapes. We handle the paging, the fingerprints, the proxies and the retries, so your lead list stays clean. 😈

ALDI Australia Products Scraper: every row says notForSale, and that is correct

Devil Scrapes — Mon, 12 Oct 2026 19:11:21 +0000

Quick answer

The ALDI Australia Products Scraper pulls ALDI Australia's product catalogue by search term into JSON, CSV or Excel: SKU, name, brand, pack size, unit price in integer cents, the per-100 g or per-unit comparison price, categories and a working image URL. Give it milk, bread or a brand name and it pages through every match, deduplicated by SKU. One thing surprises everyone the first time: every row says notForSale: true, and that is correct. ALDI Australia sells almost nothing online. This is a price and assortment feed, not a shopping cart. Pricing is pay-per-result, $0.20 per run plus $0.003 per product, $3.20 per 1,000 products.

Why is every ALDI product "not for sale"?

Because ALDI Australia is a walk-in retailer. The catalogue on aldi.com.au shows prices and pack sizes, but for the vast majority of items there is no checkout behind it, and ALDI's own product data says so with a notForSale flag. We pass that flag through exactly as ALDI reports it. We never filter it, drop it or rewrite it.

That matters more than it looks. A scraper that silently removed "not for sale" items would return an almost empty dataset. One that dropped the field would let you build a storefront that implies people can buy things they can't. Keeping it verbatim keeps your price-tracking honest.

ALDI Australia's catalogue is a price reference, not a store: nearly every product is flagged not-for-sale online, and the flag is the truth, not a bug.

What else is awkward about ALDI's catalogue data?

Image links arrive as templates. The asset URL ALDI ships contains literal {width} and {slug} placeholders. Use it as-is and every image 404s. We fill both in, so imageUrl opens a real picture.
Prices are in cents. priceAmountCents: 399 is $3.99. We keep the integer for maths and ALDI's own priceDisplay string for humans, with no float rounding drift.
The comparison price is optional. "$2.38 per 100 g" exists only where ALDI publishes one. We return null rather than computing a figure ALDI never stated.
Search terms overlap. "milk" and "chocolate" both find milk chocolate. We dedupe on sku across every term and page, so one product is emitted and billed once.
Results page 30 at a time. We walk the offsets until ALDI's own total count is reached or your cap is hit.

What the Actor gives you

One row per product, 21 fields: sku, name, brandName, urlSlugText, notForSale, discontinued, quantityMin, quantityMax, quantityInterval, quantityDefault, quantityUnit, sellingSize, priceAmountCents, priceDisplay, comparisonPriceCents, comparisonPriceDisplay, currencyCode, categories, imageUrl, searchTerm and scrapedAt.

{
  "sku": "000000000538441002",
  "name": "Milk Mouse - Milk Chocolate 168g",
  "brandName": "CHOCEUR",
  "notForSale": true,
  "discontinued": false,
  "sellingSize": "168 g",
  "priceAmountCents": 399,
  "priceDisplay": "$3.99",
  "comparisonPriceCents": 238,
  "comparisonPriceDisplay": "$2.38 per 100 g",
  "currencyCode": "AUD",
  "categories": ["Pantry", "Confectionery"],
  "searchTerm": "milk",
  "scrapedAt": "2026-10-02T00:00:00+00:00"
}

Input: searchTerms (one or more), servicePoint (the ALDI store or region code, default G452 on the east coast), maxItemsPerQuery up to 3,000 and a run-wide maxResults up to 50,000.

Honest limitations 🚧

Catalogue prices and attributes only: no live stock, no specials-week history, no purchase path. One servicePoint per run, so regional price comparisons need one run per region. The catalogue is English-only, so plain product or brand words work best. A term with no matches is a successful run that costs only the start fee.

FAQ

Do I need an ALDI account or API key?
No. The Actor reads ALDI Australia's public product search.

What about blocks and rate limits?
That's our job. We rotate real-browser TLS fingerprints (Chrome, Firefox, Safari) through curl-cffi, route through Apify Proxy, and retry with exponential backoff on 408, 429 and 5xx, up to 5 attempts per page. One bad page or malformed product never kills the run.

Can I track ALDI prices over time?
Yes. Schedule the same search terms daily and diff priceAmountCents per sku.

How does pricing work?
$0.20 per run plus $0.003 per product row: $3.20 per 1,000 products.

→ ALDI Australia Products Scraper on Apify

Built by Devil Scrapes. We handle the fingerprints, the proxies, the retries and the template URLs. We do the dirty work so your dataset stays clean. 😈

Flashscore Live Scores Scraper: the scores feed is not JSON, it is and

Devil Scrapes — Mon, 12 Oct 2026 19:10:43 +0000

Quick answer

The Flashscore Live Scores Scraper turns Flashscore's fixtures, live scores and final results for football, tennis, basketball and hockey into clean JSON, CSV or Excel rows: teams, scores, status, kickoff time in UTC, league, country and a link to the event page. Pick a sport and the days you want (from a week back to a week ahead), optionally narrow by league, and run it. Flashscore has no bulk export and no public API, and the data behind the page is not JSON at all. It is a proprietary delimited feed, and we parse it for you. Pricing is pay-per-result, $0.20 per run plus $0.002 per event, $2.20 per 1,000 events.

Does Flashscore have an API?

Not a public one. The scores you see on flashscore.com are painted by the site's own JavaScript from a feed host it calls in the background. That feed only answers requests that carry a header value the site's front-end bundle supplies, and the value can change when the bundle does. So "just call the API" means reverse-engineering a moving target and keeping up with it. That part is our job: when the feed host starts rejecting requests, the Actor reports it as a rotated header rather than as a vague network error, and we ship the update.

What does the feed actually look like?

Not JSON. One day of football is a single text body where records are split by ¬~, fields by ¬, and keys from values by ÷. A trimmed slice of a real feed:

¬~ZA÷AFRICA: Olympic Games Women - Qualification¬ZY÷Africa¬ZL÷/football/world/olympic-games-women/
¬~AA÷GvGtelQO¬AD÷1791385200¬AB÷3¬AE÷Guinea W¬AF÷Cameroon W¬WM÷GUI¬WN÷CAM

The keys are two- and three-letter codes. AA is the event id, AD the kickoff as a Unix epoch, AB the status (1 scheduled, 2 live, 3 finished), AE and AF the home and away sides, and AG and AH the scores once a match has started.

Flashscore's live scores travel as a delimited text feed with two-letter keys, and the league is a header record, not a field on the match.

What goes wrong if you parse it naively?

Three things, each covered by a test against real captured feeds.

The league is a header, not a column. A ZA record names the league, then every match below it belongs to that league until the next ZA. Split the body into records and treat each one alone, and every match loses its league. We carry the header forward and stamp league, country, league_url and tournament_id onto each event.
Keys repeat inside a record. The first value is the right one. Last-write-wins quietly swaps fields.
A finished match can carry no score keys. The Guinea vs Cameroon event above is status 3 with no AG or AH. We return null rather than inventing 0-0, because a fake score is worse than a missing one in any model you feed.

Across several days the same event can appear in more than one feed, so we dedupe on event_id and you never pay twice for one match.

What the Actor gives you

One row per event, 19 fields: event_id, sport, day_offset, league, country, league_url, tournament_id, home_team, away_team, home_team_short, away_team_short, home_score, away_score, status, stage_code, kickoff_utc, kickoff_epoch, event_url and scraped_at.

{
  "event_id": "GvGtelQO",
  "sport": "football",
  "day_offset": 0,
  "league": "AFRICA: Olympic Games Women - Qualification",
  "home_team": "Guinea W",
  "away_team": "Cameroon W",
  "home_score": null,
  "away_score": null,
  "status": "finished",
  "kickoff_utc": "2026-10-05T13:00:00+00:00",
  "event_url": "https://www.flashscore.com/match/GvGtelQO/"
}

Input: sport (1 football, 2 tennis, 3 basketball, 4 hockey), dayOffsets (for example [0, 1] for today and tomorrow, -7 to 7, up to 15 days), an optional leagueFilter substring such as premier league, and maxEvents up to 5,000.

Honest limitations 🚧

One sport per run. Fixtures, scores and status only: no odds, lineups or match statistics. Each run is a snapshot, not a stream, so schedule it to follow live matches. Tennis days are large, so fewer days means a cheaper run. A league filter that matches nothing is a successful run that costs only the start fee.

FAQ

Do I need a Flashscore account?
No. The Actor reads the same public feed the website uses.

What about blocks and rate limits?
That's our job. Every request carries a real-browser TLS fingerprint through curl-cffi, goes through Apify Proxy with a fresh session on every retry, and backs off exponentially on rate limits and server errors, honouring Retry-After. One day that still fails is logged and skipped. The run fails only if every requested day fails.

Can I track live scores?
Yes, with Apify Schedules: run it every few minutes on dayOffsets: [0] and react to status and score changes.

How does pricing work?
$0.20 per run plus $0.002 per event row: $2.20 per 1,000 events. No data, no charge beyond the start fee.

→ Flashscore Live Scores Scraper on Apify

Built by Devil Scrapes. We handle the feed format, the fingerprints, the proxies and the retries, so your scores table stays clean. 😈

Meetup Events Scraper: the search page ships its whole Apollo cache, and that is the API

Devil Scrapes — Thu, 08 Oct 2026 16:01:17 +0000

Quick answer

The Meetup Events Scraper pulls public Meetup.com events by keyword and city into JSON, CSV or Excel: title, start time with timezone, in-person or online, venue with coordinates, the aggregate "going" count, capacity, fee, and the hosting group with its Meetup URL. No login and no API key. Meetup's own API is positioned for Meetup Pro customers, so we read the data the public search page already embeds instead: a normalized Apollo GraphQL cache inside the page's __NEXT_DATA__ script tag. Pricing is pay-per-result, $0.20 per run plus $0.004 per event, $4.20 per 1,000 events.

Does Meetup have a public API?

Not for this. Meetup's help centre describes its API as a way for Meetup Pro customers to build integrations and automate the networks they manage. For the question most people actually have, "which AI meetups are happening in New York next month and how many people are going", that means a paid organiser subscription and an OAuth flow to reach data that is already printed on a public web page. The search page at meetup.com/find/?keywords=ai&location=New%20York%2C%20NY renders without a session, and the server sends more than the HTML.

Where does the data actually come from?

Meetup's site is a Next.js app, and like every Next.js page it carries a <script id="__NEXT_DATA__"> tag with the props the page was rendered from. Under props.pageProps.__APOLLO_STATE__ sits the Apollo Client cache for that render: the GraphQL result of the search, flattened into entities keyed by type and id.

In our golden fixture that cache has 11 keys: one ROOT_QUERY, five Event:* entities and five Group:* entities. The search result itself lives under a ROOT_QUERY key that embeds the filter in its name, something like eventSearch:{"filter":{"city":"New York","country":"us",...}}, and each edge's node is not an event but a pointer: {"__ref": "Event:315229748"}. The event entity points at its group the same way. Venue and RSVP data are nested inline.

Meetup's public search page embeds the GraphQL response it was rendered from, as a normalized Apollo cache, in a script tag anyone can read.

So the parser is a small dereferencer, not a DOM walker:

def _make_deref(apollo: dict) -> Callable[[Any], dict]:
    def deref(value):
        if isinstance(value, dict) and "__ref" in value:
            return apollo.get(value["__ref"]) or {}
        return value if isinstance(value, dict) else {}
    return deref

event = deref(edge["node"])          # Event:315229748
group = deref(event.get("group"))    # Group:29431902
venue = deref(event.get("venue"))    # inline dict, passes through

What goes wrong if you treat the cache as a flat list?

Four things, each of which we hit once and now test for.

The search key is not stable. It contains the serialized filter, so a hardcoded key breaks the moment the location changes. We match on the eventSearch prefix and take the first hit.
Nodes are pointers. Read edge["node"]["title"] and you get a KeyError, because the node is {"__ref": ...}. Every __ref has to be resolved against the cache, including the group inside the event.
"Free" is the absence of a field. There is no isFree boolean. A free event has feeSettings: null; a paid one carries {amount, currency}. We derive is_free from that and emit fee_amount and fee_currency only when a fee exists.
Online events still have a venue. It is a placeholder with blank strings. We collapse blanks to null, so an online event reads venue_name: null rather than "".

The RSVP count comes from rsvps.totalCount, an aggregate. The Actor emits that number and nothing about who is going. Member identities are never requested, which keeps the dataset GDPR-safe by construction rather than by a filter you have to trust.

What the Actor gives you

One row per public event, 23 fields: search_keyword, event_id, title, event_url, date_time (ISO-8601 with offset, e.g. 2026-07-21T08:30:00-04:00), event_type and is_online, going_count, max_tickets, is_free, fee_amount, fee_currency, venue_name, venue_address, venue_city, venue_state, venue_country, latitude, longitude, group_name, group_url, description and scraped_at. Pass several keywords in one run, optionally a location, and set eventType to any, physical or online.

Honest limitations 🚧

One search page is one request and yields roughly 27 events per keyword, in Meetup's own relevance order. We do not paginate past it, so breadth comes from more keywords, not deeper pages. maxEventsPerKeyword caps at 500. A keyword with no matches is a successful run and costs only the start fee. Descriptions are Meetup's markdown, kept verbatim; switch includeDescription off for a leaner row.

FAQ

Do I need a Meetup account or API key?
No. The Actor reads public search pages only.

What about blocks and rate limits?
That is our job. Every request replays a real browser TLS handshake through curl-cffi, rotating between Chrome, Firefox and Safari profiles, runs through Apify Proxy, and retries with exponential backoff on 403, 408, 429 and 5xx responses, up to 5 attempts per page with the delay capped at 30 seconds.

Does it collect attendee data?
No. Only the aggregate going count. No names, no profiles, no member IDs.

Can I get a specific group's events rather than a keyword search?
Not in this version. Use the group's name as a keyword and filter on group_url in the output.

How does pricing work?
$0.20 per run plus $0.004 per event row: $4.20 per 1,000 events. No data, no charge beyond the start fee.

→ Meetup Events Scraper on Apify

Built by Devil Scrapes. We handle the fingerprints, the proxies and the retries, and we test the parser against a real search payload before a customer ever runs it. We do the dirty work so your dataset stays clean. 😈

Bayt Jobs Scraper: the proxy country pin that was the block

Devil Scrapes — Sat, 26 Sep 2026 10:06:59 +0000

Quick answer

We pinned our Bayt.com scraper's proxy to the country it scrapes, and that pin was the block. On Apify residential exits in the UAE, cloud QA landed 0 rows in 15 attempts. We changed one field to a UK residential exit and the same build returned 20 rows, then a full 60-row run with no blocked pages. The data stayed genuinely Emirati because Bayt decides the country from the URL path (/en/uae/jobs/...), not from where your IP sits. The Bayt Jobs Scraper now defaults to RESIDENTIAL / GB, and you pick the market with the country input.

Shouldn't a proxy match the country you're scraping?

Usually, yes. A geo-random exit can quietly give you plausible but wrong data: prices in another currency, or another region's catalogue. Pinning the exit is normally the safe default, and it's our fleet rule for exactly that reason. The rule assumes the site uses your IP to pick the country. Bayt doesn't. It scopes every search by a path segment, so /en/uae/jobs/accountant-jobs/ returns UAE jobs whatever exit the request comes from. That leaves the pin with a cost and no benefit. The UAE residential pool was the one Bayt's Cloudflare layer challenged hardest, so pinning to it made us look more suspicious.

How do you tell "the pin is the block" apart from "the site blocks us"?

Change one variable. Same build, same input, same residential tier, and only the exit country different: AE went 0 for 15, GB cleared on the first attempt and kept clearing over three spaced runs. Swapping tiers or TLS profiles at the same time would have hidden which change did the work. Before you shelve a target as unreachable, try a neutral exit.

If the site picks its country from the URL, the exit country is just one more fingerprint, and the local one can be the most-watched.

What the Actor gives you

One row per job: job_id, title, company, company_url, location, career_level, salary (only on postings that publish it), a remote flag, summary, posted_at_raw (e.g. "3 days ago"), the absolute detail_url, and the keyword that found it. Pass several keywords in one run, and set country to uae, saudi-arabia, egypt or any other Bayt market path.

Honest limitations 🚧

List-page fields only, with no follow-through to detail pages. The posted date is Bayt's own relative text, kept verbatim rather than guessed into a timestamp. Salary appears on a minority of postings, and we return null rather than invent it. A search with no matches is a successful run.

FAQ

Do I need a Bayt account?
No. It reads Bayt's public search result pages.

Why a UK proxy for UAE jobs?
Because Bayt picks the market from the URL, and UK residential exits cleared where UAE ones were challenged. The rows are still the UAE (or whichever country you set).

What about blocks and captchas?
That's our job. We rotate browser fingerprints, retry with backoff on rate limits and challenges, and rotate residential sessions on every block.

$0.20 per run plus $0.002 per job row: $2.20 per 1,000 results. A zero-match search succeeds and costs only the start fee.

→ Bayt Jobs Scraper on Apify

Built by Devil Scrapes. We handle the fingerprints, the proxies and the retries, and we test the proxy default we ship against the target before a customer ever runs it.