DEV Community: Devil Scrapes

UTC Offsets Are Not Integers — A Remote Job Feed With 5.75 In It

Devil Scrapes — Thu, 10 Sep 2026 04:03:12 +0000

Quick answer

UTC offsets are not integers. If you model a remote job's timezone requirement as list[int], you will crash or silently truncate on every role that accepts candidates in India (+5:30), Newfoundland (−3:30), Nepal (+5:45), Adelaide (+9:30) or the Marquesas (−9:30). The Himalayas remote-jobs API returns these as JSON numbers in a timezoneRestrictions array, and a live pull right now shows 37 distinct offsets, six of them fractional: -9.5, -3.5, 3.5, 4.5, 5.5, 5.75. The correct type is a float — and 5.75 is the one that catches people who "fixed" it by switching to half-hour steps.

Why does a remote job board even have fractional timezones? 🕐

Because remote hiring is expressed as an overlap window, not a location. A job that says "must overlap 4 hours with UTC+5:30" is describing India, and India has never been on a whole-hour offset. Nepal is stranger still at UTC+5:45, which is why 5.75 shows up in real data.

Here is what the field actually looks like on a live pull:

distinct offsets in one page : 37
fractional offsets present   : -9.5, -3.5, 3.5, 4.5, 5.5, 5.75

Three of those are not half-hours away from anything convenient. 5.75 cannot be represented as an integer, cannot be represented as "hours plus 30 minutes", and will not round-trip through a naive int() cast — Python will happily give you 5, which is Pakistan, not Nepal. That is a silent 45-minute error in a field whose entire purpose is scheduling overlap.

So the row contract we ship is explicit about it:

timezone_restrictions: list[float]   # NOT list[int]

That single annotation is the whole lesson. We shipped this Actor with list[int] in the first draft of the spec and the live data corrected us before any customer saw it.

What does this break in practice?

Three things, in rising order of how long they take to notice:

A hard crash, if your parser is strict. int("5.75") raises; a Pydantic list[int] rejects the row. This is the good outcome — it fails loudly on day one.
A silent truncation, if your parser is permissive. You store 5 for a job that means 5.75, and every downstream "can this candidate overlap?" calculation is wrong by 45 minutes for that row only.
A dropped row, if your fault handling is coarse. This is the expensive one: a single un-parseable record takes down the whole batch, and the 249 good jobs on that page go with it.

That third failure mode is the one worth designing against, and it is not specific to timezones. Any job feed will eventually hand you one record shaped differently from the other 249 — a missing salary, a null company slug, an employment type nobody has seen before. If one bad record can end a run, then the reliability of your scrape is set by the worst row in the feed rather than the typical one.

How we handle it 🛡️

The Himalayas Remote Jobs Scraper isolates faults per item. A record that fails to parse is skipped and logged; the run continues and delivers everything else. You are never in the position of paying for a run that returned nothing because one job posting had an unusual field.

The other thing worth knowing about this API is that it paginates by cursor, not by page number. The response body says so itself:

Cursor pagination is now available and is the preferred way to page through the feed. Pass the nextCursor value from each response back as ?cursor=. It is faster than offset.

Cursor chains are also the safer choice on a feed that changes under you — new jobs are posted while you are walking it, and offset pagination on a shifting list is how you get duplicate and skipped rows. On a per-row-billed scrape, duplicates are not a cosmetic problem: they are rows a customer pays for twice.

We also do not trust a job feed's own filter parameters without checking them. Some of the documented filters on this API do not narrow the result set the way you would expect, so the Actor fetches and filters client-side. That is slightly more work per run and it means the filter you asked for is the filter you get, rather than the filter the upstream felt like applying today.

Output

One row per job posting, as JSON, CSV or Excel:

id, title, company_name, company_slug, company_logo_url, url,
excerpt, description_html, employment_type, seniority[],
categories[], parent_categories[], location_restrictions[],
timezone_restrictions[]  ← float, and now you know why
min_salary, max_salary, salary_period, currency,
published_at, expires_at, scraped_at

Pricing is $0.20 to start a run plus $0.0015 per job row — $1.70 for a thousand postings. Zero rows written means you pay only the start fee.

Built by Devil Scrapes. We publish the wire-format quirks we hit, because the failure that costs you a night is rarely the one in the docs.

Airbnb's search payload doesn't carry amenities. Here's what we do instead of guessing.

Devil Scrapes — Wed, 09 Sep 2026 21:33:17 +0000

Quick answer

Airbnb's homes-search page embeds a JSON payload with everything the front end needs to render a results grid — price, rating, host, superhost badge, bedroom count — but not everything a comps dataset wants. Amenities are usually absent from that payload entirely; they live on the listing's own detail page. Building an Airbnb scraper around that reality means returning null for a field the search payload didn't carry, instead of guessing, and pinning the residential exit IP to the market's own country so a $180 nightly rate doesn't quietly become a $180 rate in the wrong currency.

What's actually in Airbnb's search payload

Airbnb doesn't publish a documented API for this data, so the only source of truth is the page itself, as of September 2026. Each search-results page for a given place and date range ships a large embedded JSON blob that server-renders the grid — no separate XHR call needed to get the base fields. Per listing, that blob reliably carries: listing ID, title, room/listing type, coordinates, price and currency as displayed, rating, review count, host name and ID, superhost status, bedroom/bed/bathroom counts, max guest capacity, and a set of photo URLs.

What it does not reliably carry: amenities. The search grid doesn't need to render a full amenity list to sell you on a card, so Airbnb doesn't ship one in the search payload. That data lives on the listing's own /rooms/<id> page, in a different section of embedded JSON entirely, structured around amenity categories rather than a flat list.

Why we return `null` instead of inventing a field

The tempting shortcut is to infer amenities from the listing title or type — "Entire home" listings statistically have a kitchen more often than a private room does — and ship a best-guess field. We don't do that. A null amenities field tells the caller exactly what happened: the search payload didn't carry it, and enrichment wasn't requested. A guessed value that's wrong 15% of the time is worse than an honest gap, because nothing downstream flags it as uncertain.

So the Actor ships enableDetailEnrichment as an explicit opt-in. Off, amenities is always an empty list and the run only touches the search-results pages — one request per page of results. On, the Actor makes a second pass and fetches each emitted listing's own detail page for its amenity section, which roughly doubles the total request volume for that run. The customer decides whether that trade-off is worth it for their use case; we don't decide it for them by silently doing the expensive thing on every run, and we don't decide it by silently doing the cheap-but-wrong thing either.

# actors/airbnb-scraper/src/models.py (shape, not full source)
class ResultRow(BaseModel):
    price_per_night: float | None = None
    currency: str | None = None
    amenities: list[str] = Field(default_factory=list)
    # amenities stays [] unless enrichment ran — never inferred

The same discipline applies to total_price: it only populates when both checkIn and checkOut are supplied, because that's the only case where Airbnb's own page computes a stay total. Without both dates, you get the base nightly rate and nothing else — not a multiplied guess.

Pinning the proxy country to the market's currency

Airbnb resolves the currency it displays partly from the requesting IP's apparent geography. A residential exit that lands in the wrong country doesn't error — it returns a normal-looking 200 with prices in that country's currency, and the response gives no indication anything drifted. A run built around "San Francisco, CA" that happened to route through a residential exit in the UK would come back with a fully-formed dataset of listings priced in GBP, every field populated, every row internally consistent — and wrong for the customer's use case.

The fix is the same shape we use across the fleet for this failure class: never let the exit country be random when the target's response depends on it. proxyCountry is a required, non-randomized field wired straight into the proxyConfiguration — apifyProxyCountry is set explicitly per run, not left to whatever the residential pool happens to hand back. If a customer is comping a market in Spain, the proxy exits from Spain for every request in that run, and currency on every row reflects it. We don't attempt currency conversion on top of that — price_per_night and currency are passed through exactly as Airbnb's page reports them, because converting introduces an exchange-rate assumption that's stale the moment it's written.

Retrying without losing the run

Airbnb's search surface sits behind anti-bot defenses rather than a documented, exhaustible rate limit — a blocked request is a session to drop, not a quota to wait out. On 403/429/5xx, the Actor rotates to a fresh residential session and browser-fingerprint impersonation profile and retries with capped exponential backoff. A run blocked partway through surfaces as completed with a status message describing what landed, not a hard failure that discards results the customer already paid the per-event price for.

The part that generalizes

Any page whose response depends on signals it doesn't expose — currency from IP geography, amenities on a second page only, a total only when both dates are given — will hand back a plausible answer to the wrong question if you don't pin the input and check what's actually present. Pin what the response depends on, and leave a field null rather than inferring it when the source genuinely didn't say.

Pricing

$0.20 per run, $0.0029 per listing row — about $3.10 per 1,000 listings, amenities included when enableDetailEnrichment is on. A search that matches nothing costs only the start fee.

Run the Airbnb Listings & Pricing Scraper on Apify — free trial credit, no card required.

Built by Devil Scrapes. We pin what the target's response depends on and leave the rest honestly blank. 😈

How we keep 200 web scrapers alive

Devil Scrapes — Wed, 09 Sep 2026 19:09:00 +0000

Originally published on the Devil Scrapes blog.

Quick answer: Web scraper maintenance, not the first successful run, is what you're actually paying for. A scraper that returns clean rows today can return zero rows tomorrow because a target moved one field, tightened one rate limit, or started serving a different page to a different exit IP — and nothing in yesterday's green test tells you that happened. We run 202 public Actors on the Apify Store, and the daily discipline of catching that drift before a customer's run silently fails is most of what we do.

Shipping is the easy 10%

Every scraper we've ever built worked on day one. That's the least interesting fact about it. The target's HTML structure, its internal API shape, its rate-limit thresholds, and its bot-defense posture are all things the target's own team can change on a Tuesday without telling anyone — and often does. A scraper is a contract with a system you don't control, and the contract gets renegotiated constantly, silently, and without a changelog.

That's why we treat every Actor we publish as a standing commitment, not a finished artifact. Across our fleet, customers ran our Actors 16,077 times in the last 30 days, and 96.1% of those runs succeeded. That number isn't a launch-day snapshot — it's a rolling average we watch every day, because it's the only honest measure of whether a scraper is actually alive.

A "SUCCEEDED" run can still deliver nothing

The first lesson in web scraper maintenance is that the platform's own status field lies to you if you let it. A run can finish with a green SUCCEEDED status and zero rows in the dataset — the process didn't crash, it just quietly found nothing. We learned to stop trusting "it finished" as proof of "it worked," and instead check what actually landed in the dataset against what a healthy run for that target normally delivers.

The second lesson: a scraper's own recent runs aren't the fleet's health signal — the target's 30-day public run stats are, because a handful of our own QA runs passing tells you nothing about the last thousand customer runs. We watch the rolling window, not the latest tick.

Real breakages, not hypothetical ones

Three examples from our own fleet, because concrete beats abstract:

One of our job-board Actors kept every retry inside its own bounded loop, but nothing capped the sum of all the retries across a full career-site crawl. On a large site, the retries stacked up past the platform's own run timeout, and the run died with nothing written — even though every individual request handler was working exactly as designed.
A live-chat archiver opened a long-lived connection and read from it in a loop with no upper bound. Against a quiet channel with a stalled connection, that loop could hang indefinitely instead of timing out and flushing whatever it had already captured.
On a geo-sensitive retail target, a residential exit in the wrong country didn't get blocked — it got a normal-looking 200 OK with prices in the wrong currency. No error, no crash, just quietly wrong data next to correctly-labeled rows.

None of these are exotic. They're the ordinary failure modes of long-running network code, and every one of them looks identical to "everything is fine" until you go looking for it.

The loop: monitor, triage, fix, verify in the cloud

Our web scraper maintenance loop has four steps, and skipping the last one is how bugs come back:

Monitor every public Actor's rolling 30-day success rate and dataset output, not just the latest run.
Triage the failure to a specific cause — a fault that should have been isolated to one item instead of killing the whole run, a retry budget that needed a hard ceiling, or an exit IP that needs to be pinned to the right country.
Fix the root cause: catch and skip per-item failures instead of letting one bad page abort a thousand good ones, cap the run's total wall-clock budget, add a guard that treats an unexpected currency or a country-selector page as a block rather than a result.
Verify in the cloud, against a customer-sized input, before calling it done. A fix that only ran against a five-row test fixture hasn't been tested against the failure it was written for — the bug that took down a real crawl needed a real crawl's worth of pages to reproduce.

We also learned the hard way that "committed" isn't "deployed": a fix can sit correct and tested on our side while the live Actor is still running the build from before the fix. So the loop always ends by confirming the deployed version, not just the patched source, is what customers are actually running.

Price the maintenance, not the first run

This is why our pricing is pay-per-event instead of a flat license: a $0.20 actor-start fee, then a per-result price (typically $1–$5 per 1,000 rows depending on the target). If a run returns nothing, you pay the $0.20 warm-up and nothing else — you're never billed for a scraper that had a bad day. But the flip side matters too: the per-result price isn't just covering the compute for that one run. It's covering the fact that we're still watching this Actor next month, and the month after, catching the day the target changes something and fixing it before it becomes your outage.

Buyers who evaluate a scraper by running it once and checking the output are pricing the wrong thing. The right question isn't "did it work today" — it's "who's watching it tomorrow."

Two Actors from our fleet that illustrate the pattern directly: the Workday Jobs Scraper survived exactly the run-budget fix described above, and the Kick Chat Scraper & Real-Time Archive survived the unbounded-loop fix. Both are live, monitored, and priced the same way — pay only for rows that land.

Run it on Apify:

FAQ

What does "web scraper maintenance" actually mean?

It means someone is watching whether a published scraper keeps working after the target site changes — its HTML, its internal API shapes, its rate limits, or its bot defenses — and fixing it before those changes turn into failed customer runs. Without it, a scraper's success rate decays quietly over weeks.

How do you know an Actor is broken before a customer complains?

We watch each Actor's rolling 30-day public run stats — total runs, successes, failures, timeouts — rather than relying on our own last few test runs, which can look green while customer runs are failing.

Why can a run say "SUCCEEDED" but return no data?

Because the process finishing without crashing and the process actually finding data are two different things. A run can complete cleanly against a page that returned nothing useful — a geo-mismatched exit, an empty result set, a silently changed selector — and still report success unless something checks the output, not just the exit code.

Why do you price per result instead of a flat fee?

A flat fee prices the code as it exists on launch day. Pay-per-event pricing — a small start fee plus a per-result charge — means you only pay when data actually lands, and it keeps our incentives aligned with keeping the Actor working, since a broken Actor earns nothing.

Does this apply to every Actor you publish?

Yes. Every one of our 202 live Actors goes through the same monitor-triage-fix-verify loop; none of them are "launch and forget."

Devil Scrapes builds and maintains 200+ pay-per-result web scrapers on the Apify Store. Blocks, retries and proxies handled by us. Browse the full catalog or commission a custom Actor.

Actors mentioned in this post:

Why your scraper gets blocked at 3 a.m.

Devil Scrapes — Wed, 09 Sep 2026 19:08:28 +0000

Originally published on the Devil Scrapes blog.

Quick answer: Why does my scraper get blocked? Usually not because your parser is wrong — it's because the request itself doesn't look like it came from a browser. A plain HTTP client sends a different handshake than Chrome, reuses an IP with no browsing history behind it, drops session continuity between pages, and ignores the target's own rate-limit signals. Any one of those is enough to get flagged; most scrapers hit two or three at once.

It's rarely the parser

When a scraper "stops working," the instinct is to check the CSS selector or the JSON path first. That's usually not it. If a run comes back with a splash page, a stripped-down HTML shell, or a response that doesn't match what a browser would see, the request never made it past the target's front door as a trusted client. The parser was never given real data to parse. Here's what actually decides whether a request gets treated as a browser or a bot.

TLS and HTTP fingerprints

Every HTTP client — Python's requests, Node's fetch, curl — negotiates its TLS handshake and orders its HTTP headers in a way that's specific to that library, not to the browser it's pretending to be. Real Chrome, Firefox, and Safari each send a distinctive cipher-suite order, extension list, and header casing. A target that inspects the handshake (plenty do, quietly, without ever showing a challenge page) can tell "Python script" from "Chrome tab" before your request body is even read, independent of your User-Agent string.

We rotate through real Chrome, Firefox, and Safari handshake profiles per request, so the target's edge sees a browser-shaped connection rather than a library's default fingerprint.

IP reputation: residential vs. datacenter

The exit IP carries its own trust score, built from a history of prior traffic. A datacenter IP block that's been used by a thousand scrapers before yours starts with a reputation deficit, no matter how clean your request looks. A residential IP — an exit that looks like an ordinary home connection — usually starts closer to neutral. Neither is a silver bullet: datacenter exits are faster and cheaper, and they clear plenty of targets fine, so the right call is target-by-target, not a blanket "always residential" rule. We rotate through both residential and datacenter proxy pools and pick per target based on what actually clears.

Session and cookie continuity

Pagination on modern sites is usually bound to a session — a cursor, a cookie, a token issued on page one that page two expects back. Rotate your exit IP mid-session without carrying that session forward, and the target doesn't see "a new visitor," it sees a broken client and invalidates the cursor. We hold a sticky session — the same exit IP and cookie jar — for the length of a paginated crawl, and only rotate to a fresh session when a page actually needs one.

Rate limits: `Retry-After` is not a suggestion

Most targets that rate-limit tell you exactly how long to wait, in the Retry-After header on a 429 or 503 response. Ignoring it and retrying immediately is one of the fastest ways to escalate a soft rate limit into a hard IP ban. We back off exponentially on 408, 429, and 5xx responses and honor Retry-After when the target sends one, capped at a handful of attempts per page so a stuck request doesn't run forever.

The trap that doesn't look like a block at all

The failure mode that cost us the most debugging time wasn't a 403 — it was a 200 OK. On a geo-sensitive target, a residential exit in the wrong country doesn't get rejected. It gets served a normal-looking page: prices in the wrong currency, a country-selector splash screen instead of the product page, or a locale variant that silently doesn't match what was requested. There's no error, no challenge, nothing a naive status-code check would ever flag — just data that's confidently wrong sitting next to rows that are correct.

We pin the exit country on every geo-sensitive target instead of letting the proxy pick one at random, and we added a guard that treats an unexpected currency symbol or a country-selector marker as a failed fetch to retry — not a result to parse and ship.

What this looks like for you

Every one of the mechanics above is a maintenance burden if you own it yourself: fingerprint drift as browsers update, proxy pool churn, session bugs that only show up at scale, rate-limit tuning per target, and geo bugs that pass every test except the one that matters. We absorb all of it inside the Actor, so what lands in your dataset is rows, not a debugging session at 3 a.m.

Run it on Apify:

FAQ

Why does my scraper get blocked even with a correct User-Agent header?

Because the User-Agent string is just one header among many. The TLS handshake and the full HTTP header order and casing are usually what a target's edge actually inspects, and a plain HTTP client sends a library-specific fingerprint there regardless of what User-Agent you set on top of it.

Is a residential proxy always better than a datacenter proxy?

No — it depends on the target. Residential exits generally carry a better starting trust score, but datacenter exits are faster, cheaper, and clear plenty of targets without issue. The right choice is per-target, decided by what actually gets through, not a fixed rule.

Why did my scraper get a 200 response with wrong data instead of an error?

Geo-sensitive targets often don't reject a foreign exit outright — they serve a normal-looking response in the wrong locale: wrong currency, a country-selector page, or a mismatched variant. It looks like success unless you specifically check the content against what's expected for the country you asked for.

What does honoring `Retry-After` actually prevent?

It prevents a soft, temporary rate limit from escalating into a hard IP ban. Retrying immediately after a 429 tells the target your client isn't respecting its limits, which is exactly the pattern that gets an IP or session blacklisted rather than just throttled.

Can session continuity really break a scrape on its own?

Yes. If a target's pagination cursor is bound to a session cookie and you rotate the exit IP between pages without carrying the session forward, the target sees an inconsistent client and drops the cursor — the crawl stalls or restarts from page one, not because the parser broke, but because the session did.

Devil Scrapes builds and maintains 200+ pay-per-result web scrapers on the Apify Store. Blocks, retries and proxies handled by us. Browse the full catalog or commission a custom Actor.

Actors mentioned in this post:

Export a competitor's entire Google Ads library

Devil Scrapes — Wed, 09 Sep 2026 19:07:56 +0000

Originally published on the Devil Scrapes blog.

Quick answer: The Google Ads Transparency Center publicly lists every ad creative Google is currently serving for a given advertiser, across Search, YouTube, Display, Shopping, Maps, and Play — but Google ships no official API for it. Our Google Ads Transparency Scraper exports it by domain or advertiser ID to JSON or CSV, at $0.20 per run plus $3.00 per 1,000 ads landed.

A public registry with no export button

Google has quietly built one of the most useful competitive-intelligence datasets on the web and given it almost no interface. Search or paste a domain into the Transparency Center and you get a paginated UI showing every ad creative currently running for that advertiser — 40 at a time, one click at a time. For a brand the size of Nike, that's tens of thousands of creatives behind an interface built for spot-checking one ad, not exporting a library.

There's no bulk download, no CSV export, and no documented API. If you want the dataset rather than the UI, you need to talk to whatever the UI itself talks to — which is exactly what this Actor does. It replays the same internal request the browser makes, carrying a real browser's TLS handshake so the request isn't distinguishable from the page you'd load by hand, and turns the response into typed rows instead of rendered HTML.

What comes back per ad

Field	What it tells you
`advertiser_id` / `advertiser_name`	Who's running the ad
`creative_id` / `creative_url`	A stable ID and a deep link back into the Transparency Center
`landing_domain`	Where the click actually goes
`format_type`	Text, image, or video
`first_shown_ts` / `last_shown_ts`	How long the creative has been live
`impressions`	Google's own reported impression count
`preview_image_url` / `preview_content_js_url`	The creative asset itself, not just its metadata
`region`	The locale you tagged the request with
`scraped_at`	ISO-8601 timestamp of the pull

That last point matters more than it looks: plenty of scrapers in this space return who's advertising and where the click goes, but drop the creative asset entirely. Getting preview_image_url back means you can actually look at the ad, not just its metadata.

Who this is for

Agencies running competitive intel — pull a client's top three rivals weekly and diff the creative set to catch new launches before the client asks about them.
Brand protection teams — sweep your own brand domain and trademark terms to see who else is buying ads against your name, including resellers and affiliates you never authorized.
Affiliate-fraud and compliance teams — cross-reference landing_domain against advertiser_name; a mismatch is the fastest signal that an advertiser isn't who their name claims.

Running it

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")

run = client.actor("DevilScrapes/google-ads-transparency").call(
    run_input={
        "searchDomains": ["nike.com"],
        "maxResults": 500,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["advertiser_name"], item["creative_id"], item["landing_domain"])

You can pass multiple domains in one run, or drop in advertiser IDs directly if you already have them from a Transparency Center URL. Each ad lands as one dataset row, exportable as JSON, CSV, Excel, or JSONL from the run's Storage tab, or pulled straight over the API.

Pricing, exactly

Pay-per-event: a $0.20 actor-start fee once per run, then $3.00 per 1,000 ads written to the dataset. Nothing is charged for ads you don't get back.

Pull	Cost
500 ads (the snippet above)	$1.70
1,000 ads	$3.20
10,000 ads	$30.20
100,000 ads (a monthly sweep across a portfolio of brands)	$300.20

A weekly sweep of five competitors at roughly 200 ads each costs about $4 a month in per-result charges plus five runs' worth of start fees — a fraction of what a dedicated ad-intelligence subscription runs, for exactly the slice of that data you actually need.

What we're upfront about

Google's underlying endpoint ignores the region parameter server-side — we tested every plausible request shape to confirm it rather than shipping a region filter that quietly does nothing. We still let you tag exports by intended locale, but it's metadata, not a live filter. Video and rich creatives return a script bundle URL rather than a renderable video file, since actually playing the creative requires executing Google's own JS. And Google itself retains roughly 12 months of history, so a date range older than that clips to whatever Google still has.

If you need the full picture before committing to a run, the Actor's Store listing has the complete field reference and input options.

Run it on Apify: Google Ads Transparency Scraper — paste a domain, get every creative back as clean rows.

FAQ

Does Google have an official Ads Transparency API?

No. As of 2026, Google publishes no official API for the Transparency Center. This Actor replays the same internal request the browser UI makes and returns typed rows instead of rendered pages.

Is scraping the Google Ads Transparency Center legal?

The Transparency Center is a public registry Google operates under regulatory pressure in the EU and US. This Actor reads only what the public UI already exposes, at a measured pace, and doesn't collect personal data — only advertiser-level ad metadata.

Can I filter results by country or region?

Not effectively — Google's own endpoint ignores the region parameter, a limitation we verified directly rather than shipping a filter that silently does nothing. The region field is available for tagging your export, not for narrowing results.

How much does it cost to export 500 ads?

$1.70 total: the $0.20 per-run start fee plus $3.00 per 1,000 ads, prorated to $1.50 for 500 ads.

Can I automate a weekly competitor sweep?

Yes — attach the Actor to an Apify Console schedule and run it weekly. Google's Transparency Center updates at most daily, so a weekly cadence catches new creative launches without wasting runs on unchanged data.

Devil Scrapes builds and maintains 200+ pay-per-result web scrapers on the Apify Store. Blocks, retries and proxies handled by us. Browse the full catalog or commission a custom Actor.

Actors mentioned in this post:

google-ads-transparency · Run it on Apify

Pay-per-result scraping: what you actually pay

Devil Scrapes — Wed, 09 Sep 2026 19:07:24 +0000

Originally published on the Devil Scrapes blog.

Quick answer: Pay per result web scraping means you're billed for what actually lands in your dataset, not for the compute time a run consumed or a flat monthly fee regardless of usage. On Apify, that's Pay-Per-Event: a small actor-start fee charged once per run, then a per-result price — typically $1 to $5 per 1,000 rows — charged only for the rows that were actually written. A run that finds nothing costs only the start fee.

Two charges, and only two

Every Actor we publish bills exactly two kinds of event:

actor-start — a fixed fee, charged once when the run begins, regardless of how much data it finds. It covers the warm-up cost: spinning up the run, resolving a proxy, establishing the first session.
A per-result event — charged once for each row that's validated and written to the dataset. Nothing charged for rows that were requested but never landed — a blocked page, an empty search, a target with no matching data.

That second point is the one buyers underrate. If you run a scraper against a query that returns nothing — a product that's out of stock, a search term with zero hits, a page the target took down — you are not billed as if the run failed on your end. You pay the start fee and nothing else, because nothing else happened.

Three real price points

Pricing varies by target, because the engineering cost varies by target — a static RSS feed and a JavaScript-heavy conversation graph are not the same amount of work to keep working. Here's what that looks like in practice, using our own published prices:

Actor	Start fee	Per-result price	1,000 results	10,000 results
RSS Feed Scraper	$0.005	$1.00 / 1,000	$1.01	$10.01
Google Ads Transparency Scraper	$0.20	$3.00 / 1,000	$3.20	$30.20
Threads Reply Scraper	$0.20	$5.00 / 1,000	$5.20	$50.20

Three things worth noticing in that table. First, the start fee itself varies: a feed parser that runs on a schedule against a lightweight, static format costs almost nothing to warm up, while a target that requires session handling and fingerprint rotation carries a heavier fixed cost per run. Second, the per-result price tracks how much reliability engineering the target demands — a conversation-graph scrape that has to hold a session across a nested reply tree costs more per row than parsing a well-formed feed. Third, and most importantly: at zero results, every row in that table collapses to just the start fee. A misconfigured input or a target with nothing new to return costs cents, not dollars.

What an empty run actually costs

Take the RSS Feed Scraper on a daily schedule watching for new posts. Most days, nothing new has been published — zero new items, zero per-result charges, just the $0.005 start fee. Only on the days something actually publishes does the per-result charge kick in. Contrast that with a model that bills for compute time regardless of output: a scheduled run that finds nothing still burns the same runtime cost as one that finds fifty items, because the meter is measuring seconds, not results.

How to cap spend before it happens

Pay-per-event pricing doesn't remove the need to set a ceiling — it just moves the ceiling to a place you control directly:

Use the Actor's own result-limit input. Most of our Actors expose a maxResults (or equivalent) field — set it and the run stops pulling once it hits that number, capping the per-result charge at a known maximum before the run even starts.
Set Apify's account-level usage limits. Apify Console lets you cap total monthly spend across all Actors, which stops runs platform-wide once you hit it — a backstop independent of any single Actor's input.
Start small and scale the input, not the trust. Run a capped test — a few dozen results — to confirm the output shape and rate before pointing the same Actor at a 10,000-row pull.

How this compares to the alternatives

Two other pricing models exist on scraping platforms, and neither is wrong — they're just measuring something different:

Compute-unit pricing bills for the CPU and memory time a run consumes, independent of what it finds. It rewards efficient code and can be cheaper for very high-volume, low-per-item-cost targets, but it means a run that finds nothing still costs whatever compute it burned trying.
Flat subscription pricing bills a fixed monthly rate regardless of usage. It's predictable and can be cheaper at very high, steady volume, but it means you're paying the same in a month you barely use it as in a month you lean on it hard.

Pay-per-result sits between the two: you're not billed for idle compute, and you're not paying a flat rate for a month you didn't use — you're billed for what showed up in your dataset, which is usually the number a buyer actually cares about.

Run it on Apify:

FAQ

What happens if my scraper run finds zero results?

You pay only the actor-start fee — typically a few cents to $0.20 depending on the Actor — and nothing else. The per-result charge only applies to rows that were actually validated and written to the dataset.

Is pay-per-result cheaper than a flat monthly subscription?

It depends on your usage pattern. For sporadic or variable-volume use, pay-per-result usually costs less because you're never paying for a month you didn't use. For very high, steady, predictable volume, a flat rate can work out cheaper — the two models are optimized for different usage shapes.

Why do different Actors charge different per-result prices?

The price reflects the ongoing engineering cost of keeping that specific target working — session handling, proxy rotation, and reliability engineering vary a lot by target, and the price tracks that, not an arbitrary markup.

Can I set a hard spending cap before running a scraper?

Yes, two ways: most Actors expose a result-limit input (like maxResults) that caps the run's own output, and Apify Console lets you set an account-wide monthly usage limit as a platform-level backstop.

Does the actor-start fee apply every time I run an Actor?

Yes — it's charged once per run, regardless of how many results come back, because it covers the fixed cost of starting the run itself: spinning up the container, resolving a proxy, establishing the first session.

Devil Scrapes builds and maintains 200+ pay-per-result web scrapers on the Apify Store. Blocks, retries and proxies handled by us. Browse the full catalog or commission a custom Actor.

Actors mentioned in this post:

Run an Apify Actor From n8n, Make or Zapier

Devil Scrapes — Wed, 09 Sep 2026 19:06:52 +0000

Originally published on the Devil Scrapes blog.

If you want to run an Apify Actor from n8n, Make, or Zapier, the short version is: install the official connector, point it at the Actor slug, pass your input as JSON, and either wait for the run to finish inline or fire a webhook when it does. The whole setup takes about 10 minutes once you know which of those two patterns your workflow needs. Here's both, with a working example against a live Actor.

Why wire a scraper into a no-code tool at all

Most teams that reach for n8n, Make, or Zapier already have the destination sorted — a Slack channel, a Google Sheet, a CRM field, an AI agent's context window. What they're missing is the step that gets fresh, structured data into that destination on a schedule, without someone manually running a script. An Apify Actor is that step: point it at a target, get back Pydantic-validated rows, and let your automation platform do the routing.

We'll use Google Ads Transparency Scraper as the running example — it pulls a competitor's live ad creatives by domain, which is a natural fit for a recurring "check what our competitors are running this week" workflow. The pattern below applies to any of our 202 live Actors; only the input fields change.

Option A — n8n

n8n ships an official Apify node (community package, install it from the n8n node panel).

Add the Apify node to your workflow canvas and connect it with your Apify API token (create one under Apify Console → Settings → Integrations).
Pick the operation. For a short run, use Run Actor synchronously and get dataset items — this maps to Apify's run-sync-get-dataset-items endpoint and returns the rows directly into the next node, no polling required.
Set the Actor to DevilScrapes/google-ads-transparency and pass the input JSON:

{
  "advertiserDomain": "competitor.com",
  "maxAds": 100
}

Chain a downstream node — a Sheets append, a Slack message, or an AI-agent node that summarizes new creatives.
Add a Schedule Trigger upstream to run it daily or weekly.

Option B — Make

Make (formerly Integromat) has an Apify app with two modules that matter here: Run Actor and Get Dataset Items.

Add the Apify → Run Actor module, authenticate with your API token, and select the Actor.
Fill the input fields — Make renders the Actor's input schema as a form automatically, so you rarely need to hand-write JSON here.
For a fast Actor, tick wait for finish and the module blocks until the run completes, then hands the run ID to the next module.
Add Apify → Get Dataset Items, feed it the dataset ID from step 3, and route the rows into your scenario — a filter, a router, a CRM update.

Option C — Zapier

Zapier's Apify integration exposes Run Actor as an action and New Dataset Item as a trigger.

Create a Zap with your usual trigger (a form submission, a new row, a schedule).
Add the Apify: Run Actor action, select the Actor and fill the input fields from the trigger's data.
If you need the results in the same Zap, add a second Apify: Get Dataset Items action after a short delay step, since Zapier's action model doesn't block on long-running Actor runs the way n8n's synchronous option does.

run-sync-get-dataset-items vs async + webhook

This is the one decision that actually matters, and it comes down to how long the Actor takes to finish.

Use run-sync-get-dataset-items (synchronous) when:

The run typically finishes in well under the platform's timeout window (n8n, Make, and Zapier all cap how long a single step can block — usually somewhere between 30 seconds and a few minutes depending on your plan).
You're pulling a small-to-medium batch — a handful of URLs, one domain, one channel.
You want the simplest possible workflow: one node in, rows out, no state to track between steps.

curl "https://api.apify.com/v2/acts/DevilScrapes~google-ads-transparency/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"advertiserDomain": "competitor.com", "maxAds": 100}'

Use async + webhook when:

The run can take minutes to hours — a channel back-catalog pull, a multi-thousand-row batch, or anything where we're deliberately pacing requests against a target's rate limits rather than hammering it and getting the whole run blocked.
You don't want your automation platform's execution sitting idle (and possibly billed) while a run completes.
You want retry/notification logic independent of the trigger — e.g. post to Slack only on SUCCEEDED, alert on FAILED.

The async pattern: call the regular run-actor endpoint (no -sync), attach a webhooks parameter pointing at your automation platform's webhook-catching URL, and let Apify call you back when the run finishes:

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")
run = client.actor("DevilScrapes/google-ads-transparency").call(
    run_input={"advertiserDomain": "competitor.com", "maxAds": 500},
    webhooks=[{
        "eventTypes": ["ACTOR.RUN.SUCCEEDED"],
        "requestUrl": "https://hook.eu1.make.com/your-webhook-id",
    }],
)

n8n and Make both expose a Webhook trigger node — drop that URL in, and the rest of your workflow only runs once the Actor has actually finished, with the run ID and dataset ID available in the payload for a follow-up Get Dataset Items call.

What this costs

Pay-Per-Event pricing applies the same whether you trigger a run from the Apify Console or from n8n — there's no automation surcharge. Google Ads Transparency Scraper charges a $0.20 actor-start fee plus $3.00 per 1,000 ad rows returned. Pulling 100 ads through the n8n flow above costs $0.20 + $0.30 = $0.50; a 1,000-ad batch runs $3.20. If the domain has no ads live, you're charged only the $0.20 start fee — never for rows that never landed.

Run it on Apify

Grab an Apify account (free trial credit, no card required) and connect it to your automation tool of choice:

Google Ads Transparency Scraper on Apify — the Actor used in this walkthrough.
Browse the rest of the catalog for an Actor that fits your workflow — every one of our 202 live Actors works with the same n8n / Make / Zapier setup described above, just with different input fields.

FAQ

Do I need a paid n8n, Make, or Zapier plan to run an Apify Actor?

No. All three platforms' free tiers support HTTP-based integrations and the official Apify connectors. What limits you on a free tier is usually execution time or monthly task/operation count, not the ability to call Apify at all.

Which is faster to set up, the node/app or a raw HTTP request?

The official connectors (n8n's Apify node, Make's Apify app) are faster for most people because they render the Actor's input schema as a form. A raw HTTP Request node calling the REST API directly works identically and is sometimes preferable if you're already comfortable with JSON and want fewer moving parts.

What happens if the Actor run fails mid-workflow?

Apify marks the run FAILED and, if you're on the async + webhook pattern, still fires the webhook (filter on ACTOR.RUN.FAILED to catch it). We fail loud on genuine blocks rather than handing back a silent empty dataset, so a FAILED status means something upstream actually broke — check the run log for the reason.

Can I pass a schedule from n8n/Make instead of using Apify's own Scheduler?

Yes, and it's a common pattern — use your automation platform's schedule trigger to control cadence and business logic (e.g. skip weekends, only run for active accounts) while Apify just executes the single run you ask for.

Devil Scrapes builds and maintains 200+ pay-per-result web scrapers on the Apify Store. Blocks, retries and proxies handled by us. Browse the full catalog or commission a custom Actor.

Actors mentioned in this post:

google-ads-transparency · Run it on Apify

Twitch Chat Scraper and 2 More Live-Stream Chat Archives

Devil Scrapes — Wed, 09 Sep 2026 19:06:21 +0000

Originally published on the Devil Scrapes blog.

A Twitch chat scraper, a Kick chat archive, and a YouTube transcript export solve the same underlying problem from three different angles: none of the three platforms hands you a clean, structured record of what was said during a broadcast. Twitch retains a raw replay but no export tool; Kick has no chat history API at all; YouTube captions exist but aren't built for bulk pulls. Here's what each of our three Actors returns, what teams actually build with the output, and what it costs.

The platform gap, briefly

Live-stream chat and captions are some of the richest unstructured data on the internet — real-time audience reaction, sponsor call-outs, moderation events, comprehension-level language samples — and almost none of it is queryable in bulk from the platform itself. Twitch's chat replay lives behind an internal GraphQL endpoint, not the public Helix API. Kick has no chat-history endpoint whatsoever; once a stream ends, the conversation is gone from every Kick surface. YouTube's caption tracks are public but the timedtext endpoint isn't designed for hundreds of sequential pulls. We built one Actor per platform because each one needed a different transport, not because the underlying job differs.

Twitch: VOD chat, after the fact

Twitch Chat Scraper — VOD Chat Archive walks the same VOD-replay endpoint Twitch's own player uses and returns one row per chat message from a published VOD — no login required. Each row carries message_text, structured message_fragments (so emote IDs survive, not just literal emote shortcodes), user_color, badges, and message_offset_seconds, the position within the VOD the message was posted at.

That offset field is the one people underrate. Join it against the VOD timeline and you can locate exactly where chat spiked — a common proxy for "something worth clipping happened here" — without watching the broadcast back.

Pricing: $0.20 actor-start plus $0.001 per message, i.e. $1.20 per 1,000 messages. A 5,000-message stream costs about $5.20.

Kick: real-time, because there's no other option

Kick Chat Scraper & Real-Time Archive connects to the same public Pusher WebSocket Kick's own site uses to render chat, subscribes to up to 20 channel slugs at once, and archives every message live — sender, derived role (moderator, vip, subscriber, etc.), raw badges, chat color, and both the platform's send timestamp and our scrape timestamp.

The limitation that matters here isn't a limitation of the Actor, it's a limitation of Kick: this only captures chat sent while the run is active. There's no upstream to query for last week's stream, because Kick doesn't store it either. If you want a record of a future stream, the run has to be live during it.

Pricing: identical structure to Twitch — $0.20 actor-start plus $0.001 per message, so $1.20 per 1,000 messages archived across all subscribed channels combined.

YouTube: transcripts, not chat, but the same job

YouTube Transcript Scraper & Bulk Downloader is a different shape of data — one row per video, not per message — but it answers the same underlying question: what did the audience actually hear or say. It pulls manual or auto-generated caption tracks in any language and returns the full joined transcript_text, optional timed segments, channel name, title, duration, and every available_languages code on the video, from a list of URLs or bare video IDs. No API key needed.

Pricing: $0.20 actor-start plus $0.004 per transcript, i.e. $4.20 per 1,000 transcripts — a different unit from the chat Actors (per-video, not per-message), so don't compare the two rates directly.

What people actually build with this

Moderation research — pull Twitch or Kick chat across a batch of streams and quantify how often a term, slur list, or brigading pattern appears before deciding on a moderation policy change.
Sentiment and hype detection — bucket message_offset_seconds into time windows and chart message volume against the stream timeline; spikes correlate strongly with in-game or in-broadcast events worth reviewing.
Sponsor-mention auditing — search archived chat and transcript text for a brand or product name to check whether — and how — a sponsored segment actually landed with the audience, independent of what the creator's read reported.
Creator analytics — a talent agency or network archiving chat and transcripts across a roster of streamers to build engagement benchmarks that don't depend on the platform's own (often coarse) analytics dashboard.
NLP and RAG datasets — transcripts feed directly into a vector store or LLM context window for conference-talk or podcast search; chat archives are commonly used as training or evaluation data for toxicity and spam classifiers, with badges and sender_role as useful features.
Post-broadcast review — a streamer or their mod team reviewing an old VOD's chat for context around a moment, a decision, or a dispute, without scrubbing back through hours of stream.

How we handle the platform-side friction

All three Actors sit on the same infrastructure the rest of our fleet uses: we rotate curl-cffi browser TLS fingerprints (Chrome, Firefox, Safari) on every hop that could fingerprint a client, route through Apify Proxy with fresh sessions when a target pushes back, and retry with exponential backoff on 408/429/5xx up to five attempts per request, honoring Retry-After when it's sent. Every row is Pydantic-validated before it lands in your dataset, with ISO-8601 timestamps and stable IDs — so you can pull results as JSON, CSV, or Excel straight from the Apify Console, or via the API, without a cleanup pass first.

Run it on Apify

Every new Apify account starts with free trial credit, no card required, enough to test each Actor's output shape before committing spend.

FAQ

Can I get live Twitch chat, not just VOD chat, with these Actors?

Not with the Twitch Actor described here — it reads the VOD replay endpoint, which only exists once a broadcast has ended and Twitch has processed the recording. Live IRC-over-WebSocket chat is a separate protocol and a different Actor.

Why does Kick chat archiving require the run to be live during the stream?

Because Kick itself doesn't store chat history anywhere — not for us, not for anyone. There's no upstream endpoint to backfill from once a stream ends, so real-time capture is the only way to keep a record.

Do these Actors need a Twitch, Kick, or Google account to run?

No login is required for any of the three. Twitch's VOD replay and Kick's chat WebSocket are both public surfaces every viewer's browser reaches without authentication, and YouTube captions are public metadata attached to public videos.

What happens if a video has captions disabled or a VOD has expired?

Each Actor treats that as a normal, non-error outcome: the row (or VOD) is skipped, logged, and the run finishes SUCCEEDED rather than failing the whole batch over one unavailable item.

Which one should I use for sentiment analysis on a specific game or topic?

If you need audience reaction as it happened during a broadcast, use the Twitch or Kick chat Actor and bucket by message_offset_seconds or sent_at. If you need what the creator said, use the YouTube transcript Actor against their video and run your sentiment model over transcript_text.

Devil Scrapes builds and maintains 200+ pay-per-result web scrapers on the Apify Store. Blocks, retries and proxies handled by us. Browse the full catalog or commission a custom Actor.

Actors mentioned in this post:

Scrape Car Listings Europe: 22 Marketplaces, One Schema

Devil Scrapes — Wed, 09 Sep 2026 19:05:49 +0000

Originally published on the Devil Scrapes blog.

If you want to scrape car listings across Europe, the problem isn't finding a scraper for one marketplace — it's that "the European used-car market" is actually 20-plus independent national marketplaces, each with its own layout, its own currency, its own language, and its own defenses. We built a country-specific Actor for each one and normalized the output into one shared row schema, so pulling German and Serbian listings side by side doesn't mean writing two parsers and reconciling two field naming conventions yourself.

The buyer problem this actually solves

Nobody wakes up wanting "an AutoScout24 scraper" as an end in itself. The people who reach for one of these Actors are usually doing one of three things:

Cross-border price arbitrage — the same 2021 BMW 3 Series sells for a meaningfully different price in Germany versus Poland versus Romania, once you normalize for mileage and trim. Spotting that gap requires comparable data from multiple countries at once, not a single-market snapshot.
Dealer and fleet intelligence — dealer groups and export brokers track inventory turnover, pricing moves, and stock mix across competitor lots in several countries, which means pulling the same fields from several different sites on a recurring schedule.
Market research and pricing models — building a used-car valuation model that works across the EU needs training data with consistent fields (make, model, year, mileage, fuel type, price) across enough countries that a single national quirk doesn't skew the model.

All three jobs fail the moment your data pipeline has to special-case each country's field names. That's the actual product here: not 22 scrapers, one schema.

The shared schema

Every Actor in the fleet returns the same core fields, adapted to what each marketplace actually publishes. Compare leboncoin (France) and AutoScout24 (Germany) — different sites, different languages, same shape:

Field	leboncoin (FR)	AutoScout24 (DE)
`listing_id`	`"3117822207"`	`"a15406b4-db66-..."`
`make` / `model`	`"MORRIS"` / `"Autre"`	`"Opel"` / `"Corsa"`
`year`	`1968`	`2013`
`price` / `currency`	`34900` / `"EUR"`	`3490` / `"EUR"`
`mileage_km`	`101890`	`101125`
`fuel_type`	`"Essence"` (local label)	`"Benzin"` (local label)
`transmission`	`"Manuelle"`	`"Schaltgetriebe"`
`engine_power_hp`	`null` (not always published)	`101`
`body_type`	`"Citadine"`	`"Limousine"`
`seller_type`	`"private"` / `"dealer"` (normalized)	`"private"` / `"dealer"` (normalized)
`photo_urls`	array of full-size image URLs	array of full-size image URLs
`scraped_at`	ISO-8601 timestamp	ISO-8601 timestamp

Two things stay deliberately un-normalized, because normalizing them would lose information: currency (most of these markets price in EUR, but a few — Iceland's ISK, Norway's NOK, Sweden's SEK, Ukraine's USD/UAH — genuinely aren't) and raw field labels for fuel type and transmission, which we preserve verbatim in the source language alongside the normalized seller_type field, since collapsing "Essence" and "Benzin" into a single enum would need a lookup table you may want to own yourself. Everything structural — field names, types, the presence of mileage_km rather than miles, ISO timestamps — is identical across all 22 Actors, so a downstream pipeline written against one country's output works against any other with zero changes.

Country to Actor

Country	Marketplace	Actor
France	leboncoin.fr	leboncoin France Car Scraper
Germany	AutoScout24.de	AutoScout24 Germany Car Scraper
Serbia	Polovni Automobili	Polovni Automobili Car Scraper
Finland	Tori.fi	Tori.fi Marketplace Scraper
Lithuania	Autoplius.lt	Autoplius Lithuania Car Scraper
Poland	OTOMOTO.pl	OTOMOTO Poland Car Scraper
Portugal	Standvirtual.com	Standvirtual Portugal Car Scraper
Bulgaria	Mobile.bg	Mobile.bg Bulgaria Car Scraper
Belgium	2dehands.be	2dehands Belgium Car Scraper
Netherlands	Marktplaats.nl	Marktplaats Netherlands Car Scraper
Austria	willhaben.at	willhaben Austria Car Scraper
Ukraine	AUTO.RIA	AUTO.RIA Ukraine Car Scraper
Iceland	Bilasolur.is	Bilasolur Iceland Car Scraper
Norway	FINN.no	FINN.no Car Scraper
Italy	Subito.it	Subito Italy Car Scraper
Ireland	Carzone.ie	Carzone Car Scraper
Spain	coches.net	coches.net Spain Car Scraper
Romania	Autovit.ro	Autovit Romania Car Scraper
Sweden	Blocket.se	Blocket Car Scraper
Estonia	Auto24.ee	Auto24 Estonia Car Scraper
Bulgaria	Cars.bg	Cars.bg Bulgaria Car Scraper
Croatia	Njuškalo.hr	Njuškalo Car Scraper

(Bulgaria has two: Mobile.bg and Cars.bg are separate marketplaces with separate Actors, both listed above.)

What we handle so the schema stays clean

Every listing site in this fleet is a JavaScript-heavy classifieds marketplace with its own rate-limiting behavior, and several run detail-page enrichment as a second hop per listing. We rotate curl-cffi browser TLS fingerprints across requests, route through Apify Proxy with a fresh session on every block, and retry on 408/429/5xx with exponential backoff, honoring Retry-After. A search that returns genuinely zero matching listings finishes SUCCEEDED with an empty dataset and a status message describing what was searched — we fail loud only when the target itself couldn't be reached or parsed.

Pricing

Every Actor in the fleet uses the same Pay-Per-Event structure: a $0.20 actor-start fee plus $2.00 per 1,000 listings returned. A 500-listing pull from any single country costs $1.20; a 1,000-listing run costs $2.20. There's no bundled multi-country pricing tier — each Actor bills independently on Apify.

Run it on Apify

Every new Apify account gets free trial credit, no card required — enough to pull a first batch from any country in the table above before deciding whether to add more.

FAQ

Do all 22 Actors return exactly the same field names?

The core fields — listing_id, make, model, year, price, currency, mileage_km, fuel_type, transmission, seller_type, photo_urls, scraped_at — are identical across every Actor in the fleet. A handful of countries add fields the others don't publish (Iceland's engine size in cc, Germany's separate engine_power_kw, Ukraine's VIN field), which show up as extra columns rather than breaking the shared shape.

Why isn't price normalized to a single currency across countries?

Because normalizing it would require us to pick an exchange-rate source and a timestamp for the conversion, and that decision belongs to whoever is building the comparison — a snapshot rate baked into our dataset would silently go stale. We return the price in the currency the listing was actually posted in, plus the ISO currency code, so you can convert with whatever rate source your use case actually needs.

Can I search by make and model, or only by URL?

Both, depending on the country. Most Actors accept either a direct search-results URL (paste it from the marketplace and get the same filtered results back structured) or make/model/price-range filters as separate input fields. Check the individual Actor's Input table for which mode it supports.

Is detail-page enrichment (full description, extra photos) always included?

It's usually an optional input flag rather than the default, since enrichment means an extra request per listing and roughly doubles the run's page-fetch cost. Turn it on when you need the full listing description or full-resolution photo set; leave it off for a fast, list-page-only pull.

What happens if a country's marketplace changes its page layout?

Every Actor in the fleet is checked daily by our monitoring; when a target changes its markup, we triage and ship a fix — the same maintenance model that covers all 202 of our live Actors, not something specific to the car fleet.

Devil Scrapes builds and maintains 200+ pay-per-result web scrapers on the Apify Store. Blocks, retries and proxies handled by us. Browse the full catalog or commission a custom Actor.

Actors mentioned in this post:

OpenCorporates Alternative: US Business Registry Data

Devil Scrapes — Wed, 09 Sep 2026 19:05:17 +0000

Originally published on the Devil Scrapes blog.

If you're looking for an OpenCorporates alternative because its API returns a 401 without a paid key, the fix is simpler than it looks: the underlying US company data OpenCorporates re-publishes is, for a meaningful slice of states, already free and keyless at the source. New York, Colorado, Connecticut, and Oregon each publish their own Secretary-of-State business registry as an open Socrata dataset. We built three Actors on top of those open feeds — one for point-in-time company verification, one for a daily new-filings feed, and one for professional license rosters — so KYB analysts, SDRs, and sales-ops teams get structured rows without licensing a global registry they may not need.

Who actually needs this, and who doesn't

Be honest about scope before you build a pipeline on it: if you need OpenCorporates' 140-plus-jurisdiction global coverage — UK, EU, Asia-Pacific entities — nothing here replaces that. These three Actors cover US state-level registries only, currently New York, Colorado, Connecticut, and Oregon for company data, plus Delaware and Colorado for professional licenses. If your buyers, vendors, or prospect list are US companies concentrated in (or reachable through) those states, keep reading. If you need Cayman Islands holding companies, this isn't that.

The verification Actor

OpenCorporates Alternative Scraper takes a batch of company names and checks each one against all four state registries, returning one row per matched entity per jurisdiction:

{
  "query": "Acme Robotics LLC",
  "entity_id": "20251664337",
  "entity_name": "ACME ROBOTICS LLC",
  "jurisdiction": "CO",
  "entity_type": "DLLC",
  "status": "Good Standing",
  "status_normalized": "active",
  "formation_date": "2025-06-15",
  "principal_address": "9159 Anasazi Indian Trl, Highlands Ranch, CO 80129",
  "registered_agent_name": "ACME ROBOTICS LLC",
  "officers": [],
  "source_record_url": "https://data.colorado.gov/resource/4ykn-tg5h.json?entityid=20251664337",
  "scraped_at": "2026-07-18T12:00:00+00:00"
}

status_normalized is the field most buyers filter on first, because every state publishes its own raw status vocabulary — Colorado says "Good Standing," other states say something else — and we collapse that into a consistent active / inactive / unknown enum so a downstream filter doesn't need per-state logic. Connecticut is the one jurisdiction that also publishes a separate officers dataset, so officers populates there and stays [] elsewhere — documented behavior, not a gap in the code.

The new-filings feed

New Business Filings — US State Registry Leads Feed answers a different question: not "does this company exist," but "what companies newly formed in a date range." Feed it a [dateFrom, dateTo] window and a jurisdiction, and it returns every entity that registered in that window — company name, entity type, formation_date, registered agent, and address:

{
  "entity_id": "7969358",
  "entity_name": "URGB LLC",
  "jurisdiction": "NY",
  "entity_type": "DOMESTIC LIMITED LIABILITY COMPANY",
  "formation_date": "2026-07-17",
  "registered_agent_address": "32 Jagger Court, Melville, NY 11747",
  "scraped_at": "2026-07-19T12:00:00+00:00"
}

This is the shape sales and lead-gen teams actually want for outbound: a recurring, scheduled pull that surfaces brand-new LLCs and corporations the day (or week) they form, before they show up in any paid data broker's monthly refresh.

The professional license Actor

Professional License Lookup Scraper covers a related but distinct need — verifying an individual or business holds an active professional license, not that a company is registered. It exports rosters from Delaware and Colorado's open-data licensing portals across nursing, real estate, contracting, medical, and accountancy categories, with a disciplinary_flag surfaced where the state publishes disciplinary action:

{
  "jurisdiction": "DE",
  "licensee_name": "SHAHAN,BRIAN",
  "license_number": "RS-0037427",
  "profession": "Real Estate",
  "status_normalized": "active",
  "expiration_date": "2028-04-30",
  "disciplinary_flag": false
}

Who buys this and why

KYB onboarding — batch-verify a vendor or customer list before signing a contract; flag anything inactive or unmatched before finance cuts a PO.
Sales-ops list hygiene — confirm the companies on an SDR's prospect list are real, active, on-file entities before a team spends a quarter's outreach budget on dead accounts.
Lead generation from new formations — a recurring feed of brand-new LLCs is a standard top-of-funnel source for business insurance, accounting, and B2B SaaS sales teams who want to reach a company in its first weeks of existence.
Due diligence — pull incorporation date, entity type, and registered-agent data as a first-pass screen before a deeper manual review.
Compliance and licensing checks — confirm a contractor, real-estate agent, or medical professional's license is active (and clean) before onboarding them as a vendor or partner.

Why this is more than "just hit the state API yourself"

Each state runs its own Socrata dataset with its own field names, its own status vocabulary, and its own rate-limit behavior under a batch of a few hundred names in one run. New York's snapshot updates monthly, not in real time; Colorado, Connecticut, and Oregon each use different resource IDs and query parameters. None of that is hidden — these are public open-data portals — but normalizing four raw schemas into one row shape, fuzzy-matching a company name against each state's index, and handling one state's timeout without losing the other three states' results is the actual engineering work. We also deliberately never surface home addresses that Connecticut's principals dataset happens to publish alongside business addresses — a KYB dataset leaking an officer's home address because a state technically made it public is a decision we didn't want to leave to accident.

Pricing — exact numbers

All three Actors use the same Pay-Per-Event rate: $0.05 (verification and license Actors) or $0.20 (new-filings feed) actor-start fee, plus $5.00 per 1,000 matched rows. A batch of 1,000 matched company or license rows costs $5.05–$5.20 depending on the Actor; a search that returns zero matches costs only the start fee. Apify gives every new account $5 of free trial credit, which covers roughly the first 1,000 matched rows before you spend anything real.

Honest limitations

This is a four-state footprint for company data (NY, CO, CT, OR) and a two-state footprint for licenses (DE, CO) — not an 18-state or global claim. We picked these because each one publishes a genuinely free, keyless, Secretary-of-State-sourced open dataset, not a scrape of a login-gated search page. More states are a per-state adapter away, not a rewrite, and are on the roadmap. Officer data is sparse by design — only Connecticut publishes it — and freshness varies by state, with New York's feed explicitly monthly rather than real-time.

Run it on Apify

Every new Apify account starts with free trial credit, no card required.

FAQ

Is this the same data as OpenCorporates?

No. These Actors query each state's own free, keyless Secretary-of-State open-data API directly, not opencorporates.com. They serve the same category of buyer but pull from a different — and for these specific states, more current — data source.

Why only four states for company verification?

Those are the four we've confirmed publish a genuinely free, keyless Socrata dataset sourced from the state's own Secretary of State, rather than a login-gated or heavily defended search UI. Adding a state is a contained adapter, not a rewrite.

Can I search by entity ID instead of company name?

Not in the current release of the verification Actor — it's name-search only. Direct ID lookup is a straightforward addition; open an issue on the Actor's page if you need it sooner.

How current is the new-business-filings data?

It depends on the state's own publishing cadence — some states update closer to real time, New York's snapshot is explicitly monthly. Check the registry_url field in your results for the state's own dataset page, which usually states its refresh schedule.

Do I need an API key or login for any of these?

No. All three Actors query public, keyless state open-data portals. There's no account to create, no key to request, and no login step in the Actor itself.

Devil Scrapes builds and maintains 200+ pay-per-result web scrapers on the Apify Store. Blocks, retries and proxies handled by us. Browse the full catalog or commission a custom Actor.

Actors mentioned in this post:

Custom Web Scraper Development vs Building In-House

Devil Scrapes — Wed, 09 Sep 2026 19:04:45 +0000

Originally published on the Devil Scrapes blog.

The honest answer to "custom web scraper development or build in-house" is: it depends on whether the target actively defends itself, and whether scraping is core to your product or a one-time data need. Below is the framework we actually use when a prospective customer asks us to build against a target we haven't already covered — total cost of ownership, blocking risk, time-to-data, and compliance — plus exactly how our commissioned-build engagement works if you decide a custom Actor is the right call.

The framework

1. Total cost of ownership

The build is rarely the expensive part. A working scraper against an undefended target can be a weekend project. What isn't a weekend project is maintenance: the target changes its markup, rotates an internal API parameter, or ships a redesign, and your scraper silently starts returning empty or wrong data until someone notices — usually a customer, not you. In-house, that's an ongoing engineering cost with no natural stopping point: someone owns "is the scraper still working" forever, on top of whatever they were hired to build. A commissioned Actor shifts that ownership to a team whose whole job is watching for exactly that kind of breakage across a portfolio of targets, which is a materially different cost curve than one engineer's part-time attention.

2. Blocking risk

If the target has no meaningful anti-bot defenses — a plain REST API, a static site, a small internal tool — building in-house is usually the right call. The risk profile changes sharply once a target actively fingerprints clients, rate-limits aggressively, or serves different content to suspected bots. That's not a "write better code" problem; it's an ongoing arms race that needs browser-fingerprint rotation, proxy rotation, session management, and backoff tuned per target — infrastructure that's expensive to build once and genuinely expensive to keep current as the target's defenses evolve. We've run that gauntlet on 202 live targets so far; every one of them needed a different combination of the same underlying toolkit.

3. Time-to-data

In-house builds compete with your team's existing roadmap — a scraper is rarely the highest-priority ticket in the backlog, so "we'll get to it" often means weeks, not days. A commissioned build runs on a fixed timeline agreed at quote time, independent of your team's other priorities, because it's the only thing the person building it is doing that week.

4. Compliance and data handling

Neither path removes the need to think about what you're allowed to do with scraped data — that's a legal question, not an engineering one, and it doesn't change based on who writes the code. What does change: a vendor who's built compliance-conscious defaults into their process (never scraping data gated behind a login without authorization, never surfacing personal data a public dataset happens to include when it isn't the point of the ask, documenting exactly what a scraper does and doesn't touch) saves you from re-deriving those defaults yourself on a project-by-project basis.

5. When in-house wins outright

Scraping is a core, differentiating part of your product, not a supporting data feed — you want the expertise in-house long-term regardless of this one target.
The target is genuinely simple and stable (a documented public API, a slow-moving static site).
You need extremely tight coupling with proprietary internal systems that a third party can't reasonably access or shouldn't need to.
You already have engineers with free capacity and the specific anti-blocking expertise this requires — which is rarer than it sounds, since it's a narrow specialty most product engineering teams don't maintain day to day.

How our commissioned-build engagement works

If the framework above points toward "outsource this," here's the actual process, start to finish:

Brief — you describe the target, the fields you need, expected volume, and how fresh the data has to be. We tell you upfront if the target looks straightforward or if it's the kind that needs heavier anti-blocking investment.
Fixed quote — a fixed price and timeline based on the brief, not a time-and-materials open tab. You know the cost before we write a line of code.
Build — the same stack and standards behind our 202 live Actors: Pydantic-validated input and output, browser-fingerprint rotation, proxy rotation through Apify Proxy, retry with backoff on transient failures, and a clean typed dataset schema documented from day one.
Cloud QA — before handover, the Actor runs a real cloud smoke test against the live target — not just a local test suite — verifying it actually reaches the target, handles a block gracefully, and produces the schema you asked for.
Handover — the finished Actor deploys to Apify under your account or ours, whichever you prefer. Either way you get the source, the documentation, and a working, tested Actor you can run on demand or on a schedule.
Maintenance retainer (optional) — targets change. A retainer keeps the Actor monitored and patched when the source site updates, so you're not the one who finds out it broke when a customer complains.

What we bring that a from-scratch build doesn't start with

Two Actors from our own fleet are useful reference points for the kind of target this process handles well. Google Ads Transparency Scraper pulls structured ad-creative data from a JavaScript-heavy transparency center with no public API — 77 users and over 9,400 runs to date. YouTube Transcript Scraper & Bulk Downloader bulk-extracts captions from an endpoint that isn't designed for repeated automated pulls, handling rate-limit pacing and endpoint drift so a batch of hundreds of videos doesn't fall over halfway through. Both started as exactly this kind of build: a target with no clean API, a defined schema, and a customer who needed the data on a recurring basis rather than a one-off pull.

Run it on Apify

If your target isn't already in our catalog, that's exactly what a commissioned build is for. Read more about the engagement at /custom-actors/ or reach out directly via /contact/ with your target and expected volume, and we'll tell you honestly whether it's a fixed-quote build or a weekend project you can handle yourself.

FAQ

How long does a typical custom Actor build take?

It depends entirely on the target's defenses and the schema's complexity — a straightforward site with a stable layout can turn around in days; a heavily defended or JavaScript-rendered target with a large field set takes longer. You get a specific timeline at quote time, not an estimate that shifts later.

Who owns the Actor after handover?

Whichever account you specify — yours or ours. If it deploys under your Apify account, you have full control over pricing, scheduling, and access from day one. If it deploys under ours, you get run access and the same dataset output without managing an Apify account yourself.

What happens if the target changes its site after handover?

Without a maintenance retainer, a change to the target is your team's responsibility to catch and report, same as any in-house scraper. With a retainer, we monitor the Actor and ship a fix as part of the ongoing agreement — the same daily-monitoring model that covers all 202 of our live Actors.

Is there a minimum volume or contract length for a custom build?

No minimum volume — the fixed quote scales with the actual scope of the brief, whether that's a single target or several. The maintenance retainer is separate and optional, and only makes sense once you've decided you want the Actor to keep working without you watching it.

Can you build against a target that requires a login?

We evaluate this case by case at the brief stage — some authenticated targets are reasonable to automate on your behalf with your own credentials, others raise terms-of-service or compliance questions we'll flag before quoting rather than after building.

Devil Scrapes builds and maintains 200+ pay-per-result web scrapers on the Apify Store. Blocks, retries and proxies handled by us. Browse the full catalog or commission a custom Actor.

Actors mentioned in this post:

The US sanctions list still ships with a 1980s DOS end-of-file byte

Devil Scrapes — Wed, 09 Sep 2026 08:26:17 +0000

Quick answer

The US Treasury's OFAC sanctions list still ships with a DOS end-of-file marker. The last three bytes of SDN.CSV are 0d 0a 1a — CR, LF, then 0x1A, the Ctrl-Z character MS-DOS used to mark end-of-text in the 1980s. Python's splitlines() treats that trailing \x1a as content and hands you a final one-column row that is not a record. If your parser validates column counts — and a sanctions parser absolutely should — it will reject the entire file on the last line, after correctly parsing 19,364 real ones.

Why is there a 1980s EOF byte in a live compliance feed? 🔍

Because the file format is old and stable, and stability is the point. SDN.CSV is a fixed-shape export that downstream compliance systems have consumed for decades, so nobody is going to modernise the trailer and break them. Here are the actual last bytes:

00000000: 2041 4e4f 4e49 4d20 5349 524b 4554 492e   ANONIM SIRKETI.
00000010: 220d 0a1a                                 "...

That final 1a is the whole problem. It is invisible in a text editor, survives every naive .strip() that only removes whitespace (\x1a is not whitespace), and produces a phantom row precisely at the point where most parsers stop paying attention.

The fix is a guard clause, not a heuristic: strip the EOF marker explicitly before splitting, and keep the strict column-count check for every remaining row. What you must not do is relax the column-count validation to make the error go away — that check is the thing standing between you and silently mis-parsing a sanctions record.

What else is unusual about the SDN format?

-0- is null. OFAC does not use empty fields. A missing value is the literal three-character string -0-:

36,"AEROCARIBBEAN AIRLINES",-0- ,"CUBA",-0- ,-0- ,-0- ,-0- ,-0- ,-0- ,-0- ,-0-

Note the trailing space after each one, too. If you load this into a dataframe without translating the sentinel, you get a column full of the string "-0-" that is not null, not empty, and will happily pass an if value: check. Every one of those fields then reads as populated.

The download is a redirect to a signed, expiring URL. A request to the export path answers 302 with a Location header pointing at a presigned S3 object carrying X-Amz-Expires=3600 and a signature. You cannot cache that URL — it dies within the hour. You have to re-request the export path each time and follow the hop. Worth also not forwarding your original headers onto the S3 leg: presigned URLs can reject requests carrying unexpected headers.

The OFAC SDN Sanctions List Scraper handles all three — the EOF marker, the -0- sentinel, and the redirect chain — and joins ALT.CSV (aliases) and ADD.CSV (addresses) onto the primary records by entity number.

How big is the list, and how fast does it move?

SDN.CSV is roughly 5.6 MB across about 19,365 rows as of 2026-09-09, with a separate alias file of comparable size. That matters for a practical reason people discover in production rather than in testing: if you pull the primary list, the aliases and the addresses concurrently through a single network egress, they share bandwidth, and a per-request timeout tuned on a laptop will be far too tight. A 30-second total timeout that passes locally every time will die in a datacenter having received 991,217 of 1,062,698 bytes — 93% of the way through the smallest of the three files.

The lesson generalises past this API: for multi-megabyte concurrent downloads, a total request timeout is not the same knob as a connect timeout, and the number you need is set by your slowest egress, not your fastest.

FAQ

Do I need an API key?
No. The OFAC sanctions list is published by the US Treasury as open data, with no key, no registration and no IP whitelisting.

Is screening a name against the SDN list enough for compliance?
No, and please do not treat it as such. The SDN list is one list among several — consolidated non-SDN lists, EU and UK regimes, and local requirements all exist. Fuzzy name matching also produces both false positives and false negatives, especially across transliterated names, which is why match scores need human review rather than being wired straight into a block.

Why do aliases live in a separate file?
Because one entity can have many. ALT.CSV maps alias records back to the primary entity by ent_num, so a single sanctioned individual can appear under a dozen spellings. Screening only against primary names in SDN.CSV will miss the alias that a counterparty actually uses — which is, in practice, the whole reason alias data exists.

How current is the file?
OFAC republishes on change, and the presigned URL path includes a dated segment. Treat the list as something to re-pull rather than cache; a stale sanctions list is worse than no sanctions list, because it feels like diligence.

The short version

A 5.6 MB CSV, a 1980s EOF byte on the last line, -0- where a null should be, and a one-hour signed URL in front of it. None of these are hard once you know; all of them are silent when you don't.

Byte-level details in this post were verified against sanctionslistservice.ofac.treas.gov on 2026-09-09.

DEV Community: Devil Scrapes

UTC Offsets Are Not Integers — A Remote Job Feed With 5.75 In It

Quick answer

Why does a remote job board even have fractional timezones? 🕐

What does this break in practice?

How we handle it 🛡️

Output

Airbnb's search payload doesn't carry amenities. Here's what we do instead of guessing.

Quick answer

What's actually in Airbnb's search payload

Why we return null instead of inventing a field

Pinning the proxy country to the market's currency

Retrying without losing the run

The part that generalizes

Pricing

How we keep 200 web scrapers alive

Shipping is the easy 10%

A "SUCCEEDED" run can still deliver nothing

Real breakages, not hypothetical ones

The loop: monitor, triage, fix, verify in the cloud

Price the maintenance, not the first run

FAQ

What does "web scraper maintenance" actually mean?

How do you know an Actor is broken before a customer complains?

Why can a run say "SUCCEEDED" but return no data?

Why do you price per result instead of a flat fee?

Does this apply to every Actor you publish?

Why your scraper gets blocked at 3 a.m.

It's rarely the parser

TLS and HTTP fingerprints

IP reputation: residential vs. datacenter

Session and cookie continuity

Rate limits: Retry-After is not a suggestion

The trap that doesn't look like a block at all

What this looks like for you

FAQ

Why does my scraper get blocked even with a correct User-Agent header?

Is a residential proxy always better than a datacenter proxy?

Why did my scraper get a 200 response with wrong data instead of an error?

What does honoring Retry-After actually prevent?

Can session continuity really break a scrape on its own?

Export a competitor's entire Google Ads library

A public registry with no export button

What comes back per ad

Who this is for

Running it

Pricing, exactly

What we're upfront about

FAQ

Does Google have an official Ads Transparency API?

Is scraping the Google Ads Transparency Center legal?

Can I filter results by country or region?

How much does it cost to export 500 ads?

Can I automate a weekly competitor sweep?

Pay-per-result scraping: what you actually pay

Two charges, and only two

Three real price points

What an empty run actually costs

How to cap spend before it happens

How this compares to the alternatives

FAQ

What happens if my scraper run finds zero results?

Is pay-per-result cheaper than a flat monthly subscription?

Why do different Actors charge different per-result prices?

Can I set a hard spending cap before running a scraper?

Does the actor-start fee apply every time I run an Actor?

Run an Apify Actor From n8n, Make or Zapier

Why wire a scraper into a no-code tool at all

Option A — n8n

Option B — Make

Option C — Zapier

run-sync-get-dataset-items vs async + webhook

What this costs

Run it on Apify

FAQ

Do I need a paid n8n, Make, or Zapier plan to run an Apify Actor?

Which is faster to set up, the node/app or a raw HTTP request?

What happens if the Actor run fails mid-workflow?

Can I pass a schedule from n8n/Make instead of using Apify's own Scheduler?

Twitch Chat Scraper and 2 More Live-Stream Chat Archives

Why we return `null` instead of inventing a field

Rate limits: `Retry-After` is not a suggestion

What does honoring `Retry-After` actually prevent?