DEV Community: Devil Scrapes

Bayt Jobs Scraper: the proxy country pin that was the block

Devil Scrapes — Sat, 26 Sep 2026 10:06:59 +0000

Quick answer

We pinned our Bayt.com scraper's proxy to the country it scrapes, and that pin was the block. On Apify residential exits in the UAE, cloud QA landed 0 rows in 15 attempts. We changed one field to a UK residential exit and the same build returned 20 rows, then a full 60-row run with no blocked pages. The data stayed genuinely Emirati because Bayt decides the country from the URL path (/en/uae/jobs/...), not from where your IP sits. The Bayt Jobs Scraper now defaults to RESIDENTIAL / GB, and you pick the market with the country input.

Shouldn't a proxy match the country you're scraping?

Usually, yes. A geo-random exit can quietly give you plausible but wrong data: prices in another currency, or another region's catalogue. Pinning the exit is normally the safe default, and it's our fleet rule for exactly that reason. The rule assumes the site uses your IP to pick the country. Bayt doesn't. It scopes every search by a path segment, so /en/uae/jobs/accountant-jobs/ returns UAE jobs whatever exit the request comes from. That leaves the pin with a cost and no benefit. The UAE residential pool was the one Bayt's Cloudflare layer challenged hardest, so pinning to it made us look more suspicious.

How do you tell "the pin is the block" apart from "the site blocks us"?

Change one variable. Same build, same input, same residential tier, and only the exit country different: AE went 0 for 15, GB cleared on the first attempt and kept clearing over three spaced runs. Swapping tiers or TLS profiles at the same time would have hidden which change did the work. Before you shelve a target as unreachable, try a neutral exit.

If the site picks its country from the URL, the exit country is just one more fingerprint, and the local one can be the most-watched.

What the Actor gives you

One row per job: job_id, title, company, company_url, location, career_level, salary (only on postings that publish it), a remote flag, summary, posted_at_raw (e.g. "3 days ago"), the absolute detail_url, and the keyword that found it. Pass several keywords in one run, and set country to uae, saudi-arabia, egypt or any other Bayt market path.

Honest limitations 🚧

List-page fields only, with no follow-through to detail pages. The posted date is Bayt's own relative text, kept verbatim rather than guessed into a timestamp. Salary appears on a minority of postings, and we return null rather than invent it. A search with no matches is a successful run.

FAQ

Do I need a Bayt account?
No. It reads Bayt's public search result pages.

Why a UK proxy for UAE jobs?
Because Bayt picks the market from the URL, and UK residential exits cleared where UAE ones were challenged. The rows are still the UAE (or whichever country you set).

What about blocks and captchas?
That's our job. We rotate browser fingerprints, retry with backoff on rate limits and challenges, and rotate residential sessions on every block.

$0.20 per run plus $0.002 per job row: $2.20 per 1,000 results. A zero-match search succeeds and costs only the start fee.

→ Bayt Jobs Scraper on Apify

Built by Devil Scrapes. We handle the fingerprints, the proxies and the retries, and we test the proxy default we ship against the target before a customer ever runs it.

InfoJobs Spain Scraper: the proxy default that 407'd every run

Devil Scrapes — Fri, 25 Sep 2026 09:11:38 +0000

Quick answer

We shipped a proxy default that read as reasonable and failed every single run. The config was {"useApifyProxy": true, "apifyProxyCountry": "ES"} — a bare useApifyProxy with no named group. That resolves to Apify's datacenter tier, and our datacenter pack is a static US-only pool, so pinning ES against it asked for an exit country the pack simply doesn't own. Cloud QA caught it before any customer did: CONNECT 407, proxy auth rejected. The fix was naming the group — apifyProxyGroups: ["RESIDENTIAL"] alongside the country pin — and the InfoJobs Spain Jobs Scraper now ships with both, not just one.

If you pin a proxy country, do you also need to name the group?

Yes, and skipping it isn't a degraded result — it's a guaranteed tunnel failure. A country pin tells Apify Proxy which exit to hand you; the group tells it which pool to hand it from. Name only the country and Apify defaults you into datacenter, because that's the cheaper tier absent other instructions. If your account's datacenter pack has no IPs in the country you pinned — ours is a static US-only 27-IP pack — the tunnel refuses to open at all. Every InfoJobs run against the old default would have failed before fetching a single page. This is a config-shape bug, not a scraping-difficulty bug, and it's exactly the class of thing a five-minute local test never surfaces because local runs don't route through Apify's proxy infrastructure.

Did the shipped defaults actually get exercised in the cloud before going live?

No — and that was the second, separate finding on this Actor. The prefill that ships with every new Actor is what Apify's own automated QA runs, and what a customer sees pre-filled if they click Start without touching the form. Ours was maxResults: 5, maxPages: 1 — a single page. Pagination is the entire point of a job-board scraper, and it had never actually run in the cloud, by us or by Apify. We widened it to maxResults: 40, maxPages: 2 and re-ran: 40 rows landed, status message read "2/2 page(s) parsed". Two separate bugs, same root cause — a config nobody had actually pointed at the live target.

A prefill that returns one page is a prefill that never tests pagination — no matter how much the pagination code itself has been reviewed.

What the Actor gives you

One row per deduplicated job posting: job_id, title, description, city, url, contract_type, salary_min/salary_max/salary_period/salary_currency (null when unpublished), workday, teleworking, published_at (ISO-8601), company_name, company_logo_url, company_url, states, and is_executive. Province filters accept either a plain name ("Madrid") or InfoJobs' own numeric id ("33") — both resolve against the site's own province table, not a hand-typed lookup.

Honest limitations 🚧

One keyword plus an optional province per run — no multi-keyword batching yet. maxResults and maxPages are independent caps; whichever fires first stops the run, and the status message says which. List-page fields only, no detail-page follow-through. A narrow search returning zero rows is a successful run, not a failure.

FAQ

Why did the Actor 407 on every run before this fix?
Because the default proxy config named a country but not a group, which silently resolved to a US-only datacenter pack that has no Spanish exits. Naming apifyProxyGroups: ["RESIDENTIAL"] fixed it.

Do I need an InfoJobs account or API key?
No — this reads InfoJobs' public search result pages, parsed from the page's own window.__INITIAL_PROPS__ JSON blob.

Can I filter by province?
Yes — pass a plain Spanish province name or its numeric InfoJobs id; the input param is provinceIds under the hood, resolved by testing live requests rather than guessed from the UI.

$0.20 per run plus $0.0025 per deduplicated job row — $2.70 per 1,000 results. A zero-match search still succeeds and costs only the start fee.

→ InfoJobs Spain Jobs Scraper on Apify

Built by Devil Scrapes. We rotate browser fingerprints, retry with backoff, and route every request through Apify Proxy — and we test our own proxy configuration against the pack we actually own before we ship it as a default.

Fotocasa Scraper: the null field that hid every discounted listing

Devil Scrapes — Fri, 25 Sep 2026 09:11:06 +0000

Quick answer

We assumed reducedPrice was a raw integer, because in the single listing we'd probed it, the field was null — and null has no type to infer from. A mandated live run against real listings surfaced the truth: when a Fotocasa listing actually carries a discount, reducedPrice is a formatted string, "75.000 €", the exact same shape as the regular price field. Ship the integer assumption and the parser silently skips every discounted listing — no error, no warning, just a quietly incomplete dataset on precisely the rows a buyer cares about most. The Fotocasa Spain Property Scraper parses it as a formatted string, same as price, before converting to reduced_price_eur.

Can you infer a field's type from a `null` sample?

No — and that's the whole lesson here. A field that reads null in your one probed sample has no observable shape; you cannot know from an absence whether the real value, when it exists, is an integer, a formatted string, or something else entirely. The only fix is running against enough live data that the field actually appears populated at least once. We did — and reducedPrice turned out to share the exact same "75.000 €"-style formatting as the primary price field, not the raw integer a schema guess would produce. reduced_price_eur in our output is legitimately null on listings with no discount; it's only populated, as an integer in euros, on the listings that actually have one.

Does this scraper need a browser to run?

No. Fotocasa.es server-renders each listing's full structured data into a <script id="__initial_props__"> block on the search-results page itself — price, address, coordinates, features, photo URLs, and the description all arrive in that one JSON payload. No separate API calls, no client-side rendering to wait on. We read it with curl-cffi under browser TLS impersonation, no headless browser required, which keeps the run cheap and the dataset arriving fast.

A field that's null in your sample carries no type information at all — the only way to learn its real shape is a live run against data where it's actually populated.

What the Actor gives you

One validated row per listing: price in euros (price_eur, plus the human-formatted price_display) and reduced_price_eur when discounted, building_type/building_subtype, rooms, bathrooms, surface_m2, full address (address_province, address_city, address_district, address_neighborhood, address_zip_code), latitude/longitude, dynamic_features, up to ~18 image_urls, the full Spanish-language description, and agency_alias/agency_type/phone when the seller publishes them.

Honest limitations 🚧

Search-results-page fields only in v1 — no per-listing detail-page fetch, since the results page already embeds everything above. Rent (alquiler) URLs are supported best-effort; sale (comprar) is the wire-confirmed path. Price filters apply after fetching, so a narrow band still costs the request for out-of-range listings on that page. propertyTypeId passes through as Fotocasa's raw internal ID — there's no confirmed ID-to-label table yet.

FAQ

Why did the original spec get reducedPrice's type wrong?
The only sample probed at spec time had it as null, and null doesn't reveal a type. A live run against listings that actually carry a discount showed it's a formatted string like price, not a raw integer.

Do I need a Fotocasa account or API key?
No — Fotocasa publishes no public API for this data; this Actor reads the same public search-results pages a browser loads and parses the JSON already embedded in them.

Why is the proxy pinned to residential Spain (ES) by default?
Because a Spain-only portal can render the wrong currency, or block outright, from a foreign exit — we force the ES residential exit regardless of other settings.

$0.20 per run plus $0.004 per validated listing — $4.20 per 1,000 results. No data, no charge beyond the start fee.

→ Fotocasa Spain Property Scraper on Apify

Built by Devil Scrapes. We rotate browser fingerprints, rotate residential proxies, retry with backoff, and ship clean typed rows — and we don't guess a field's shape from a sample where it's empty.

XING's job data ships as a JS object, not JSON. Here's how we parse it without breaking on the first undefined.

Devil Scrapes — Fri, 25 Sep 2026 08:56:33 +0000

Quick answer

XING ships a complete Apollo GraphQL cache inline in every search-result page — <script id="runtime-config">window.crate={...} — so the job data never needs DOM scraping. The catch: that blob is a JavaScript object literal, not JSON, so a naive json.loads throws on it. We normalize it first, then resolve ROOT_QUERY's jobSearchByQuery collection into typed VisibleJob/Company nodes. Verified cloud run: 50 deduplicated rows across two keywords ("developer" 40, "marketing" 10), Berlin, pagination confirmed across 2 pages per keyword. The XING Jobs Scraper is built on that resolved graph, not a screen-scrape.

Isn't `window.crate={...}` just JSON you can `json.loads`?

No, and treating it like JSON is the fastest way to ship a scraper that dies on page two. window.crate is a live JavaScript object literal assigned in a <script> tag — it's what the browser's own runtime evaluates, not a JSON serialization of it. It contains bare undefined tokens in place of missing fields, which are valid JavaScript but not valid JSON syntax; json.loads throws on the first one it hits. We normalize the blob — swapping the JS-only tokens for their JSON equivalents — before we ever try to parse it, which is the difference between "this works on the one listing I tested" and "this works on the thousandth listing that happens to have a hole in it."

Where does the company name actually come from?

Not where you'd expect. The Apollo cache is normalized: every job entity holds a reference like Company:<id> instead of the company's data inline, and resolving that reference on its own gets you a logo URL and nothing else — no name. The name lives on a different node entirely, companyInfo.companyNameOverride, which we join back onto the resolved job. Skip that join and you ship a scraper that returns a logo and a blank company column — technically a "success," practically useless to a recruiter filtering by employer.

Why is `salaryMedian` sparser than `salaryMin`/`salaryMax`?

Because XING serves two structurally different salary shapes under one field name, and only one of them carries a median. Its own SalaryEstimate type includes a computed median; an employer-stated SalaryRange gives you a min and max only, no median, because the employer never published one. Counted live on a single result page: 44 SalaryEstimate nodes against 28 SalaryRange nodes. So on a page where roughly two-thirds of salaried postings carry a median, expect that ratio to hold in your dataset too — the sparsity isn't a scraping gap, it's what XING itself published.

The blob at window.crate looks like JSON and isn't — bare undefined tokens break json.loads before you ever get to the graph you actually want.

What the Actor gives you

One deduplicated row per job across one or more keywords: id, slug, title, url, companyName, companyLogoUrl, locationCity, locations, employmentType, salaryMin/salaryMax, salaryMedian (when the posting is a SalaryEstimate), salaryCurrency, keyResponsibilities, refreshedAt, activeUntil, paid, topJob, plus sourceKeyword/sourceLocation tagging which search found each row. Run-wide dedup means the same job id never appears twice, even across pages or across keywords batched into one run.

Honest limitations 🚧

List-page fields only — full job-description text isn't fetched; follow the returned url for a detail-page job if you need it. XING ranks by relevance, not strict keyword match, so a niche keyword can surface adjacent roles instead of an empty page. A narrow search legitimately returning zero rows is a successful run, not a failure.

FAQ

Do I need a XING account or API key?
No — this reads XING's public search result pages.

Can I search multiple keywords in one run?
Yes — pass an array to keywords; every row is tagged via sourceKeyword, and dedup applies across the whole run.

Why did I get fewer rows than maxResults?
Either the search genuinely has fewer matches, or maxPagesPerKeyword stopped the run first — the run's status message says which.

$0.20 per run plus $0.0025 per deduplicated job row — $2.70 per 1,000 results. A zero-match search still succeeds and costs only the start fee.

→ XING Jobs Scraper on Apify

Built by Devil Scrapes. We rotate Chrome/Firefox TLS fingerprints, retry with backoff, and route every request through residential proxy pinned to Germany — and we normalize XING's inline cache into a real object graph instead of guessing at a screen-scrape.

PropertyFinder.ae's category parameter wasn't what the spec assumed. We tested it before shipping.

Devil Scrapes — Fri, 25 Sep 2026 08:56:03 +0000

Quick answer

We tested PropertyFinder.ae's c= search parameter before trusting it, and it wasn't what the spec assumed. The written spec read c= as a plain residential-vs-commercial switch. The wire disagreed: c=1 is residential-sale, c=2 is residential-rent, c=3 is commercial-sale, c=4 is commercial-rent — one parameter carries both category and transaction type. Shipped on the original reading, the PropertyFinder UAE Property Scraper would have quietly handed a customer asking for for-sale listings a page of for-rent ones. We caught it before it shipped, not after a refund request.

Does PropertyFinder.ae's `c=` parameter just mean residential vs commercial?

No, and assuming so is the mistake worth flagging for anyone building against this target directly. c= is a combined category-and-transaction axis, not two independent filters bolted together — category and purpose in this Actor's input map onto that single parameter internally so you never have to know the encoding. The listings themselves live in the page's embedded __NEXT_DATA__ JSON blob — roughly 584,000 characters of it — at props.pageProps.searchResult.listings, which is a far more reliable source than parsing rendered DOM, but the query encoding sitting in front of that JSON still has to be verified against real requests, not inferred from the parameter's name.

What happens when you ask for a bedroom filter that doesn't exist on the wire?

There is no server-side bedroom parameter on this target — we checked, and it isn't there. That left two options: invent a query param the server silently ignores, which looks like it worked and does nothing, or filter client-side on the parsed rows and say so plainly in the docs. We chose the second. An unverified filter is worse than a missing one, because a filter the server ignores returns a plausible, confidently wrong result set, and nothing in the output tells you it was never applied. We'd rather ship an honest client-side filter than a fake server-side one. We also confirmed the failure mode on the other end: an absurd price range returns a genuine zero-match SUCCEEDED run with a status message — not a crash dressed up as a bug.

An unverified filter is worse than a missing one — a filter the server ignores returns a plausible, confidently wrong result set, and nothing in the output tells the buyer it was never applied.

What the Actor gives you

One row per listing: listing_id, title, price_amount + price_currency, property_type, bedrooms, bathrooms, size_value + size_unit, location and location_hierarchy (city → community → sub-community), agent_name, agency_name, listing_url, image_urls, posted_at / updated_at, and latitude/longitude where PropertyFinder's own JSON carries them. A real sampled row: a 4-bedroom villa in Meadows 7, Meadows, Dubai, AED 23,500,000, location_hierarchy ["Dubai", "Meadows", "Meadows 7"], with agent name and coordinates attached. Proxy is RESIDENTIAL, pinned to AE — forced in code, not trusted from whatever the run's prefill happens to say, because a geo-random exit on a country-scoped portal is a known way to get wrong-currency or wrong-inventory data back.

Honest limitations 🚧

v1 is search-results only — full description text, amenities lists, and floor plans live on the detail page and aren't fetched. location_hierarchy, latitude/longitude, and agency_name are null when PropertyFinder's own JSON doesn't carry them for a given listing — we don't backfill guesses. bedrooms is applied client-side, as covered above, not as a confirmed server-side filter. Non-UAE PropertyFinder country portals are out of scope for this Actor.

FAQ

Does category/purpose map onto one parameter or two on the real site?
One — c= is a combined category-and-transaction axis (1–4), not two independent switches. This Actor encodes it correctly so you just pick category and purpose and never touch the raw parameter.

Is the bedroom filter applied by PropertyFinder's server or by the Actor?
By the Actor, client-side, on the parsed rows — there's no confirmed server-side bedroom parameter on this target, and we'd rather filter honestly after the fact than send a query param the server silently ignores.

Does a zero-result search count as a failed run?
No — an absurd price range verified as a genuine SUCCEEDED run with zero rows and a status message, not a crash.

$0.20 per run plus $0.008 per listing row — $8.20 per 1,000 results.

→ PropertyFinder UAE Property Scraper on Apify

Built by Devil Scrapes 😈. We rotate Chrome/Firefox TLS fingerprints, retry with backoff, pin the proxy exit country where the target needs it, and verify every filter against the real wire before we ship it as working.

A green QA run hid an untested pagination path. Here's the check that caught it before we shipped.

Devil Scrapes — Fri, 25 Sep 2026 08:55:32 +0000

Quick answer

Our own QA harness silently clamped maxResults from the prefill's 48 down to 3. The run passed, returned rows, and looked completely green — but it had only ever fetched page one, so pagination shipped untested behind a passing check. We caught it because the harness prints a DEPTH UNPROVEN warning when a run's actual depth falls short of its own prefill, and we fired a deliberate deep run at the prefill's own value: 48 results, charged result: 48 times, pagination proven across pages before this Actor shipped. The Realtor.ca Listings Scraper sells depth — a curated city or bounding box of normalized Canadian listings — so a capped smoke test couldn't be allowed to stand in for proof of it.

Can a "successful" QA run still hide an untested feature?

Yes, and that's the uncomfortable part — success and coverage are two different claims, and a green checkmark only speaks to the first one. A run that exits 0, writes rows to the dataset, and matches its expected schema has proven the code executes without crashing. It has not proven the code executes the path a customer will actually exercise. Here, the harness's own safety clamp — meant to keep smoke tests cheap — quietly capped maxResults at 3 instead of running the prefill's real value of 48. Three results fit on page one of Realtor.ca's api2.realtor.ca search response. The pagination logic that walks page two, page three, and beyond simply never ran, and nothing about the run's exit code said so.

What made this catchable?

A second signal that doesn't just ask "did it succeed" but "did it prove what it's supposed to prove." The harness compares the run's actual depth against the prefill's own stated depth and prints DEPTH UNPROVEN when they diverge — in this case, 3 delivered against 48 promised. That's the difference between a binary pass/fail and a coverage check: it forced the question "did this run test the feature we're charging for," not just "did it crash."

What did the deep run actually prove?

We reran at the prefill's real value — 48 results, no clamp — and the run billed result 48 times and returned distinct listings spanning multiple response pages, confirming the pagination path that page-one-only testing had never touched. A sampled row from that run: a Markham, Ontario condo listed at CAD 990,000, 5+1 bedrooms, with the agent's name and the brokerage's name and address attached — the full shape of what this Actor is built to return, not just what page one happens to contain.

A green run tells you the code ran, not that it ran the feature you are selling. If an Actor's product is depth, a capped smoke test cannot prove it.

What the Actor gives you

One normalized row per listing, deduplicated by id: mlsNumber, price, currency, transactionType, streetAddress, latitude/longitude, propertyType, bedrooms, bathroomTotal, sizeInterior, amenities, photoUrls, agentName, agentPhone, brokerageName, brokerageAddress, listingDetailUrl, plus searchCity, page, and scrapedAt tagging where and when each row was found.

Honest limitations 🚧

One search — one city or bounding box, one transaction type — per run, capped at the server's own 600-result ceiling. Residential listings only; commercial and land property groups are out of scope for v1. No agent email and no listing description text, because Realtor.ca's search response carries neither (we don't emit empty columns pretending otherwise). The curated city list covers 10 major Canadian cities; anywhere else needs an explicit bounding box.

FAQ

Do I need a Realtor.ca account or API key?
No — this reads Realtor.ca's own publicly listed search-result data.

Can I search a custom area instead of a city?
Yes — pass an explicit bounding box (latitudeMax/latitudeMin/longitudeMax/longitudeMin) instead of a city name.

How do you know pagination actually works, not just page one?
We measured it directly: a deep run at 48 results billed 48 result events and returned listings spanning multiple response pages, after our QA harness flagged a shallower run as DEPTH UNPROVEN.

$0.20 per run plus $0.003 per deduplicated listing row — $3.20 per 1,000 results. A zero-match search still succeeds and costs only the start fee.

→ Realtor.ca Listings Scraper on Apify

Built by Devil Scrapes. We rotate residential proxies pinned to Canada, retry with backoff, and — before we call a run "proven" — we check that it actually exercised the depth we're charging for.

A probe and a production scraper saw two different pages for the same missing Snapchat user

Devil Scrapes — Thu, 24 Sep 2026 09:08:36 +0000

Quick answer

A plain curl at a nonexistent Snapchat username — no browser impersonation, no proxy — returns a bare 404. The same username, fetched through the path the Snapchat Profile Scraper actually ships (impersonated Chrome/Firefox TLS, US-proxied), returns 200 with a doubly nested __NEXT_DATA__ payload whose real signal is buried at pageProps.pageProps.pageMetadata.pageType == "NOT_FOUND". Those are two different documents describing the same fact. Our classifier was originally written against the curl-shaped probe and called the production response MALFORMED instead of NOT_FOUND — a clean "this user doesn't exist" was misreported as a parse failure.

Why would the same missing username produce two different response shapes?

Because "the page your probe sees" and "the page your scraper sees" are only the same document if you fetched them the same way, and a bare curl and an impersonated, proxied browser session are not the same way. Snapchat serves a lighter, flatter error response to unauthenticated, unfingerprinted requests — a 404 status with a small body — and serves the full Next.js application shell, __NEXT_DATA__ and all, to requests that look like a real browser hitting the page. Once you're getting the full shell, the "not found" state isn't a status code anymore, it's a field three levels deep inside a JSON payload that otherwise looks exactly like a live profile's payload structure. If your test fixtures were captured with a convenient curl command during development and your classifier logic was written against that shape, it will correctly handle a probe and incorrectly handle the thing you actually ship, because production traffic never takes the path you tested.

The nesting itself is a small, specific trap: pageProps appears twice, not once. The outer pageProps is the Next.js page wrapper; the inner pageProps is where Snapchat puts the actual page data, including pageMetadata.pageType. Reading data.pageProps.pageMetadata instead of data.pageProps.pageProps.pageMetadata gets you None on every single request, live profile or not — which looks exactly like a malformed-page symptom, not a missing-key symptom, and sends you debugging the wrong layer.

What's the second finding, on top of the fixture mismatch?

userProfile in that same payload is a discriminated union keyed on a $case field, not a single fixed shape. The obvious-looking implementation assumes publicProfileInfo is always present and reads it directly — which works for the common case and silently drops every profile whose $case resolves to something else, private accounts and non-standard profile types included. Switching on $case explicitly, the way a discriminated union is meant to be consumed, is what keeps those profiles correctly classified (and reported) instead of vanishing from the run with no trace they were ever requested.

Capture your fixtures from the path you actually ship — impersonation, proxy, headers, all of it — not from whatever curl command was fastest to type while developing. A probe and a production request can return structurally different documents for the identical fact, and a classifier trained on the wrong one fails exactly where it matters most: distinguishing "doesn't exist" from "couldn't be read."

What the Actor gives you

One row per resolved username: is_public_profile, display_name, bio, website_url, subscriber_count, profile_picture_url, snapcode_image_url, has_active_story, story_snap_count, and story_snaps (per-snap media_url, media_preview_url, media_type, posted_at) when the account currently has a live story. page_title and profile_url are included for traceability. A private, removed, or nonexistent username is a per-item skip reported in the run's status message — it never fails the whole run.

Honest limitations 🚧

v1 covers the profile card and current story only — no lenses, no curated/Spotlight highlights, no tagged-tab search, no pagination. story_snaps[].media_url and media_preview_url are Snapchat CDN links that expire — Snapchat stories are ephemeral by design, so treat story media as a snapshot at scrape time, not a durable asset you can fetch later. No authenticated views, DMs, or friend-graph data — public profile pages only.

FAQ

Why did a username that clearly exists come back as not found in my own quick test?
Check how you tested it. A bare, unimpersonated request to Snapchat's profile URL can return a different response shape than the one this Actor uses in production — the production path is impersonated and proxied specifically because that's the path that reliably resolves real profiles.

Do I need a Snapchat account?
No — this reads only the public profile page at snapchat.com/@<username>, no login, no API key.

Can I rely on story_snaps media URLs after the run finishes?
Not indefinitely. They're Snapchat CDN links tied to ephemeral story content and will expire — download or re-host anything you need to keep before the story itself expires on Snapchat's side.

$0.20 per run plus ~$0.003 per resolved public profile — ~$3.00 per 1,000 profiles.

→ Snapchat Profile Scraper on Apify

Built by Devil Scrapes. We rotate browser fingerprints, pin the proxy exit the payload actually needs, and retry with backoff — and we test our classifiers against the page our own scraper sees, not the page a quick curl sees.

Per-profile or per-link: the billing unit on a link-in-bio scraper differs by 10x

Devil Scrapes — Thu, 24 Sep 2026 09:08:05 +0000

Quick answer

On a link-in-bio page, "one row per profile" and "one row per link" aren't a formatting preference — they're two different products, and they diverge by roughly 10x on a typical creator profile. A profile with 68 links bills as 1 row under the first model and 68 rows under the second. The Linktree Profile Scraper bills per link, on purpose: the link is the unit you actually analyze — "where does this creator send traffic" — not the profile.

Why does the billing unit matter more than the price-per-result headline?

Because "price per result" is meaningless until you know what a result is. A scraper billing $1/1,000 profiles and a scraper billing $4/1,000 links look like the first is 4x cheaper — until you fetch the same 68-link profile from both and realize the first one gave you one flat JSON blob you'll spend engineering time re-parsing into link-level rows yourself, while the second one already gave you 68 clean, typed, filterable rows. The per-profile scraper's "1,000 results" is 1,000 profiles, which could be tens of thousands of underlying links; the per-link scraper's "1,000 results" is 1,000 analyzable facts. Compare the cost per useful fact, not the cost per row, and the ranking can flip entirely.

We chose per-link billing for this Actor because a link-in-bio page's value is almost entirely in its outbound destinations — link type, position, group membership, gate status — not in the container profile around them. link_url, link_type, position, group_id/group_title, and gated/gate_type are all first-class columns precisely because those are the fields a link-in-bio analysis actually filters and joins on. Denormalizing the profile identity fields (username, profile_title, profile_is_verified, profile_tier) onto every link row means you never have to join back to a separate profile table to answer "which verified creators link to a specific commerce product."

What's the second finding, below the billing question?

Linktree's __NEXT_DATA__ payload — the JSON blob the page ships instead of server-rendered HTML — carries a per-link rules.gate object: age, passcode, password, or NFT. That means some links inside a single profile's payload are ones a visitor literally cannot click through without clearing a gate, and the payload doesn't mark that at the top level — it's per link, buried in rules. A scraper that emits link_url and link_title without reading rules.gate hands you a row that looks exactly like every other clickable link and isn't one. We surface that as gated (boolean) and gate_type (which gate) on every row, without attempting to clear the gate — you get an accurate picture of what a real visitor can and can't reach, not a false completeness count.

Before comparing "price per result" across two scrapers on the same target, check what a result is. A cheaper price on a coarser unit can cost more per fact you actually need, and it's the buyer's job to check the unit — a listing rarely states it plainly.

What the Actor gives you

One row per link (not per profile): link_id, link_title, link_url, link_type (CLASSIC, GROUP, YOUTUBE_VIDEO, COMMERCE_PRODUCT, and any newer type Linktree adds — never dropped), position, group_id/group_title for child-of-group links, thumbnail_url, gated/gate_type, meta_title/meta_description, and scraped_at. Every row also carries the parent profile's profile_title, profile_description, profile_avatar_url, profile_is_verified, profile_tier, and profile_has_password, denormalized so you never need a join to answer a profile-level question from a link-level table.

Honest limitations 🚧

Public profiles only — no login-required data, analytics, or edit-mode fields. Gated links are flagged, never bypassed; this Actor tells you a gate exists, it doesn't try to get past one. Group membership is a flat group_id/group_title join, not a nested tree — reconstruct hierarchy yourself if you need it. Zero-link profiles emit no rows for that profile (logged, not a failure); the run only fails if every requested profile yields zero rows.

FAQ

Why does this bill per link instead of per profile?
Because the link is the analyzable unit on a link-in-bio page — you're asking "where does this account send traffic," and a per-profile price hides how many or how few links you actually got for it.

Does this need a Linktree account or API key?
No — it reads the same public __NEXT_DATA__ JSON payload the profile page itself serves.

What happens to age/passcode/password/NFT-gated links?
They're returned with gated: true and the specific gate_type set, but the gate itself is never cleared — you know a link exists and is protected, without us attempting to see behind it.

At 1,000 link rows, that's $4.20 / 1,000 ($0.20 start + $4.00 for the rows).

→ Linktree Profile Scraper on Apify

Built by Devil Scrapes. We rotate browser fingerprints and keep the dataset clean — Pydantic-validated, ISO-8601 timestamps — so the unit you're billed on is exactly the unit you can build on.

We tested the 'always pin the proxy country' rule instead of trusting it. Same data came back from two exit countries.

Devil Scrapes — Thu, 24 Sep 2026 09:07:34 +0000

Quick answer

We tested "pin the proxy exit country on a country-scoped site" before paying for it, instead of assuming it. Same URL, same minute, two proxy tiers: a US-exit datacenter request returned 200, title "Python Jobs und Stellenangebote", totalCount: 1727, top result inovex GmbH. A RESIDENTIAL/DE-pinned request returned 200, the same title, totalCount: 1750, the same top result. Identical German job data from two different exit countries. The StepStone Jobs Scraper ships on the cheaper US datacenter pack by default, because the measurement — not the instinct — decided the proxy tier.

Doesn't a country-scoped site require a matching exit country?

Often, yes — that's a good enough prior that we start every new country-scoped target by assuming it. It's also a prior worth pricing before you commit to it, because getting it wrong costs twice. First, this account's datacenter pack (BUYPROXIES94952) is a static US-only pool — pinning apifyProxyCountry: DE against it doesn't get you German data, it fails the proxy tunnel outright, because the pack simply has no German IPs to hand out. Second, escalating to RESIDENTIAL to satisfy the pin isn't free: it's a more expensive proxy tier per request, and Apify's own review process treats an Actor that defaults to residential as belonging to the anti-bot-prone class, which comes with more publish scrutiny. So before eating either cost, we ran the actual comparison: identical query, identical minute, US-datacenter exit against RESIDENTIAL/DE exit. totalCount differed by 23 out of 1,727 — well inside normal listing churn for a live job board between two requests — and the top-ranked result was identical on both. stepstone.de does not vary its result set by the requester's exit geography. The country pin was protecting against a failure mode that doesn't exist here.

So when should you actually pin a proxy country?

When you've checked, not when you've assumed. A geo pin is a testable hypothesis about the target, not a blanket best practice — some sites genuinely do serve different prices, stock, or catalogs per exit country (we've shipped Actors where that's exactly the finding), and some just don't vary at all. The only way to know which one you're looking at is to run the same query through two tiers and diff the response. Guessing wrong costs you either a failed tunnel (wrong pack, right instinct) or a needlessly expensive and more heavily scrutinized default (right instinct, unnecessary cost) — two request's worth of measurement is cheaper than either.

Below the proxy question, the parsing question had its own thing worth stating precisely: StepStone's job data lives in window.__PRELOADED_STATE__["app-unifiedResultlist"], not in server-rendered HTML you'd need to reconstruct from DOM structure. And pagination distinctness — the thing that actually protects your dataset from silent duplicate rows across pages — was verified directly: page 1 and page 2 share zero job ids, page 2 and page 3 share zero job ids.

A proxy country pin is a hypothesis about the target, and testing it costs one extra request. Skipping the test costs you either a failed tunnel against a pack that can't serve the country, or an unnecessarily expensive and more heavily scrutinized tier protecting against a difference that isn't there.

What the Actor gives you

One deduplicated row per job listing across one or more keywords: id (dedupe key), title, companyName, companyId, location, datePosted (ISO-8601), url, salary when published, isSponsored, isHighlighted, isAnonymous, postCode, textSnippet, skills, labels, plus sourceKeyword / sourceLocation tagging which input search found each row. Run-wide dedup means the same job id never appears twice, even across pages or across multiple keywords batched into one run.

Honest limitations 🚧

List-page fields only — full job-description text isn't fetched; use the returned url for a follow-up detail-page job if you need it. .de only in this version; .at and .ch StepStone variants aren't covered. A narrow keyword + location combination can legitimately return zero rows, and that's a successful run, not a failure.

FAQ

Do you pin the proxy exit country to Germany by default?
No. We measured it: US-datacenter and RESIDENTIAL/DE returned the same totalCount, the same top result, and the same title for an identical query. stepstone.de doesn't vary by exit geo, so the default ships on the cheaper, less-scrutinized US datacenter pack.

Do I need a StepStone account or API key?
No — this reads StepStone's public search result pages.

Can I search multiple keywords in one run?
Yes — pass an array to keywords; every row is tagged with the keyword (and location) that produced it, and dedup applies across the whole run, not per keyword.

$0.20 per run plus $0.003 per deduplicated job row — $3.20 per 1,000 results. A zero-match search still succeeds and costs only the start fee.

→ StepStone Jobs Scraper on Apify

Built by Devil Scrapes. We rotate browser fingerprints, retry with backoff, and route every request through Apify Proxy — and before we spend your money on a more expensive proxy tier, we measure whether the target actually needs it.

nbHits said 6,360. The API only let us page through 1,000 of them.

Devil Scrapes — Thu, 24 Sep 2026 09:07:03 +0000

Quick answer

Querying Welcome to the Jungle's search index with no filter reports nbHits: 6360 in the response envelope — then refuses to page past result 1,000. Page 40 of that same query comes back HTTP 200 with a body that reads {"message": "you can only fetch the 1000 hits for this query"}. That's not a rate limit and it's not a block; it's Algolia's hard pagination ceiling, and it's baked into every Algolia-backed site, not just this one. The Welcome to the Jungle Jobs Scraper treats that ceiling as a first-class fact: it surfaces it in the run's status message and gives you query, country, contractType, and remote as facets so you can slice a 6,360-hit corpus into several sub-1,000 queries instead of silently losing 84% of it.

Why does a 200 response contain an error?

Because Algolia's own index — the one Welcome to the Jungle's frontend calls directly from the browser — enforces a 1,000-result cap per query on the free/standard tier, and it enforces it at the pagination layer, not the response status layer. The first page of any query looks completely healthy: 200, real JSON, a nbHits field truthfully reporting the full match count, and 20 or so listings in hits. Nothing about that first page tells you the query has more matches than the index will ever hand you. Only when you request page 40+ (at 25 hits/page, that's the 1,000-result boundary) does the response change shape — still 200, but now the payload is the error: you can only fetch the 1000 hits for this query.

A scraper that treats "response received, parse hits" as its only success condition will parse that error body as if it were a results page, get an empty or malformed hits array, and stop. The run finishes green. The dataset looks complete because nothing in the run log says otherwise — it just quietly tops out at 1,000 rows on any query broad enough to have more. That's a worse failure than a crash: a crash gets investigated, a silently-truncated dataset gets shipped to a customer's pipeline and trusted.

Is the ceiling really about the search index and not the target blocking us?

Yes, and the distinction matters because they call for opposite fixes. A block calls for better fingerprinting, proxy rotation, or backoff. A 1,000-hit ceiling calls for narrower queries. We confirmed which one we were looking at by checking the same query with a tighter facet: narrowing by country=FR plus a keyword cut nbHits from 6,360 to under 1,000, and paging that narrower query all the way through returned every result with no error body — same site, same session, same browser fingerprint, no block anywhere in the run. The ceiling moved when the query got smaller; a block wouldn't have.

The second finding sits one layer below the pagination cap: the endpoint gates on the request's Origin / Referer header, not on the User-Agent or TLS fingerprint. A request with no origin header returns 403 {"message":"Method not allowed with this referer"} regardless of which browser you're impersonating. Both chrome131 and firefox133 impersonation profiles pass cleanly once the header is set to the site's own origin — the fingerprint was never the gate here, the header was. Worth knowing before you spend a debugging session rotating TLS profiles against a header check.

A 200 status code tells you the transport succeeded, not that the payload is a results page. On any Algolia-backed search, check whether page N+1 still parses as a hit list before you trust that a broad query returned everything it claimed to have.

What the Actor gives you

One row per job listing, with the site's own facets available as scrape-time filters (not just export-time ones): objectId, title, slug, jobUrl, companyName, companySlug, offices (city/state/country per office), contractType + contractTypeLabel, remote policy, publishedAt (ISO-8601), salary range and currency when the recruiter published one, sectors, profession, and department. Set query, country, contractType, or remote — alone or combined — to keep any single search's true match count under the 1,000-hit ceiling.

Honest limitations 🚧

The 1,000-hit-per-query cap is real and not something we bypass — you narrow around it with facets, you don't get past it with retries or a different proxy. Full job-page HTML (benefits copy, application forms) isn't included; the search index already carries everything relevant to filtering and ranking, and a detail-page fetch would be a different, heavier Actor. Company-profile enrichment (reviews, culture pages) is out of scope for v1.

FAQ

Why did my broad search return exactly 1,000 rows when the listing count looked higher?
You hit the Algolia index's pagination ceiling. Split the query by country, contractType, or remote and run it again — each narrower query gets its own 1,000-hit budget.

Does this need a login or API key?
No — listings are public, and we call the same index endpoint the site's own frontend calls.

Why did my run return zero rows instead of failing?
A query + filter combination with no matches is a successful search, not a broken one. The run status message says exactly what was searched.

$0.20 per run plus $0.0025 per result — $2.70 per 1,000 results. A zero-match search still succeeds and costs only the start fee.

→ Welcome to the Jungle Jobs Scraper on Apify

Built by Devil Scrapes. We rotate browser fingerprints, retry with backoff, and route every request through Apify Proxy — and when a target's own search index caps what any single query can return, we tell you in the run status instead of shipping you a quietly truncated dataset.

Filter the CFTC COT report for WHEAT and you get three different markets

Devil Scrapes — Thu, 24 Sep 2026 03:32:39 +0000

Quick answer

Ask the CFTC's Commitments of Traders data for commodity_name = 'WHEAT' on the 2026-09-15 report and you don't get one market back. You get three: soft red winter wheat on the Chicago Board of Trade, hard red winter wheat on the same exchange, and hard red spring wheat on the MIAX Futures Exchange. They are different contracts with different open interest and different traders, and speculators can be positioned in opposite directions in them. Add them up and you get a number that describes no real market.

The CFTC Commitments of Traders Scraper pulls the COT reports straight from the CFTC's own public data portal. Most of the engineering is in the dull parts. Most of what you need to know to use the data well is in that one gotcha.

Why does one commodity filter return several markets?

Because commodity_name is a family, not a contract. In the CFTC's legacy dataset, the wheat contracts all carry commodity_name: "WHEAT". The field that tells them apart is market_and_exchange_names (WHEAT-SRW - CHICAGO BOARD OF TRADE), and the one that identifies a contract for good is cftc_contract_market_code (001602 for SRW, 001612 for HRW).

Our own cloud QA run shows why this matters. It used the Actor's default input (wheat, Chicago Board of Trade, report dates 2026-06-30 to 2026-09-15) and returned 24 rows: 12 weekly reports × 2 contracts. Speculator net positioning (non-commercial long minus short), straight from those rows:

report date   contract          open interest   speculator net
2026-06-30    WHEAT-SRW (CBOT)        406,723          -55,004
2026-06-30    WHEAT-HRW (CBOT)        260,913          -10,339
2026-09-15    WHEAT-SRW (CBOT)        485,138           +1,228
2026-09-15    WHEAT-HRW (CBOT)        310,836          +30,335

Speculators ended the window net long in both contracts, but by very different amounts. HRW moved to +30,335 while SRW barely crossed zero. A chart summed on commodity_name blurs those two into one line. Group on cftc_contract_market_code and you keep them apart.

So the Actor's commodity input is an exact match on the CFTC's family name, and market is a substring match on the market-and-exchange name. You can use market to narrow a family down to the contract you actually trade.

What does the raw CFTC API actually hand you?

Strings. This is a real row from the CFTC's Socrata endpoint:

{
  "id": "260630001612F",
  "commodity_name": "WHEAT",
  "open_interest_all": "260913",
  "noncomm_positions_long_all": "66398",
  "report_date_as_yyyy_mm_dd": "2026-06-30T00:00:00.000"
}

Every position count is a JSON string, and the field named ..._yyyy_mm_dd holds a full timestamp. Both are small problems, and both break a spreadsheet or a pandas pipeline quietly: string columns sort lexically, and "66398" > "100000". The Actor coerces every position field to a number and trims the date to real YYYY-MM-DD. If a value won't parse, it's set to null and a warning goes in the log. We don't guess a number for you.

The row id isn't random, either. 260630001612F is the report date (260630), the contract code (001612), and F for futures-only. It's stable, so you can dedupe on it when you re-pull overlapping windows.

How much history is there?

A lot. When we queried the legacy futures-only dataset it held 289,274 rows, running from 1986-01-15 to the 2026-09-15 report. The Actor covers all six COT variants: Legacy, Disaggregated, and Traders in Financial Futures, each as futures-only or futures-and-options-combined. You pick one with the reportType field. A second cloud run backfilled wheat from 2024 onward, returning 100 rows with 100 distinct IDs: 50 consecutive weekly reports for each of the two CBOT contracts.

What the Actor gives you

One row per market per report date, with the Socrata id, report_type, market-and-exchange name, report date, commodity name, contract market code, total open interest, non-commercial and commercial long and short positions, week-over-week change in open interest, and total trader count. You can filter by report type, commodity, market, and date range. There's no API key. We handle the pagination, retries with backoff on 408 / 429 / 503, and the type coercion, so the dataset exports to JSON, CSV, or Excel ready to chart.

Honest limitations 🚧

A capped run returns the oldest weeks, not the newest. The wheat backfill capped at 100 rows over 2024-01-01 to 2026-09-15 returned January through December 2024. To get this week's report, set dateFrom close to today rather than relying on the cap.
pct_of_open_interest_all is always 100 in the legacy dataset. We checked all 289,274 rows and it held one value. It's passed through for completeness, but it tells you nothing.
The CFTC's schedule is the Actor's schedule. Reports come out weekly and describe the prior Tuesday's positions. The Actor can't return a report before the CFTC publishes it.
Raw positions only. The Actor doesn't compute net positioning, COT index, or z-scores. Those take one subtraction or a rolling window downstream.
Use the Legacy report for position fields. The v1 row shape follows the Legacy report. The Disaggregated and TFF datasets split traders into different categories (managed money, swap dealers, leveraged funds and so on) and don't publish the Legacy commercial/non-commercial columns. On those report types, you get open interest, the week-over-week change, and trader count, but the long/short position fields come back null.

FAQ

Do I need a CFTC or Socrata API key?
No. The CFTC's public reporting portal is keyless open data.

What does it cost?
$3.20 per 1,000 rows under Pay-Per-Event: a $0.20 start fee plus $0.003 per row written to your dataset. A year of weekly reports for one contract is 52 rows.

Why did my WHEAT query return more markets than I expected?
Because WHEAT is a family of contracts. Add a market filter, or group on cftc_contract_market_code.

What if my filters match nothing?
The run succeeds with zero rows and a status message listing the filters it used, and you're charged only the start fee. If the CFTC portal can't be reached at all, the run fails loudly instead of returning a quiet, empty success.

→ CFTC Commitments of Traders Scraper on Apify

Built by Devil Scrapes. We handle the paging, retries and type coercion, so the numbers in your dataset are numbers.

Our two-state UCC run returned 200 rows, 127 filings, and zero from Colorado

Devil Scrapes — Thu, 24 Sep 2026 03:32:08 +0000

Quick answer

Before the UCC Lien Filing Scraper ever went public, a deep cloud run asked it for Connecticut and Colorado filings from the last 120 days, capped at 200 rows. The run came back SUCCEEDED with 200 rows. Every one of them was from Connecticut. Colorado — which had real filings in that exact window — contributed zero. And those 200 rows described only 127 distinct filings.

Both numbers were telling us something. One was a bug. The other was the data being honest about its own shape, and the fix there was to our copy, not our code.

How does a two-state run return rows from only one state?

By handing the whole budget to whoever asks first. The first version of the run loop gave the entire remaining maxResults to the first state it fetched. Connecticut went first, had well over 200 matching records in 120 days, and spent the whole cap before Colorado was ever queried.

Nothing failed. There was no error to log. The run did exactly what it was told, and the result looked like "Colorado had nothing this quarter" — which is a plausible-sounding lie. We only caught it because we checked Colorado's portal directly for the same window and it returned real filings. Running the Connecticut fetch alone with a budget of 200 then reproduced the exact 200-rows / 127-filings result, which pinned it on the cap, not on a broken Colorado adapter.

The fix: maxResults is now split evenly across the states you request, up front, and any share a state doesn't use rolls over to the next one so the cap is never wasted. The same failing input now returns 100 Connecticut rows and 100 Colorado rows. Our post-fix cloud QA runs (daysBack: 14, maxResults: 25) landed 13 CT and 12 CO rows each time.

Why were 200 rows only 127 filings?

Because a UCC filing can name more than one debtor, and Connecticut publishes one record per debtor per filing. We checked filing 0005384782 against data.ct.gov by hand: four genuine source records, same filing number, four different debtors. In that same run, one filing — 0005384859 — accounted for 43 rows on its own.

Colorado was the side out of step. Its adapter joined each filing to its debtor dataset and kept only the first debtor it saw, silently discarding the rest. So the two states disagreed on what a row even meant.

We had a choice: collapse Connecticut into one row per filing with a debtor list, or expand Colorado to one row per debtor. We expanded Colorado. Connecticut's native shape is per-debtor, so aggregating would mean inventing a structure the source doesn't publish. And the people who buy this data — lenders, collections shops, lead-gen teams — want each debtor as its own row with its own address, ready to import as a contact.

That decision is visible on your bill, so we put it next to the price rather than in a footnote: a row is a filing–debtor pair. A filing with three debtors is three rows and three result events. Rows that share a filing_id are the same lien.

Why does Colorado need a three-way join?

Connecticut ships one flat dataset. Colorado splits the same information across three linked Socrata datasets — a filings index, a debtor table, and a secured-party table — joined on fileid. The filings index alone held 2,594,585 rows when we measured it, so an unbounded pull is never acceptable. Every query is date-bounded server-side (daysBack, or dateFrom/dateTo) and paged.

Here is a real Colorado row from a cloud run, after the join:

{
  "state": "CO",
  "filing_id": "2594901",
  "filing_date": "2026-09-08T00:00:00.000",
  "lapse_date": "2031-09-08T00:00:00.000",
  "filing_type": "ucc",
  "filing_description": "UCC financing statement",
  "transaction_type": "original",
  "debtor_address": "5045 BEACH CT",
  "debtor_city": "DENVER",
  "secured_party_name": "GoodLeap, LLC",
  "secured_party_state": "CA",
  "status": "active"
}

Some Colorado filings join to nothing — a termination amendment we sampled (2595634) had no debtor or secured-party rows at all. Those still ship, with the missing fields set to null, instead of one empty join sinking the whole run.

What the Actor gives you

One normalized row per filing–debtor pair across Connecticut and Colorado in a single run: state, filing_id, filing_date, lapse_date, filing_type, transaction_type / is_amendment, debtor name and address, secured party name and address, a derived status (active, lapsed, terminated, unknown), and a source_record_url that queries the state portal for that exact filing. You can filter by debtor name with debtorNameContains. There is no API key and no login. We handle the pacing, retries with backoff on 408 / 429 / 5xx, and the join between the two states' schemas, so what you get is one dataset rather than two government schemas to reconcile.

Honest limitations 🚧

Two states only. Connecticut and Colorado. It is not a national UCC search. If a debtor only filed in a third state, you correctly get zero rows.
Colorado individual debtors come back without a name. Colorado stores business debtors in organizationname and people in separate firstname / lastname fields, and v1 reads only the first. In our two post-fix QA samples, 6 of 12 and 8 of 12 Colorado rows had an empty debtor_name. The address is still there and business debtors are unaffected. We found this while writing this post and it's the next fix on the list.
Rows are not sorted newest-first. In both 14-day QA runs, every row was dated the first day of the window. If maxResults is smaller than the window, you get the oldest filings in it. To catch fresh filings, shrink daysBack; don't just lower the cap.
Dates are Socrata timestamps (2026-09-08T00:00:00.000), not bare YYYY-MM-DD.
UCC filings only. Colorado's IRS and hospital lien types are out of scope, and so are scanned UCC-1 images and collateral text.

FAQ

Do I need an API key or a state portal login?
No. Both states publish this as public, keyless open data.

What does it cost?
$5.20 per 1,000 rows under Pay-Per-Event: a $0.20 start fee plus $0.005 per row written to your dataset. A row is a filing–debtor pair, so budget by debtors, not filings.

Can I monitor new liens against a specific business?
Yes. Set debtorNameContains and a short daysBack, then run it on a schedule. A new financing statement against a business means it just took on secured credit.

What if my search matches nothing?
The run succeeds with zero rows and a status message saying what was searched. An empty result is still an answer.

→ UCC Lien Filing Scraper on Apify

Built by Devil Scrapes. We handle the pacing, retries and cross-state schema joins, and when a run returns something that looks plausible, we check it against the source before we believe it.

DEV Community: Devil Scrapes

Bayt Jobs Scraper: the proxy country pin that was the block

Quick answer

Shouldn't a proxy match the country you're scraping?

How do you tell "the pin is the block" apart from "the site blocks us"?

What the Actor gives you

Honest limitations 🚧

FAQ

InfoJobs Spain Scraper: the proxy default that 407'd every run

Quick answer

If you pin a proxy country, do you also need to name the group?

Did the shipped defaults actually get exercised in the cloud before going live?

What the Actor gives you

Honest limitations 🚧

FAQ

Fotocasa Scraper: the null field that hid every discounted listing

Quick answer

Can you infer a field's type from a null sample?

Does this scraper need a browser to run?

What the Actor gives you

Honest limitations 🚧

FAQ

XING's job data ships as a JS object, not JSON. Here's how we parse it without breaking on the first undefined.

Quick answer

Isn't window.crate={...} just JSON you can json.loads?

Where does the company name actually come from?

Why is salaryMedian sparser than salaryMin/salaryMax?

What the Actor gives you

Honest limitations 🚧

FAQ

PropertyFinder.ae's category parameter wasn't what the spec assumed. We tested it before shipping.

Quick answer

Does PropertyFinder.ae's c= parameter just mean residential vs commercial?

What happens when you ask for a bedroom filter that doesn't exist on the wire?

What the Actor gives you

Honest limitations 🚧

FAQ

A green QA run hid an untested pagination path. Here's the check that caught it before we shipped.

Quick answer

Can a "successful" QA run still hide an untested feature?

What made this catchable?

What did the deep run actually prove?

What the Actor gives you

Honest limitations 🚧

FAQ

A probe and a production scraper saw two different pages for the same missing Snapchat user

Quick answer

Why would the same missing username produce two different response shapes?

What's the second finding, on top of the fixture mismatch?

What the Actor gives you

Honest limitations 🚧

FAQ

Per-profile or per-link: the billing unit on a link-in-bio scraper differs by 10x

Quick answer

Why does the billing unit matter more than the price-per-result headline?

What's the second finding, below the billing question?

What the Actor gives you

Honest limitations 🚧

FAQ

We tested the 'always pin the proxy country' rule instead of trusting it. Same data came back from two exit countries.

Quick answer

Doesn't a country-scoped site require a matching exit country?

So when should you actually pin a proxy country?

What the Actor gives you

Honest limitations 🚧

FAQ

nbHits said 6,360. The API only let us page through 1,000 of them.

Quick answer

Why does a 200 response contain an error?

Is the ceiling really about the search index and not the target blocking us?

What the Actor gives you

Honest limitations 🚧

FAQ

Filter the CFTC COT report for WHEAT and you get three different markets

Quick answer

Why does one commodity filter return several markets?

What does the raw CFTC API actually hand you?

How much history is there?

What the Actor gives you

Honest limitations 🚧

Can you infer a field's type from a `null` sample?

Isn't `window.crate={...}` just JSON you can `json.loads`?

Why is `salaryMedian` sparser than `salaryMin`/`salaryMax`?

Does PropertyFinder.ae's `c=` parameter just mean residential vs commercial?