DEV Community

Devil Scrapes
Devil Scrapes

Posted on

A note said the run scraped 0 rows. The platform's own record said 6.

Quick answer

A note in this repo declared the FINRA BrokerCheck Scraper unpublishable: its own shipped prefill, run verbatim, "SUCCEEDS with 0 rows." That's a serious defect if true — a customer running the exact default input would get a clean, billed, empty run. When we went back to fix it, we pulled the actual platform record for the cited run instead of trusting the note. GET /v2/actor-runs/dTWAgCsoPf6DgFkP0 came back status: SUCCEEDED, statusMessage: "Processed 2 queries, skipped 0 due to transport failures; emitted 6 row(s)", chargedEventCounts: {"result-emitted": 6}. The dataset had itemCount: 6, and the six items were real BrokerCheck records for CRD 5998211 and "john smith." The Actor was fine. The note was wrong, and it sat unchallenged for weeks.

How does a wrong finding survive that long?

Because a written verdict is easy to trust and expensive to re-check, and once one agent files "0 rows" against a run ID, every downstream decision treats that as settled fact rather than a claim worth re-opening. The original note presumably reflected something real at the moment it was written — maybe a genuine platform blip, maybe a mis-cited run ID — but nothing forced a second look before the Actor got shelved on the strength of it. A cached verdict is not the same thing as the primary record, and the two only diverge when someone actually goes back to the source.

Re-checking took two independent paths, and both disagreed with the note. First, the platform's own run record for the exact ID the finding cited — status, charged events, and dataset item count, pulled straight from the API, not summarized secondhand. Second, a fresh local apify run against the live FINRA endpoint with the unmodified pre-fix code and the identical prefill: 6 rows back in about 9 seconds. Same build, same input, same result both times, and neither matched "0 rows."

So what was actually wrong with it?

Something real, just not that. proxyConfiguration shipped as a bare {"useApifyProxy": true} — no group named — which on this account silently resolves to the datacenter pack rather than the residential one the build brief called for. The platform's own run logs don't echo proxyConfiguration back, so this couldn't even be confirmed from a log read; it had to be traced from the input schema itself. That's now fixed: apifyProxyGroups: ["BUYPROXIES94952"] and apifyProxyCountry: "US" are pinned explicitly in the default, the prefill, and the QA fixture, with a regression test guarding against a silent revert to the bare form.

We also added a guard the original finding's concern deserved even though its specific claim didn't hold up: a whole-run zero-rows check. If every query in a run gets a real HTTP answer but the run as a whole produces zero rows, it now fails loud rather than reporting a clean SUCCEEDED — a deliberate narrowing from this codebase's usual "empty search still succeeds" default, because a batch of CRD numbers or names that all miss is far more likely to be a customer typo than a real "no such broker exists."

A finding that says "this run returned 0 rows" is a claim about that run, not a fact about the Actor — and the fastest way to know which one you're looking at is to pull the platform's own record for the run ID, not the summary of it.

What the Actor gives you

One merged row per broker or firm hit — search by name or by CRD number, mixed freely in the same list, with purely numeric entries resolved as direct CRD lookups and everything else searched. Each row carries registration scope, disclosure flag, current and previous employments, registered states and SROs, exam counts, and — when fetchFullDetail is on — the full disciplinary disclosure history merged onto the same row instead of left on a separate detail page you'd otherwise have to fetch yourself.

Honest limitations 🚧

This is a lookup-and-merge tool against FINRA's own public API, not a discovery engine — you supply names or CRD numbers, it doesn't enumerate brokers on its own. A name search that matches nothing for a given query still succeeds for that query (only a whole-run zero-match batch fails loud).

FAQ

Do I need a FINRA account or API key?
No — this uses FINRA BrokerCheck's public, keyless JSON API, the same data backing brokercheck.finra.org.

Can I mix broker names and CRD numbers in one run?
Yes. Purely numeric entries are fetched directly by CRD; everything else is name-searched.

What happens if one name in my batch doesn't match anything?
That query succeeds with zero rows and is named in the run's status message. Only a batch where every single query comes back empty fails loud.

Does the disclosure history come from a separate lookup?
Not one you have to make yourself — with fetchFullDetail on, disclosures, employments, and registration counts are merged from the CRD detail endpoint onto the same row as the search hit.

Pricing

$0.20 per run plus $0.0028 per broker/firm row — $3.00 per 1,000 results. A run that matches nothing costs only the start fee.

FINRA BrokerCheck Scraper on Apify


Built by Devil Scrapes. We rotate fingerprints, retry with backoff, and — as this one shows — we re-check the primary record before we trust a verdict about our own Actor.

Top comments (0)