Quick answer
A browser-based scraper has two completely different ways to not get a page: the site refuses you (429/403 — a real bot wall), or the proxy tunnel dies before the request ever leaves your machine (NS_ERROR_PROXY_CONNECTION_REFUSED from Page.goto). Both surface as "the navigation failed." If you spend one shared retry budget on both, the transport faults eat the attempts the bot wall needed, every failure gets reported as "blocked," and you tune the wrong knob for a week. Splitting the budget in two took the Manta Business Directory Scraper from a 60% success rate to 5 out of 5, at $1.25 per 1,000 results all-in.
Why does one retry budget hide the actual failure? 🧱
The retry loop looked reasonable. Four attempts, exponential backoff, and on exhaustion it raised "bot-wall exhausted retries." Measured properly across real runs it was 3 green and 2 failed — roughly a 40% failure rate.
Reading the actual run logs rather than reasoning about the code, the eight failed attempts split into two populations that have nothing to do with each other:
-
Manta answered.
429or403from manta.com. That is a genuine anti-bot response, and the right reaction is to back off and come back with a different exit IP and a fresh browser page. -
Manta never heard from us.
Page.goto: NS_ERROR_PROXY_CONNECTION_REFUSED— the residential proxy CONNECT tunnel refused before any request was issued. The site has no idea we exist. The right reaction is to grab a different tunnel and retry almost immediately, because nothing about our fingerprint is implicated.
Five of the eight failed attempts were the second class. They were consuming a budget sized for the first. Both failing runs died on page 1, which also killed the maxResults theory that had been the leading suspect — the run never got far enough for a result cap to matter.
And the diagnostic was actively lying. Whichever class ran out, the error message said "bot-wall exhausted retries," so every log read like an anti-bot problem. Along the way a geoip=True browser LeakWarning got investigated as a suspect and exonerated: the failures happen at the proxy CONNECT layer, before a single fingerprint is ever presented. It was a real warning about a non-problem.
What does the fix actually look like? 🔧
Two independent budgets instead of one:
- Bot-wall budget: 4 attempts. Slow backoff, fresh page and fresh exit IP each time.
- Transport budget: 6 attempts. The 6 is not a guess — it is tuned from a measured ~67% per-attempt transport-fault rate on this proxy path. At that rate, four attempts is not enough to be confident you tried.
And the failure message now names which budget ran out. That one line is worth as much as the retry change: the next person to read a failed run learns whether the site refused them or the plumbing did, instead of inheriting the same wrong assumption.
Result on the rebuilt Actor: 5 runs out of 5 SUCCEEDED, 145 rows, 0 of which failed the manta.com/c/ URL-shape check, and 120 unique listing URLs on a scale run. Residential proxy usage confirmed from the platform's own billing record, not from the config we asked for — requesting a proxy tier and actually using one are different things, and a degraded helper falls back to a direct connection silently.
Why check the URL shape at all? 🕵️
Because "the run succeeded and produced rows" is not proof that the rows are real. A blocked-but-200 page can feed generic fallback selectors nav links and page furniture, and if your only validation is "has a name and a link," that garbage passes. We have had exactly that happen on another Actor in this fleet: a SUCCEEDED run, three rows, every one fabricated from page chrome.
So the delivery check here asserts something only a genuine Manta listing can satisfy — every row's listing_url must match Manta's real /c/<id>/<slug> listing shape. 145 rows, 0 failures. That is the difference between "rows exist" and "rows are businesses."
What you get per row
| Field | Example |
|---|---|
business_name |
Austin Local Plumbers Expert Team |
phone |
(888) 388-6407 |
street_address / city / state
|
600 N Lamar Blvd / Austin / TX
|
category / sic_code
|
firmographic classification |
employee_count_estimate / revenue_estimate
|
size signal most directory scrapers drop |
website / year_established
|
|
listing_url |
https://www.manta.com/c/m1x0tvw/... |
Search by category or keyword plus a city or state. Export as JSON, CSV or Excel.
FAQ
What does it cost?
$0.05 per run start plus $0.0012 per row — about $1.25 for 1,000 businesses. Measured platform cost on a 120-row residential run was ~$0.76 per 1,000 rows, so the price is set against a real measurement rather than a guess. Tiny test runs are the reason the start fee is $0.05 and not $0.01: at a penny, a 25-row run billed $0.04 against ~$0.05 of cost, which is below cost.
Do I need a proxy or an account?
No. Proxy rotation, the browser engine, retries and backoff are all handled inside the Actor.
Why do so many listings share a phone number?
That is Manta, not the scraper. Its directory carries a lot of aggregator-published SMB listings that share a call-tracking number. Each row still resolves to its own distinct listing URL, which is why the URL-shape check is the honest delivery test.
Will it fail if a search finds nothing?
No. A search that legitimately matches zero businesses finishes SUCCEEDED with a status message naming exactly what was searched. Only real errors fail.
Built by Devil Scrapes. We do the dirty work so your dataset stays clean. 😈
Top comments (0)