If you want to scrape Yellow Pages business listings by city and keyword, the actual blocker isn't finding the data — it's getting past the front door. yellowpages.com sits behind an anti-bot layer that fingerprints your client at the TLS handshake, and it doesn't treat every browser fingerprint the same way. I found that out the hard way while building a search-results scraper for it, and the fix ended up being more interesting than the scraper itself.
The setup: a directory site everyone assumes is "easy"
Yellow Pages is one of those targets that looks trivial on paper. Public search-results pages, no login wall, no infinite-scroll JavaScript rendering — just GET /search?search_terms=plumbers&geo_location_terms=Austin,+TX and parse the HTML. A quick look at the Apify Store confirms plenty of people have tried: there's a crowded field of community scrapers for this exact site, including at least one with well over 100,000 lifetime runs.
So I started where you'd expect: a plain HTTP client, a standard set of headers, and a request to the search endpoint. Immediate 403. Tried again with a different, more "modern" browser User-Agent string. Still 403. That's the tell that you're not looking at a missing header or a malformed query string — you're looking at a fingerprinting layer that's inspecting something below the HTTP layer entirely.
What the data looks like once you're through
Before getting into the fingerprinting story, here's the target: a single row per business listing on a search-results page.
{
"business_name": "ARS / Rescue Rooter",
"phone": "(512) 837-9500",
"street_address": "1500 W Anderson Ln",
"city": "Austin",
"state": "TX",
"zip_code": "78757",
"categories": ["Plumbers", "Air Conditioning Contractors & Systems", "Heating Contractors & Specialties"],
"website": "https://www.ars.com/austin",
"rating": 4.5,
"review_count": 128,
"listing_url": "https://www.yellowpages.com/austin-tx/mip/ars-rescue-rooter-473194899",
"search_term": "plumbers",
"location_query": "Austin, TX",
"scraped_at": "2026-07-30T12:00:00Z"
}
Name, phone, structured address (not one opaque blob), category list, website, a parsed star rating, review count, and the canonical listing URL. Straightforward, if you can get the HTML in the first place.
The naive approach, and why it falls apart
Most people scraping a directory site reach for requests or httpx, set a User-Agent header that says "Chrome," and call it a day. That works against sites with no anti-bot layer. It does not work here.
The reason is that a User-Agent header is just a string — trivial to fake, and every anti-bot vendor knows it. What's much harder to fake is the shape of the TLS handshake itself: the cipher suite order, the TLS extension list, the ALPN negotiation, the HTTP/2 SETTINGS frame. Real Chrome, real Firefox, and real Safari each produce a distinctive, consistent fingerprint at that layer — commonly called a JA3/JA3S or JA4 fingerprint — regardless of what User-Agent string you bolt on top. Python's default TLS stack produces its own distinctive (and instantly recognizable) fingerprint, no matter what headers you set on top of it. If the target is inspecting the handshake rather than just the headers, spoofing the User-Agent string does nothing.
The actual finding: not all impersonation profiles are equal
Once I understood that, the fix seemed obvious: use a library that replays a real browser's TLS fingerprint, not just its headers. I reached for curl-cffi, which does exactly that — it impersonates specific, named browser builds at the TLS/HTTP2 layer.
Here's where it got genuinely interesting. I ran the same search request through four different impersonation profiles from the same machine, same IP, same everything else:
-
chrome131→403 -
chrome124→403 -
safari180→403 -
firefox147→ clean200, full 294 KB of result HTML
Three out of four "real browser" fingerprints still got blocked. Only the Firefox profile cleared. That's not a fluke you'd catch by assuming "impersonate a browser" is a solved problem and moving on — it's a live, empirical finding you only get by testing multiple profiles against the actual target and watching what comes back. Anti-bot rules get retuned regularly, which means a fingerprint profile that clears today can get flagged next month. This is exactly why we don't hard-code a single profile and hope: our scraper biases toward the profile that verifiably clears, then rotates and retries with backoff the moment it sees a block, instead of assuming yesterday's answer still holds.
The Actor
This is the part where I stop describing the problem and show the fix in production. We built Yellow Pages Business Directory Scraper to handle exactly this: search by keyword and city, get back structured rows, without babysitting fingerprint rotation yourself.
We rotate browser fingerprints across requests, retry with exponential backoff on 408/429/5xx, and rotate proxy sessions on every block — fresh session, fresh exit IP, before the next attempt. If the target still stops the run partway through, it reports exactly how many rows it collected before stopping. No silent empty dataset with a green checkmark next to it.
Run it via the Apify Python SDK:
from apify_client import ApifyClient
client = ApifyClient("APIFY_TOKEN")
run = client.actor("DevilScrapes/yellow-pages-business-scraper").call(
run_input={
"searchTerm": "plumbers",
"location": "Austin, TX",
"maxResults": 300,
}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item["business_name"], item["phone"], item["rating"])
Or paste the same shape straight into the Console:
{
"searchTerm": "plumbers",
"location": "Austin, TX",
"maxResults": 300,
"proxyConfiguration": { "useApifyProxy": true }
}
Rows stream into the run's dataset as they're found — export as JSON, CSV, or Excel, or pull them via the API into whatever CRM or spreadsheet your pipeline feeds.
What you'd actually use this for
- Call/email list building — pull every plumber, roofer, or dentist in a target city into an outreach list, ready for your own SDR sequence.
- Local-market research — compare business density and category mix across cities before opening a new location or committing ad spend.
- CRM enrichment feeds — ingest directory data as a structured input for a broader prospecting pipeline instead of hand-copying listings.
- Competitive scans — see how many competitors in a category operate in a given city, and how they stack up on rating and review count.
Pricing — the actual numbers
Pay-per-event: $0.01 flat per run (the warm-up charge) plus $0.0012 per business listing row written to the dataset. A 1,000-row scrape costs $1.21 total. No subscription, no minimum spend. Apify gives every new account $5 of free trial credit, no card required — that covers roughly 4,000 rows before you spend a cent of your own money.
Honest limitations
v1 covers yellowpages.com's US search-results pages — not individual business detail/profile pages, not yellowpages.ca or any other country's directory, and it doesn't geocode or validate the location input against a real gazetteer (it checks the "City, ST" shape, not whether that city actually exists). The website field is captured verbatim, exactly as Yellow Pages lists it — sometimes that's a tracked redirect or a micro-site rather than the business's own domain, and we don't try to silently "fix" that. Each run is a fresh scrape; there's no cross-run deduplication built in yet.
Try it
Live on the Apify Store: apify.com/DevilScrapes/yellow-pages-business-scraper. Free $5 trial credit, no card required to start.
If you've hit a similar fingerprinting wall on a different directory site, I'd genuinely like to hear which profile cleared it for you — drop it in the comments.
Further reading:
- curl-cffi documentation — the browser-impersonation library referenced above.
- Apify SDK for Python docs — how Actors, datasets, and proxy configuration fit together.
- How TLS fingerprinting (JA3/JA4) works — background on the handshake-level signal anti-bot systems key off.
Top comments (0)