Clutch.co is one of the richest public directories of B2B service providers out there. Agencies for development, marketing, design, and more, each with ratings, verified reviews, hourly rates, team size, and location. If you do lead generation, competitive research, or market sizing, it's a goldmine.
It's also genuinely annoying to scrape. If you've tried, you already know why.
Why most Clutch scrapers fail
Send a plain HTTP request to a Clutch listing page and you get back a 403 with a Cloudflare challenge instead of the data. It happens on almost every request, even for robots.txt. Tools built on simple HTTP clients (requests, axios, cheerio-only actors) hit this wall and either return nothing or break at random.
That's the real reason most Clutch scrapers on the market are unreliable. They never actually get past the front door. What you usually see is a scraper that "succeeds" with an empty result set, which is worse than failing outright, because you think you have data and you don't.
What actually works
The fix isn't a smarter HTTP header. It's using a real browser. A headless Chromium instance (via Playwright) solves Cloudflare's managed challenge on its own within a few seconds, the same way a normal visitor's browser does. After that, the full rendered page is right there.
Two things keep it reliable at scale:
- A real browser, not raw HTTP. Playwright or Chromium passes the challenge. A bare request never will.
- Residential proxies. Datacenter IPs get flagged fast. Rotating residential IPs keep the success rate high across a lot of pages.
Once you're in, the data is clean and structured. Roughly 60 companies per listing page, each with a name, profile URL, review count, rating, minimum project size, hourly rate, description, locations, and the agency's own website.
The extraction, step by step
- Open the listing URL (say,
https://clutch.co/agencies/digital-marketing) in a headless browser with a residential proxy. - Wait for the company cards to render. The Cloudflare challenge clears on its own.
- Read each card's fields from the page.
- Paginate with
?page=N, and stop when a page returns no new companies. Clutch clamps out-of-range pages to the last valid one, so naive pagination just loops on duplicates. De-dupe by the stable provider id. - Fail loudly if a run returns zero results, so you never pay for an empty dataset.
That last point matters more than it looks. The most common complaint about existing Clutch scrapers is "it ran, said success, returned nothing." A scraper you can trust treats an empty run as an error, not a success.
A ready-made option
If you'd rather not build and maintain the browser, proxy, and pagination logic yourself, I published a Clutch.co scraper on the Apify Store that does all of the above:
→ Clutch.co Agency Directory Scraper
Give it a category or location URL and it returns clean, structured company data: name, rating, reviews, min project size, hourly rate, team size, location, and website. It uses a real browser with residential proxies to get past Cloudflare, paginates on its own, de-dupes, and fails loudly on an empty run. Pricing is pay-per-result ($3.00 per 1,000 companies), so you only pay for data you actually get.
When to use it
- Lead generation: build a targeted list of agencies by service and location.
- Competitive research: track how competitors are rated and priced.
- Market analysis: size a category by number of providers, ratings spread, and typical rates.
Build your own or use the actor, the takeaway is the same. On Clutch the wall is Cloudflare, the key is a real browser plus residential proxies, and what separates a scraper you can trust from one you can't is whether it fails loudly when it comes back with nothing.
Top comments (0)