Turning a list of company websites into a contact list with a plain HTTP crawl

Anyone who has done outbound sales or recruiting knows this task. You have a list of company websites and you need the contact info off each one. Open the Contact page, copy the email, check LinkedIn, move to the next site. A 50 domain list eats most of an afternoon done by hand.

The approach

Give it a start URL and it crawls within that same domain, up to a depth you choose, using Crawlee's CheerioCrawler. On each page it checks for:

• Email addresses in the raw HTML
• Phone numbers
• Social profile links (LinkedIn, X, Facebook, Instagram, GitHub)

Everything found gets returned as one record per page, with the URL, domain, and whatever contact info was on it.

{
  "url": "https://example.com/contact",
  "domain": "example.com",
  "emails": ["[e-mail]"],
  "phones": ["[phone]"],
  "socialProfiles": {
    "linkedin": "https://linkedin.com/company/example",
    "twitter": "https://x.com/example"
  }
}

What it deliberately doesn't do

No headless browser. It only reads what the site already renders in the raw HTML response to a plain request, the same thing any visitor's browser would see before JavaScript runs. That keeps it fast and cheap to run, and it means it only extracts information the site is already publishing publicly. Nothing behind a login, nothing a normal visitor couldn't already find on a Contact or About page.

Where it falls short, on purpose

Sites that render their content client side with JavaScript are a real weak spot here. A plain HTTP crawl never sees content that only shows up after JS runs, so a heavily client rendered site might come back with far less than it actually has. That's the honest tradeoff for staying fast, cheap, and proxy free. A browser based crawler would handle those sites but costs more to run and adds real complexity. For static or server rendered company sites, which is most small and mid size business sites, the plain HTTP approach works well.

Pricing shape

Billed per page where contact info was actually found. A domain with nothing public anywhere costs nothing to check, and a domain where three pages all list contact info costs about the same as pulling it from all three yourself, just a lot faster.

Try it

• Source: github.com/timmKal01/website-lead-extractor
• Hosted version (Apify actor, no setup): apify.com/m_ctim/website-lead-extractor

If you run it against a list of your own domains, curious to hear how the hit rate looks on real sites.

Comments

[Raknaos]

Cheerio-first with a browser fallback only when the HTML has nothing is the right cost ordering — full headless crawling a 50-domain list turns an afternoon job into a GPU bill.

In our runs the yield collapses exactly where you'd expect: Cloudflare challenges and emails injected by JS. What's your trigger for the fallback, zero contacts after N pages, or a per-site blocklist you maintain by hand?

[Timothy Kelvin]

Good question, and honestly you're ahead of where the actor actually is right now. There's no fallback yet. It's Cheerio only, so a Cloudflare challenge or a JS injected email just comes back empty for that page instead of triggering anything smarter.

I picked that on purpose to keep it fast and cheap for the common case, static or server rendered company sites, which covers most small and mid size businesses. But you clearly have real runs to compare against, which is more than I have. When you say the yield collapses at Cloudflare and JS injected emails, how big a chunk of your domain list is that in practice? If it's a small tail I'd lean toward your zero contacts after N pages idea. If it's a third of the list or more, a per-site blocklist starts looking like the safer bet. What does it look like on your end?