DEV Community

Tùng Nguyễn
Tùng Nguyễn

Posted on Edited on

How to Build a High-Speed B2B Lead, Email & Social Profile Scraper in Python (Open-Source + Cloud API)

#python #automation #opensource #webdev

Building a reliable B2B lead enrichment pipeline in 2026 is much harder than simply running requests.get() on a list of URLs. Most modern company websites split their contact details across /contact, /about-us, footer obfuscation (info [at] company.com), and mailto: / tel: attributes.

In this guide, I will walk through how to build a high-concurrency Python B2B Website Email, Phone & Social Media Scraper that extracts decision-maker contact info and exports directly to a formatted Excel / CSV dashboard.

âš¡ Quick Links (Cloud API, Open-Source & Pro Suite)

If you want to run this immediately in the cloud or download the full multi-threaded CLI + Excel VBA reporting suite:

🧠 Architecture of a High-Accuracy B2B Contact Crawler

A production B2B contact extractor needs 4 stages:

  1. Homepage Fetch & Metadata Extraction: Grab <title>, <meta name="description">, and visible text from the landing page.
  2. Smart Contact Page Discovery: Automatically detect internal links matching /contact, /about, /team, or /support and crawl them concurrently.
  3. Regex + De-obfuscation + False-Positive Filtering: Strip out junk asset matches (logo@2x.png, sentry@domain.com, example@domain.com) while capturing mailto: and visible emails.
  4. Social Profile Normalization: Match canonical company profiles on LinkedIn (/company/), Twitter/X, Facebook, Instagram, YouTube, and TikTok.

Core Python Implementation 

import re
from urllib.parse import urljoin, urlparse
import requests
from bs4 import BeautifulSoup

EMAIL_REGEX = re.compile(r'[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}')
IGNORE_DOMAINS = {'sentry.io', 'example.com', 'domain.com', 'wixpress.com', 'cloudflare.com'}
IGNORE_EXTS = ('.png', '.jpg', '.jpeg', '.gif', '.svg', '.webp', '.css', '.js')

SOCIAL_PATTERNS = {
    'linkedin': re.compile(r'https?://(?:www\.)?linkedin\.com/(?:company|in)/[a-zA-Z0-9_\-]+', re.I),
    'twitter': re.compile(r'https?://(?:www\.)?(?:twitter\.com|x\.com)/[a-zA-Z0-9_]+', re.I),
    'facebook': re.compile(r'https?://(?:www\.)?facebook\.com/[a-zA-Z0-9.\-_]+', re.I),
    'instagram': re.compile(r'https?://(?:www\.)?instagram\.com/[a-zA-Z0-9._]+', re.I),
}

def clean_emails(raw_text: str) -> list[str]:
    found = set()
    for match in EMAIL_REGEX.findall(raw_text):
        email = match.lower().strip('.')
        if email.endswith(IGNORE_EXTS):
            continue
        domain = email.split('@')[-1]
        if domain in IGNORE_DOMAINS:
            continue
        found.add(email)
    return sorted(found)

def scrape_b2b_domain(url: str) -> dict:
    resp = requests.get(url, timeout=15, headers={'User-Agent': 'Mozilla/5.0'})
    soup = BeautifulSoup(resp.text, 'html.parser')
    combined_html = resp.text

    # Follow /contact or /about link if present
    for a in soup.find_all('a', href=True):
        href = a['href'].lower()
        if any(k in href for k in ('/contact', '/about', '/impressum')):
            sub_url = urljoin(url, a['href'])
            if urlparse(sub_url).netloc == urlparse(url).netloc:
                try:
                    combined_html += "\n" + requests.get(sub_url, timeout=10).text
                except Exception:
                    pass
                break

    emails = clean_emails(combined_html)
    socials = {
        name: sorted(set(pattern.findall(combined_html)))
        for name, pattern in SOCIAL_PATTERNS.items()
    }
    return {
        'url': url,
        'title': soup.title.string.strip() if soup.title and soup.title.string else '',
        'emails': emails,
        'socials': socials,
    }
Enter fullscreen mode Exit fullscreen mode

🚀 Scaling to 10,000+ Domains via Apify Cloud API

If you don't want to manage proxies, concurrency, or headless browsers locally, I published a ready-to-run Cloud Actor on the Apify Store that handles batch URLs, automatic contact page traversal, and structured JSON/CSV/Excel dataset export:

Or grab the Standalone Python + Excel VBA Pro Suite on Gumroad:
👉 Download Python B2B Lead Scraper & Excel VBA Suite Pro ($14.99)

🌐 Live REST APIs on RapidAPI Hub & Instant Download Store

Top comments (0)