Stop Paying $99/mo for Scraping SaaS: Build a Stealth Dual-Engine Web Scraper in Python

If you have ever needed to collect B2B leads, track competitor pricing, or build an internal dataset, you know the modern web scraping dilemma:

On one hand, you have cloud scraping SaaS platforms charging between $60 and $150 every single month. They give you metered credits, throttle your concurrency, and charge exorbitant surcharges the moment you need JavaScript rendering.

On the other hand, you have simple Python scripts with requests and BeautifulSoup. You run them on a directory site, and within 10 requests, you hit a 403 Forbidden, a Cloudflare challenge screen, or a blank page because the site is an infinite-scroll React SPA.

You do not need an expensive SaaS subscription to pull clean, structured web data.

In this guide, we will build a Dual-Engine Python Web Scraper from scratch. It uses HTTPX for blazing-fast static page extraction and seamlessly switches to Playwright Headless with anti-bot stealth spoofing for dynamic SPAs, auto-extracts emails, phones, and WhatsApp links, and exports boardroom-ready Excel spreadsheets.

---

Architecture: Why Single-Engine Scrapers Fail

A single-tool scraping strategy always hits a wall:

• HTTPX / Requests alone: Extremely fast (100+ pages/minute), but completely blind to JavaScript hydration, shadow DOMs, and infinite scroll.
• Headless Browsers (Selenium / Puppeteer / Playwright) alone: Renders everything, but consumes high CPU and RAM. Using a headless browser to scrape 5,000 static HTML pages is an enormous waste of compute.

The optimal approach is a Dual-Engine Orchestrator:

[Target Target URL / Directory]
               │
      Is page dynamic JS?
         ├── NO ──► [HTTPX Async Engine] ──► (0.2s response)
         └── YES ─► [Playwright Stealth]  ──► (Bypasses bot flags & renders SPA)
                           │
                           ▼
             [Deep Regex & DOM Extractor]
         (Emails, Clean Phones, WhatsApp Links)
                           │
                           ▼
             [In-Memory Deduplication]
             (Hash check on domain/email)
                           │
                           ▼
          [Styled Excel (.xlsx) / JSON Exporter]

---

Step 1: Setting Up the Anti-Detection Playwright Stealth Context

Most basic scrapers get blocked instantly because default automation flags scream "bot" to modern WAFs (Web Application Firewalls).

The primary flag is navigator.webdriver = true. Here is how you initialize an authentic, stealth browser context in Playwright:

import asyncio
from playwright.async_api import async_playwright

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
]

async def create_stealth_browser(p, proxy: str = None):
    # Launch Chromium with automation flags masked
    browser = await p.chromium.launch(
        headless=True,
        args=[
            "--disable-blink-features=AutomationControlled",
            "--no-sandbox",
            "--disable-infobars"
        ]
    )

    # Configure authentic browser context
    context = await browser.new_context(
        user_agent=USER_AGENTS[0],
        viewport={"width": 1920, "height": 1080},
        locale="en-US",
        timezone_id="America/New_York"
    )

    # Strip navigator.webdriver signature before any page script executes
    await context.add_init_script("""
        Object.defineProperty(navigator, 'webdriver', {
            get: () => undefined
        });
    """)

    return browser, context

---

Step 2: High-Speed Async Static Scraping with HTTPX

When scraping directories where the HTML is delivered server-side (WordPress, standard directories, static HTML), do not launch Chromium. Use asynchronous HTTPX with browser-grade headers:

import httpx
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": USER_AGENTS[0],
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Sec-Ch-Ua": '"Chromium";v="128", "Not;A=Brand";v="24", "Google Chrome";v="128"',
    "Sec-Ch-Ua-Mobile": "?0",
    "Sec-Ch-Ua-Platform": '"Windows"',
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1"
}

async def fetch_static_page(url: str) -> str:
    async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
        response = await client.get(url, headers=HEADERS)
        response.raise_for_status()
        return response.text

---

Step 3: Deep Contact & Social Profile Extraction

Raw HTML is useless without structured extraction. We want to pull clean business emails, phone numbers, direct WhatsApp links, and social profiles:

import re

EMAIL_REGEX = r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+'
PHONE_REGEX = r'(\+?[0-9]{1,3}[-.\s]?)?(\(?\d{3}\)?[-.\s]?)[\d-.]{6,10}'

def extract_leads_from_html(html: str, base_url: str) -> dict:
    soup = BeautifulSoup(html, 'html.parser')
    text = soup.get_text(separator=' ')

    # Extract unique, non-asset emails
    raw_emails = set(re.findall(EMAIL_REGEX, text))
    clean_emails = [
        email for email in raw_emails 
        if not email.endswith(('.png', '.jpg', '.jpeg', '.svg', '.webp', '.gif'))
    ]

    # Extract phone numbers
    raw_phones = list(set([match[0] + match[1] for match in re.findall(PHONE_REGEX, text)]))
    clean_phones = [p.strip() for p in raw_phones if len(p.strip()) >= 7]

    # Find WhatsApp direct chat links
    whatsapp_links = []
    for a in soup.find_all('a', href=True):
        href = a['href']
        if 'wa.me' in href or 'api.whatsapp.com/send' in href:
            whatsapp_links.append(href)

    return {
        "url": base_url,
        "title": soup.title.string.strip() if soup.title else "",
        "emails": clean_emails,
        "phones": clean_phones,
        "whatsapp": whatsapp_links[0] if whatsapp_links else ""
    }

---

Step 4: Deduplication & Clean Excel Exporting

Clients and sales teams do not want messy JSON files or broken CSVs where commas corrupt column boundaries. They want a formatted Excel file with clickable WhatsApp links and auto-fitted columns.

Using openpyxl, you can generate native .xlsx files with zero manual touch:

from openpyxl import Workbook
from openpyxl.styles import Font, PatternFill, Alignment
from openpyxl.utils import get_column_letter

def export_to_styled_excel(records: list, output_filename: str):
    wb = Workbook()
    ws = wb.active
    ws.title = "Verified Leads"

    # Header row setup
    headers = ["Company / Title", "Source URL", "Primary Email", "Phone", "WhatsApp Action Link"]
    ws.append(headers)

    header_fill = PatternFill(start_color="1F2937", end_color="1F2937", fill_type="solid")
    header_font = Font(name="Segoe UI", size=11, bold=True, color="FFFFFF")

    for col_num, header in enumerate(headers, 1):
        cell = ws.cell(row=1, column=col_num)
        cell.fill = header_fill
        cell.font = header_font
        cell.alignment = Alignment(horizontal="center", vertical="center")

    # Populate rows with deduplicated records
    seen_domains = set()
    for item in records:
        domain = item.get("url", "").split("//")[-1].split("/")[0]
        if domain in seen_domains:
            continue
        seen_domains.add(domain)

        ws.append([
            item.get("title", "N/A"),
            item.get("url", ""),
            item.get("emails", [""])[0] if item.get("emails") else "",
            item.get("phones", [""])[0] if item.get("phones") else "",
            item.get("whatsapp", "")
        ])

    # Auto-adjust column widths for crisp formatting
    for col in ws.columns:
        max_len = max(len(str(cell.value or '')) for cell in col)
        col_letter = get_column_letter(col[0].column)
        ws.column_dimensions[col_letter].width = max(max_len + 3, 14)

    wb.save(output_filename)
    print(f"Successfully exported {len(seen_domains)} verified leads to {output_filename}")

---

4 Rules to Avoid Scraping Bans

1. Match Your Sec-CH-UA Headers with Your User-Agent: If your User-Agent claims Chrome 128 on Windows, but your Sec-Ch-Ua-Platform header sends Linux or is missing, Cloudflare flags you immediately.
2. Never Use Predictable Delays: Instead of time.sleep(2.0), use randomized human jitter: await asyncio.sleep(random.uniform(1.2, 3.8)).
3. Respect Rate Limits: Run with max 3–5 concurrent tasks per domain. Crashing a target server with 50 threads will get your IP subnet blacklisted.
4. Scrape Off-Peak Hours: Target directories during night hours in their local timezone when site traffic is lowest.

---

Want the Complete, Battle-Tested CLI Tool?

If you don't want to spend your weekend writing custom parsers, managing proxy rotation configs, and debugging selector edge cases across 10 different website layouts:

Check out OmniScraper AI — Headless Lead & Competitor Intelligence CLI:
👉 Download OmniScraper AI on Gumroad ($9 USD — Special Launch Price)

What’s included in the production repository:

• 🚀 Dual-Engine CLI: Single-command execution (python scraper.py --mode dynamic --target presets/b2b.json).
• 📁 10 Pre-Configured Industry Presets: Ready-to-run configurations for B2B directories, local services, and e-commerce listings.
• 🛡️ 22 User-Agent Profiles & Viewport Presets: Fully configured stealth headers JSON out of the box.
• 📊 Automated Tri-Format Export: Generates clean CSV, structured JSON, and styled .xlsx with automatic deduplication.
• ⚡ 1-Minute Quickstart Guide: Zero complex configuration. Clone, install requirements, and pull leads in under 60 seconds.

Prefer a modular pipeline for WhatsApp direct extraction and B2B enrichment?
👉 Production Python Scraper Toolkit v2.0 on Gumroad ($9 USD)

Running an agency or need workflows for n8n as well?
Get our entire developer suite:
👉 All-in-One AI Developer Automation Suite ($49 USD).