DEV Community

Karan Kumar
Karan Kumar

Posted on

How to Scrape and Verify Local Business Leads with Python (Without Paying $100/mo for SaaS)

#webdev #python #programming #automation

If you do B2B marketing, freelance outreach, or local client acquisition, you know the pain: lead platforms charge $50 to $100+ per month just to export basic public contact info.

For local businesses (clinics, shops, trades), much of that data already sits in open databases like OpenStreetMap (OSM). Here's how to build a lightweight, self-hosted lead pipeline with Python.

The Architecture: 3 Stages

  1. Location querying: pull businesses by category and city.
  2. Website crawling: extract public contact details from their sites.
  3. Verification and segmentation: filter out bad data before you send anything.

Stage 1: Pull Places from OpenStreetMap (Overpass API)

Overpass is OSM's public query language. It's free, and you can query any city or bounding box without paying Google Places API fees.

import requests

OVERPASS_URL = "https://overpass-api.de/api/interpreter"
HEADERS = {"User-Agent": "local-lead-pipeline/1.0 (your@email.com)"}

def fetch_local_businesses(city, key="amenity", value="dentist"):
    query = f"""
    [out:json][timeout:60];
    area["name"="{city}"]["boundary"="administrative"]->.a;
    nwr["{key}"="{value}"](area.a);
    out center tags;
    """
    r = requests.post(OVERPASS_URL, data={"data": query},
                      headers=HEADERS, timeout=90)
    r.raise_for_status()
    elements = r.json().get("elements", [])
    # Keep only records that actually have a website
    return [e for e in elements
            if e.get("tags", {}).get("website")
            or e.get("tags", {}).get("contact:website")]
Enter fullscreen mode Exit fullscreen mode

Each result includes the name, address, website, and coordinates, when the mapper filled them in. Swap key/value for other tags (shop=hairdresser, craft=plumber, healthcare=clinic).

Heads up: OSM coverage varies a lot by region and category. Expect gaps.

Stage 2: Crawl for Public Contact Details

For each website, check the homepage plus common contact pages, and read both mailto: links and visible text.

import re
from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup

EMAIL_RE = re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")
IMAGE_EXT = (".png", ".jpg", ".jpeg", ".gif", ".webp", ".svg")
PATHS = ["", "/contact", "/contact-us", "/about"]

def extract_emails(base_url):
    if not base_url.startswith("http"):
        base_url = "https://" + base_url
    found = set()
    for path in PATHS:
        try:
            r = requests.get(urljoin(base_url, path),
                             headers=HEADERS, timeout=5)
            r.raise_for_status()
        except requests.RequestException:
            continue
        soup = BeautifulSoup(r.text, "html.parser")
        for a in soup.select('a[href^="mailto:"]'):
            found.add(a["href"][7:].split("?")[0].strip().lower())
        found.update(e.lower() for e in EMAIL_RE.findall(soup.get_text(" ")))
    return sorted(e for e in found if not e.endswith(IMAGE_EXT))
Enter fullscreen mode Exit fullscreen mode

Be a polite crawler: respect robots.txt, add delays between requests, and keep timeouts short.

Stage 3: Verify and Export Clean CSVs

Never cold-email unverified addresses. At minimum:

  1. Syntax check: the regex above.
  2. MX record check: does the domain accept mail?
  3. API verification: a service like ZeroBounce for the rest. 
import dns.resolver  # pip install dnspython

def has_mx(email):
    try:
        return bool(dns.resolver.resolve(email.split("@")[-1], "MX"))
    except Exception:
        return False
Enter fullscreen mode Exit fullscreen mode

Layered verification keeps bounce rates low (aim for under 2%), though no method guarantees zero bounces or spam traps.

Then split the output into four files:

  • deliverable_emails.csv: ready for email campaigns
  • phone_only_leads.csv: for calling or SMS
  • risky_review.csv: catch-all or uncertain domains
  • master_leads.csv: full archive

Stay Legal

  • OSM data is under ODbL, so attribute "© OpenStreetMap contributors."
  • Cold email is regulated (CAN-SPAM, GDPR, and local equivalents). Include an unsubscribe option and a real sender identity.
  • Scraping public pages is not a free pass. Check each site's terms and your local laws.

Want the Ready-Made Version?

Disclosure: I'm the author of this kit.

If you'd rather skip wiring up the scripts, dashboard, and launchers, I packaged everything into the LocalLeads Kit:

  • Streamlit dashboard
  • One-click launchers (run.bat for Windows, run.sh for Mac/Linux)
  • Commercial license included

👉 Get the LocalLeads Kit on Gumroad ($39 one-time)

Questions about Overpass rate limits or crawler timeouts? Drop them in the comments.

Top comments (0)