DEV Community

tienlq2011
tienlq2011

Posted on Edited on

Find the email behind any website with 70 lines of Python (incl. Cloudflare-hidden ones)

You have a list of businesses with websites, maybe from a Google Maps export, and you need their email addresses. The obvious regex over the homepage misses a lot of them. Here's what catches most of the rest, and a small standard-library script you can copy.

1. Most emails aren't on the homepage

Small business sites usually put the email on a contact, about or imprint page. So read the homepage first, collect its links, and follow the ones that look like contact pages.

Don't only check for contact. Your list is probably not all English:

CONTACT_WORDS = ["contact", "kontakt", "contacto", "contato", "contatti",
                 "impressum", "imprint", "about", "a-propos", "nosotros", "lien-he"]
Enter fullscreen mode Exit fullscreen mode

German sites are required to have an Impressum, and it almost always lists an email. Three or four pages per site is enough for most small businesses.

2. [email protected] is Cloudflare, and it's easy to decode

If you've scraped a few hundred sites, you've seen this in the HTML:

<a href="/cdn-cgi/l/email-protection#4a222f2626250a2f322b273a262f64292527">[email protected]</a>
Enter fullscreen mode Exit fullscreen mode

Cloudflare's email obfuscation swaps the address for a hex string, and a script in the browser decodes it. A plain HTTP scraper never runs that script, so it sees no email.

That hex decodes to hello@example.com. The encoding is a single-byte XOR: the first byte is the key, the rest is the address.

def decode_cfemail(hexstr):
    b = bytes.fromhex(hexstr)
    return bytes(c ^ b[0] for c in b[1:]).decode()
Enter fullscreen mode Exit fullscreen mode

The hex shows up in two places: the data-cfemail attribute, and after # in /cdn-cgi/l/email-protection#... links. Check both.

3. Filter the junk

A naive regex returns things like:

  • logo@2x.png, which is a retina image file name, not an email
  • name@example.com and your@email.com, which are form placeholders
  • abc123@sentry.io and ...@sentry.wixpress.com, which are error tracking IDs

Drop anything whose "TLD" is a file extension, and keep a short blocklist of placeholder domains and users.

4. Don't forget structured data

Many local business sites ship schema.org JSON-LD for Google:

<script type="application/ld+json">{"@type": "Dentist", "email": "hello@smile.example", "telephone": "+1 512 555 0100"}</script>
Enter fullscreen mode Exit fullscreen mode

When it's there, it's the cleanest source of an email and a phone number you'll find.

The script

Standard library only. Pass it one or more domains:

import json, re, sys, urllib.request
from html.parser import HTMLParser
from urllib.parse import urljoin, urlparse

EMAIL = re.compile(r"[a-z0-9][a-z0-9._%+-]{0,63}@(?:[a-z0-9-]{1,63}\.)+[a-z]{2,24}", re.I)
JUNK = re.compile(r"\.(png|jpe?g|gif|svg|webp|css|js)$|example\.|sentry|wixpress|your@|name@|email@", re.I)
WORDS = ["contact", "kontakt", "contacto", "contato", "impressum", "imprint", "about"]

def decode_cfemail(h):
    b = bytes.fromhex(h)
    return bytes(c ^ b[0] for c in b[1:]).decode()

class Page(HTMLParser):
    def __init__(self):
        super().__init__()
        self.links, self.cf, self.ld, self._in_ld = [], [], [], False
    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if a.get("data-cfemail"):
            self.cf.append(a["data-cfemail"])
        href = a.get("href") or ""
        if tag == "a" and href:
            self.links.append(href)
            if "/cdn-cgi/l/email-protection#" in href:
                self.cf.append(href.rsplit("#", 1)[1])
        self._in_ld = tag == "script" and a.get("type") == "application/ld+json"
    def handle_data(self, data):
        if self._in_ld:
            self.ld.append(data)

def fetch(url):
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
    with urllib.request.urlopen(req, timeout=15) as r:
        return r.read(3_000_000).decode("utf-8", "ignore")

def emails_on(url):
    html = fetch(url)
    p = Page()
    p.feed(html)
    found = set(EMAIL.findall(html))
    found |= {decode_cfemail(h) for h in p.cf}
    found |= set(EMAIL.findall(" ".join(p.ld)))
    return {e.lower() for e in found if not JUNK.search(e)}, p.links

def scrape(site, max_pages=4):
    home = site if site.startswith("http") else f"https://{site}"
    emails, links = emails_on(home)
    host = urlparse(home).netloc
    pages = []
    for link in links:
        url = urljoin(home, link).split("#")[0]
        if urlparse(url).netloc == host and any(w in url.lower() for w in WORDS) and url not in pages:
            pages.append(url)
    # contact pages first, then imprint, then about
    pages.sort(key=lambda u: min(i for i, w in enumerate(WORDS) if w in u.lower()))
    for url in pages[: max_pages - 1]:
        try:
            emails |= emails_on(url)[0]
        except Exception:
            pass
    return sorted(emails)

for site in sys.argv[1:]:
    print(site, scrape(site))
Enter fullscreen mode Exit fullscreen mode
$ python3 emails.py postgresql.org
postgresql.org ['coc@postgresql.org', 'funds-group@postgresql.org', 'security@postgresql.org', 'usergroups@postgresql.org', 'webmaster@postgresql.org']
Enter fullscreen mode Exit fullscreen mode

What this script doesn't do

It's a starting point. Before running it on a real lead list, you'll want:

  • Fallbacks. Plenty of small business sites have broken HTTPS or only work on www.. Try https://, http:// and www. before giving up.
  • Phones and socials. Use tel: links rather than guessing from digits in the text, which mostly finds order numbers and dates. For socials, keep profile links and skip share buttons (sharer.php, intent/tweet).
  • Concurrency and a size cap. Some "websites" are 50 MB PDFs.
  • Dedup by domain. Chains with 40 branches on Google Maps share one website.

If you don't want to maintain it

I packaged the full version as an Apify actor: Website Contact Scraper. Paste a list of websites, or the dataset ID of a Google Maps scrape, and you get emails, phones and social profiles joined back to each place. It costs $3 per 1,000 websites where contacts were found. Websites with nothing found, or that don't load, are free.

On a test list of 10 dentists in Austin from Google Maps, it found contacts for all 10 and emails for 5. That's about what to expect from local businesses: many only have a contact form.


A note on use: stick to contact details businesses publish themselves. If you collect personal data, like a named employee's email, GDPR and similar laws apply to how you store and use it.

Top comments (0)