You have a list of businesses with websites, maybe from a Google Maps export, and you need their email addresses. The obvious regex over the homepage misses a lot of them. Here's what catches most of the rest, and a small standard-library script you can copy.
1. Most emails aren't on the homepage
Small business sites usually put the email on a contact, about or imprint page. So read the homepage first, collect its links, and follow the ones that look like contact pages.
Don't only check for contact. Your list is probably not all English:
CONTACT_WORDS = ["contact", "kontakt", "contacto", "contato", "contatti",
"impressum", "imprint", "about", "a-propos", "nosotros", "lien-he"]
German sites are required to have an Impressum, and it almost always lists an email. Three or four pages per site is enough for most small businesses.
2. [email protected] is Cloudflare, and it's easy to decode
If you've scraped a few hundred sites, you've seen this in the HTML:
<a href="/cdn-cgi/l/email-protection#4a222f2626250a2f322b273a262f64292527">[email protected]</a>
Cloudflare's email obfuscation swaps the address for a hex string, and a script in the browser decodes it. A plain HTTP scraper never runs that script, so it sees no email.
That hex decodes to hello@example.com. The encoding is a single-byte XOR: the first byte is the key, the rest is the address.
def decode_cfemail(hexstr):
b = bytes.fromhex(hexstr)
return bytes(c ^ b[0] for c in b[1:]).decode()
The hex shows up in two places: the data-cfemail attribute, and after # in /cdn-cgi/l/email-protection#... links. Check both.
3. Filter the junk
A naive regex returns things like:
-
logo@2x.png, which is a retina image file name, not an email -
name@example.comandyour@email.com, which are form placeholders -
abc123@sentry.ioand...@sentry.wixpress.com, which are error tracking IDs
Drop anything whose "TLD" is a file extension, and keep a short blocklist of placeholder domains and users.
4. Don't forget structured data
Many local business sites ship schema.org JSON-LD for Google:
<script type="application/ld+json">{"@type": "Dentist", "email": "hello@smile.example", "telephone": "+1 512 555 0100"}</script>
When it's there, it's the cleanest source of an email and a phone number you'll find.
The script
Standard library only. Pass it one or more domains:
import json, re, sys, urllib.request
from html.parser import HTMLParser
from urllib.parse import urljoin, urlparse
EMAIL = re.compile(r"[a-z0-9][a-z0-9._%+-]{0,63}@(?:[a-z0-9-]{1,63}\.)+[a-z]{2,24}", re.I)
JUNK = re.compile(r"\.(png|jpe?g|gif|svg|webp|css|js)$|example\.|sentry|wixpress|your@|name@|email@", re.I)
WORDS = ["contact", "kontakt", "contacto", "contato", "impressum", "imprint", "about"]
def decode_cfemail(h):
b = bytes.fromhex(h)
return bytes(c ^ b[0] for c in b[1:]).decode()
class Page(HTMLParser):
def __init__(self):
super().__init__()
self.links, self.cf, self.ld, self._in_ld = [], [], [], False
def handle_starttag(self, tag, attrs):
a = dict(attrs)
if a.get("data-cfemail"):
self.cf.append(a["data-cfemail"])
href = a.get("href") or ""
if tag == "a" and href:
self.links.append(href)
if "/cdn-cgi/l/email-protection#" in href:
self.cf.append(href.rsplit("#", 1)[1])
self._in_ld = tag == "script" and a.get("type") == "application/ld+json"
def handle_data(self, data):
if self._in_ld:
self.ld.append(data)
def fetch(url):
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
with urllib.request.urlopen(req, timeout=15) as r:
return r.read(3_000_000).decode("utf-8", "ignore")
def emails_on(url):
html = fetch(url)
p = Page()
p.feed(html)
found = set(EMAIL.findall(html))
found |= {decode_cfemail(h) for h in p.cf}
found |= set(EMAIL.findall(" ".join(p.ld)))
return {e.lower() for e in found if not JUNK.search(e)}, p.links
def scrape(site, max_pages=4):
home = site if site.startswith("http") else f"https://{site}"
emails, links = emails_on(home)
host = urlparse(home).netloc
pages = []
for link in links:
url = urljoin(home, link).split("#")[0]
if urlparse(url).netloc == host and any(w in url.lower() for w in WORDS) and url not in pages:
pages.append(url)
# contact pages first, then imprint, then about
pages.sort(key=lambda u: min(i for i, w in enumerate(WORDS) if w in u.lower()))
for url in pages[: max_pages - 1]:
try:
emails |= emails_on(url)[0]
except Exception:
pass
return sorted(emails)
for site in sys.argv[1:]:
print(site, scrape(site))
$ python3 emails.py postgresql.org
postgresql.org ['coc@postgresql.org', 'funds-group@postgresql.org', 'security@postgresql.org', 'usergroups@postgresql.org', 'webmaster@postgresql.org']
What this script doesn't do
It's a starting point. Before running it on a real lead list, you'll want:
-
Fallbacks. Plenty of small business sites have broken HTTPS or only work on
www.. Tryhttps://,http://andwww.before giving up. -
Phones and socials. Use
tel:links rather than guessing from digits in the text, which mostly finds order numbers and dates. For socials, keep profile links and skip share buttons (sharer.php,intent/tweet). - Concurrency and a size cap. Some "websites" are 50 MB PDFs.
- Dedup by domain. Chains with 40 branches on Google Maps share one website.
If you don't want to maintain it
I packaged the full version as an Apify actor: Website Contact Scraper. Paste a list of websites, or the dataset ID of a Google Maps scrape, and you get emails, phones and social profiles joined back to each place. It costs $3 per 1,000 websites where contacts were found. Websites with nothing found, or that don't load, are free.
On a test list of 10 dentists in Austin from Google Maps, it found contacts for all 10 and emails for 5. That's about what to expect from local businesses: many only have a contact form.
A note on use: stick to contact details businesses publish themselves. If you collect personal data, like a named employee's email, GDPR and similar laws apply to how you store and use it.
Top comments (0)