You have a list of company domains (a CRM export, a conference attendee list, the output of a tech stack lookup) and you need the public way to reach each one: the info@ or sales@ inbox and the LinkedIn company page. Most companies publish these on the homepage footer or a contact, about or impressum page, so you don't need a search engine or a headless browser. A short Python script with requests can do it.
Below is a full working script (about 100 lines). I'll walk through the parts that are easy to get wrong.
1. Only visit pages that are likely to have contacts
Crawling a whole site is slow and rude. Instead, fetch the homepage, then follow only same-host links whose path looks like a contact page:
CONTACT_HINTS = re.compile(r"contact|about|impressum|imprint|kontakt|team|support", re.I)
def contact_links(page, base_url):
host = urlparse(base_url).netloc
links = []
for href in re.findall(r'href=["\']([^"\'#]+)', page, re.I):
url = urljoin(base_url, href)
if urlparse(url).netloc == host and CONTACT_HINTS.search(urlparse(url).path):
links.append(url.split("?")[0])
return list(dict.fromkeys(links))
The impressum / kontakt hints matter: German and Austrian sites are legally required to have an imprint page, and it almost always contains an email.
2. Respect robots.txt
The standard library already has a parser, so there's no excuse to skip it:
def robots_for(base):
rp = robotparser.RobotFileParser()
try:
r = requests.get(urljoin(base, "/robots.txt"), headers=UA, timeout=10)
rp.parse(r.text.splitlines() if r.ok else [])
except requests.RequestException:
rp.parse([])
return rp
3. Find emails, including the obfuscated ones
A plain regex catches mailto: links and visible addresses. Two things trip it up:
- HTML entities (
@instead of@), so runhtml.unescapefirst. - Hand-obfuscated addresses like
sales [at] acme [dot] com.
Image file names such as logo@2x.png also match an email regex, so drop those.
EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,24}")
OBFUSCATED_RE = re.compile(
r"([A-Za-z0-9._%+-]+)\s*[\[\(]\s*at\s*[\]\)]\s*([A-Za-z0-9-]+(?:\s*[\[\(]\s*dot\s*[\]\)]\s*[A-Za-z0-9-]+)+)",
re.I,
)
def extract_emails(page):
text = html.unescape(page).replace("\\/", "/")
found = set(EMAIL_RE.findall(text))
for user, dom in OBFUSCATED_RE.findall(text):
dom = re.sub(r"\s*[\[\(]\s*dot\s*[\]\)]\s*", ".", dom, flags=re.I)
found.add(f"{user}@{dom}")
return {e.lower().rstrip(".") for e in found if not e.lower().endswith(JUNK_TLDS)}
4. Social links without the share buttons
Every page has "share on X" and "share on Facebook" buttons, which are not the company's profile. Negative lookaheads skip them. Links that sit inside inline JSON come escaped as https:\/\/..., so unescape and normalize before deduplicating, or you'll get the same LinkedIn page four times:
SOCIAL = {
"linkedin": re.compile(r"https?://(?:[a-z]{2,3}\.)?linkedin\.com/(?:company|in|school)/[^\s\"'<>?#]+", re.I),
"twitter": re.compile(r"https?://(?:www\.)?(?:twitter|x)\.com/(?!share|intent|home)[A-Za-z0-9_]{1,15}(?=[/\"'?#\s<]|$)", re.I),
"facebook": re.compile(r"https?://(?:www\.)?facebook\.com/(?!sharer|share|dialog|tr\b)[^\s\"'<>?#]+", re.I),
"github": re.compile(r"https?://(?:www\.)?github\.com/(?!sponsors|features|about|topics)[A-Za-z0-9-]+(?=[/\"'?#\s<]|$)", re.I),
"youtube": re.compile(r"https?://(?:www\.)?youtube\.com/(?:@|c/|channel/|user/)[^\s\"'<>?#]+", re.I),
}
def clean_url(url):
# Links inside inline JSON come escaped (https:\/\/...), so drop the backslashes,
# then normalize scheme, "www." and trailing slashes so duplicates collapse.
url = url.replace("\\", "").rstrip("/")
p = urlparse(url)
host = p.netloc.lower().removeprefix("www.")
return f"https://{host}{p.path.rstrip('/')}"
5. Role inboxes vs personal-looking addresses
For outreach you usually want info@ or sales@, not a named employee. A small allowlist of role prefixes splits them:
ROLE = {"info", "hello", "contact", "sales", "support", "office", "team",
"press", "media", "jobs", "careers", "privacy", "legal", "admin", "help",
"webmaster", "conduct"}
def is_role(email):
return email.split("@")[0].split("+")[0] in ROLE
This also matters for privacy rules: a personal work address is personal data in the EU, a role inbox much less so. Check the email marketing rules of the recipient's country before contacting anyone.
The full script
import csv, html, re, sys
from urllib.parse import urljoin, urlparse
from urllib import robotparser
import requests
UA = {"User-Agent": "contact-tutorial/1.0 (+https://dev.to)"}
CONTACT_HINTS = re.compile(r"contact|about|impressum|imprint|kontakt|team|support", re.I)
EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,24}")
OBFUSCATED_RE = re.compile(
r"([A-Za-z0-9._%+-]+)\s*[\[\(]\s*at\s*[\]\)]\s*([A-Za-z0-9-]+(?:\s*[\[\(]\s*dot\s*[\]\)]\s*[A-Za-z0-9-]+)+)",
re.I,
)
SOCIAL = {
"linkedin": re.compile(r"https?://(?:[a-z]{2,3}\.)?linkedin\.com/(?:company|in|school)/[^\s\"'<>?#]+", re.I),
"twitter": re.compile(r"https?://(?:www\.)?(?:twitter|x)\.com/(?!share|intent|home)[A-Za-z0-9_]{1,15}(?=[/\"'?#\s<]|$)", re.I),
"facebook": re.compile(r"https?://(?:www\.)?facebook\.com/(?!sharer|share|dialog|tr\b)[^\s\"'<>?#]+", re.I),
"github": re.compile(r"https?://(?:www\.)?github\.com/(?!sponsors|features|about|topics)[A-Za-z0-9-]+(?=[/\"'?#\s<]|$)", re.I),
"youtube": re.compile(r"https?://(?:www\.)?youtube\.com/(?:@|c/|channel/|user/)[^\s\"'<>?#]+", re.I),
}
ROLE = {"info", "hello", "contact", "sales", "support", "office", "team",
"press", "media", "jobs", "careers", "privacy", "legal", "admin", "help",
"webmaster", "conduct"}
JUNK_TLDS = (".png", ".jpg", ".jpeg", ".gif", ".svg", ".webp", ".css", ".js")
def clean_url(url):
# Links inside inline JSON come escaped (https:\/\/...), so drop the backslashes,
# then normalize scheme, "www." and trailing slashes so duplicates collapse.
url = url.replace("\\", "").rstrip("/")
p = urlparse(url)
host = p.netloc.lower().removeprefix("www.")
return f"https://{host}{p.path.rstrip('/')}"
def robots_for(base):
rp = robotparser.RobotFileParser()
try:
r = requests.get(urljoin(base, "/robots.txt"), headers=UA, timeout=10)
rp.parse(r.text.splitlines() if r.ok else [])
except requests.RequestException:
rp.parse([])
return rp
def fetch(url):
try:
r = requests.get(url, headers=UA, timeout=15)
except requests.RequestException:
return None, url
if not r.ok or "html" not in r.headers.get("content-type", ""):
return None, r.url
return r.text, r.url
def extract_emails(page):
text = html.unescape(page).replace("\\/", "/")
found = set(EMAIL_RE.findall(text))
for user, dom in OBFUSCATED_RE.findall(text):
dom = re.sub(r"\s*[\[\(]\s*dot\s*[\]\)]\s*", ".", dom, flags=re.I)
found.add(f"{user}@{dom}")
return {e.lower().rstrip(".") for e in found if not e.lower().endswith(JUNK_TLDS)}
def contact_links(page, base_url):
host = urlparse(base_url).netloc
links = []
for href in re.findall(r'href=["\']([^"\'#]+)', page, re.I):
url = urljoin(base_url, href)
if urlparse(url).netloc == host and CONTACT_HINTS.search(urlparse(url).path):
links.append(url.split("?")[0])
return list(dict.fromkeys(links))
def scan(domain, max_pages=6):
base = domain if domain.startswith("http") else f"https://{domain}"
rp = robots_for(base)
page, final = fetch(base) if rp.can_fetch(UA["User-Agent"], base) else (None, base)
row = {"domain": urlparse(final).netloc or domain, "emails": set(), "pages": 0,
**{k: set() for k in SOCIAL}}
if page is None:
return row
queue, seen = [final] + contact_links(page, final), set()
for url in queue[:max_pages]:
if url in seen or not rp.can_fetch(UA["User-Agent"], url):
continue
seen.add(url)
body = page if url == final else fetch(url)[0]
if not body:
continue
row["pages"] += 1
row["emails"] |= extract_emails(body)
for name, rx in SOCIAL.items():
row[name] |= {clean_url(m) for m in rx.findall(body.replace("\\/", "/"))}
return row
def is_role(email):
return email.split("@")[0].split("+")[0] in ROLE
if __name__ == "__main__":
domains = sys.argv[1:] or ["python.org", "example.com"]
out = csv.writer(sys.stdout)
out.writerow(["domain", "pages", "role_emails", "other_emails", *SOCIAL])
for d in domains:
r = scan(d)
role = sorted(e for e in r["emails"] if is_role(e))
other = sorted(e for e in r["emails"] if not is_role(e))
out.writerow([r["domain"], r["pages"], " ".join(role), " ".join(other),
*(" ".join(sorted(r[k])) for k in SOCIAL)])
Run it with any domains:
pip install requests
python contacts.py python.org example.com apify.com > contacts.csv
Output from a real run today:
domain,pages,role_emails,other_emails,linkedin,twitter,facebook,github,youtube
www.python.org,6,webmaster@python.org,psf@python.org python-announce@python.org,https://linkedin.com/company/python-software-foundation,https://twitter.com/ThePSF,,https://github.com/python,
example.com,1,,,,,,,
apify.com,5,hello@apify.com support@apify.com,john.wick@thecontinental.com,https://linkedin.com/company/apify,https://x.com/apify,,https://github.com/apify,
Notice john.wick@thecontinental.com on apify.com. It's a placeholder in a demo form, not a real contact. Any regex-based extractor will pick up example addresses like this, so it's worth a quick filter (or a manual glance) before you import results anywhere.
Limits of this approach
- Sites that render everything with JavaScript return almost empty HTML. You'll need a browser for those, or accept a miss.
- Some sites block non-browser user agents. Treat a blocked or empty site as "no data", don't retry aggressively.
- Phone numbers and postal addresses are much messier than emails (formats differ per country), so I left them out to keep the script readable.
If you'd rather not run it yourself
Disclosure: I built and maintain a hosted version of this on Apify, Contact Details Extractor. It adds phones, addresses and the source page for each value, runs on a schedule and exports CSV. Domains where nothing is found are free. The script above is enough for a few hundred domains, though.
Top comments (0)