Finding Contact Emails on Company Websites Politely: robots.txt, mailto, JSON-LD and the Traps

Disclosure: I'm Ben, and I built the Apify Actor mentioned later in this post. I charge per result for it. Everything in the tutorial part works with free tools too.

You have a list of company domains (a CRM export, a lead list, a spreadsheet from an event) and you need a contact email, a phone number and maybe the company's Instagram or LinkedIn. The details are usually right there on the company's own site. Getting them reliably, and politely, takes more care than a regex over the homepage.

Step 1: A minimal, polite extractor

This is roughly the smallest version I'd be comfortable running against small-business sites. It checks robots.txt with its own User-Agent, reads the homepage, pulls mailto: and tel: links, and lists same-domain contact/about pages to visit next. It ran as-is on 25 Oct 2026:

import re, urllib.request, urllib.robotparser
from urllib.parse import urljoin, urlparse

UA = "MyContactBot/0.1 (+mailto:[e-mail])"

def fetch(url):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=15) as r:
        return r.geturl(), r.read().decode("utf-8", "ignore")

def contacts(site):
    rp = urllib.robotparser.RobotFileParser()
    try:  # fetch robots.txt with *your* UA; RobotFileParser.read() uses urllib's default
        rp.parse(fetch(urljoin(site, "/robots.txt"))[1].splitlines())
    except Exception:
        rp.parse([])  # no robots.txt (404) means allowed
    if not rp.can_fetch(UA, site):
        return {"site": site, "status": "blocked_by_robots_txt"}
    final, html = fetch(site)
    emails = set(re.findall(r'mailto:([^"\'?>\s]+)', html, re.I))
    phones = set(re.findall(r'tel:([+\d()\-. ]{7,})', html, re.I))
    host = urlparse(final).netloc
    contact_links = {urljoin(final, h) for h in re.findall(r'href="([^"]+)"', html)
                     if re.search(r"contact|about|impressum", h, re.I)
                     and urlparse(urljoin(final, h)).netloc == host}
    return {"site": final, "emails": sorted(emails), "phones": sorted(phones),
            "contactPages": sorted(contact_links)[:5]}

print(contacts("https://www.aloha-dental.com/"))

Output:

{'site': 'https://www.aloha-dental.com/', 'emails': [], 'phones': ['[phone]'],
 'contactPages': ['https://www.aloha-dental.com/about-us/', 'https://www.aloha-dental.com/contact-us/']}

Two lessons from that one run.

First, the homepage had no email. The phone was there as a tel: link, but the address lives deeper in the site. My packaged tool (below) found [e-mail] on the same site after visiting 5 pages. You almost always need to follow the contact/about links.

Second, my first version wrongly reported this site as blocked by robots.txt. Python's RobotFileParser.read() fetches robots.txt with urllib's default User-Agent, the server answered 403, and the parser treats an access error as "disallow everything". Fetching robots.txt yourself with your real UA fixed it. If your crawler says "blocked" a lot, check that before blaming the sites.

Step 2: The traps you'll hit next

• Emails aren't only in mailto:. Look in page text, and in schema.org JSON-LD ("email", "telephone", "address"), which a lot of small-business sites ship through their CMS or SEO plugin.
• Obfuscation. Cloudflare's email protection replaces addresses with a data-cfemail hex string (a simple XOR you can decode). People also write name [at] domain [dot] com.
• Junk addresses. Placeholder and tracking addresses (example.com, Sentry DSNs, wixpress) show up in page source. Filter them.
• Dead domains. An email whose domain has no MX record won't receive mail. A DNS lookup is cheap and doesn't touch anyone's mailbox.
• Phone false positives. Order numbers, dates and prices look like phone numbers in text. Trust tel: links and schema.org more than regexes.
• No contact link at all. Fall back to trying /contact, /contact-us and /about.
• Be slow. These are often tiny shared-hosting sites. One request per second per site, an identifiable User-Agent, and robots.txt Crawl-delay if they set one.

Step 3: Or use the packaged version

I packaged this into an Apify Actor, Website Contact Scraper. You paste domains, URLs or emails, and get one row per company:

• Emails from mailto:, text, JSON-LD, Cloudflare-protected and "[at]/[dot]" forms. Each is tagged sameDomain, role (info@, sales@…) and domainAcceptsMail (the MX check).
• Phones from tel: and schema.org. Numbers found only in text go into a separate phonesUncertain list.
• 13 social networks (LinkedIn, Facebook, Instagram, X, YouTube, TikTok, Pinterest, GitHub, Threads, Telegram, WhatsApp, Discord, Yelp), plus contactPageUrls
• Company name and postal address when the site publishes them in schema.org or OpenGraph

It crawls the homepage plus up to 19 contact/about/impressum/team pages on the same domain, honours robots.txt (including Crawl-delay), goes 1 request per second per site and identifies itself as ContactEnrichmentBot. It doesn't run JavaScript.

The real result for the site above (trimmed):

{
  "domain": "aloha-dental.com",
  "status": "ok",
  "companyName": "Aloha Dental",
  "primaryEmail": "[e-mail]",
  "phones": ["[phone]"],
  "phonesUncertain": ["[phone]"],
  "address": "1614 E. Riverside DR., Austin, Texas, 78741, US",
  "contactPageUrls": ["https://www.aloha-dental.com/contact-us/"],
  "instagram": ["https://www.instagram.com/alohadentaltx"],
  "pagesCrawled": 5,
  "charged": true
}

Real fill rates

From live runs on 17 Oct 2026, measured over sites that were actually reached:

Test	Websites	Reached	Email	Phone	Instagram	LinkedIn	Contact page
Manchester restaurant/café sites	90	72	68%	62%	76%	11%	70%
Mixed: US trades/dentists, Paris cafés, 12 SaaS companies	55	49	55%	57%	69%	59%	89%

The sites that weren't reached were blocked by robots.txt (7 and 4 sites, never crawled), returned HTTP 403/404, had no DNS, or timed out.

Limits, honestly

• No JavaScript rendering. Sites that only render contact details client-side return fewer results.
• Some sites block data-centre IPs. You can set an Apify Proxy, which Apify bills separately.
• Email isn't guaranteed. Between a third and almost half of the reached sites in the tests had no email at all (32% and 45%). Many small businesses only offer a contact form, which is why contactPageUrls is there.
• The MX check isn't verification. It confirms the domain accepts mail, not that the mailbox exists. The tool never sends mail or probes mailboxes.

What it costs

$0.002 per website where at least one email, phone or social profile was found ($2 per 1,000). Websites that are unreachable, blocked by robots.txt or empty are free (charged: false). In the mixed test, 47 of the 55 sites were charged, which is 9.4 cents. With a $0.01 max-cost cap, it charged exactly 5 sites and stopped. The script above is free.

Play fair

Named employees' emails and phones are personal data under GDPR, UK GDPR and CCPA, and you're the data controller. Have a lawful basis, honour opt-outs, follow CAN-SPAM, CASL and PECR, and if you want to minimise personal data, keep only the role addresses (role: true). Please don't build spam lists with any of this.

What obfuscation trick have you run into that I haven't listed? Tell me in the comments and I'll add a decoder for it.