"Find the contact email on each of these websites" sounds like one regex. In practice a naive scraper gets a third of the emails wrong or misses them. This post walks through the four problems that cause most of the misses, with code for each. Disclosure: SpellzCode, the brand behind this account, sells a tool that does all of this (linked at the end), but the techniques below stand on their own.
1. The email is not on the home page
Most companies put their address on a Contact, About, Team or Imprint page. Look at the home page's links and open the few most likely ones, including other languages:
const CONTACT_WORDS = /contact|kontakt|about|om-oss|team|imprint|impressum|mentions-legales|contacto|contatti|yhteystiedot/i;
function likelyContactPages(html, baseUrl, max = 4) {
const links = [...html.matchAll(/href="([^"#]+)"/g)].map((m) => new URL(m[1], baseUrl));
const same = links.filter((u) => u.hostname === new URL(baseUrl).hostname);
return [...new Set(same.filter((u) => CONTACT_WORDS.test(u.pathname)).map(String))].slice(0, max);
}
Four or five pages per site finds the address on nearly every site that publishes one.
2. The email is hidden by Cloudflare
Sites behind Cloudflare often have "email address protection" on. The HTML then contains <a data-cfemail="...">[email protected]</a> instead of the address. It is a simple XOR: the first byte is the key.
function decodeCfEmail(hex) {
const key = parseInt(hex.slice(0, 2), 16);
let out = '';
for (let i = 2; i < hex.length; i += 2) {
out += String.fromCharCode(parseInt(hex.slice(i, i + 2), 16) ^ key);
}
return out;
}
const hidden = [...html.matchAll(/data-cfemail="([0-9a-f]+)"/gi)].map((m) => decodeCfEmail(m[1]));
Also read name [at] company [dot] com spellings, which many small sites use.
3. Your regex matches junk
An email regex over raw HTML returns things like logo@2x.png, Sentry and error-tracking addresses, you@example.com placeholders in forms, and firstname.lastname@company.com from instructions. Filter them:
const JUNK = [
/\.(png|jpe?g|gif|svg|webp)$/i, // retina image names
/@(sentry|wixpress|example)\./i, // tracking and placeholders
/^(you|your|name|firstname|email)[.@]/i, // form hints
];
const clean = (emails) => [...new Set(emails.map((e) => e.toLowerCase()))].filter((e) => !JUNK.some((r) => r.test(e)));
Then rank what is left: an address at the site's own domain beats a Gmail address or the web agency's address in the footer, and info@ or contact@ beats a named person for a first message.
4. The domain cannot receive mail
Old addresses on old pages point at domains that no longer have a mail server. One DNS lookup catches them before they bounce:
import { resolveMx } from 'node:dns/promises';
async function domainAcceptsMail(email) {
try {
const mx = await resolveMx(email.split('@')[1]);
return mx.length > 0;
} catch {
return false;
}
}
This confirms the domain takes mail, not that the mailbox exists. Checking a mailbox means talking SMTP to the server, which many servers now refuse or answer falsely, so treat the MX check as the reliable part.
Phones and socials
Take phone numbers from tel: links and from the site's JSON-LD ("telephone") rather than regexing digits out of text, which picks up VAT numbers and dates. Social profiles are just links to linkedin.com/company, facebook.com, instagram.com and so on.
Using the tool instead
Contact Details Scraper on Apify does all of the above, returning one row per website with primaryEmail, every email with its type and source page, phones, social links and the contact page URL:
// npm i apify-client
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('spellzcode/website-contact-extractor').call({
websites: ['apify.com', 'letsencrypt.org'],
maxPagesPerSite: 5,
onlyDomainEmails: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const s of items) console.log(s.domain, s.primaryEmail, s.phones?.[0] ?? '');
You pay $1.50 per 1,000 websites where contacts were found; sites with nothing, or that refuse the request, are free. Apify's free $5 monthly credit covers about 3,300 sites with contacts. It reads the HTML a site serves, so contacts that only appear after JavaScript runs can be missed.
Before you send anything
Addresses of named people are personal data under GDPR, and many countries restrict unsolicited e-mail even to businesses. Prefer role addresses, have a lawful basis, honour opt-outs, and respect each site's terms and robots.txt (the tool does by default).
Links
- Contact Details Scraper
- Email List Cleaner for checking a list you already have
- All SpellzCode Apify actors
Top comments (0)