DEV Community

SpellzCode
SpellzCode

Posted on Edited on

Getting the contact email from 500 company websites: hidden addresses, junk matches and dead domains

"Find the contact email on each of these websites" sounds like one regex. In practice a naive scraper gets a third of the emails wrong or misses them. This post walks through the four problems that cause most of the misses, with code for each. Disclosure: SpellzCode, the brand behind this account, sells a tool that does all of this (linked at the end), but the techniques below stand on their own.

1. The email is not on the home page

Most companies put their address on a Contact, About, Team or Imprint page. Look at the home page's links and open the few most likely ones, including other languages:

const CONTACT_WORDS = /contact|kontakt|about|om-oss|team|imprint|impressum|mentions-legales|contacto|contatti|yhteystiedot/i;

function likelyContactPages(html, baseUrl, max = 4) {
  const links = [...html.matchAll(/href="([^"#]+)"/g)].map((m) => new URL(m[1], baseUrl));
  const same = links.filter((u) => u.hostname === new URL(baseUrl).hostname);
  return [...new Set(same.filter((u) => CONTACT_WORDS.test(u.pathname)).map(String))].slice(0, max);
}
Enter fullscreen mode Exit fullscreen mode

Four or five pages per site finds the address on nearly every site that publishes one.

2. The email is hidden by Cloudflare

Sites behind Cloudflare often have "email address protection" on. The HTML then contains <a data-cfemail="...">[email protected]</a> instead of the address. It is a simple XOR: the first byte is the key.

function decodeCfEmail(hex) {
  const key = parseInt(hex.slice(0, 2), 16);
  let out = '';
  for (let i = 2; i < hex.length; i += 2) {
    out += String.fromCharCode(parseInt(hex.slice(i, i + 2), 16) ^ key);
  }
  return out;
}

const hidden = [...html.matchAll(/data-cfemail="([0-9a-f]+)"/gi)].map((m) => decodeCfEmail(m[1]));
Enter fullscreen mode Exit fullscreen mode

Also read name [at] company [dot] com spellings, which many small sites use.

3. Your regex matches junk

An email regex over raw HTML returns things like logo@2x.png, Sentry and error-tracking addresses, you@example.com placeholders in forms, and firstname.lastname@company.com from instructions. Filter them:

const JUNK = [
  /\.(png|jpe?g|gif|svg|webp)$/i,          // retina image names
  /@(sentry|wixpress|example)\./i,          // tracking and placeholders
  /^(you|your|name|firstname|email)[.@]/i,  // form hints
];
const clean = (emails) => [...new Set(emails.map((e) => e.toLowerCase()))].filter((e) => !JUNK.some((r) => r.test(e)));
Enter fullscreen mode Exit fullscreen mode

Then rank what is left: an address at the site's own domain beats a Gmail address or the web agency's address in the footer, and info@ or contact@ beats a named person for a first message.

4. The domain cannot receive mail

Old addresses on old pages point at domains that no longer have a mail server. One DNS lookup catches them before they bounce:

import { resolveMx } from 'node:dns/promises';

async function domainAcceptsMail(email) {
  try {
    const mx = await resolveMx(email.split('@')[1]);
    return mx.length > 0;
  } catch {
    return false;
  }
}
Enter fullscreen mode Exit fullscreen mode

This confirms the domain takes mail, not that the mailbox exists. Checking a mailbox means talking SMTP to the server, which many servers now refuse or answer falsely, so treat the MX check as the reliable part.

Phones and socials

Take phone numbers from tel: links and from the site's JSON-LD ("telephone") rather than regexing digits out of text, which picks up VAT numbers and dates. Social profiles are just links to linkedin.com/company, facebook.com, instagram.com and so on.

Using the tool instead

Contact Details Scraper on Apify does all of the above, returning one row per website with primaryEmail, every email with its type and source page, phones, social links and the contact page URL:

// npm i apify-client
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('spellzcode/website-contact-extractor').call({
  websites: ['apify.com', 'letsencrypt.org'],
  maxPagesPerSite: 5,
  onlyDomainEmails: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const s of items) console.log(s.domain, s.primaryEmail, s.phones?.[0] ?? '');
Enter fullscreen mode Exit fullscreen mode

You pay $1.50 per 1,000 websites where contacts were found; sites with nothing, or that refuse the request, are free. Apify's free $5 monthly credit covers about 3,300 sites with contacts. It reads the HTML a site serves, so contacts that only appear after JavaScript runs can be missed.

Before you send anything

Addresses of named people are personal data under GDPR, and many countries restrict unsolicited e-mail even to businesses. Prefer role addresses, have a lawful basis, honour opt-outs, and respect each site's terms and robots.txt (the tool does by default).

Links

Top comments (0)